ceph.spec.in: split out ceph-common as in Debian

Move files, postun scriptlet, and add dependencies on ceph-common
where appropriate

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Merge pull request #2074 from onlyjob/docs

doc: mention kernel support for rbd format 2

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2077 from ceph/wip-8754

tests: don't depend on 'data' pool in rbd test

Reviewed-by: Sage Weil <sage@redhat.com>

tests: don't depend on 'data' pool in rbd test

Since we removed the default 'data' and 'metadata' pools,
tests which need a pool should create it themselves.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/release-notes: clarify CRUSH notes about tunables

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2075 from ceph/wip-8745

mds: use client-provided time stamp for user-visible file metadata

Reviewed-by: Sage Weil <sage@redhat.com>

mds: use client-provided time stamp for user-visible file metadata

Fixes: #8745
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

doc: mention kernel support for rbd format 2

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

Merge remote-tracking branch 'gh/next'

Merge pull request #2072 from ceph/wip-8738-next

Patch up the CRUSH map compatibility guards

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: Fix a typo regarding requiretty for RHEL based platforms

Merge pull request #2058 from ceph/wip-refs

refcount debugging for RefCountedObject

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

mon: check changes to the whole CRUSH map and to tunables against cluster features

When we change the tunables, or set a new CRUSH map, we need to make sure it's
supported by all the monitors and OSDs currently participating in the cluster.

Fixes: #8738
Signed-off-by: Greg Farnum <greg@inktank.com>

OSDMonitor: fix quorum_features comparison in check_cluster_features

We need to see if there's a feature which is not in the quorum_features,
not if there are no features in common!

Signed-off-by: Greg Farnum <greg@inktank.com>

msg: debug refs on incoming Messages

Signed-off-by: Sage Weil <sage@inktank.com>

common/RefCountedObject: make nref, cct private

Signed-off-by: Sage Weil <sage@inktank.com>

msg/Message: use RefCountedObject ctor to set initial ref count

Also, drop the useless assert in the dtor (the RefCountedObject dtor does
the same).

Signed-off-by: Sage Weil <sage@inktank.com>

msg/Pipe: debug Pipe refs

Signed-off-by: Sage Weil <sage@inktank.com>

msg: debug Connection refs

Signed-off-by: Sage Weil <sage@inktank.com>

osd: debug Session refs

Signed-off-by: Sage Weil <sage@inktank.com>

common/RefCountedObject: assert nref == 0 in dtor

Signed-off-by: Sage Weil <sage@inktank.com>

common/RefCountedObject: optionally take initial ref count to ctor

Signed-off-by: Sage Weil <sage@inktank.com>

common/RefCountedObject: optionally debug

Introduce a 'refs' subsys for logging.  Print log ref count inc and dec
for any RefCountedObject that gets a cct passed to its ctor.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2059 from ceph/wip-osd-dumpres

osd: allow dumping reservation state via asok

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2069 from thorstenb/thb-buildconvenience-1

Automagically setup submodules on first run.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2066 from thorstenb/wip-janitorial-clang-2

[werror] Fix c++11-compat-reserved-user-defined-literal

Merge pull request #2062 from somnathr/wip-sd-replicated-pg

ReplicatedPG: Removed the redundant register_snapset_context call

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2064 from somnathr/wip-sd-share-map

OSD: adjust share_map() to handle the case that the osd is down

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2065 from somnathr/wip-sd-optracker

Pull request for optracker changes

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2063 from athanatos/wip-8691

OSD: wake_pg_waiters after dropping pg lock

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2047 from dachary/wip-osd-config

osd: improve tests for configuration updates

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

Merge pull request #2036 from ceph/wip-osd-map-cache-size

osd: allow osd map cache size to be adjusted at runtime

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2068 from thorstenb/wip-janitorial-clang-4

ceph_argparse_flag has no regular 3rd parameter.

Reviewed-by: Loic Dachary <loic@dachary.org>

qa: support running under non privileged user

If the test is run against a cluster started with vstart.sh (which is
the case for make check), the --asok-does-not-need-root disables the use
of sudo and allows the test to run without requiring privileged user
permissions.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 522174b066044e51a7019bd6cad81117e83c394e)

Merge pull request #2061 from ceph/wip-8728

qa/workunits/rest/test.py: make osd create test idempotent

Reviewed-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #2046 from ceph/wip-8670

mon: OSDMonitor: 'osd pool' - if we can set it, we must be able to get it

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Sage Weil <sage@redhat.com>

Automagically setup submodules on first run.

I mean, who's reading the README, anyway?

Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

ceph_argparse_flag has no regular 3rd parameter.

With clang warning: 'va_start' has undefined behavior with reference
types, noticing copy-paste mistake on ceph_argparse_flag.

Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

[werror] Fix c++11-compat-reserved-user-defined-literal

c++11 readiness, avoid clash with
http://www.stroustrup.com/C++11FAQ.html#UD-literals

Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

OSD: adjust share_map() to handle the case that the osd is down

The assert was hitting while OSd is waiting for becoming healthy
in handle_osd_map(). This can happen while io is going on and
OSDs are made down forcefully by say osd thrash command.
So, the fix could be instead of asserting just return from here.

Fixes: #8646
Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

mon: OSDMonitor: 'osd pool' - if we can set it, we must be able to get it

Add support to get the values for the following variables:
 - target_max_objects
 - target_max_bytes
 - cache_target_dirty_ratio
 - cache_target_full_ratio
 - cache_min_flush_age
 - cache_min_evict_age

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

ReplicatedPG: Removed the redundant register_snapset_context call

In the get_object_context(), the get_snapset_context is been called
and the register_snapset_context is already been invoked from there.
This call seems to be redundant.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

OpTracker: The optracker enabled/disabled check is added

The optracker enabled/disabled check is now added in the
mark_event(const string&) interface.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

OpTracker: use mark_event rather than _mark_event

mark_event() interfaced changed to accept time and it is default
to 'now'. The mark_event is the wrapper function of _mark_event()
and it has the check for optracking enabled/disabled as well.
The _mark_event() is now a private function.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

qa/workunits/rest/test.py: make osd create test idempotent

Avoid possibility that we create multiple OSDs do to retries by passing in
the optional uuid arg.  (A stray osd id will make the osd tell tests a
few lines down fail.)

Fixes: #8728
Signed-off-by: Sage Weil <sage@inktank.com>

osd: add dump_reservations asok command

Allow an admin to inspect pending and in-progress reservations.

Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

common/AsyncReserver: add dump()

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2055 from ceph/wip-8692

mon: clear osd request queue latency info on down or up

Reviewed-by: Samuel Just <sam.just@inktank.com>

mon: clear osd request queue latency info on down or up

It doesn't make sense to warn about slow requests on OSDs that are down;
clear those stats in the monitor when we see an OSD go down (or up).

Fixes: #8692
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2054 from ceph/wip-fs-cmds-oops

Apply some accidentally-dropped commits from the wip-fs-cmds branch to be a little friendlier and pass tests.

Reviewed-by: Greg Farnum <greg@inktank.com>

OSD: wake_pg_waiters after dropping pg lock

Otherwise, we dispatch_session_waiting while still holding the pg lock,
which is obviously wrong.  Unfortunately, this places an additional
burden on any user of _create_lock_pg, but I think it's unavoidable
since that method must atomically add the pg to the map and lock it.

Fixes: #8961
Introduced in:
  25466839589813047c975e44e67e14f34e32139e
  ecda2fef8ce982df3581a3b47ba74ae581d82479
Signed-off-by: Samuel Just <sam.just@inktank.com>

mds: Update default FS name

From 'default' to 'cephfs' so that it looks
at least vaguely meaningful.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update cephtool EC pools test to respect IDs

...of data and metadata pools which are no longer
constant.

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MDSMonitor: EC check in 'fs new' like newfs

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add a check for crash_replay_interval autoset

... that after a fs new on fresh pools, crash_replay_interval
is set to the default on the data pool.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: Set crash_replay_interval automatically

Signed-off-by: John Spray <john.spray@redhat.com>

qa: Update data pool ID for vxattrs test

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #2053 from dachary/wip-signed-warnings

test: use 0U with gtest to avoid spurious warnings

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

Merge pull request #2052 from dachary/wip-vstart-wrapped

qa: support running under non privileged user

Reviewed-by: Florent Flament <florent.flament@cloudwatt.com>

test: use 0U with gtest to avoid spurious warnings

Signed-off-by: Loic Dachary <loic@dachary.org>

qa: support running under non privileged user

If the test is run against a cluster started with vstart.sh (which is
the case for make check), the --asok-does-not-need-root disables the use
of sudo and allows the test to run without requiring privileged user
permissions.

Signed-off-by: Loic Dachary <loic@dachary.org>

OSD: await_reserved_maps() prior to calling mark_down

send_message_osd_cluster() et al are *trying* to protect their Connection
lookups (and not re-open zapped Connections) via map reservations, but
that only works if we know that we haven't already called mark_down() on
the entities they might be looking up. So we need to await_reserved_maps
before we do any mark_down calls.
Since the waiting might take some time (fast dispatch in progress), only do
so if we are actually going to mark somebody down.

Fixes: #8512
Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1942 from ceph/wip-fs-cmds

Wip fs cmds

Reviewed-by: Greg Farnum <greg@inktank.com>

osd: allow osd map cache size to be adjusted at runtime

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #2050 from ceph/wip-cephtool

qa/workunits/cephtool/test.sh: sudo ceph daemon

qa/workunits/cephtool/test.sh: sudo ceph daemon

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2049 from ceph/wip-fix-pglog-unittest

PGLog: fix clear() to avoid the IndexLog::zero() asserts

Reviewed-by: Sage Weil <sage@redhat.com>

PGLog: fix clear() to avoid the IndexLog::zero() asserts

Introduced in:
  c5b8d8105d965da852c79add607b69d5ae79a4d4
  ac11ca40b4f4525cbe9b1778b1c5d9472ecb9efa
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2003 from ferustigris/hotfix/rgw_memory_leaks

RGW memory leaks have been fixed

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2041 from ceph/wip-map-unmap

Wip map unmap

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2040 from ceph/wip-krbd-settle

krbd: rework the unmap retry loop

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2038 from ceph/wip-fsx-random-ctx

test_librbd_fsx: use private RNG context

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2048 from ceph/wip-8699

rgw: fix uninit ofs in RGWObjManifect::obj_iterator

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

rgw: fix uninit ofs in RGWObjManifect::obj_iterator

Valgrind picked this up:

  <kind>UninitCondition</kind>
  <what>Conditional jump or move depends on uninitialised value(s)</what>
  <stack>
    <frame>
      <ip>0x5145B8</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWObjManifest::obj_iterator::seek(unsigned long)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.82-354-g62027ec/src/rgw</dir>
      <file>rgw_rados.cc</file>
      <line>562</line>
    </frame>
    <frame>
      <ip>0x5672A4</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>list_multipart_parts(RGWRados*, req_state*, std::string const&amp;, std::string&amp;, int, int, std::map&lt;unsigned int, RGWUploadPartInfo, std::less&lt;unsigned int&gt;, std::allocator&lt;std::pair&lt;unsigned int const, RGWUploadPartInfo&gt; &gt; &gt;&amp;, int*, bool*, bool)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.82-354-g62027ec/src/rgw</dir>
      <file>rgw_rados.h</file>
      <line>217</line>
    </frame>
    <frame>
      <ip>0x5688EE</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWListMultipart::execute()</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.82-354-g62027ec/src/rgw</dir>
      <file>rgw_op.cc</file>
      <line>2956</line>
    </frame>
...

Fixes: #8699
Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2045 from ceph/wip-8542

qa/workunits/suites/fsx.sh: don't use zero range

Reviewed-by: Greg Farnum <greg@inktank.com>

osd: improve tests for configuration updates

Extract the default values from the actual configuration instead of
having them hardcoded. Also check that lowering the osd_map_cache_size
triggers the expected warning.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #2026 from athanatos/wip-8438-master

Wip 8438 master

Fixes: #8438
Backport: Firefly
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #2027 from athanatos/wip-7999-master

Wip 7999 master

Fixes: #7999
Backport: Firefly
Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

qa/workunits/suites/fsx.sh: don't use zero range

Zero range is not supported by cephfs.

Fixes: #8542
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2039 from dachary/wip-erasure-code-doc

erasure-code: add definitions to the glossary

Reviewed-by: Florent Flament <florent.flament@cloudwatt.com>

mon/MDSMonitor: log warning while MDS up but no FS

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MDSMonitor: fix incorrect comment

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MDSMonitor: s/enabled/get_enabled()/

Signed-off-by: John Spray <john.spray@redhat.com>

mds: Handle setting 'enabled' in upgrades

Previously was always defaulting to disabled,
should be enabled if MDS map epoch is >1 (i.e.
if an MDS was ever started).

Signed-off-by: John Spray <john.spray@redhat.com>

mds: journal-related debug messages

Signed-off-by: John Spray <john.spray@redhat.com>

test/qa: update for MDSMonitor changes

Accomodate changes:
 * data and metadata pools no longer exist by default
 * filesystem-using tests must use `fs new` to create
   the filesystem first.

Signed-off-by: John Spray <john.spray@inktank.com>

vstart: update for newfs/rmfs changes

Signed-off-by: John Spray <john.spray@inktank.com>

mon: add `ceph fs [new|rm|ls]`

Signed-off-by: John Spray <john.spray@inktank.com>

erasure-code: add definitions to the glossary

and fix a typo

Signed-off-by: Loic Dachary <loic@dachary.org>

mon: hide mdsmap in 'ceph status' if not enabled

Previously we checked if the MDS map had ever been
updated (epoch > 1), now we have an explicit flag
for whether it's enabled.

Signed-off-by: John Spray <john.spray@inktank.com>

mon: stop MDSMonitor when FS disabled

Where 'stopped' means ignoring beacons and
not performing 'tick' activities.

Signed-off-by: John Spray <john.spray@inktank.com>

mon: enable deleting pools when FS disabled

Signed-off-by: John Spray <john.spray@inktank.com>

mon: warn in newfs if crash_replay_interval=0

This is the setting we would apply to data pools
created automatically, so notify the user if they're
failing to use it for data pools they have created
by hand.

Signed-off-by: John Spray <john.spray@inktank.com>

mon: don't require 'sure' flag for new fs

Only prompt for --yes-i-really-mean-it if there
is an existing FS.

Signed-off-by: John Spray <john.spray@inktank.com>

osdmap: Don't create FS pools by default

Because many Ceph users don't use the filesystem,
don't create the 'data' and 'metadata' pools by
default -- they will be created by newfs if
they are needed.

Signed-off-by: John Spray <john.spray@inktank.com>

mds: no initial filesystem

Because not everyone uses CephFS, we would
like to avoid initially creating any data/metadata
pools for CephFS.

To avoid creating those pools, we must avoid initially
populating the MDSMap.

Signed-off-by: John Spray <john.spray@inktank.com>

mds: add 'enabled' flag to MDSMap

When 'enabled' is false, the MDSMap is effectively
null.  This allows Ceph clusters with no filesystem
or filesystem data/metadata pools.

Signed-off-by: John Spray <john.spray@inktank.com>

doc: RPM instructions correction

Signed-off-by: John Spray <john.spray@inktank.com>

krbd: rework the unmap retry loop

The retry loop in the unmap path turned out to be insufficient for
doing long fsx -K runs.  Replace it with a single retry and then call
out to 'udevadm settle' if the retry doesn't help.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>