qa/workunits: cephtool: test get/set on both tier and non-tier pools

Make sure gets and sets of tiering-specific variables succeed on tier
pools and fail on non-tier pools.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: split get/set on tier pools from get/set tests

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: be scary about inconsistent pool tier ids

We may not crash your cluster, but you'll know that this is not something
that should have happened.  Big letters makes it obvious.  We'd make them
red too if we bothered to look for the ANSI code.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

osd: pg_pool_t: clear tunables on clear_tier()

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: limit tier-specific pool set/get on non-tier pools

Fixes: 8696
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

doc/changelog/v0.80.2: include tag

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'master' of https://github.com/ceph/ceph

doc: Added CentOS/RHEL install for git.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

rgw: modelines

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Added keyring location note to resolve pull abandoned pull request #1946.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Clean up formatting, usage and removed duplicate section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc/release-notes: v0.80.2

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1905 from leseb/doc-rbd-icehouse

doc: rbd icehouse updated. Need to remove duplicate section.

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2087 from theanalyst/doc/typo-fix

doc: Fixing a few typos in docs

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

Revert "qa: add an fsx run which turns on kernel debugging"

This reverts commit 29c33f0c057acc4e0f4e5022c97553a2dc095b21.
We don't need the debugging any more, and having two separate fsx runners
already caused one update-in-the-wrong-place issue.

Signed-off-by: Greg Farnum <greg@inktank.com>

doc: fix a typo in quickstart doc

* doc/start/documenting-ceph.rst: s/along side/alongside

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: fix a few typos in radosgw docs

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: fix a few typos in rados docs

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: fix a few typos in the dev docs

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: fix a few typos in rbd docs

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: Fix a few typos in the install docs

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: fix a few typos in architecture page

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: Fix a typo in the rbd man page

's/specifially/specifically'

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #2029 from ceph/wip-8523

mon: OSDMonitor: add 'osd pool get-quota' command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

qa/workunits: cephtool: test for 'osd pool {get,set}-quota'

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: add 'osd pool get-quota' command

Enables us to obtain current quotas for a given pool.

Fixes: #8523
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #2076 from cernceph/wip-wbthrottle-perfcounters

decrement WBThrottle perfcounters in clear_object

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2082 from ceph/wip-8727

messages: MForward: fix compat version

Reviewed-by: Sage Weil <sage@redhat.com>

messages: MForward: fix compat version

A while ago we bumped the head version and reset the compat version to 0.
Doing this so happens to make the messenger assume that the message does
not support the compat versioning and sets the compat version to the head
version -- thus making compat = 2 when it should have been 1.

The nasty side-effect of this is that upgrading from emperor to firefly
will have emperor-leaders being unable to decode forwarded messages from
firefly-peons.

Fixes: #8727
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #2081 from ceph/wip-osd-leaks

osd: fix several memory leaks from fast dispatch

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd: clear sessions_waiting_on_map on shutdown

Wake up these ops and drop the Session refs on shutdown.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: fix session leak when waiting on map

We don't want to leak the Session if this Session is already waiting on
a map.

Introduced by eb30f88c942d80cbc951d4d0f375479069d15624.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2037 from ceph/wip-async-log

MDS: Encode log events in a separate thread from main execution.

This is a first small step in multi-threading the MDS daemon.

Reviewed-by: Greg Farnum <greg@inktank.com>

osd: clear Sessions for loopback Connections on shutdown

Starting with the fast dispatch patches, we are calling the handle_connect
on loopback.  Make sure we zap them on shutdown to break the Session <->
Connection ref cycle.

Signed-off-by: Sage Weil <sage@inktank.com>

Revert "OSD: move waiting_for_pg into Session"

This reverts commit ecda2fef8ce982df3581a3b47ba74ae581d82479.

This leaves Session* refs indefinitely in the map.

This was one source of #7995.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2080 from ceph/wip-8306-rebase

Wip 8306 rebase

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

OSD: fix debug logging output

The snapmapper and infos statements were backwards. Now they aren't.

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #2078 from ceph/wip-8751

test: generalise default_pools in test_rados

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: add bash completion file for radosgw-admin

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

ceph.spec.in: rhel7-related changes:

udev rules: /lib -> /usr/lib
/sbin binaries move to /usr/sbin or %{_sbindir}

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Fix/add missing dependencies:

- rbd-fuse depends on librados2/librbd1
- ceph-devel depends on specific releases of libs and libcephfs_jni1
- librbd1 depends on librados2
- python-ceph does not depend on libcephfs1

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

ceph.spec.in: whitespace fixes

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

ceph.spec.in: split out ceph-common as in Debian

Move files, postun scriptlet, and add dependencies on ceph-common
where appropriate

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

test: generalise default_pools in test_rados

Instead of fragile assumptions about what pools
do or do not exist by default, simply use what
exists at test setup time as a baseline.

Fixes: #8751
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #2074 from onlyjob/docs

doc: mention kernel support for rbd format 2

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2077 from ceph/wip-8754

tests: don't depend on 'data' pool in rbd test

Reviewed-by: Sage Weil <sage@redhat.com>

tests: don't depend on 'data' pool in rbd test

Since we removed the default 'data' and 'metadata' pools,
tests which need a pool should create it themselves.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/release-notes: clarify CRUSH notes about tunables

Signed-off-by: Sage Weil <sage@inktank.com>

decrement WBThrottle perfcounters in clear_object

We observed that the WBThrottle perfcounters are leaking upwards
at a rate of around 50-100 ios_dirtied per day. The counters are
currently not decremented in clear_object, so that's the likely
explanation. Decrement them like elsewhere in WBThrottle.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #2075 from ceph/wip-8745

mds: use client-provided time stamp for user-visible file metadata

Reviewed-by: Sage Weil <sage@redhat.com>

mds: use client-provided time stamp for user-visible file metadata

Fixes: #8745
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

doc: mention kernel support for rbd format 2

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

Merge remote-tracking branch 'gh/next'

Merge pull request #2072 from ceph/wip-8738-next

Patch up the CRUSH map compatibility guards

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: Fix a typo regarding requiretty for RHEL based platforms

Merge pull request #2058 from ceph/wip-refs

refcount debugging for RefCountedObject

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

mon: check changes to the whole CRUSH map and to tunables against cluster features

When we change the tunables, or set a new CRUSH map, we need to make sure it's
supported by all the monitors and OSDs currently participating in the cluster.

Fixes: #8738
Signed-off-by: Greg Farnum <greg@inktank.com>

OSDMonitor: fix quorum_features comparison in check_cluster_features

We need to see if there's a feature which is not in the quorum_features,
not if there are no features in common!

Signed-off-by: Greg Farnum <greg@inktank.com>

msg: debug refs on incoming Messages

Signed-off-by: Sage Weil <sage@inktank.com>

common/RefCountedObject: make nref, cct private

Signed-off-by: Sage Weil <sage@inktank.com>

msg/Message: use RefCountedObject ctor to set initial ref count

Also, drop the useless assert in the dtor (the RefCountedObject dtor does
the same).

Signed-off-by: Sage Weil <sage@inktank.com>

msg/Pipe: debug Pipe refs

Signed-off-by: Sage Weil <sage@inktank.com>

msg: debug Connection refs

Signed-off-by: Sage Weil <sage@inktank.com>

osd: debug Session refs

Signed-off-by: Sage Weil <sage@inktank.com>

common/RefCountedObject: assert nref == 0 in dtor

Signed-off-by: Sage Weil <sage@inktank.com>

common/RefCountedObject: optionally take initial ref count to ctor

Signed-off-by: Sage Weil <sage@inktank.com>

common/RefCountedObject: optionally debug

Introduce a 'refs' subsys for logging.  Print log ref count inc and dec
for any RefCountedObject that gets a cct passed to its ctor.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2059 from ceph/wip-osd-dumpres

osd: allow dumping reservation state via asok

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2069 from thorstenb/thb-buildconvenience-1

Automagically setup submodules on first run.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2066 from thorstenb/wip-janitorial-clang-2

[werror] Fix c++11-compat-reserved-user-defined-literal

Merge pull request #2062 from somnathr/wip-sd-replicated-pg

ReplicatedPG: Removed the redundant register_snapset_context call

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2064 from somnathr/wip-sd-share-map

OSD: adjust share_map() to handle the case that the osd is down

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2065 from somnathr/wip-sd-optracker

Pull request for optracker changes

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2063 from athanatos/wip-8691

OSD: wake_pg_waiters after dropping pg lock

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2047 from dachary/wip-osd-config

osd: improve tests for configuration updates

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

Merge pull request #2036 from ceph/wip-osd-map-cache-size

osd: allow osd map cache size to be adjusted at runtime

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2068 from thorstenb/wip-janitorial-clang-4

ceph_argparse_flag has no regular 3rd parameter.

Reviewed-by: Loic Dachary <loic@dachary.org>

qa: support running under non privileged user

If the test is run against a cluster started with vstart.sh (which is
the case for make check), the --asok-does-not-need-root disables the use
of sudo and allows the test to run without requiring privileged user
permissions.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 522174b066044e51a7019bd6cad81117e83c394e)

Merge pull request #2061 from ceph/wip-8728

qa/workunits/rest/test.py: make osd create test idempotent

Reviewed-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #2046 from ceph/wip-8670

mon: OSDMonitor: 'osd pool' - if we can set it, we must be able to get it

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Sage Weil <sage@redhat.com>

Automagically setup submodules on first run.

I mean, who's reading the README, anyway?

Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

ceph_argparse_flag has no regular 3rd parameter.

With clang warning: 'va_start' has undefined behavior with reference
types, noticing copy-paste mistake on ceph_argparse_flag.

Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

[werror] Fix c++11-compat-reserved-user-defined-literal

c++11 readiness, avoid clash with
http://www.stroustrup.com/C++11FAQ.html#UD-literals

Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

OSD: adjust share_map() to handle the case that the osd is down

The assert was hitting while OSd is waiting for becoming healthy
in handle_osd_map(). This can happen while io is going on and
OSDs are made down forcefully by say osd thrash command.
So, the fix could be instead of asserting just return from here.

Fixes: #8646
Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

mon: OSDMonitor: 'osd pool' - if we can set it, we must be able to get it

Add support to get the values for the following variables:
 - target_max_objects
 - target_max_bytes
 - cache_target_dirty_ratio
 - cache_target_full_ratio
 - cache_min_flush_age
 - cache_min_evict_age

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

ReplicatedPG: Removed the redundant register_snapset_context call

In the get_object_context(), the get_snapset_context is been called
and the register_snapset_context is already been invoked from there.
This call seems to be redundant.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

OpTracker: The optracker enabled/disabled check is added

The optracker enabled/disabled check is now added in the
mark_event(const string&) interface.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

OpTracker: use mark_event rather than _mark_event

mark_event() interfaced changed to accept time and it is default
to 'now'. The mark_event is the wrapper function of _mark_event()
and it has the check for optracking enabled/disabled as well.
The _mark_event() is now a private function.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

qa/workunits/rest/test.py: make osd create test idempotent

Avoid possibility that we create multiple OSDs do to retries by passing in
the optional uuid arg.  (A stray osd id will make the osd tell tests a
few lines down fail.)

Fixes: #8728
Signed-off-by: Sage Weil <sage@inktank.com>

mds: defer encoding/submitting log events to separate thread

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: use mutex to protect log segment list

prevent race between creating new log segment and trimming old
segment.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: add thread to encode/submit log events

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

osd: add dump_reservations asok command

Allow an admin to inspect pending and in-progress reservations.

Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

common/AsyncReserver: add dump()

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2055 from ceph/wip-8692

mon: clear osd request queue latency info on down or up

Reviewed-by: Samuel Just <sam.just@inktank.com>

mon: clear osd request queue latency info on down or up

It doesn't make sense to warn about slow requests on OSDs that are down;
clear those stats in the monitor when we see an OSD go down (or up).

Fixes: #8692
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2054 from ceph/wip-fs-cmds-oops

Apply some accidentally-dropped commits from the wip-fs-cmds branch to be a little friendlier and pass tests.

Reviewed-by: Greg Farnum <greg@inktank.com>

OSD: wake_pg_waiters after dropping pg lock

Otherwise, we dispatch_session_waiting while still holding the pg lock,
which is obviously wrong.  Unfortunately, this places an additional
burden on any user of _create_lock_pg, but I think it's unavoidable
since that method must atomically add the pg to the map and lock it.

Fixes: #8961
Introduced in:
  25466839589813047c975e44e67e14f34e32139e
  ecda2fef8ce982df3581a3b47ba74ae581d82479
Signed-off-by: Samuel Just <sam.just@inktank.com>

mds: Update default FS name

From 'default' to 'cephfs' so that it looks
at least vaguely meaningful.

Signed-off-by: John Spray <john.spray@redhat.com>