Merge pull request #2932 from nilamdyuti/wip-doc-documenting-ceph

doc: Adds build-doc guidelines for Fedora and CentOS/RHEL.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Adds build-doc guidelines for Fedora and CentOS/RHEL.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Extended discussion for building docs on CentOS / RHEL.

Fixes: #7722
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added a script to build docs on CentOS / RHEL.

Fixes: #7722
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2098 from ceph/wip-dump-new-crush

crush: include CRUSH_V3, v2/v3 rules checks in dump_tunables()

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #2102 from ceph/wip-8823

ceph_test_rados_api_tier: fix [EC] HitSet{Read,Write,Trim} tests

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #2097 from sjahl/master

doc: Fixes a broken link on the rados deploy osd page.

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

doc/release-notes: some additional warnings and recommendations against adjusting tunables

Several users are having troubles related to the overhead of the data
rebalance from adjusting tunables.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_tier: fix [EC] HitSet{Read,Write,Trim} tests

The hit_set_ fields can only be set on tier pools as of
f131dfbaedf6f451572e7aa3a83f653912122953.

Fixes: #8823
Signed-off-by: Sage Weil <sage@inktank.com>

osd: fix confusing debug output for op_applied

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Fixes a broken link on the rados deploy osd page.

Signed-off-by: Stephen Jahl <stephenjahl@gmail.com>

Merge pull request #1999 from yuyuyu101/keyvaluestore-cache

Keyvaluestore cache

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2096 from ceph/wip-test-post-file

qa/workunits/post-file.sh

Reviewed-by: Greg Farnum <greg@inktank.com>

doc/release-notes: v0.80.3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/post-file.sh

Test ceph-post-file.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2090 from ceph/wip-nuke-dlist

nuke dlist

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2089 from lpabon/lpabon_readforward

Support for READFORWARD in the caching tier

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2093 from theanalyst/doc/format-fix

doc: minor format fix for radosgw admin docs

Merge pull request #2042 from ceph/wip-port-fixes

Portability updates

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2086 from ceph/wip-8696

mon: OSDMonitor: limit tier-specific pool set/get on non-tier pools

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2091 from ceph/wip-kinetic-os

prototype kinetic osd backend

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #2071 from somnathr/wip-sd-cpu-optimized

Wip sd cpu optimized

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2094 from ceph/wip-8815

mon: avoid scrub noise due to divering OSDMap encoding versions

Reviewed-by: Greg Farnum <greg@inktank.com>

osd/osd_types: be pedantic about encoding last_force_op_resend without feature bit

The addition of the value is completely backward compatible, but if the
mon feature bits don't match it can cause monitor scrub noice (due to the
parallel OSDMap encoding).  Avoid that by only adding the new field if the
feature (which was added 2 patches after the encoding, see
3152faf79f498a723ae0fe44301ccb21b15a96ab and
45e79a17a932192995f8328ae9f6e8a2a6348d10.

Fixes: #8815
Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove useless encode_compat local var for pg_pool_t::encode

This is left over from db3fd1152a87f8f152facdf141f8367296b371df
when we made the HitSet stuff incompat, but that behavior was reverted by
dc3ce58add130ca5a0a20b097b0ac934a1b486c7, leaving this cruft behind.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2088 from guangyy/wip-8733

Fix the PG listing issue which could miss objects for EC pool

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2092 from onlyjob/rbdmap

rbdmap: per-device mount (Closes: #8538)

Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits: cephtool: adjust pool name where missing as it has changed

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

crush: include CRUSH_V3, v2/v3 rules checks in dump_tunables()

Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

doc: minor format fix for radosgw admin docs

Missing backtick fixed

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

rbdmap: per-device mount (Closes: #8538)

`/etc/init.d/rbdmap start` was doing `mount -a`. Although (arguably)
`mount -a -O _netdev` could be less disruptive, it's not RBD mapping job to
mount unrelated devices and potentially do it at the wrong time.

Solution is to call `mount {device}` which works as expected and mounts
device even if it given in form `mount /dev/rbd/pool/imagename` while
`/etc/fstab` uses UUID or LABEL notation.

Furthermore this commit

 * fixes global exit code (it was always 0): now it is 0 only when
   all devices were (un)mounted successfully; otherwise non-zero.
 * replaces `mount -a` with per-device post-mapping `mount {dev}`
 * show mapping progress using LSB functions per device instead of for
   {start|stop} invocation.
 * capture output of `(u)mount` (if any) and report it as "info".

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

rbd.cc: Check io-size avoid floating point exception.

In func do_bench_write if io_size is zero,it can cause floating point execption.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

qa/workunits: cephtool: cleanup after pool creation

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: pool needs to be a tier to be used as such

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: test erroneous 'tier remove'

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: test get/set on both tier and non-tier pools

Make sure gets and sets of tiering-specific variables succeed on tier
pools and fail on non-tier pools.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: split get/set on tier pools from get/set tests

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: be scary about inconsistent pool tier ids

We may not crash your cluster, but you'll know that this is not something
that should have happened.  Big letters makes it obvious.  We'd make them
red too if we bothered to look for the ANSI code.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

osd: pg_pool_t: clear tunables on clear_tier()

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: limit tier-specific pool set/get on non-tier pools

Fixes: 8696
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

doc/changelog/v0.80.2: include tag

Signed-off-by: Sage Weil <sage@inktank.com>

os: add prototype KineticStore

Implement the KeyValueDB interface using libkinetic_client,
and allow it to be configured as the backend for the KeyValueStore,
running the entire OSD on it.

This prototype implementation has no transaction safety, and is
only suitable as a proof of concept. Since the libkinetic_client
API does not provide reverse iteration over keys without also reading
the value off disk, it implements iterators in a very slow but correct way.
These are used heavily by the KeyValueDB callers, so this is a bottleneck
in performance.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

PG::op_must_wait_for_map: pass the epoch rather than the map

have_same_or_newer_map is also modified to accept epoch instead of
map.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Merge branch 'master' of https://github.com/ceph/ceph

doc: Added CentOS/RHEL install for git.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

rgw: modelines

Signed-off-by: Sage Weil <sage@inktank.com>

rgw: fix RGWObjManifestRule decoder

Only decode the new field if it is a new struct.

Fixes: #8804
Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

doc: Added keyring location note to resolve pull abandoned pull request #1946.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

nuke dlist

No users.

Signed-off-by: Sage Weil <sage@inktank.com>

Add random_cache.hpp to Makefile.am

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test: fix make_pair() for c++11

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

Remove some not-resolving make_pair() invocations.

There's nothing wrong with make_pair(), but there is some type
resolution issue in these instances, at least with GCC 4.8.

Signed-off-by: Matt Benjamin <matt@linuxbox.com>

Add required spacing after string literals (PRIu64).

The common formatting for these 64bit types is not C++ 2011
clean.

Signed-off-by: Matt Benjamin <matt@linuxbox.com>
Conflicts:
src/common/perf_counters.cc

Convert ConnectionRef explicitly to bool.

Signed-off-by: Matt Benjamin <matt@linuxbox.com>

Work around an apparent binding bug (GCC 4.8).

A reference to h->seq passed to std::pair ostensibly could not bind
because the header structure is packed.  At first this looked like
a more general unaligned access problem, but the only location the
compiler rejects is a false positive.

Signed-off-by: Matt Benjamin <matt@linuxbox.com>

Fix the PG listing issue which could miss objects for EC pool (where there is object shard and generation).
Backport: firefly
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

doc: Clean up formatting, usage and removed duplicate section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc/release-notes: v0.80.2

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1905 from leseb/doc-rbd-icehouse

doc: rbd icehouse updated. Need to remove duplicate section.

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2087 from theanalyst/doc/typo-fix

doc: Fixing a few typos in docs

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

Revert "qa: add an fsx run which turns on kernel debugging"

This reverts commit 29c33f0c057acc4e0f4e5022c97553a2dc095b21.
We don't need the debugging any more, and having two separate fsx runners
already caused one update-in-the-wrong-place issue.

Signed-off-by: Greg Farnum <greg@inktank.com>

Revert "qa: add an fsx run which turns on kernel debugging"

This reverts commit 29c33f0c057acc4e0f4e5022c97553a2dc095b21.
We don't need the debugging any more, and having two separate fsx runners
already caused one update-in-the-wrong-place issue.

Signed-off-by: Greg Farnum <greg@inktank.com>

doc: fix a typo in quickstart doc

* doc/start/documenting-ceph.rst: s/along side/alongside

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: fix a few typos in radosgw docs

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: fix a few typos in rados docs

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: fix a few typos in the dev docs

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: fix a few typos in rbd docs

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: Fix a few typos in the install docs

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: fix a few typos in architecture page

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc: Fix a typo in the rbd man page

's/specifially/specifically'

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Use submit_transaction_sync to make change durable

submit_transaction will use asynchronous write operation, there is no journal
for keyvaluestore so we need to make it sync

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Add random cache and replace SharedLRU in KeyValueStore

SharedLRU plays pool performance in KeyValueStore with large header cache size,
so a performance optimized RandomCache could improve it.

RandomCache will record the lookup frequency of key. When evictint element,
it will randomly compare several elements's frequency and evict the least
one.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2029 from ceph/wip-8523

mon: OSDMonitor: add 'osd pool get-quota' command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2085 from ceph/wip-7891

osd: fix pg ref leaks on shutdown

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd: cancel agent_timer events on shutdown

We need to cancel all agent timer events on shutdown.  This also needs to
happen early so that any in-progress events will execute before we start
flushing and cleaning up PGs.

Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

osd: s/applying repop/canceling repop/

The 'applying' language dates back to when we would wait for acks from
replicas before applying writes locally.  We don't do any of that any more;
now, this loop just cancels the repops with remove_repop() and some other
cleanup.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: clear PGBackend state on shutdown

This was leaking state on shutdown whenever there were in-flight repops
that were canceled.

Fixes: #7891
Signed-off-by: Sage Weil <sage@inktank.com>

osd: separate cleanup from PGBackend::on_change()

The generic portion of on_change() cleaned up temporary on-disk objects
and requires a Transaction.  The rest is clearing out in-memory state and
does not.  Separate the two.

Signed-off-by: Sage Weil <sage@inktank.com>

Support for READFORWARD in the caching tier

READFORWARD caching policy uses writeback for writes
and redirects reads to the back end starage tier.

Signed-off-by: Luis Pabón <lpabon@redhat.com>

Merge remote-tracking branch 'gh/next'

qa/workunits: cephtool: test for 'osd pool {get,set}-quota'

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: OSDMonitor: add 'osd pool get-quota' command

Enables us to obtain current quotas for a given pool.

Fixes: #8523
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #2076 from cernceph/wip-wbthrottle-perfcounters

decrement WBThrottle perfcounters in clear_object

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2082 from ceph/wip-8727

messages: MForward: fix compat version

Reviewed-by: Sage Weil <sage@redhat.com>

messages: MForward: fix compat version

A while ago we bumped the head version and reset the compat version to 0.
Doing this so happens to make the messenger assume that the message does
not support the compat versioning and sets the compat version to the head
version -- thus making compat = 2 when it should have been 1.

The nasty side-effect of this is that upgrading from emperor to firefly
will have emperor-leaders being unable to decode forwarded messages from
firefly-peons.

Fixes: #8727
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #2081 from ceph/wip-osd-leaks

osd: fix several memory leaks from fast dispatch

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd: clear sessions_waiting_on_map on shutdown

Wake up these ops and drop the Session refs on shutdown.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: fix session leak when waiting on map

We don't want to leak the Session if this Session is already waiting on
a map.

Introduced by eb30f88c942d80cbc951d4d0f375479069d15624.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2037 from ceph/wip-async-log

MDS: Encode log events in a separate thread from main execution.

This is a first small step in multi-threading the MDS daemon.

Reviewed-by: Greg Farnum <greg@inktank.com>

osd: clear Sessions for loopback Connections on shutdown

Starting with the fast dispatch patches, we are calling the handle_connect
on loopback.  Make sure we zap them on shutdown to break the Session <->
Connection ref cycle.

Signed-off-by: Sage Weil <sage@inktank.com>

Revert "OSD: move waiting_for_pg into Session"

This reverts commit ecda2fef8ce982df3581a3b47ba74ae581d82479.

This leaves Session* refs indefinitely in the map.

This was one source of #7995.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2080 from ceph/wip-8306-rebase

Wip 8306 rebase

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

OSD: fix debug logging output

The snapmapper and infos statements were backwards. Now they aren't.

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #2078 from ceph/wip-8751

test: generalise default_pools in test_rados

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: add bash completion file for radosgw-admin

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

ceph.spec.in: rhel7-related changes:

udev rules: /lib -> /usr/lib
/sbin binaries move to /usr/sbin or %{_sbindir}

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Fix/add missing dependencies:

- rbd-fuse depends on librados2/librbd1
- ceph-devel depends on specific releases of libs and libcephfs_jni1
- librbd1 depends on librados2
- python-ceph does not depend on libcephfs1

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

ceph.spec.in: whitespace fixes

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

ceph.spec.in: split out ceph-common as in Debian

Move files, postun scriptlet, and add dependencies on ceph-common
where appropriate

Signed-off-by: Sandon Van Ness <sandon@inktank.com>