common/blkdev: use realpath instead of readlink to resolve the recursion symbol link

Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #12354 from umullan/correction

doc: correcting the object name

Merge pull request #12347 from gregsfortytwo/wip-18157-fix

ceph-fuse: start up log on parent process before shutdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12259 from liupan1111/wip-fix-nbd-max-part

rbd-nbd: support partition for rbd-nbd mapped raw block device

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12359 from tchaikov/wip-cmake

cmake: detect keyutils if WITH_LIBCEPHFS OR WITH_RBD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12361 from dachary/wip-18030-pglog

tests: remove TestPGLog EXPECT_DEATH tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12208 from ukernel/wip-18047

mds: don't access mdsmap from log submit thread

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12077 from jcsp/wip-17954

mds: fix dropping events in standby replay

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #12133 from jcsp/wip-16924

mds: don't maintain bloom filters in standby replay

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12360 from wido/docs-cephfs-datascan-workers

doc: Fix for worker arguments to cephfs-data-scan tool

Reviewed-by: John Spray <john.spray@redhat.com>

tests: remove TestPGLog EXPECT_DEATH tests

As per Sam Just advice, remove the EXPECT_DEATH tests to avoid
intermittent hang because they do not play well with threads.

Fixes: http://tracker.ceph.com/issues/18030
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #12327 from iain-buclaw-sociomantic/pybind-validate-args

pybind/rados: Add @requires for all aio methods

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix for worker arguments to cephfs-data-scan tool

The docs were incorrect and these arguments should work.

Signed-off-by: Wido den Hollander <wido@42on.com>

cmake: turn krbd into a static lib

* simplify the link dependencies.
* s/keyutils/${KEYUTILS_LIBRARIES}/

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: find keyutils if rbd or libcephfs is enabled

* find_package(keyutils REQUIRED) if (WITH_LIBCEPHFS OR WITH_RBD)
  prior to this change, we detect keyutils if the building platform is not
  FreeBSD, we should instead check the WITH_* options, and let the
  maintainer to decided what is the best for his/her platform, and error
  out if the building host cannot fulfill the requirement to build the
  asserts.
* build krbd.cc if (WITH_RBD)

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: correcting the object name
Signed-off-by: umullangi@walmartlabs.com

Merge pull request #12350 from chanyoung/struct-OSDOp-typos

osd: fix typos in "struct OSDOp" comments

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fix typos in "struct OSDOp" comments

"nembers" -> "members"

Signed-off-by: Chanyoung Park <park910113@gmail.com>

Merge pull request #12344 from trociny/wip-resync-fixup

rbd-mirror: fix gmock warnings in bootstrap request unit tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11369 from theanalyst/kraken-release-notes

doc: release notes for 11.0.2

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add salient features of kraken 11.1.0 rc

Reordering to add Sage's release note comments on top

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: add release notes for 11.1.0

Release notes as of 7b1293e70db51f620210332bde1f83ade15bd9d3

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: bump release notes to 11.0.2

bumping release notes to 11.0.2 kraken and adding the single commit that
differs between the changes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: release notes for 11.0.1

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #12346 from theanalyst/doc/changelog/v10.2.4

doc: adding changelog for v10.2.4

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-fuse: start up log on parent process before shutdown

Otherwise, we hit an assert in the Ceph context and logging teardown.

Fixes: http://tracker.ceph.com/issues/18157
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

doc: adding changelog for v10.2.4

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #12328 from liewegas/wip-bluestore-flush

os/bluestore: flush before enumerating omap values

Merge pull request #12265 from liewegas/wip-bluestore-precondition

os/bluestore: drop old bluestore preconditioning; replace with wal preextension of file size

Merge pull request #12281 from dachary/wip-17743-memstore

tests: do not use memstore.test_temp_dir in two tests

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #12298 from chhabaramesh/bit_alloc_mempool

os/bluestore: bitmap allocator dump functionality

Reviewed-by: Sage Weil <sage@redhat.com>

rbd-mirror: fix gmock warnings in bootstrap request unit tests

Fixes: http://tracker.ceph.com/issues/18156
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12338 from liewegas/wip-kraken-notes

doc/release-notes: kraken release notes (draft)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc/release-notes: kraken release notes (draft)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12340 from athanatos/wip-18127

osd/PGLog::proc_replica_log,merge_log: fix bound for last_update

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12339 from kylinstorage/wip-optimize-assert

msg/async: assert(0) -> ceph_abort()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12329 from ktdreyer/wip-doc-spelling

doc: fix grammar/spelling in RGW sections

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

msg/async/rdma/RDMAStack.cc: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

msg/async/rdma/Infiniband.h: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

msg/async/rdma/Infiniband.cc: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

rbd-nbd: support partition for rbd-nbd mapped raw block device.

Fixes: http://tracker.ceph.com/issues/18115
Signed-off-by: Pan Liu pan.liu@istuary.com

Merge pull request #12125 from ukernel/wip-17990

mds: properly commit new dirfrag before splitting it

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12334 from Adirl/fix_log_level

msg/async/rdma: change log level: 0 -> 1

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: change log level: 0 -> 1

this will avoid irrelevant messages to console in case of "ceph -s"

issue: 914828

Change-Id: Ia220c62ff7a9f80fbe6068e1dd9d4a9a05ed01db
Signed-off-by: Avner BenHanoch <avnerb@mellanox.com>

msg/async/dpdk/TCP.h: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

msg/async/dpdk/DPDKStack.cc: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

msg/async/dpdk/DPDK.cc: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

common/map_cacher.hpp: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

tests: do not use memstore.test_temp_dir in two tests

unittest_memstore_clone and test_objectstore_memstore.sh both
use memstore.test_temp_dir and when running in parallel they
interfere with each other.

src/test/objectstore/store_test_fixture.h will create

   memstore.test_temp_dir

when running via src/test/test_objectstore_memstore.sh

src/test/objectstore/test_memstore_clone.cc will also create

   memstore.test_temp_dir

when running via unittest_memstore_clone.

Fixes: http://tracker.ceph.com/issues/17743
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #9837 from zhjwpku/typos

doc: network-protocol typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

PGLog::proc_replica_log,merge_log: fix bound for last_update

If olog.tail > log.tail, olog.tail is also a bound on last_update.
See the changed comment in the commit for details.

Also adds unit tests.

Fixes: http://tracker.ceph.com/issues/18127
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12053 from theanalyst/v10.2.4/release-notes

doc: release notes for 10.2.4

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: fix capitalization in RGW admin ops

lower-case second word in this sentence.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: fix spelling in RGW s3 compliance

mentionned -> mentioned

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

os/bluestore: flush before enumerating omap values

For a few update ops we enumerate omap keys (to remove them).
Make sure we flush the pipeline before doing so or else we may
lose keys inserted by a previous transaction that hasn't yet
committed.

Fixes: http://tracker.ceph.com/issues/18140
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: pre-extend file size for WAL (.log) files

When rocksdb has log recycling on (this is required!), it will
do robust checksums on log records and change playback behavior
to tolerate trailing garbage in the log file.  This normally
allows it to overwrite previous log files, but it can also let
us overwrite arbitrary garbage on the device.

If we allocate some new space for a .log file (already indicated
by the WRITER_WAL hint), extend the size immediately so that each
subsequent append doesn't have to (unless/until we do another
allocation).

This is safe as long as rocksdb recycling is enabled (which it is
by default).

This is faster because we don't have to flush the bluefs log on
every log append during the period after startup before rocksdb
starts recycling log files.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove kludgey rocksdb preconditioning

This doesn't work, it's slow, and it's hacky.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/rados: Add @requires for all aio methods

Signed-off-by: Iain Buclaw <iain.buclaw@sociomantic.com>

Merge pull request #12275 from xiexingguo/xxg-wip-bluestore-1202

os/bluestore: dedup omap_head, reuse nid instead

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12305 from athanatos/wip-ec-rbd-fixes

osd: fixes to make rbd on ec work

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12306 from athanatos/wip-ec-scrub

osd: fix ec scrub errors

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12212 from trociny/wip-16991

rbd-mirror: recovering after split-brain

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12307 from athanatos/wip-18042

msg/simple/Pipe::stop_and_wait: unlock pipe_lock for stop_fast_dispatching()

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

bitmap allocator dump functionality

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

doc: release notes: add the last 5 prs making it in 10.2.4

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #12310 from wjin/change_info

mailmap: change personal info

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12318 from tchaikov/wip-18128

test/ceph_crypto: do not read ceph.conf in global_init()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12317 from xiexingguo/xxg-wip-dedup-perfnotes

doc/dev/perf: fix dittography

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/ceph_crypto: do not read ceph.conf in global_init()

ForkDeathTest.MD5 expect an empty output while global_init() complains
if ceph.conf is missing if 0 is passed in as the `flags`. this test
passes if ceph.conf is in current working directory, but jenkins does
not prepare this file for this test.

Fixes: http://tracker.ceph.com/issues/18128
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: move empty-check of osr-queue out of loop

which is optimal.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc/dev/perf: fix dittography

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12311 from wjin/fix

src/common: remove nonused config option

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: kill deprecated get_preferred_csum_order() method

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: dedup omap_head, reuse nid instead

This is faster and save us some memory as well as space.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12260 from tchaikov/wip-dh-python

debian: apply dh_python to python-rgw also

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: change personal info

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

src/common: remove nonused config option

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

Merge pull request #12163 from ovh/wip-throttle-get-started-perf-counter

common: Add throttle_get_started perf counter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12285 from liewegas/wip-bluestore-allocator

os/bluestore: use bitmap allocator for bluefs

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #9714 from wido/mirror-docs-fix

doc: Small styling fix to mirror documentation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12299 from Adirl/fix_max_values

msg/async/rdma: set correct value to memory manager

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: set correct value to memory manager

set the correct max_send_wr and max_recv_wr
print log messages accordingly

Change-Id: I97a562bc994f14b87724fcf84baca4cdbf0f8e0d
Signed-off-by: Adir Lev <adirl@mellanox.com>

Merge pull request #12270 from yuyuyu101/wip-rdma-polling

msg/async/rdma: event polling thread can block on event

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #11987 from Liuchang0812/patch-2

doc: Update layout.rst, move commands to CODE block

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

ReplicatedPG: add return value to RepGather

Now that repops can represent an error, we need to remember
the error code so that we can reply to waiting_for_* messages
with the right value.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::record_write_error: send only the flags requested by the op

This isn't really important, but it's nice to be consistent with
OSD::reply_op_error.  Yet another reason to do away with the ack flag.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12297 from SUSE/wip-18134

tests: check hostname --fqdn sanity before running make check

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ReplicatedPG: add debugging to EINVAL returns

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: allow repops to apply with commit

Up to now, the repop machinery depended on all repops commiting and
applying in order.  For MOSDPGUpdateLogMissing operations, however,
we don't really want to send two messages in the ECBackend case.
Instead, just allow those repops to skip the applied stage and be
completed in order once the repops ahead of them finish.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: always send the ack with the commit

This was always a pretty silly concept and it has gotten in
the way of something else.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::submit_log_entries: fill in repop->v

This way, we can deal with waiting_for_commit appropriately.
Also, add more debugging to new_repop and already_*.

Signed-off-by: Samuel Just <sjust@redhat.com>

OSD::init_op_flags: don't consider PING to be a write

Signed-off-by: Samuel Just <sjust@redhat.com>

tests: check hostname --fqdn sanity before running make check

Fixes: http://tracker.ceph.com/issues/18134
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd-mirror: make 'rbd mirror image resync' work after split-brain

Fixes: http://tracker.ceph.com/issues/18051
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: split-brain issues should be clearly visible in mirror status

Fixed: http://tracker.ceph.com/issues/16991
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: test_status_in_pool_dir: explicitly check grep return value

Otherwise, it does not work as supposed to work in statements like below:

 set -e
 test_status_in_pool_dir ... && ...

(e.g. in wait_for_status_in_pool_dir)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12282 from liewegas/wip-bluestore-dump

os/bluestore: clean up Allocator::dump

Merge pull request #12287 from liewegas/wip-store-kv-stats

kv: dump rocksdb stats

ReplicatedPG::do_op: wait_for_commit even for results < 0 entries

This way, the replies come back in order and we don't expose
uncommitted results.

Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend: ops must consistently use or not use the cache

If the state is flipped, ops already in the pipeline need to
continue to maintain it.

Signed-off-by: Samuel Just <sjust@redhat.com>