ceph-fuse: start up log on parent process before shutdown

Otherwise, we hit an assert in the Ceph context and logging teardown.

Fixes: http://tracker.ceph.com/issues/18157
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12338 from liewegas/wip-kraken-notes

doc/release-notes: kraken release notes (draft)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc/release-notes: kraken release notes (draft)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12340 from athanatos/wip-18127

osd/PGLog::proc_replica_log,merge_log: fix bound for last_update

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12339 from kylinstorage/wip-optimize-assert

msg/async: assert(0) -> ceph_abort()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12329 from ktdreyer/wip-doc-spelling

doc: fix grammar/spelling in RGW sections

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

msg/async/rdma/RDMAStack.cc: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

msg/async/rdma/Infiniband.h: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

msg/async/rdma/Infiniband.cc: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

Merge pull request #12125 from ukernel/wip-17990

mds: properly commit new dirfrag before splitting it

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12334 from Adirl/fix_log_level

msg/async/rdma: change log level: 0 -> 1

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: change log level: 0 -> 1

this will avoid irrelevant messages to console in case of "ceph -s"

issue: 914828

Change-Id: Ia220c62ff7a9f80fbe6068e1dd9d4a9a05ed01db
Signed-off-by: Avner BenHanoch <avnerb@mellanox.com>

msg/async/dpdk/TCP.h: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

msg/async/dpdk/DPDKStack.cc: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

msg/async/dpdk/DPDK.cc: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

common/map_cacher.hpp: assert(0) -> ceph_abort()

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

Merge pull request #9837 from zhjwpku/typos

doc: network-protocol typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

PGLog::proc_replica_log,merge_log: fix bound for last_update

If olog.tail > log.tail, olog.tail is also a bound on last_update.
See the changed comment in the commit for details.

Also adds unit tests.

Fixes: http://tracker.ceph.com/issues/18127
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12053 from theanalyst/v10.2.4/release-notes

doc: release notes for 10.2.4

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: fix capitalization in RGW admin ops

lower-case second word in this sentence.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: fix spelling in RGW s3 compliance

mentionned -> mentioned

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #12275 from xiexingguo/xxg-wip-bluestore-1202

os/bluestore: dedup omap_head, reuse nid instead

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12305 from athanatos/wip-ec-rbd-fixes

osd: fixes to make rbd on ec work

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12306 from athanatos/wip-ec-scrub

osd: fix ec scrub errors

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12212 from trociny/wip-16991

rbd-mirror: recovering after split-brain

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12307 from athanatos/wip-18042

msg/simple/Pipe::stop_and_wait: unlock pipe_lock for stop_fast_dispatching()

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: release notes: add the last 5 prs making it in 10.2.4

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #12310 from wjin/change_info

mailmap: change personal info

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12318 from tchaikov/wip-18128

test/ceph_crypto: do not read ceph.conf in global_init()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12317 from xiexingguo/xxg-wip-dedup-perfnotes

doc/dev/perf: fix dittography

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/ceph_crypto: do not read ceph.conf in global_init()

ForkDeathTest.MD5 expect an empty output while global_init() complains
if ceph.conf is missing if 0 is passed in as the `flags`. this test
passes if ceph.conf is in current working directory, but jenkins does
not prepare this file for this test.

Fixes: http://tracker.ceph.com/issues/18128
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: move empty-check of osr-queue out of loop

which is optimal.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc/dev/perf: fix dittography

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12311 from wjin/fix

src/common: remove nonused config option

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: kill deprecated get_preferred_csum_order() method

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: dedup omap_head, reuse nid instead

This is faster and save us some memory as well as space.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12260 from tchaikov/wip-dh-python

debian: apply dh_python to python-rgw also

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: change personal info

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

src/common: remove nonused config option

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

Merge pull request #12163 from ovh/wip-throttle-get-started-perf-counter

common: Add throttle_get_started perf counter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12285 from liewegas/wip-bluestore-allocator

os/bluestore: use bitmap allocator for bluefs

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #9714 from wido/mirror-docs-fix

doc: Small styling fix to mirror documentation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12299 from Adirl/fix_max_values

msg/async/rdma: set correct value to memory manager

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: set correct value to memory manager

set the correct max_send_wr and max_recv_wr
print log messages accordingly

Change-Id: I97a562bc994f14b87724fcf84baca4cdbf0f8e0d
Signed-off-by: Adir Lev <adirl@mellanox.com>

Merge pull request #12270 from yuyuyu101/wip-rdma-polling

msg/async/rdma: event polling thread can block on event

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #11987 from Liuchang0812/patch-2

doc: Update layout.rst, move commands to CODE block

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

ReplicatedPG: add return value to RepGather

Now that repops can represent an error, we need to remember
the error code so that we can reply to waiting_for_* messages
with the right value.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::record_write_error: send only the flags requested by the op

This isn't really important, but it's nice to be consistent with
OSD::reply_op_error.  Yet another reason to do away with the ack flag.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12297 from SUSE/wip-18134

tests: check hostname --fqdn sanity before running make check

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ReplicatedPG: add debugging to EINVAL returns

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: allow repops to apply with commit

Up to now, the repop machinery depended on all repops commiting and
applying in order.  For MOSDPGUpdateLogMissing operations, however,
we don't really want to send two messages in the ECBackend case.
Instead, just allow those repops to skip the applied stage and be
completed in order once the repops ahead of them finish.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: always send the ack with the commit

This was always a pretty silly concept and it has gotten in
the way of something else.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::submit_log_entries: fill in repop->v

This way, we can deal with waiting_for_commit appropriately.
Also, add more debugging to new_repop and already_*.

Signed-off-by: Samuel Just <sjust@redhat.com>

OSD::init_op_flags: don't consider PING to be a write

Signed-off-by: Samuel Just <sjust@redhat.com>

tests: check hostname --fqdn sanity before running make check

Fixes: http://tracker.ceph.com/issues/18134
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd-mirror: make 'rbd mirror image resync' work after split-brain

Fixes: http://tracker.ceph.com/issues/18051
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: split-brain issues should be clearly visible in mirror status

Fixed: http://tracker.ceph.com/issues/16991
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: test_status_in_pool_dir: explicitly check grep return value

Otherwise, it does not work as supposed to work in statements like below:

 set -e
 test_status_in_pool_dir ... && ...

(e.g. in wait_for_status_in_pool_dir)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12282 from liewegas/wip-bluestore-dump

os/bluestore: clean up Allocator::dump

Merge pull request #12287 from liewegas/wip-store-kv-stats

kv: dump rocksdb stats

ReplicatedPG::do_op: wait_for_commit even for results < 0 entries

This way, the replies come back in order and we don't expose
uncommitted results.

Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend: ops must consistently use or not use the cache

If the state is flipped, ops already in the pipeline need to
continue to maintain it.

Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend::try_state_to_reads: fix debug spacing

Signed-off-by: Samuel Just <sjust@redhat.com>

ECTransaction::get_write_plan: don't create empty to_read entries

Signed-off-by: Samuel Just <sjust@redhat.com>

ECTransaction: fix projected_size for delete->create in get_write_plan

Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend::try_state_to_reads: assert that rmw ops require hacky_overwrites enabled

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::_scrub_finish: check stats for ec overwrites pools

This should work fine now that we are doing an update-in-place
with rollback information off to the side.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: merge apply_ctx_scrub_stats into apply_stats

The only way the two aren't called essentially atomically is when the
ECBackend needs to delay a write for rmw.  In such a case, we actually
want to delay the scrub_cstat application as well since otherwise, we'd
have to flush the backend before settling up at the end of scrub.

Perhaps this should be a separate function to distinguish between the
replica+primary info update and the primary scrub_cstat update, but that
seems more complicated than it's really worth.

Fixes: http://tracker.ceph.com/issues/17999
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12284 from ceph/wip-cmake-args

build: dump env during build

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #12224 from yehudasa/wip-rgw-list-all

rgw/rgw_rados: do not omap_getvals with (u64)-1 max (updated)
also approve, passed teuthology (many false positives in several classes)

Merge pull request #12187 from Jing-Scott/dev/slave-master-sync-info

rgw: metadata sync info should be shown at master zone of slave zoneg…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: dump kv stats

Signed-off-by: Sage Weil <sage@redhat.com>

osd: dump kv stats for ObjectStores (if any)

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

kv/RocksDBStore: dump extended stats

Signed-off-by: Varada Kari <varada.kari@sandisk.com>
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

debian/rules: dump env during build

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: dump env during build

Signed-off-by: Sage Weil <sage@redhat.com>

kv: make kvdb can print itself statistics.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #12245 from ifed01/wip-fix-global-init-refcount

test/fio: fix global CephContext life cycle

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12241 from dachary/wip-17889-systemd-order

ceph-disk: enable --runtime ceph-osd systemd units

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.cz>

os/bluestore/BlueFS: default to bitmap allocator

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: dump allocator state on allocation failure

This is probably noisy for an *actual* ENOSPC, but the OSD should
actually prevent those from happening in general, and we are
very interested in fixing spurious allocation failures due to
allocator problems.  See http://tracker.ceph.com/issues/18054

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12075 from ceph/wip-rgw-zonegroup-config

rgw: fix for unsafe change of rgw_zonegroup

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12280 from trociny/wip-17127

rbd-mirror: set SEQUENTIAL and NOCACHE advise flags on image sync

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: dump allocator state at lowest debug level

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unused ostream arg to Allocator::dump

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12262 from cbodley/wip-17888

rgw: fix for versioned delete_multi_object
ftw

Merge pull request #12256 from liewegas/wip-18093

osd: fix memory leak from EC write workload

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12238 from gaurav36/bug-rbd-feature-enable-error

librbd: enabling/disabling rbd feature should report missing dependency

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12195 from trociny/wip-qa-rbd-wait_for_image_present

qa/workunits/rbd: use image id when probing for image presence

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11942 from wjwithagen/wip-wjw-freebsd-boost

do_freebsd.sh: Build with SYSTEM Boost on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

common: Add throttle_get_started perf counter

This change adds 'get_started' performance counter which
is increased at the beginning of Throttle::get() call,
before any potential wait is started. It's purpose is
to measure in realtime the number of jobs waiting for the
throttle (get_started - get) and get the characteristic of
get cals before throttler does it's job.

Signed-off-by: Bartłomiej Święcki <bartlomiej.swiecki@corp.ovh.com>

rbd-mirror: set SEQUENTIAL and NOCACHE advise flags on image sync

Fixes: http://tracker.ceph.com/issues/17127
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Cmake/FreeBSD: use the SYSTEM_BOOST

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #12209 from tchaikov/wip-enable-unittest_dns_resolve

test: enable unittest_dns_resolve

Reviewed-by: Loic Dachary <ldachary@redhat.com>

msg/async/rdma: make rdma dispatcher can sleep if inactive enough

Signed-off-by: Haomai Wang <haomai@xsky.com>

test: enable unittest_dns_resolve

guess this test was missed when we were moving to cmake. let's re-enable
it!

Signed-off-by: Kefu Chai <kchai@redhat.com>

PGTransaction,ReplicatedPG: clarify handling of noop operations

The offending transaction was [call rbd.copyup,delete] on a non-existent
object.  PGTransaction incorrectly ended up with Create and delete_first
causing a transaction beginning with trying to collection_move_rename a
non-existent head object.  In fact, if we delete an object which the
transaction currently claims to be creating, the transaction should show
as empty (for that object).  Rather than going through the normal write
pipeline for that case, let's just record a 0 error code in the log and
call it a day.  That way, the transaction generating code only needs to
worry about about updates to actual objects.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12263 from Adirl/fix_max_buffers

msg/async/rdma: don't use more buffers than what device capabilities …

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12140 from iain-buclaw-sociomantic:pybind-aioexec

python-rados: implement new aio_execute

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind: Fix comment for exec in Ioctx

Signed-off-by: Iain Buclaw <iain.buclaw@sociomantic.com>