os/bluestore: fix narrow osr->flush() race

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13803 from trociny/wip-18785

rbd-mirror: separate ImageReplayer handling from Replayer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14406 from wjwithagen/wip-wjw-freebsd.readme

README.FreeBSD: Update the status

Reviewed-by: Kefu Chai <kchai@redhat.com>

README.FreeBSD: Update the status

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #14397 from liewegas/wip-19541

osd/PG: only correct filestore collection bits on load

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14382 from zealoussnow/wip-0407

messages: fix return type name of MOSDMap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14405 from wjwithagen/wip-wjw-gitignore

gitignore: Ignore rejects by patch

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14261 from liewegas/wip-15386

ceph_test_librados_api_misc: fix stupid LibRadosMiscConnectFailure.ConnectFailure test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14130 from liewegas/wip-json-slash

common/escape: do not escape / in json

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #14198 from liewegas/wip-fs

qa/suites: drop 'fs' facet, and add 'objectstore' facet where missing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14255 from liewegas/wip-19430

osd/PrimaryLogPG: do not expect FULL_TRY ops to get resent

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/cephtool/test.sh: handle non-escaped-/ json

Signed-off-by: Sage Weil <sage@redhat.com>

gitignore: Ignore rejects by patch

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #14386 from tchaikov/wip-19515

mon/OSDMonitor: update creating epoch if target osd changed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14227 from ivancich/wip-perf-counter-race

common/perf_counters: fix race condition with atomic variables

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14365 from tchaikov/wip-another-mgr-command

mgr: move "osd pool stats" to mgr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14370 from chardan/jfw-wip-simple_spinlock_atomic

common: Implements simple_spin_t in terms of std::atomic_flag.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14379 from tchaikov/wip-cmake-ctags

cmake: support optional argument for overriding default ctag excludes

Reviewed-by: Dan Mick <dmick@redhat.com>

tests: ceph_objectstore_tool.py: run vstart with MGR=1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14374 from liewegas/wip-mgr-init

mgr: fix bugs in init, beacons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14306 from liewegas/wip-bluestore-throttle-model

os/bluestore: unify throttling model

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

os/bluestore: make deferred_aggressive a counter

Multiple threads may run _osr_drain_preceding; use a count to
prevent the first finisher from prematurely disabling
aggressive mode.

Fixes: http://tracker.ceph.com/issues/19542
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14387 from wjwithagen/wip-wjw-test-osd-dup

test/osd/CMakeLists.txt: osd-dup.sh require BlueStore/AIO

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14348 from rzarzynski/wip-bs-bitmap-no-virts-in-hotspot

os/bluestore: avoid the VTABLE-related burden in BitMapAllocator's hotspot

Reviewed-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: send mgr beacons to leader

The leader has the last_beacon map that matters; make sure all
beacons go there.

At the same time drop the somewhat redundant checks in
preprocess_beaon and just send *everything* to prepare_beacon,
which is smart enough to do no update if there is none.

Fixes: http://tracker.ceph.com/issues/19502
Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror A/A: separate ImageReplayer handling from Replayer

Fixes: http://tracker.ceph.com/issues/18785
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror A/A: InstanceReplayer class to acquire and release images

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osd/PG: only correct filestore collection bits on load

PG::update_store_with_options() is called on load *and* when the pool
properties change, but if we only need to fix bits right at load time.
More importantly, but doing a second check on pool change, we may race
with a previously queued collection create that is not yet readable via
filestore, try to (synchronously) set the property again, and deadlock.

Fixes: http://tracker.ceph.com/issues/19541
Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: add set_remote_images method to ImageReplayer

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14366 from wjin/clean_mgr

mgr: remove unused function declarations

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/osd/CMakeLists.txt: osd-dup.sh require BlueStore/AIO

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

common: Implements simple_spin_t in terms of std::atomic_flag

Removes the requirement of machine-specific code in favor of the standard
library. See also PR #14337.

Signed-off-by: Jesse Williamson <jwilliamson@suse.de>

mon/OSDMonitor: update creating epoch if target osd changed

there is chance that the acting_primary of a PG being created changes
across osdmap updates, in that case we should update the new creating_pgs
with the mapping's epoch, not the stale one when the PG was originally
created. otherwise the new target OSD won't get the pg-create message.

Fixes: http://tracker.ceph.com/issues/19515
Signed-off-by: Kefu Chai <kchai@redhat.com>

messages: fix return type name of MOSDMap

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

Merge pull request #14029 from dachary/wip-osd-crushmap

mon: osd crush set crushmap need sanity check

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: support optional argument for overriding default ctag excludes

so instead of excluding all submodules, one can just exclude boost and
rocksdb using:

  cmake -DCTAG_EXCLUDES="boost;rocksdb" ..

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14334 from tchaikov/wip-cmake-ctags

cmake: Add simple recursive ctags target for Ceph source only

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #14320 from Adirl/poll_blocking

msg/async/rdma: Make poll_blocking() poll for async events in additio…

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14340 from oritwas/wip-rgw-fix-zone-create

radosgw-admin: use zone id when creating a zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14262 from guihecheng/rgw_file-fix-rmdir

rgw_file: fix missing unlock in unlink

Merge pull request #14216 from dongbula/add-multi-zonegroup-for-test

rgw: add --num-zonegroups option for multi test

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14354 from mogeb/cmake-lttng

cmake: build tracepoint libraries for vstart target

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/Mgr: unlock lock while blocking for metadata load

Fixes: http://tracker.ceph.com/issues/19516
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14115 from liewegas/wip-bluestore-metadata

os/bluestore: implement collect_metadata

Merge pull request #14304 from rchagam/18576_bug_fix

tracing: fix segv

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14328 from liewegas/wip-mgrmap-features

mgr: fix several init/re-init bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14333 from liewegas/wip-bluestore-misc

os/bluestore: misc fixes

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

msg/async/rdma: Make poll_blocking() poll for async events in addition to tx/rx

Issue: 995322
Change-Id: I62e75f4c4e574ba97e24b20a4d7b94c56c605bb5
Signed-off-by: Amir Vadai <amir@vadai.me>

Merge pull request #14210 from liewegas/wip-osd-dup

os: allow offline conversion of filestore -> bluestore (or anything else)

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #12175 from dongbula/fix-slavezonegroup-cannot-enable-bucket-versioning

rgw: fix slave zonegroup cannot enable the bucket versioning

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14342 from theanalyst/cleanup/rgw/rgw-lc-inc

rgw_lc: drop a bunch of unused headers

Merge pull request #13622 from zhangsw/feature-lifecycle-multipart

rgw: add support for multipart upload expiration.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #14337 from smithfarm/wip-19492

common: support s390 and unknown architectures in spin-wait loop

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14339 from haklein/master

add RGW ldap auth documentation

cmake: build tracepoint libraries for vstart target

Avoid a segfault in TracepointProvider::verify_config() when building
for the 'vstart' target. TracepointProvider tries to load the tracepoint
providers libraries at runtime, but these aren't built as part of the
'vstart' target.

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

mgr: clean nonused function declaration

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

mgr: move "osd pool stats" to mgr

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: support s390 and unknown architectures in spin-wait loop

This fixes the s390 build which was broken by 34481768cbce0991bfba1d511dad81893a9004a9

It also reinstates the previous architecture-independent for-loop
implementation as a catch-all for "unknown" (not-explicitly-supported)
architectures.

Fixes: http://tracker.ceph.com/issues/19492
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #14240 from dillaman/wip-rbd-mirror-peer-uuids

rbd-mirror: pool watcher should track mirror uuid

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14293 from badone/wip-ceph-disk-list-selinux

ceph-disk: Populate mount options when running "list"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cmake: Add simple recursive ctags target for Ceph source only

Excludes submodules and random JavaScript.

Signed-off-by: Dan Mick <dan.mick@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14032 from joscollin/wip-18511-returntype

librbd: changed the return type of ImageRequestWQ::discard()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13957 from wangzhengyong/import

rbd: error out if import image format failed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13836 from gaurav36/bug-14535-rbd-snap-pool-name-validatation

rbd: validate pool and snap name optionals

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14282 from wjwithagen/wip-wjw-freebsd-ceph-fuse

build: let FreeBSD build ceph-fuse

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14308 from badone/SyntheticClient_random_walk_warning

client/SyntheticClient.cc: Fix warning in random_walk

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10146 from yonghengdexin735/wip-zzz-filestore-write

os/filestore: handle error returned from write_fd()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13815 from theanalyst/cleanup/rgw/rest-read-input

rgw: don't read all user input for a few param requests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon/MgrMonitor: make debug prefix useful

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: don't use virtual for accessing BitMapZone::is_exhausted.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

os/bluestore: do not wake kv thread if only deferred events pending

No need to wake up if there is only deferred work; we can do
it lazily.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bluestore_deferred_batch_ops = 32 (not 8)

A bit more batching.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: unify throttling model

Implement a super simple model for the cost of a transaction
for the purposes of throttling.  This replaces two independent
throttles (one for transaction ops, one for bytes) and puts
them both under the 'bytes' throttle.  The txc model cost is
expressed in terms of bytes as this is probably the simplest
thing for users to reason about.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: release throttle before commit

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: error via derr too

This helps establish timing relative to the log.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: increase aio queue depth

We've seen some reports of the aio EAGAIN backoff in logs; use a larger
aio ring buffer to reduce the risk of this.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: reduce some debug level

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: put fsck structures in mempools

This lets us track them individually in debug mode.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: do not skip first 2 mempools

This broke when we sorted the pool definitions in the header.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: flush osr even on failure in _deferred_replay

This is one less thing for fsck to worry about.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: trim cache during fsck

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: be noisy about bitmap init (it's slow!)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14316 from liewegas/wip-bluestore-fsck-unused

os/bluestore: fsck: verify blob.unused field

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use typedef for blob unused (uint16_t)

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: verify blob.unused field

Two checks:

- verify that no logical extents reference the portion of the
  blob marked unused, and
- verify that the csum (if present) is zero for any unused
  region.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14157 from liewegas/wip-19379

os/bluestore: fix deferred writes vs collection split race

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14258 from liewegas/wip-bluestore-tool

ceph-bluestore-tool: rename from bluefs-tool; improve usage

Merge pull request #14294 from tchaikov/wip-check-last-osd-report-if-luminous

mon/OSDMonitor: check last_osd_report only when the whole cluster is lu…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14303 from liewegas/wip-more-mgr-commands

mgr: move 'osd perf' and 'osd blocked-by' to mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14010 from zhangsw/fix-rgw-fetchremote-bug

rgw: version id doesn't work in fetch_remote_obj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13310 from ceph/wip-ceph-disk-fix

ceph-disk: Add fix subcommand

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw_lc: drop a bunch of unused headers

Dropping headers that are already included as a part of rgw_common

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: add support for multipart upload expiration.

Fixes: http://tracker.ceph.com/issues/19088
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

radosgw-admin: use zone id when creating a zone

Fixes: http://tracker.ceph.com/issues/19498
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

add RGW ldap auth documentation

Signed-off-by: Harald Klein <hari@vt100.at>

Merge pull request #14314 from rzarzynski/wip-rgw-bulkupload-freebsd

rgw: switch to std::array in RGWBulkUploadOp due to C++11 and FreeBSD.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14252 from cbodley/wip-19409

radosgw-admin: new 'global quota' commands update period config
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14109 from cbodley/wip-rgw-stat-manifest

rgw: stat requests skip compression, manifest handling, etc
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore/KernelDevice: assume rotational if we can't get devname

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14281 from tchaikov/wip-19429

qa/tasks/workunit.py: use "overrides" as the default settings of workunit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>