os: do not inject read EIO on meta pool objects

Avoid EIO on, say, osdmaps until we fix
http://tracker.ceph.com/issues/23029

Signed-off-by: Sage Weil <sage@redhat.com>

test_filestore_idempotent_sequence: put txn object in each collection

The transactions are idependent in each collection/sequencer, so we
can't record to a single txn object with racing transactions.  Fix
it by doing one in each collection, and when reading the latest op,
use the highest txn value we see.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix split ordering

We need to flush between split.  This requirement unfortunately
doesn't quite go away with the FileStore tracking.

Also, flush for each batch.  This is just because the test environment
may have a low open file ulimit.  (The old code did apply_transaction,
so it's functionally equivalent to this.)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: keep PG alive until the final onreadable context under old ch is done

The onreadable completions go through a finisher; add a final event
in that stream that keeps the PG alive while prior events flush.

flush() isn't quite sufficient since it doesn't wait for the finisher
events to flush too--only for the actual apply to have happened.

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: wait_for_apply on get_omap_iterator

Note that we don't need to worry about the internal get_omap_iterator
callrs (e.g., omap_rmkeyrange) because the apply thread does these
ops sequentially and in order.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: flush ch before pg delete

Make sure SnapMapper's ContainerContexts don't outlive the mapper
itself.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: fix SnapMapper vs import

- keep mapper around for duration of import
- flush in-flight requests before tearing it down.  This is necessary
because the mapper still uses onreadable.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: remove last_updated_applied, op_applied()

No longer used or needed, yay!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGBackend: drop on_applied callback for submit_transaction

This removes a ton of tracking for ReplicatedBackend.  ECBackend needs
to keep most of it so that it can track in-flight applies on legacy
peer OSDs.  We can remove this post-nautilus.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGBackend: remove on_applied() from interface

PrimaryLogPG calls it synchronously, on its own, after
submit_transaction.  That means the backends no longer need to
track it or call back to it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop scrub machinery to wait for last_updated_applied

This is no longer needed--queued writes are immediately readable.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: clean out applied members of RepGather

on_applied
rep_done (same as all_committed)
applies_with_commit (no longer relevant)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: drop repop_all_applied()

We don't need to eval_repop() since we know nothing has committed yet.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: remove obsolete comment in _delete_some()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ECBackend: unify commit+applied msg; drop applied callback

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedBackend: do applied work synchronously

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: call op_applied directly after queueing txn

There is still more cleanup to be done, here!  Later.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: stop holding PG ref until on_applied on PG removal

Not necessary--the backend should manage the lifecycle.  We just own
one ref anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make C_DeleteMore called only once (on safe, not applied)

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop C_UpdateLastRollbackInfoTrimmedToApplied

The variable can probably be removed as well, but leaving that for another
time.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGBackend: drop on_local_applied_async

Unused!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop unnecessary OSDMap pinning

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: remove ondisk_{read,write}_lock infra

This is no longer needed.  FileStore was the only backend doing async
applies, and it now blocks until apply all on its own.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: drop most queue_transactions() variants

Register completions on the Transaction(s) in the caller!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: synchronously open collection handles

bluestore and memstore are the only backends to implement
open_collection, and both of them can issue a handle immediately
after queue_transaction.  Do that!

Signed-off-by: Sage Weil <sage@redhat.com>

os: apply_transaction -> queue_transaction

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: keep OpSequencers alive

Prevent a collection delete + recreate sequence from allowing two
conflicting OpSequencers for the same collection to exist as this
can lead to racing async apply threads.

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: more efficient waiter tracking

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: wait_for_apply on read ops

On any read, wait for any updates to the object to apply first.

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: track objects with writes being applied

Note that this is *slight* overkill in that a *source* object of a clone
will also appear in the applying map, even though it is not being
modified.  Given that those clone operations are normally coupled with
another transaction that does write (which is why we are cloning in the
first place) this should not make any difference.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19939 from liewegas/wip-osd-new-class

mon,osd: do not use crush_device_class file to initalize class for new osds

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20363 from dongbula/e-rados-get-put-command

make rados get/put/append command help txt clear

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

tools/rados: make rados get/put/append command help txt clearer

Fixes: http://tracker.ceph.com/issues/22958
Signed-off-by: lvshuhua <lvshuhua@cmss.chinamobile.com>

Merge pull request #20392 from adamemerson/wip-clangtastic-voyage

cmake, test/fs, client: fix build with clang

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20379 from tchaikov/wip-test-fix-freebsd

test/librados: s/invoke_result_t/result_of_t/

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #20360 from ifed01/wip-ifed-extents-cleanup

os/bluestore: cleanup around ExtentList, AllocExtent and bluestore_extent_t classes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20320 from rzarzynski/wip-common-md_config_cacher_t

common: introduce md_config_cacher_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20198 from wido/zabbix-health-checks

mgr/zabbix: Implement health checks

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20378 from tchaikov/wip-os-perf-stat-in-ns

osd: check feature bits when encoding objectstore_perf_stat_t

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20275 from badone/wip-osdmonitor-unused-function

mon/OSDMonitor: Comment out unused function

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20371 from tchaikov/wip-admin_socket_output

test/admin_socket_output: add --vstart=path/to/asok option

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #20227 from rzarzynski/wip-bs-final-aiocontexts

os/bluestore: mark derivatives of AioContext as final.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20376 from dillaman/wip-22873

librbd: auto-remove trash snapshots when image is deleted

Reviewed-by: Mykola Golub <mgolub@suse.com>

build: Use the right experimental library

Test whether we're building with libstdc++ or libc++.

Use stdc++filesystem if the former and c++experimental if the later.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

{client,test}: Explicitly initialize empty inode_t

Using an explicit {} initializer. This unbreaks Clang.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

librbd: auto-remove trash snapshots when image is deleted

Fixes: http://tracker.ceph.com/issues/22873
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados: s/invoke_result_t/result_of_t/

to workaround buggy libc++ implementation. see
https://cplusplus.github.io/LWG/issue2807 . we should switch to
invoke_result_t once issue2807 is fixed in the latest release of
libc++.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: cleanup around ExtentList class.

Also move it out of bluestore_types.h.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: kill AllocExtent[Vector] to simplify code a bit

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20387 from ifed01/wip-ifed-fio-build-fix

test/fio: fix build failure caused by sequencer replacement.

Merge pull request #20043 from Rubab-Syed/configuration_setting_browser

mgr/dashboard: add configuration setting browser

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20389 from trociny/wip-cli_generic

qa/workunits/rbd: fix cli_generic test_purge for rbd default format 1

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd/cli_generic: fix test_purge for rbd default format 1

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20364 from Songweibin/wip-fix-test-rbd

test/rbd: cli_generic fails if v1 image format or deep-flatten disabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

tools/ceph-objectstore-tool: do not fail on lack of OI_ATTR for an
object.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/fio: create superblock on mkfs to provide proper tools functioning

E.g. ceph-objectstore-tool requires that

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/fio: fix build failure caused by sequencer replacement.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20372 from tchaikov/wip-mgr-doc

doc/mgr: remove non user-facing code from doc

Reviewed-by: John Spray <john.spray@redhat.com>

mgr/zabbix: Implement health checks in Zabbix module

Should make it easier for users to see why the module is failing.

Signed-off-by: Wido den Hollander <wido@42on.com>

test/rbd: cli_generic fails if v1 image format or deep-flatten disabled

Fixes: http://tracker.ceph.com/issues/22950
Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #20382 from tchaikov/wip-fix-ftbfs-store-test

test/store_test: fix FTBFS as Sequencer is removed

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

check feature bits when encoding objectstore_perf_stat_t

OS_PERF_STAT_NS feature bit is added. it's sharing the same bit with
QOS_DMC.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/store_test: fix FTBFS as Sequencer is removed

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20294 from rzarzynski/wip-bs-drop-std_function

os/bluestore: avoid overhead of std::function in blob_t.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19232 from socketpair/precision

mgr: increase time resolution of Commit/Apply OSD latencies.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18804 from majianpeng/bluestore-collection-prealloc

os/bluestore: Prealloc memory avoid realloc in list_collection.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18494 from ifed01/wip-stupidalloc-fix2

os/bluestore: do not assert if BlueFS rebalance is unable to allocate sufficient space

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18343 from shinobu-x/sk-remove-osdmap

mon/PGMap: Remove unnecessary header

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #19954 into master

* refs/pull/19954/head:
test/encoding: refactor to avoid escaping shell magic
mds: minor refactor of SimpleLock
mds: track Capability in mempool
mds: move CInode container members to mempool
mds: move CDentry container members to mempool
mds: move CDir container members to mempool
mds: put MDSCacheObject compact_map in mempool
common: use size_t for object size
mds: convert to allocator agnostic string_view
mds: simplify initialization
compact_*: support mempool allocated containers

Reviewed-by: Zheng Yan <zyan@redhat.com>

osd: PrimaryLogPG::do_osd_ops() uses md_config_cacher_t.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #20007 from mogeb/steady-clock-librbd

librbd: use steady clock to measure elapsed time in AioCompletion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20008 from mogeb/steady-clock-tools-rbd

tools/rbd: use steady clock in bencher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20218 from shun-s/wip-speedup-diskusage-resize

librbd: speed up object map disk usage and resize

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20311 from Songweibin/wip-group-snap-ls

rbd: do not show title if there is no group snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20349 from trociny/wip-22932

rbd-mirror: fix potential infinite loop when formatting status message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #20310 into master

* refs/pull/20310/head:
qa: adjust cephfs full test for kclient

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20148 into master

* refs/pull/20148/head:
mds: reset connection's priv when marking down connection
mds: fix session reference leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20155 into master

* refs/pull/20155/head:
osdc/Journaler: make sure flush() writes enough data

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20190 into master

* refs/pull/20190/head:
mon: allow removal of tier of ec overwritable pool

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #20200 into master

* refs/pull/20200/head:
client: add cap_dirtier_uid/gid to CapSnap

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #20246 into master

* refs/pull/20246/head:
mds: remove extra 0x in ino prints
mds: print inode number not CInode ptr

mon/OSDMonitor: accept crush_device_class in 'osd new' json input

If provided, set the OSD device_class at OSD creation time.  This is
simpler than writing a file that the OSD has to read in and use to
set its initial device class, and also avoids a bit of sticky state
at the OSD that will make it keep trying to reset its device class on
startup if it ever gets cleared.

Note that we now ignore json input fields we don't understand, so remove
a test case.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr: remove none user-facing code from doc

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/test_admin_socket_output: allow different asok directory

the asok files are not necessary located in ./out, actually, they are
put in $TMPDIR/ceph-asok.$random_stuff .

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/admin_socket_output: no need to create a copy of fs::path

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20367 from ceph/simple-custom-cluster

ceph-volume: adds custom cluster name support to simple

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: use a custom cluster name in simple functional tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20305 from xiexingguo/wip-more-balancer-fixes

pybind/mgr/balancer: more specific command outputs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20355 from luomuyao/typo

rgw: fix a typo in comment

Reviewed-by: Jos Collin <jcollin@redhat.com>

test/encoding: refactor to avoid escaping shell magic

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: minor refactor of SimpleLock

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: track Capability in mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: move CInode container members to mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore: avoid overhead of std::function in blob_t.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: fix a typo in comment

Signed-off-by: luomuyao <luo.muyao@zte.com.cn>

mon/OSDMonitor: Remove unused functions

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #20161 from tchaikov/wip-20086

test/librados: be more tolerant with timed lock tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20173 from liewegas/wip-os-ch

os: unify Sequencer and CollectionHandle

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: set conf.cluster name during simple activate

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20144 from adamemerson/wip-cache-register

Cache Register!

Provide the ability to examine and delete elements from the cache.

rgw: Add cache introspection and manipulation

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>