check feature bits when encoding objectstore_perf_stat_t

OS_PERF_STAT_NS feature bit is added. it's sharing the same bit with
QOS_DMC.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20294 from rzarzynski/wip-bs-drop-std_function

os/bluestore: avoid overhead of std::function in blob_t.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19232 from socketpair/precision

mgr: increase time resolution of Commit/Apply OSD latencies.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18804 from majianpeng/bluestore-collection-prealloc

os/bluestore: Prealloc memory avoid realloc in list_collection.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18494 from ifed01/wip-stupidalloc-fix2

os/bluestore: do not assert if BlueFS rebalance is unable to allocate sufficient space

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18343 from shinobu-x/sk-remove-osdmap

mon/PGMap: Remove unnecessary header

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #19954 into master

* refs/pull/19954/head:
test/encoding: refactor to avoid escaping shell magic
mds: minor refactor of SimpleLock
mds: track Capability in mempool
mds: move CInode container members to mempool
mds: move CDentry container members to mempool
mds: move CDir container members to mempool
mds: put MDSCacheObject compact_map in mempool
common: use size_t for object size
mds: convert to allocator agnostic string_view
mds: simplify initialization
compact_*: support mempool allocated containers

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #20007 from mogeb/steady-clock-librbd

librbd: use steady clock to measure elapsed time in AioCompletion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20008 from mogeb/steady-clock-tools-rbd

tools/rbd: use steady clock in bencher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20218 from shun-s/wip-speedup-diskusage-resize

librbd: speed up object map disk usage and resize

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20311 from Songweibin/wip-group-snap-ls

rbd: do not show title if there is no group snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20349 from trociny/wip-22932

rbd-mirror: fix potential infinite loop when formatting status message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #20310 into master

* refs/pull/20310/head:
qa: adjust cephfs full test for kclient

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20148 into master

* refs/pull/20148/head:
mds: reset connection's priv when marking down connection
mds: fix session reference leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20155 into master

* refs/pull/20155/head:
osdc/Journaler: make sure flush() writes enough data

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20190 into master

* refs/pull/20190/head:
mon: allow removal of tier of ec overwritable pool

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #20200 into master

* refs/pull/20200/head:
client: add cap_dirtier_uid/gid to CapSnap

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #20246 into master

* refs/pull/20246/head:
mds: remove extra 0x in ino prints
mds: print inode number not CInode ptr

Merge pull request #20367 from ceph/simple-custom-cluster

ceph-volume: adds custom cluster name support to simple

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: use a custom cluster name in simple functional tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20305 from xiexingguo/wip-more-balancer-fixes

pybind/mgr/balancer: more specific command outputs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20355 from luomuyao/typo

rgw: fix a typo in comment

Reviewed-by: Jos Collin <jcollin@redhat.com>

test/encoding: refactor to avoid escaping shell magic

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: minor refactor of SimpleLock

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: track Capability in mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: move CInode container members to mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore: avoid overhead of std::function in blob_t.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: fix a typo in comment

Signed-off-by: luomuyao <luo.muyao@zte.com.cn>

Merge pull request #20161 from tchaikov/wip-20086

test/librados: be more tolerant with timed lock tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20173 from liewegas/wip-os-ch

os: unify Sequencer and CollectionHandle

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: set conf.cluster name during simple activate

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20144 from adamemerson/wip-cache-register

Cache Register!

Provide the ability to examine and delete elements from the cache.

rgw: Add cache introspection and manipulation

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20176 from dillaman/wip-librbd-clone-v2-1

librbd: initial hooks for clone v2 support

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: Add cache introspection and manipulation

Provide the ability to examine and delete elements from the cache.

Fixes: http://tracker.ceph.com/issues/22603
Fixes: http://tracker.ceph.com/issues/22604
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd-mirror: fix potential infinite loop when formatting status message

The improvements include:

- tag_tid values should always be increasing, so loop only if
  master.tag_tid > mirror_tag_tid in calculate_behind_master_or_send_update;
- in send_update_tag_cache don't refetch a tag if it is already in the
  cache;
- make fake tags with tag_data.predecessor.tag_tid set to zero;
- make sure the new tag is inserted to the cache if an old entry
  with this id happens to exist.

Fixes: http://tracker.ceph.com/issues/22932
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20318 from ovh/bp-limit-normalize-key

common/config: limit calls to normalize_key_name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19958 from renhwztetecs/renhw-wip-osd-sighup

osd: Sign in early SIGHUP signal

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: move CDentry container members to mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: move CDir container members to mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: put MDSCacheObject compact_map in mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: use size_t for object size

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: convert to allocator agnostic string_view

This is necessary to allow many interfaces to take mempool allocated strings.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20264 from ceph/wip-rm22620

ceph-volume dmcrypt support for simple

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20307 from tchaikov/wip-std-filesystem

test/admin_socket_output: switch to std::experimental::filesystem

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #20324 from tchaikov/wip-cmake-build-boost

cmake: fix the include dir for building boost::python

Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

pybind/mgr/balancer: cancel plan if distribution is already perfect

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind/mgr/balancer: allow customized optimization threshold

In practice it is hard to achieve a 100% (perfect) balanced distribution.
This patch introduces a user visiable threshold configuration here, which
allow user to define the minimal deviation to trigger an optimization.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind/mgr/balancer: be more specific on creating an optimization plan

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20347 from Songweibin/wip-osd-cleanup

osd: drop redundant comment

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: drop redundant comment

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge PR #20288 into master

* refs/pull/20288/head:
ptl-tool: open githubmap with utf-8 code

Reviewed-by: Jos Collin <jcollin@redhat.com>

mds: remove extra 0x in ino prints

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/objectstore/TestObjectStoreState: make objects match collection

- start pool at 1
- make hash fall within pg (just match it for now)

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: remove void *osr from Transaction

No longer used!  (It was only needed for an ugly FileStore hack.)

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: do not rely on ObjectStore::Transaction::osr

This was used only to name the osr for the purposes of the tracepoint. We
can do that ourselves!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop explicit calls to osr->flush

These are no longer needed as the implementations do it themselves.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: drop unnecessary flush() on shutdown

The ObjectStore implementation should handle this.

Signed-off-by: Sage Weil <sage@redhat.com>

os: make implementation flush before collection_{list,empty}

This takes the (awkward) burden off the caller.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: make all read operations CollectionHandle-based

Drop the coll_t shims, and convert the last few that weren't already
handle-based.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20313 from guzhongyan/fix-22350

osd/OSDMap: misleading message in print_oneline_summary()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20317 from liewegas/wip-pg-scrub-priority

osd/PG: pass scrub priority to replica

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20319 from tchaikov/wip-ops_blocked_by_scrub-cleanup

osd: pass ops_blocked_by_scrub() to requeue_scrub()

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: do not assert on insufficient allocation during bluefs
rebalance

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/store_test: add test case to for excessive fragmentation

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: fix lack of extent demotion in StupidAllocator::init_rm_free

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

ceph-volume: do not test custom cluster names with simple

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20279 from liewegas/wip-idempotent-sequence

ceph_test_filestore_idempotent_sequence: many fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: sleep 2 minutes after reboot in simple tests

This will allow for some time for OSD services to start back up after
reboot. If not we see race conditions where tests fail because they
were run before the OSD finished startup.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

os/ObjectStore: combine Sequencer and CollectionImpl in ObjectStore

In practice we have a sequencer per collection. Combine them in the interface
to simplify our lives!

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore_workloadgen: remove

Stale test code, never run.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/recovery_bench: drop recovery_bench task

This relies on smalliobench, which is gone now.  The task could
pretty easily be resurrected if we substitute in another tool.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_xattr_bench: drop

Signed-off-by: Sage Weil <sage@redhat.com>

test/bench: remove smallio bench tools

These are stale and superceded by the fio ObjectStore (and other)
backends.  Shed the weight!

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: drop unused variable from flatten state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20267 from liewegas/wip-22847

mon/OSDMonitor: filter out pgs that shouldn't exist from force-create-pg

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20259 from guzhongyan/rados-cleanup

rados: clean up rados_getxattrs() and rados_striper_getxattrs()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20260 from guzhongyan/fix-22042

librados: invalid free() in rados_getxattrs_next()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20255 from david-z/wip-fix-kv-max-option

config: Change bluestore_cache_kv_max to type INT64

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20301 from bigclouds/wip-fix-declaration-of-rados-striper-trunc

libradosstriper: fix the function declaration of rados_striper_trunc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20233 from cbodley/wip-test-log-nullptr

test/log: fix for crash with libc++

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20223 from Yan-waller/wip-walle-180124cephosd

osd: use existing osd_required variable for messenger policy

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20201 from majianpeng/mosdrepop-encode-log

osd: avoid encoding the same log entries repeatedly for different peers

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: speed up diskusage and resize

  speed up diskusage and resize

Signed-off-by: shun-s <song.shun3@zte.com.cn>

os/filestore: add replay guards for omap operations

These were not guarded, meaning they could get replayed onto an object
that was cloned or renamed.  Eek!

Fixes: http://tracker.ceph.com/issues/22920
Signed-off-by: Sage Weil <sage@redhat.com>

test/objectstore/run_seed_to_range: stop on first error

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_idempotent_sequence: use coll_entry_t

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_filestore_idempotent_sequence: fix up clone, move ops; cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/filestore_idempotent: run failure tests later in sequence

We want to get past the initial colleciton and object creation steps,
which don't include interesting ops.

Signed-off-by: Sage Weil <sage@redhat.com>

cls/rbd: store original namespace type within trash snapshot namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20016 from cbodley/wip-18885

rgw: log unlink_instance mtime as object's mtime

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

ceph-volume: adds the simple dmcrypt_plain and dmcrypt_luks tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix ceph-volume simple scan help menu test

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20145 from linuxbox2/rgwf-deref-readdir-off

rgw_file: avoid evaluating nullptr for readdir offset

Merge pull request #20188 from cbodley/wip-rgw-frontend-multimap

rgw: beast frontend can listen on multiple endpoints

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #20013 from cbodley/wip-rgw-auth-skew

rgw: share time skew check between v2 and v4 auth

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #20017 from cbodley/wip-22248

rgw: unlink deleted bucket from bucket's owner

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20120 from cbodley/wip-22797

rgw: fix for empty query string in beast frontend

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #20038 from oritwas/wip-rgw-22742

rgw: bucket resharding should not update bucket ACL or user stats
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20295 from trociny/wip-test_deep_copy_clone

test/pybind/rbd: skip test_deep_copy_clone if layering not enabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: tweak handling for moving snapshots to trash namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>