Merge pull request #20161 from tchaikov/wip-20086

test/librados: be more tolerant with timed lock tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20173 from liewegas/wip-os-ch

os: unify Sequencer and CollectionHandle

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20144 from adamemerson/wip-cache-register

Cache Register!

Provide the ability to examine and delete elements from the cache.

rgw: Add cache introspection and manipulation

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20176 from dillaman/wip-librbd-clone-v2-1

librbd: initial hooks for clone v2 support

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: Add cache introspection and manipulation

Provide the ability to examine and delete elements from the cache.

Fixes: http://tracker.ceph.com/issues/22603
Fixes: http://tracker.ceph.com/issues/22604
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20318 from ovh/bp-limit-normalize-key

common/config: limit calls to normalize_key_name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19958 from renhwztetecs/renhw-wip-osd-sighup

osd: Sign in early SIGHUP signal

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20264 from ceph/wip-rm22620

ceph-volume dmcrypt support for simple

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20307 from tchaikov/wip-std-filesystem

test/admin_socket_output: switch to std::experimental::filesystem

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #20324 from tchaikov/wip-cmake-build-boost

cmake: fix the include dir for building boost::python

Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #20347 from Songweibin/wip-osd-cleanup

osd: drop redundant comment

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: drop redundant comment

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge PR #20288 into master

* refs/pull/20288/head:
ptl-tool: open githubmap with utf-8 code

Reviewed-by: Jos Collin <jcollin@redhat.com>

test/objectstore/TestObjectStoreState: make objects match collection

- start pool at 1
- make hash fall within pg (just match it for now)

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: remove void *osr from Transaction

No longer used!  (It was only needed for an ugly FileStore hack.)

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: do not rely on ObjectStore::Transaction::osr

This was used only to name the osr for the purposes of the tracepoint. We
can do that ourselves!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop explicit calls to osr->flush

These are no longer needed as the implementations do it themselves.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: drop unnecessary flush() on shutdown

The ObjectStore implementation should handle this.

Signed-off-by: Sage Weil <sage@redhat.com>

os: make implementation flush before collection_{list,empty}

This takes the (awkward) burden off the caller.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: make all read operations CollectionHandle-based

Drop the coll_t shims, and convert the last few that weren't already
handle-based.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20313 from guzhongyan/fix-22350

osd/OSDMap: misleading message in print_oneline_summary()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20317 from liewegas/wip-pg-scrub-priority

osd/PG: pass scrub priority to replica

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20319 from tchaikov/wip-ops_blocked_by_scrub-cleanup

osd: pass ops_blocked_by_scrub() to requeue_scrub()

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume: do not test custom cluster names with simple

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20279 from liewegas/wip-idempotent-sequence

ceph_test_filestore_idempotent_sequence: many fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: sleep 2 minutes after reboot in simple tests

This will allow for some time for OSD services to start back up after
reboot. If not we see race conditions where tests fail because they
were run before the OSD finished startup.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

os/ObjectStore: combine Sequencer and CollectionImpl in ObjectStore

In practice we have a sequencer per collection. Combine them in the interface
to simplify our lives!

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore_workloadgen: remove

Stale test code, never run.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/recovery_bench: drop recovery_bench task

This relies on smalliobench, which is gone now.  The task could
pretty easily be resurrected if we substitute in another tool.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_xattr_bench: drop

Signed-off-by: Sage Weil <sage@redhat.com>

test/bench: remove smallio bench tools

These are stale and superceded by the fio ObjectStore (and other)
backends.  Shed the weight!

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: drop unused variable from flatten state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20267 from liewegas/wip-22847

mon/OSDMonitor: filter out pgs that shouldn't exist from force-create-pg

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20259 from guzhongyan/rados-cleanup

rados: clean up rados_getxattrs() and rados_striper_getxattrs()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20260 from guzhongyan/fix-22042

librados: invalid free() in rados_getxattrs_next()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20255 from david-z/wip-fix-kv-max-option

config: Change bluestore_cache_kv_max to type INT64

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20301 from bigclouds/wip-fix-declaration-of-rados-striper-trunc

libradosstriper: fix the function declaration of rados_striper_trunc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20233 from cbodley/wip-test-log-nullptr

test/log: fix for crash with libc++

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20223 from Yan-waller/wip-walle-180124cephosd

osd: use existing osd_required variable for messenger policy

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20201 from majianpeng/mosdrepop-encode-log

osd: avoid encoding the same log entries repeatedly for different peers

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/filestore: add replay guards for omap operations

These were not guarded, meaning they could get replayed onto an object
that was cloned or renamed.  Eek!

Fixes: http://tracker.ceph.com/issues/22920
Signed-off-by: Sage Weil <sage@redhat.com>

test/objectstore/run_seed_to_range: stop on first error

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_idempotent_sequence: use coll_entry_t

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_filestore_idempotent_sequence: fix up clone, move ops; cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/filestore_idempotent: run failure tests later in sequence

We want to get past the initial colleciton and object creation steps,
which don't include interesting ops.

Signed-off-by: Sage Weil <sage@redhat.com>

cls/rbd: store original namespace type within trash snapshot namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20016 from cbodley/wip-18885

rgw: log unlink_instance mtime as object's mtime

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

ceph-volume: adds the simple dmcrypt_plain and dmcrypt_luks tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix ceph-volume simple scan help menu test

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20145 from linuxbox2/rgwf-deref-readdir-off

rgw_file: avoid evaluating nullptr for readdir offset

Merge pull request #20188 from cbodley/wip-rgw-frontend-multimap

rgw: beast frontend can listen on multiple endpoints

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #20013 from cbodley/wip-rgw-auth-skew

rgw: share time skew check between v2 and v4 auth

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #20017 from cbodley/wip-22248

rgw: unlink deleted bucket from bucket's owner

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20120 from cbodley/wip-22797

rgw: fix for empty query string in beast frontend

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #20038 from oritwas/wip-rgw-22742

rgw: bucket resharding should not update bucket ACL or user stats
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20295 from trociny/wip-test_deep_copy_clone

test/pybind/rbd: skip test_deep_copy_clone if layering not enabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: tweak handling for moving snapshots to trash namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cmake: fix the include dir for building boost::python

the "includes" directories are separated by spaces not semicolons.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rbd: force use of v1 clone when testing orphan

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: image deleter now handles clone v2

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd/OSDCap: rbd-read-only profile now permits use of clone class methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: fix negative test cases for clone

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: snapshot remove request now handles clone v2

The state machine was converted to the current style and the
final removal of the snapshot was delayed until the end to
permit the retry of a previously failed snapshot removal.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: flatten request now handles clone v2

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: flatten operation should detach child first

That will permit the flatten operation to be replayed if there
is an issue.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove request now handles clone v2

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper state machine for detaching children for v1/v2 clones

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: list children now includes v2 clones

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: clone request now handles clone v2

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: removed child cleanup from failed clone

This logic is already included in the remove image state machine
and will be executed when the clone is deleted post-failure.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: added 'get_min_compatible_client' API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/options: added override for RBD clone format handling

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: removing an in-use snapshot should move it to the trash

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: new cls methods for handling clone v2

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: add child reference counter to snapshot

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: restrict snapshot rename to user-generated snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: helper method for iterating over all image snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: fix incorrect error log upon success

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: resolve names of snapshot namespaces

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new trash snapshot namespace

Snapshots will be moved to the trash if they are still in-use
by clones when deleted.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: separate clone v2 op feature bit for parent/child

librbd needs to be able to handle the case when flattening a v2 clone
that is also a v2 parent of other images.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20309 from pritha-srivastava/wip-rgw-roles-error-codes

rgw: Reinstating error codes mapping for Roles.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20316 from smithfarm/wip-doc-log

doc: log-and-debug: fix default value of "log max recent"

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: pass ops_blocked_by_scrub() to requeue_scrub()

less nesting levels, and less repeatings.

Signed-off-by: Kefu Chai <kchai@redhat.com>

 common/config: limit calls to normalize_key_name

It's not the most expensive thing in that module, but it's redundant
in _get_val, as it's later done also in _get_val_generic. And it's
still expensive.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

osd/PG: pass scrub priority to replica

If we are scrubbing with high priority on the primary, pass that along
to the replica so that it can schedule its scrub work accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: log-and-debug: fix default value of "log max recent"

Doc says 1000000, but the real figure is two orders of magnitude less:

    Option("log_max_recent", Option::TYPE_INT, Option::LEVEL_ADVANCED)
      .set_default(500)
      .set_daemon_default(10000)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20229 from zhsj/fix-dashboard-prefix

pybind/mgr/dashboard: fix duplicated slash in html href

Reviewed-by: John Spray <john.spray@redhat.com>

osd/OSDMap: misleading message in print_oneline_summary()
The output scared people as showing so many osds were full/nearfull

Improve the output to make the message clearer

Fixes: http://tracker.ceph.com/issues/22350
Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

common: Sign in early SIGHUP signal

Contains the following process:
ceph-mon ceph-mgr ceph-mds ceph-fuse radosgw.

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

rgw: Reinstating error codes mapping for Roles.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

test/admin_socket_output: switch to std::experimental::filesystem

so we don't need to compile boost::filesystem anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20114 from cethikdata/ceph-disk-lock

ceph-disk: reduce the scope of activate_lock.

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/pybind/test_rados.py: expand test_xattrs/test_obj_xattrs
to cover attribute with empty value case.

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

librados: invalid free() in rados_getxattrs_next()

Invalid free() can cause corruption when getting an object
attribute with empty value.

Check the validity of the pointer before free(). Also move
the free() call at the start of rados_getxattrs_next() to
avoid memory leak.

Fixes: http://tracker.ceph.com/issues/22042
Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

libradosstriper: fix the function declaration of rados_striper_trunc
change the type of first parameter from rados_ioctx_t to rados_striper_t

Signed-off-by: yuelongguang <yuelongguang@gmail.com>

Merge pull request #20270 from ivancich/wip-fix-bad-dmclock-sim-change

osd,dmclock: fix dmclock test simulator change

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20258 from tchaikov/wip-balancer-cleanup

 mgr/balancer: replace magic value of -1 for DEFAULT_CHOOSE_ARGS

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20230 from ifed01/wip-ifed-store-test-improve

test/store_test: update Many4KWritesTest* test cases to finalize with…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20221 from fangyuxiangGL/max

osd/ReplicatedBackend: 'osd_deep_scrub_keys' doesn't work

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>