Merge PR #24191 into master

* refs/pull/24191/head:
src/include:fix unused parameter

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23979 into master

* refs/pull/23979/head:
crush/CrushWrapper: fix crush tree json dumper

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23978 into master

* refs/pull/23978/head:
mgr: add get_latest_counter() to C++ -> Python interface.
mgr: remove PerfCounterInstance::get_current()
mgr: remove PyFormatter copy constructor; shouldn't be called

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24150 from alfredodeza/wip-rm24794

ceph-volume: earlier detection for --journal and --filestore flag requirements

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24107 into master

* refs/pull/24107/head:
common: Lock -> lock, Unlock -> lock, TryLock -> try_lock
common: Mutex::Locker -> std::lock_guard<Mutex>
common/Mutex: typedef lock_guard<Mutex> Locker
common/Mutex: Lock -> lock, Unlock -> unlock
common/Mutex: kill mutex_perf_counter

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #24181 from dillaman/wip-36084

rbd-mirror: instantiate the status formatter before changing state

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23843 from cbodley/wip-rgw-dpp-datasync

rgw: log refactoring for data sync

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge PR #24197 into master

* refs/pull/24197/head:
doc: remove Calamari content from ceph-deploy

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24201 from alfredodeza/wip-rm36088

ceph-volume  batch: allow journal+block.db sizing on the CLI

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24156 into master

* refs/pull/24156/head:
qa/tests: removed knfs suite

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24153 into master

* refs/pull/24153/head:
qa/tests: removed knfs from crontab

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23960 from votdev/feature_35695

mgr/dashboard: Refactor role management

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #23829 from dillaman/wip-34534

librbd: blacklisted client might not notice it lost the lock

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #24185 into master

* refs/pull/24185/head:
qa: install build dependencies for cfuse_workunit_kernel_untar_build.yaml

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/man/ceph-volume update batch command with new sizing flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices update fixture to include new size flags for strategies

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch.filestore consume --journal-size

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch.bluestore consume --block-db-size

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #23415 into master

* refs/pull/23415/head:
msgr/async: huge refactoring of protocol V1
msgr/async: fix forward declaration of DispatchQueue

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24175 from xiaomanh/master

doc: Fix Spelling Error In File dynamicresharding.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Increase usability of role management by enabling the user to check/uncheck all rows/columns of permissions

Fixes: https://tracker.ceph.com/issues/35695
Signed-off-by: Volker Theile <vtheile@suse.com>

common: Lock -> lock, Unlock -> lock, TryLock -> try_lock

Use the Lockable convention.

Signed-off-by: Sage Weil <sage@redhat.com>

common: Mutex::Locker -> std::lock_guard<Mutex>

Signed-off-by: Sage Weil <sage@redhat.com>

common/Mutex: typedef lock_guard<Mutex> Locker

Signed-off-by: Sage Weil <sage@redhat.com>

common/Mutex: Lock -> lock, Unlock -> unlock

This aligns us with the Lockable concept, which means we can use
lock_guard<>, unique_lock<>, etc.

Signed-off-by: Sage Weil <sage@redhat.com>

common/Mutex: kill mutex_perf_counter

This has a measurable overhead even when turned off, and we do not use it.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23429 from joke-lee/wip-website-rule-num-limit

rgw: website routing rules num limit

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #24028 from votdev/fix_rgw_status

mgr/dashboard: Catch LookupError when checking the RGW status

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: remove Calamari content from ceph-deploy

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #23823 from dillaman/wip-namespace-osd-check

librbd: prevent use of namespaces on pre-nautilus OSDs

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22579 from dillaman/wip-pybind-rados

pybind/rados: new methods for manipulating self-managed snapshots

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22739 from majianpeng/osd-shardthread-do-bluestore-oncommits

os/bluestore: make osd shard-thread do oncommits

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/OSD: choose a fixed thread do oncommits callback function

Now bluestore oncommit callback exec by osd op threads.
If there are multi threads of shard, it will cause out-of order.
For example, threads_per_shard=2
              Thread1                                 Thread2
    swap_oncommits(op1_oncommit)
                                            swap_oncommits(op2_oncommit)
    OpQueueItem.run(Op3)
                                            op2_oncommit.complete();
    op1_oncommit.complete()

This make oncommits out of order.
To avoiding this, we choose a fixed thread which has the smallest
thread_index of shard to do oncommit callback function.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/Finisher: only queue empty only wake up waiter.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/Finisher: only queue empty only wake up waiter.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #24004 from xiexingguo/wip-yet-more-async-fixes

osd/PG: async-recovery should respect historical missing objects

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

src/include:fix unused parameter

fixes unused parameter

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #24130 from tchaikov/wip-gcc-7.3

rpm: bump up required GCC version to 7.3.1

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: website routing rules num limit
According to AWS S3 , an website routing rules can
have up to 50 rules.

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

qa: install build dependencies for cfuse_workunit_kernel_untar_build.yaml

Fixes: https://tracker.ceph.com/issues/36076
Signed-off-by: Neha Ojha <nojha@redhat.com>

ceph-volume lvm.batch add sizing flags for journal and block.db LVs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23779 from Devp00l/wip-table-actions-component

mgr/dashboard: Add table actions component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

librbd: use the correct error code when the exclusive lock isn't locked

If the client is currently blacklisted, use -EBLACKLISTED, otherwise
use -EROFS.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper to retrieve the correct error code for read-only op

When the exclusive lock is unlocked, the error code should be
-EBLACKLISTED when the client is blacklisted, otherwise -EROFS.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reacquire lock should properly handle failed watcher

If the watch has been lost, assume the lock has been lost but attempt
to reacquire it if and when the watch is re-established.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix improper indentation of 'ceph_assert' statements

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: assume lock is unlocked if blacklisted or object deleted

This will ensure that it's possible to potentially re-acquire the
lock should the blacklist expire before the image is closed.

Fixes: http://tracker.ceph.com/issues/34534
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: watcher should internally track blacklisted state

Since it will periodically attempt to re-acquire the watch,
it will know when the RADOS client has been blacklisted and
when the blacklist has been removed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: attempt to recover lost image watcher upon all failures

For example, if an image is blacklisted and the blacklist eventually
expires, the image should recover its watch.

Fixes: http://tracker.ceph.com/issues/34534
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24160 from jdurgin/wip-rados-lead

doc/dev/index: update rados lead

Reviewed-by: Sage Weil <sage@redhat.com>

rbd-mirror: instantiate the status formatter before changing state

This will avoid a possible race between pre-queued status updates
firing between the time the state has been changed and the formatter
has been instantiated.

Fixes: http://tracker.ceph.com/issues/36084
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23662 from dillaman/wip-24412

librbd: support v2 cloning across namespaces

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #24059 from cbodley/wip-rgw-opstate-rm

rgw, cls: remove cls_statelog and rgw opstate tracking

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tests: removed knfs suite

Fixes: http://tracker.ceph.com/issues/36075
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #24143 into master

* refs/pull/24143/head:
qa/workunits/cephtool/test_kvstore_tool.sh: run test in ., not /tmp

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Use table actions component for roles

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for users

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add HTML coverage reports

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Fixes ts warning in RBD snapshot tests

Typescript has thrown an warning because a private method is called in the
RBD snapshot unit tests.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for RBD snapshots

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for OSDs

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #24007 from tianshan/fix_35905

rgw: async sync_object and remove_object does not access coroutine me…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23610 from XiangxiangMeng/fix-rgw-bucket-policy

rgw: policy: fix s3:x-amz-grant-read-acp keyword error

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #24128 from ricardoasmarques/wip-optional-user-password

mgr/dashboard: User password should be optional

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #24145 into master

* refs/pull/24145/head:
qa/workunits/mon/caps.sh: wait longer to time out

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #23544 from joke-lee/wip-multiobjdelete-num-limit

rgw: delete multi object num limit

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #24001 from joke-lee/doc_life_cycle

rgw: update cls_rgw.cc and cls_rgw_const.h

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24114 from robbat2/wip-35990

rgw: fix chunked-encoding for chunks >1MiB

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #24110 from cbodley/wip-35986

rgw: abort_bucket_multiparts() ignores individual NoSuchUpload errors

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

cls: remove cls_statelog and tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

osd: remove statelog from osd_class_load_list config

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: remove config variables related to rgw opstate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove dependencies on cls_statelog

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove RGWOpState and related classes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: remove opstate commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: remove test for /admin/opstate rest apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove rest handlers for /admin/opstate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused opstate params from copy/fetch/stat

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove opstate tracking from fetch_remote_obj()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18578 from gaosibei/wip-cleanup-duplicate-include

RGW: remove duplicate include header files in rgw_rados.cc

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #23927 from joke-lee/WIP-RGW-x-amz-version-id

rgw: return x-amz-version-id: null when delete obj in versioning suspended bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Use table actions component for RGW users

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for RGW buckets

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for RBDs

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add table actions component

With the help of this component it is easy to create all actions for a
table with the right permission for each table action with a simple
configuration.

Per default action button and action drop down list are shown. All
actions will be filtered depending on the permissions on
initialization. Per default all actions that have the permission
'update' or 'delete' are only enabled if an item is selected which
holds no running task.

You can use change the default behavior, to show actions only in a
named drop down.
For each item you can change it's default behaviors by using custom
functions to enable/disable an action or show/hide it
as action button or drop down item.

Please look at the tests to understand how to use it in practice.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: User password should be optional

Fixes: https://tracker.ceph.com/issues/36031
Signed-off-by: Ricardo Marques <rimarques@suse.com>

msgr/async: huge refactoring of protocol V1

This huge commit extracts the protocol V1 logic from the AsyncConnection
class into its own class called ProtocolV1.
The new protocol class extends from a Protocol base class that defines
an interface to allow the implementation of different wire protocols
in the future, such as the msgr V2 protocol.

Signed-off-by: Ricardo Dias <rdias@suse.com>

msgr/async: fix forward declaration of DispatchQueue

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge PR #23092 into master

* refs/pull/23092/head:
common: drop RefCountObject.cc and improve inlineability.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24109 into master

* refs/pull/24109/head:
doc: update docs for device management
mgr: make devicehealth always-on
mgr/devicehealth: do not create metrics pool on get-device-metrics
mgr/devicehealth: converge OPTIONS and DEFAULTS
mgr/devicehealth: squelch health warnings for unused devices
mgr/devicehealth: show-health-metrics -> get-health-metrics

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24005 from p-na/refactor-deletion-modal-component

mgr/dashboard: Refactoring of `DeletionModalComponent`

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #24149 into master

* refs/pull/24149/head:
unittest_bufferlist: add unit test for sha1 from multiple raw buffers

Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
Reviewed-by: Myoungwon Oh <omwmw@sk.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: test clone v2 across namespaces

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: permit cross-namespace when using clone v2

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use helper method to create librados::IoCtxs

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: replace librbd::ParentSpec with cls::rbd::ParentImageSpec

The newer struct includes support for pool namespaces.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: invoke new parent class API methods

If the OSDs don't support the new methods, revert to the legacy
class API methods.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: replace existing attach parent calls with new state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new attach parent state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>