Merge pull request #23960 from votdev/feature_35695

mgr/dashboard: Refactor role management

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #23829 from dillaman/wip-34534

librbd: blacklisted client might not notice it lost the lock

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #24185 into master

* refs/pull/24185/head:
qa: install build dependencies for cfuse_workunit_kernel_untar_build.yaml

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23415 into master

* refs/pull/23415/head:
msgr/async: huge refactoring of protocol V1
msgr/async: fix forward declaration of DispatchQueue

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24175 from xiaomanh/master

doc: Fix Spelling Error In File dynamicresharding.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Increase usability of role management by enabling the user to check/uncheck all rows/columns of permissions

Fixes: https://tracker.ceph.com/issues/35695
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #23429 from joke-lee/wip-website-rule-num-limit

rgw: website routing rules num limit

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #24028 from votdev/fix_rgw_status

mgr/dashboard: Catch LookupError when checking the RGW status

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #23823 from dillaman/wip-namespace-osd-check

librbd: prevent use of namespaces on pre-nautilus OSDs

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22579 from dillaman/wip-pybind-rados

pybind/rados: new methods for manipulating self-managed snapshots

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22739 from majianpeng/osd-shardthread-do-bluestore-oncommits

os/bluestore: make osd shard-thread do oncommits

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/OSD: choose a fixed thread do oncommits callback function

Now bluestore oncommit callback exec by osd op threads.
If there are multi threads of shard, it will cause out-of order.
For example, threads_per_shard=2
              Thread1                                 Thread2
    swap_oncommits(op1_oncommit)
                                            swap_oncommits(op2_oncommit)
    OpQueueItem.run(Op3)
                                            op2_oncommit.complete();
    op1_oncommit.complete()

This make oncommits out of order.
To avoiding this, we choose a fixed thread which has the smallest
thread_index of shard to do oncommit callback function.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/Finisher: only queue empty only wake up waiter.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/Finisher: only queue empty only wake up waiter.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #24004 from xiexingguo/wip-yet-more-async-fixes

osd/PG: async-recovery should respect historical missing objects

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #24130 from tchaikov/wip-gcc-7.3

rpm: bump up required GCC version to 7.3.1

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: website routing rules num limit
According to AWS S3 , an website routing rules can
have up to 50 rules.

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

qa: install build dependencies for cfuse_workunit_kernel_untar_build.yaml

Fixes: https://tracker.ceph.com/issues/36076
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23779 from Devp00l/wip-table-actions-component

mgr/dashboard: Add table actions component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

librbd: use the correct error code when the exclusive lock isn't locked

If the client is currently blacklisted, use -EBLACKLISTED, otherwise
use -EROFS.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper to retrieve the correct error code for read-only op

When the exclusive lock is unlocked, the error code should be
-EBLACKLISTED when the client is blacklisted, otherwise -EROFS.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reacquire lock should properly handle failed watcher

If the watch has been lost, assume the lock has been lost but attempt
to reacquire it if and when the watch is re-established.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix improper indentation of 'ceph_assert' statements

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: assume lock is unlocked if blacklisted or object deleted

This will ensure that it's possible to potentially re-acquire the
lock should the blacklist expire before the image is closed.

Fixes: http://tracker.ceph.com/issues/34534
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: watcher should internally track blacklisted state

Since it will periodically attempt to re-acquire the watch,
it will know when the RADOS client has been blacklisted and
when the blacklist has been removed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: attempt to recover lost image watcher upon all failures

For example, if an image is blacklisted and the blacklist eventually
expires, the image should recover its watch.

Fixes: http://tracker.ceph.com/issues/34534
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24160 from jdurgin/wip-rados-lead

doc/dev/index: update rados lead

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23662 from dillaman/wip-24412

librbd: support v2 cloning across namespaces

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #24059 from cbodley/wip-rgw-opstate-rm

rgw, cls: remove cls_statelog and rgw opstate tracking

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #24143 into master

* refs/pull/24143/head:
qa/workunits/cephtool/test_kvstore_tool.sh: run test in ., not /tmp

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Use table actions component for roles

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for users

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add HTML coverage reports

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Fixes ts warning in RBD snapshot tests

Typescript has thrown an warning because a private method is called in the
RBD snapshot unit tests.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for RBD snapshots

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for OSDs

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #24007 from tianshan/fix_35905

rgw: async sync_object and remove_object does not access coroutine me…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23610 from XiangxiangMeng/fix-rgw-bucket-policy

rgw: policy: fix s3:x-amz-grant-read-acp keyword error

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #24128 from ricardoasmarques/wip-optional-user-password

mgr/dashboard: User password should be optional

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #24145 into master

* refs/pull/24145/head:
qa/workunits/mon/caps.sh: wait longer to time out

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #23544 from joke-lee/wip-multiobjdelete-num-limit

rgw: delete multi object num limit

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #24001 from joke-lee/doc_life_cycle

rgw: update cls_rgw.cc and cls_rgw_const.h

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24114 from robbat2/wip-35990

rgw: fix chunked-encoding for chunks >1MiB

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #24110 from cbodley/wip-35986

rgw: abort_bucket_multiparts() ignores individual NoSuchUpload errors

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

cls: remove cls_statelog and tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

osd: remove statelog from osd_class_load_list config

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: remove config variables related to rgw opstate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove dependencies on cls_statelog

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove RGWOpState and related classes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: remove opstate commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: remove test for /admin/opstate rest apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove rest handlers for /admin/opstate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused opstate params from copy/fetch/stat

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove opstate tracking from fetch_remote_obj()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18578 from gaosibei/wip-cleanup-duplicate-include

RGW: remove duplicate include header files in rgw_rados.cc

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #23927 from joke-lee/WIP-RGW-x-amz-version-id

rgw: return x-amz-version-id: null when delete obj in versioning suspended bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Use table actions component for RGW users

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for RGW buckets

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use table actions component for RBDs

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add table actions component

With the help of this component it is easy to create all actions for a
table with the right permission for each table action with a simple
configuration.

Per default action button and action drop down list are shown. All
actions will be filtered depending on the permissions on
initialization. Per default all actions that have the permission
'update' or 'delete' are only enabled if an item is selected which
holds no running task.

You can use change the default behavior, to show actions only in a
named drop down.
For each item you can change it's default behaviors by using custom
functions to enable/disable an action or show/hide it
as action button or drop down item.

Please look at the tests to understand how to use it in practice.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: User password should be optional

Fixes: https://tracker.ceph.com/issues/36031
Signed-off-by: Ricardo Marques <rimarques@suse.com>

msgr/async: huge refactoring of protocol V1

This huge commit extracts the protocol V1 logic from the AsyncConnection
class into its own class called ProtocolV1.
The new protocol class extends from a Protocol base class that defines
an interface to allow the implementation of different wire protocols
in the future, such as the msgr V2 protocol.

Signed-off-by: Ricardo Dias <rdias@suse.com>

msgr/async: fix forward declaration of DispatchQueue

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge PR #23092 into master

* refs/pull/23092/head:
common: drop RefCountObject.cc and improve inlineability.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24109 into master

* refs/pull/24109/head:
doc: update docs for device management
mgr: make devicehealth always-on
mgr/devicehealth: do not create metrics pool on get-device-metrics
mgr/devicehealth: converge OPTIONS and DEFAULTS
mgr/devicehealth: squelch health warnings for unused devices
mgr/devicehealth: show-health-metrics -> get-health-metrics

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24005 from p-na/refactor-deletion-modal-component

mgr/dashboard: Refactoring of `DeletionModalComponent`

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #24149 into master

* refs/pull/24149/head:
unittest_bufferlist: add unit test for sha1 from multiple raw buffers

Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
Reviewed-by: Myoungwon Oh <omwmw@sk.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: test clone v2 across namespaces

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: permit cross-namespace when using clone v2

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use helper method to create librados::IoCtxs

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: replace librbd::ParentSpec with cls::rbd::ParentImageSpec

The newer struct includes support for pool namespaces.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: invoke new parent class API methods

If the OSDs don't support the new methods, revert to the legacy
class API methods.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: replace existing attach parent calls with new state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new attach parent state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: replace existing detach parent calls with new state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new detach parent state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refactor flatten state machine to new code style

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add new namespace compatible parent image cls methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: break cls_rbd_snap on-disk struct backwards compatability

The cls methods are now storing normalized parent overlaps within the snapshot
if the OSDs are required to be Nautilus or later. Therefore, we can safely
break the on-disk format compatibility for those newer OSDs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: migrate legacy snapshot on-disk format to normalized parent

The parent reference should be written once under the "parent" key,
so all updates to a snapshot should ensure the new normalized format
is used if Nautilus or later OSDs are required.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: cls_rbd_snap on-disk struct supports parent overlap

This is the first step to fix the denormalization of the parent
image spec references between the HEAD revision and the snapshot
revisions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: cls_rbd_parent on-disk struct supports namespaces

If the OSDs are required to be Nautilus or later, it will automatically break
backwards compatibility of the structure to support pool namespaces.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ChildImageSpec now optionally includes child pool namespace

It will break version compatibility if used against older OSDs that
don't support RBD pool namespace support.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new ParentImageSpec type to support parent namespaces

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: break composite cls rbd helpers into individual helpers

With the forthcoming need to use new methods to retrieve a clone's parent,
use per-method helpers to invoke the correct method within the image
refresh state machine.

Also used this time to merge newer cls rbd methods that should be supported
by all non-EOLed OSDs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: corrected namespace indentation within cls_client

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: not valid to have different parents between image snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

objclass: expose the required OSD release setting

This can be utilized to determine when it's safe to enable new
cls features.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: Fix Spelling Error In File dynamicresharding.rst

Signed-off-by: xiaomanh <huangxiaoman@cmss.chinamobile.com>

Merge pull request #24076 from ifed01/wip-ifed-fix-repairer

os/bluestore: fix bloom filter num entry miscalculation in repairer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24085 from liewegas/wip-32731

osd/osd_types: fix pg_t::contains() to check pool id too

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23721 from batrick/log-entry-stack

log: avoid heap allocations for most log entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24111 from gregsfortytwo/wip-35987-connection-buffers

msg/async: clean up local buffers on dispatch

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #24124 from tchaikov/wip-35969

rpm: use updated gperftools

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

doc/dev/index: update rados lead

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23754 from p-na/down-osd-raises-error

mgr/dashboard: Fix OSD down error display

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

osd/PG: explicitly clear num_objects_missing when approximate

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: async-recovery should respect historical missing objects

Peers with async-recovery enabled are usually having a update-to-date
last-update iterator and hence might be moved out from the __async_recovery_targets__
set during the next peering circles.

7de35629f562436d2bdb85788bdf97b10db3f556 makes num_objects_missing
trace historical missing objects correctly, hence we could take
num_objects_missing into account when determing __async_recovery_targets__.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #24098 into master

* refs/pull/24098/head:
Rename "include/assert.h"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>