qa: update mds_pre_upgrade to no longer stop standbys

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 2cd3494771b41216ad99a3bf462e2462b30254b6)

qa: update mds_pre_upgrade to disable standby-replay

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 8e0b9bcad67352a5ad0d91993e8fb26f482a08de)

qa: add tests for compat manipulation and upgrade

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 295971b9c6520389b6780bfa3bddc3606eaba6dd)

Conflicts:
qa/tasks/cephfs/test_admin.py: trivial

doc: remove deprecated compat commands

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 54b649af9b43b8ff48f39b033cc69f4d4e3b1b30)

doc: update MDS upgrade procedure

Now that CompatSet changes to the FSMap no longer cause old MDS to
suicide.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit efb70f2b336b119ca99fe2abb79428ab4d3453a2)

mon,mds: use per-MDS compat to inform replacement

This diff makes the following changes:

- FSMap::compat is now just a "default compat" of currently unknown
  utility. It is used when constructing a new file system but does
  not really have any effect or current use.

- The `mds compat *` CLI commands are deprecated. They manipulate
  the default compat which has no useful effect.

- Each MDS sends its compat to the mons in its beacon. This is from
  MDSMap::get_compat_set_all() at MDS boot. This CompatSet does not
  change for the duration of the MDS lifetime.

- Mons record each MDS compat in the FSMap to inform standby failover.
  An MDS is only promoted if it is compatible with the file system
  compat.

- Mons upgrade (merge) the file system compat when (a) the number of
  *in* MDS is 1 (effected by max_mds=1) and (b) the mons are promoting a
  standby with a new compat. A file system is never upgraded when there
  is more than 1 rank to prevent two MDS with incompatible compat.

- A suite of `fs compat` commands exist to manipulate the file system
  compat. These exist mostly for testing.

The consequence of these changes is that the upgrade procedure for MDS
can be updated to no longer require turning off all MDS but rank 0
before performing any upgrades. A CompatSet change would cause all MDS
receiving the new MDSMap to suicide due to incompatibility (if so).
Instead, the monitors will no longer assign an incompatible MDS to a
file system and enforce an upgrade procedure if incompatibilities exist.

Fixes: https://tracker.ceph.com/issues/49720
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 58eaa237b0a16d3c934ded77ed4dc53137d9b4a1)

Conflicts:
src/mds/FSMap.h: trivial
src/mon/MDSMonitor.cc: trivial
src/mon/MonCommands.h: work around removed OBSOLETE commands

mon: do not update inline incompat except via mds

The MDS_FEATURE_INCOMPAT_INLINE feature indicates that an MDS knows how
to read/write inline data and that the file system may have it. The
separate setting for inline_data protects this file system feature.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 8cdc36c89da224eb8e86ee6ed80cd2eb65354587)

mds: add MDSMap method for creating null MDSMap

It's not necessary to distribute a CompatSet with the null mdsmap. We
only need to communicate that the MDS is not part of any map.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 4a10b6016ff76f80e27574ad2ab40ac5b0266264)

mds: only update beacon epoch if newer

This is a defensive programming change. We don't want the beacon epoch
to ever go backwards.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 0256ae010f61520a67f95fcd3e8522784ea56f8b)

mds: harden standby_mds lookup

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit b8ad8a8c821ba57d64aa349a9da7117867b0f401)

mon/FSCommands: accept generic ostream rather than stringstream

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 31c8edd603e7db84529eacde1771aef01dbde089)

Conflicts:
src/mon/FSCommands.cc: trivial

include: add less verbose CompatSet dump

For printing in `fs dump`.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 56b36e69fae0fcfb5081ede3934f1d63cde0494a)

include: add dump operator for Feature

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5da02036eefe60c89e11f53d4bacb0e41e03e794)

include: add const qualifier to appropriate CompatSet methods

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit d179fa2bc0262854b1e001e0c9db3d2f508bf647)

mon/MDSMonitor.cc: fix join fscid not applied with pending fsmap at boot

Fixes: https://tracker.ceph.com/issues/49157
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>
(cherry picked from commit 4ab87e2eecbdcb4932775e871c3ae2f793880c20)

Merge pull request #42536 from batrick/i51940

pacific: MDSMonitor: monitor crash after upgrade from ceph 15.2.13 to 16.2.4

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42773 from neha-ojha/wip-52244-pacific

pacific: os/bluestore: make deferred writes less aggressive for large writes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42135 from dang/wip-dang-41991-backport

pacific: RGW - Bucket Remove Op: Pass in user

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #40696 from singuliere/wip-50156-pacific

pacific: rgw/notification: fix persistent notification hang when ack-levl=none

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #40695 from singuliere/wip-50157-pacific

pacific: rgw/notification: fixing the "persistent=false" flag

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #40693 from singuliere/wip-50214-pacific

pacific: rgw: objectlock: improve client error messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/store_test: more testing for deferred writes

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 324ea6b425e5898eca6e86264fa4d34cb868a6ae)

os/bluestore: enforce one more non-inclusive comparision against prefer_deferred_size

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit cb30c9965854e7dbfe851ed0cfaf113b16bcab5e)

os/bluestore: Better handling of deferred write trigger

Now deferred write in _do_alloc_write does not depend on blob size,
but on size of extent allocated on disk.
It is now possible to set bluestore_prefer_deferred_size way larger than
bluestore_max_blob_size and still get desired behavior.
Example: for deferred=256K, blob=64K : when op write is 128K both blobs will be
written as deferred. When op write is 256K then all will go as regular write.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 06ac561a72a93bd24d05b6c2266d66fee49228b8)

os/bluestore: account for alignment with max_blob_size when splitting
write I/O into chunks.

Without the fix the following write seq:
0~4M
4096~4M

produces tons of deferred writes at the second stage.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit b06bcf99f50cf7db5856eb1f6cd56100a0d0bbbc)

os/bluestore: introduce l_bluestore_write_deferred_bytes perf counter.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f38f18c149b9347a26d845539eaeef29979e9006)

os/bluestore: use non-inclusive comparision against prefer_deferred_size

Fixes: https://tracker.ceph.com/issues/52089
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f1c4448fc549ed58bb58f920dfa16886e4910315)

os/bluestore: improve logging around deferred writes

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 89a20d63ec7e255d36363bd8977c3fca1d277877)

os/bluestore: fix missing 'l_bluestore_write_deferred' perf counter increment

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dc367c4bbc701b04a5bde562e30ab5b9c1705657)

os/bluestore: cosmetic cleanups

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 080531d6813b868eda562cf27babe18d8a974413)

Merge pull request #42679 from callithea/wip-51992-pacific

pacific: monitoring/grafana/cluster: use per-unit max and limit values

Merge pull request #42755 from guits/wip-50896-pacific

pacific: ceph-volume: Consider /dev/root as mounted

ceph-volume: consider /dev/root mounted

Fixes: https://tracker.ceph.com/issues/50604
Signed-off-by: David Caro <david@dcaro.es>
(cherry picked from commit c7b959c35ff463cfbffa2abf0f56a1c1a5f594c8)

Merge pull request #42493 from dsavineau/wip-51852-pacific

pacific: ceph-volume: lvm batch: fast_allocations(): avoid ZeroDivisionError

Merge pull request #42546 from dsavineau/wip-51958-pacific

pacific: ceph-volume/tests: retry when destroying osd

Merge pull request #42606 from s0nea/wip-dashboard-pacific-translations

mgr/dashboard: update translations for pacific

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #42298 from callithea/wip-51636-pacific

pacific: monitoring: fix Physical Device Latency unit

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: clwluvw <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #42642 from cfsnyder/wip-51696-pacific

pacific: rgw: fail as expected when set/delete-bucket-website attempted on a non-exis…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42566 from cfsnyder/wip-51514-pacific

pacific: rgw/notifications: support metadata filter in CompleteMultipartUploa…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41367 from cbodley/wip-50845

pacific: rgw: deprecate the civetweb frontend

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42637 from cfsnyder/wip-51779-pacific

pacific: rgw : add check for tenant provided in RGWCreateRole

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42635 from cfsnyder/wip-50645-pacific

pacific: rgw: allow rgw-orphan-list to process multiple data pools

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42634 from cfsnyder/wip-51014-pacific

pacific: rgw: remove quota soft threshold

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42633 from cfsnyder/wip-50679-pacific

pacific: rgw: fix segfault related to explicit object manifest handling

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42631 from cfsnyder/wip-50465-pacific

pacific: rgw/notifications: delete bucket notification object when empty

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42626 from cfsnyder/wip-52007-pacific

pacific: rgw: avoid occuring radosgw daemon crash when access a conditionally …

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42625 from cfsnyder/wip-51751-pacific

pacific: RGW Zipper - Make sure bucket list progresses

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42638 from cfsnyder/wip-50708-pacific

pacific: rgw: fix bucket object listing when marker matches prefix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42649 from cfsnyder/wip-50379-pacific

pacific: rgw/amqp/test: fix mock prototype for librabbitmq-0.11.0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42647 from cfsnyder/wip-49555-pacific

pacific: rgw/notification: add exception handling for persistent notification thread

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42646 from cfsnyder/wip-50463-pacific

pacific: rgw/multisite: return correct error code when op fails

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42645 from cfsnyder/wip-52051-pacific

pacific: rgw: when deleted obj removed in versioned bucket, extra del-marker added

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42644 from cfsnyder/wip-51804-pacific

pacific: rgw/http/notifications: support content type in HTTP POST messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42641 from cfsnyder/wip-50367-pacific

pacific: rgw: during reshard lock contention, adjust logging

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42640 from cfsnyder/wip-51801-pacific

pacific: rgw: radosgw-admin errors if marker not specified on data/mdlog trim

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #42639 from cfsnyder/wip-51781-pacific

pacific: rgw : modfiy error XML for deleterole

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42656 from cfsnyder/wip-51785-pacific

pacific: rgw multisite: metadata sync treats all errors as 'transient' for retry

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42655 from cfsnyder/wip-50803-pacific

pacific: radosgw-admin: skip GC init on read-only admin ops

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42654 from cfsnyder/wip-50731-pacific

pacific: rgw/rgw_file: Fix the return value of read() and readlink()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42653 from cfsnyder/wip-50728-pacific

pacific: rgw : add check empty for sync url

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42652 from cfsnyder/wip-50711-pacific

pacific: rgw: fix for mfa resync crash when supplied with only one totp_pin.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42651 from cfsnyder/wip-51771-pacific

pacific: test/rgw: fix use of poll() with timers in unittest_rgw_dmclock_scheduler

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42650 from cfsnyder/wip-50642-pacific

pacific: rgw/sts: read_obj_policy() consults iam_user_policies on ENOENT

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42648 from cfsnyder/wip-50094-pacific

pacific: librgw/notifications: initialize kafka and amqp

Reviewed-by: Casey Bodley <cbodley@redhat.com>

monitoring/grafana/cluster: use per-unit max and limit values

The value we get is a perunit, so the limits and the max value should
be over 1, not 100. Note that the value being shown was correct, it
was the gauge that was not showing the correct indicators.

Signed-off-by: David Caro <david@dcaro.es>
(cherry picked from commit c98129803935e720bb068fcf3700812c550f61b4)

qa/rgw: run multisite tests with metadata sync error injection

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit cff6bf37e65964d2d987231598659f08a84b9410)

rgw: limit concurrency of metadata sync

limit the number of concurrent RGWMetaSyncSingleEntryCRs that each mdlog
shard is allowed to spawn. use META_SYNC_SPAWN_WINDOW=20 to match data-
and bucket sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a07add0a5408f0480e93ea5920bc013119ba6401)

rgw: metadata sync treats all errors as 'transient'

collect_children() had a special case for EAGAIN that it treated as
a 'transient' error, which set the can_adjust_marker = false to bail out
of RGWMetaSyncShardCR and retry from the previous marker

but the http client doesn't return EAGAIN - rgw_http_error_to_errno()
defaults to EIO - so this retry logic based on can_adjust_marker never
runs. on any other error, RGWMetaSyncSingleEntryCR would not call
marker_tracker->finish() to advance the sync status marker, and
RGWMetaSyncShardCR would continue on with full- or incremental sync
without ever attempting to retry the failed entries

a detailed comment in collect_children() describes a different strategy
for handling 'permanent' errors, but that was never fully elaborated.
i also don't think there's a reasonable way to differentiate between
transient and permanent errors, so this treats all errors as transient
to be retried

if an error really is permanent for a given metadata key, metadata sync
will get stuck there and require manual intervention

Fixes: https://tracker.ceph.com/issues/39657
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 866d66b8749b28ec626a8d0adba3d14fdd8abead)

radosgw-admin: skip GC init on read-only admin ops

Fixes: https://tracker.ceph.com/issues/50520
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit 9ac1991fc798af7e0ba0fac18209b71b5ae3f02b)

Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rados.h
src/rgw/rgw_sal.cc
src/rgw/rgw_sal.h

Cherry-pick notes:
- Conflicts due to move from of RGWStoreManager in rgw_sal_rados.h to StoreManager in rgw_sal.h

rgw/rgw_file: Fix the return value of read() and readlink()

Fixes: https://tracker.ceph.com/issues/49189
Signed-off-by: Dai zhiwei <daizhiwei3@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>
(cherry picked from commit bfd83e8fa142873a0bdf09a4d1ad1b04127f5885)

rgw : add check empty for sync url

Fixes: https://tracker.ceph.com/issues/50103
Signed-off-by: caolei <halei15848934852@163.com>
(cherry picked from commit 3a4e0b79310b21eeee37043d5419887bb41c0cf6)

rgw: fix for mfa resync crash when supplied with only one totp_pin.

The fix returns an appropriate error message.

fixes: https://tracker.ceph.com/issues/50394

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit 1b943419c45a9b17be71668bd023aeb814171b2c)

test/rgw: use spawn library for test_rgw_dmclock_scheduler

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a8e3589a2c875b6fadc853c75f20cb9256f294ca)

test/rgw: fix use of poll() with timers in unittest_rgw_dmclock_scheduler

the AsyncScheduler uses an asio timer to dispatch work to its executor
with an optional delay. when no delay is requested, it waits on the
timer with an expiration time in the past (crimson::dmclock::TimeZero)

tests are failing here because poll() is returning without executing the
handlers of those expired timers

asio implements these timers with timerfd and epoll. debugging with
strace, i see that these timers armed with timerfd_settime() are not
always immediately ready according to epoll_wait():

  eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK)   = 3
  epoll_create1(EPOLL_CLOEXEC)            = 4
  timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC) = 5
  epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLERR|EPOLLET, data={u32=14164052, u64=14164052}}) = 0
  epoll_ctl(4, EPOLL_CTL_ADD, 5, {events=EPOLLIN|EPOLLERR, data={u32=14164064, u64=14164064}}) = 0
  timerfd_settime(5, TFD_TIMER_ABSTIME, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=1}}, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=0}}) = 0
  epoll_wait(4, [{events=EPOLLIN, data={u32=14164052, u64=14164052}}], 128, 0) = 1
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [{events=EPOLLIN, data={u32=14164064, u64=14164064}}], 128, 0) = 1

in this example, it took 6 calls to context.poll() before it was ready
to execute the timer's handler

to work around this, replace calls to context.poll() with calls to
context.run_for() with a very short duration

Fixes: https://tracker.ceph.com/issues/42788
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 21baed999e31c5e69c75f0cbb8757ef91585d917)

rgw: read_obj_policy() consults iam_user_policies on ENOENT

when the head object doesn't exist, read_obj_policy() has to decide
whether to return ENOENT or EACCES

when there's a bucket policy, we check whether it has s3ListBucket
permissions. when there's an assumed role, we also need to check
against the role's policies in s->iam_user_policies

Fixes: https://tracker.ceph.com/issues/49780
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 5dc9375fa1888242f388f8b502f445f3ddc891f7)

rgw/amqp/test: fix mock prototype for librabbitmq-0.11.0

also use extern C for to get compilation errors when
function prototype change

Fixes: https://tracker.ceph.com/issues/50291
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 2ba598ec4c294bd09d2df18ccd2096382e303d39)

librgw/notifications: initialize kafka and amqp

Fixes: https://tracker.ceph.com/issues/49738
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 178f6bdac97b57300bbe0956633cf686a7e3ccee)

rgw/notification: add exception handling for persistent notification thread

Fixes: https://tracker.ceph.com/issues/49322
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 915963ecb9effcb1f2d38f444c1bb9307f8ffbe1)

Conflicts:
src/rgw/rgw_notify.cc

rgw/multisite: return correct error code when op fails

when trying to disable/enbale sync on non-master zone

Fixes: https://tracker.ceph.com/issues/50201
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 83e89dfa3358fe91597d6714483f96b21a234ae6)

rgw: when deleted obj removed in versioned bucket, extra del-marker added

After initial checks are complete, this will read the OLH earlier than
previously to check the delete-marker flag and under the bug's
conditions will return -ENOENT rather than create a spurious delete
marker.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 69d7589fb1305b7d202ffd126c3c835e7cd0dda3)

rgw/http/notifications: support content type in HTTP POST messages

Fixes: https://tracker.ceph.com/issues/51530
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 6a1688b57d1e329cecd0c26c494cb08c9a4c3079)

Merge pull request #42629 from rhcs-dashboard/wip-52049-pacific

pacific: mgr/dashboard: show perf. counters for rgw svc. on Cluster > Hosts

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42628 from rhcs-dashboard/wip-52048-pacific

pacific: mgr/dashboard: fix ssl cert validation for rgw service creation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

rgw: fail as expected when set/delete-bucket-website attempted on a non-existent bucket, rgw should return HTTP 404 and NoSuchBucket.

Fixes: https://tracker.ceph.com/issues/51536
Signed-off-by: xiangrui meng <mengxr@chinatelecom.cn>
(cherry picked from commit c623aa45d35b269c6701a57e44ac05bb29a79dc8)

rgw: during reshard lock contention, adjust logging

When RGW fails to get a lock on a reshard log, we log it in such a way
that it looks like an error. Instead we'll make sure that the log
message is informational.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 6d3dee37791ad427a3435c493a1d7874ba075674)

rgw: remove quota soft threshold

Remove quota soft threshold, which causes expensive checks for sharded buckets

Fixes: 14eabd4aa7b8a2e2c0c43fe7f877ed2171277526
Signed-off-by: Zulai Wang <wangzl31@outlook.com>
(cherry picked from commit 32a39705765af0f87bec9101e5d337b797e05fea)

Conflicts:
src/common/options/rgw.yaml.in

Cherry-pick notes:
- Options defined in src/common/options.cc in Pacific vs src/common/options/rgw.yaml.in

rgw: radosgw-admin errors if marker not specified on data/mdlog trim

Check that a marker was specified and trim if we don't have one.

Also: In a world where we're parsing for generation, it doesn't really
make sense to have a 'no marker specified' as separate from a marker
that is just an empty string.

Also: Successful datalog trim returns zero, not
-ENODATA, and radosgw-admin should expect this.

Fixes: https://tracker.ceph.com/issues/51712
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 4cb6fcf7f4e9548a8a0dd017b49a6ea23bedffd6)

Conflicts:
src/rgw/rgw_admin.cc

Cherry-pick notes:
- static_cast for RadosStore not needed in Pacific

rgw : modfiy error XML for deleterole

Fixes: https://tracker.ceph.com/issues/51157
Signed-off-by: caolei <halei15848934852@163.com>
(cherry picked from commit c7ab6579c7655352d08a4c12fc3a6951217dbe6f)

rgw: fix bucket object listing when marker matches prefix

When an iniitial marker that ends with a delimiter is provided, it
prevents listing of that "subdirectory" due to new logic at the cls
level to make listing more efficient. The fix catches that situation.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 220ef4b22d1d1667eb4f2c300a0b788e87b9067d)

rgw : add check for tenant provided in RGWCreateRole

Fixes: https://tracker.ceph.com/issues/51206
Signed-off-by: caolei <halei15848934852@163.com>
(cherry picked from commit 3c99ac14080c9f5b1611c9bbe4a223a9fd2927a0)

Conflicts:
src/rgw/rgw_rest_role.cc

Cherry-pick notes:
- Pacific constructs role explicitly vs. using store->get_role(), and does not wrap in a unique_ptr

rgw: allow rgw-orphan-list to process multiple data pools

We now allow multiple data pools to be specified either on the
command-line or at the prompt. The list should be space-separated.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit bde1278a5de3e9363c0f136b6ebc913bbb5e269d)

rgw: fix segfault related to explicit object manifest handling

Fixes: https://tracker.ceph.com/issues/50467
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit ae00453515b17d25a5d42430e744855b020ccec6)

rgw/notifications: delete bucket notification object when empty

Fixes: https://tracker.ceph.com/issues/49650
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 08e9846fd5a040887e2ddbfca36da97777fc76f6)

Conflicts:
src/rgw/rgw_pubsub.cc

Cherry-pick notes:
- RGWPubSub methods take a DoutPrefixProvider as first argument in Pacific

mgr/dashboard: show perf. counters for rgw svc. on Cluster > Hosts

- Use service_map_id to retrieve perf. counters for rgw services.
- Move logic from controller to CephService.

Fixes: https://tracker.ceph.com/issues/52022
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 69a87a1dea6fe148f2a81b1d7775f5ade9626c09)

mgr/dashboard: Cluster > Hosts > Performance Counters: do not show context bar.

Do not show the context bar (Object Gateway selection).

Fixes: https://tracker.ceph.com/issues/52022
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 73acb9b60d23e79bdcdd03437b0a813487420e33)

mgr/dashboard: fix ssl cert validation for rgw service creation

Fixes: https://tracker.ceph.com/issues/51965
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 5f52c00e5291b53b4fe9ab7f9fae5d499017f91b)

rgw: avoid occuring radosgw daemon crash when access a conditionally redirected static website

Fixes: https://tracker.ceph.com/issues/51715
Signed-off-by: xiangrui meng <mengxr@chinatelecom.cn>
Signed-off-by: yupeng chen <chenyupeng@chinatelecom.cn>
(cherry picked from commit 865d13e9fa043954dce5110f4f8ae693af9604ce)

RGW Zipper - Make sure bucket list progresses

The bucket list wrapper lost the automatic advancement of the marker for
each iteration.  Some places manually set that marker, and it worked
fine, but some depended on it being automatically set.  Fix it so that
the marker advances automatically each iteration.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
(cherry picked from commit 9cd74053b8bd95d3dee6cac6723cb051768865c3)

Merge pull request #42585 from rhcs-dashboard/wip-51994-pacific

pacific: mgr/dashboard: cephadm-e2e job script: improvements

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: update translations for pacific

Note: it's not a backport because the change is not intended to land in
master. It's pacific related only.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>