mon/FSCommands: accept generic ostream rather than stringstream

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 31c8edd603e7db84529eacde1771aef01dbde089)

Conflicts:
src/mon/FSCommands.cc: trivial

include: add less verbose CompatSet dump

For printing in `fs dump`.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 56b36e69fae0fcfb5081ede3934f1d63cde0494a)

include: add dump operator for Feature

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5da02036eefe60c89e11f53d4bacb0e41e03e794)

include: add const qualifier to appropriate CompatSet methods

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit d179fa2bc0262854b1e001e0c9db3d2f508bf647)

mon/MDSMonitor.cc: fix join fscid not applied with pending fsmap at boot

Fixes: https://tracker.ceph.com/issues/49157
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>
(cherry picked from commit 4ab87e2eecbdcb4932775e871c3ae2f793880c20)

Merge pull request #42536 from batrick/i51940

pacific: MDSMonitor: monitor crash after upgrade from ceph 15.2.13 to 16.2.4

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42773 from neha-ojha/wip-52244-pacific

pacific: os/bluestore: make deferred writes less aggressive for large writes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42135 from dang/wip-dang-41991-backport

pacific: RGW - Bucket Remove Op: Pass in user

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #40696 from singuliere/wip-50156-pacific

pacific: rgw/notification: fix persistent notification hang when ack-levl=none

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #40695 from singuliere/wip-50157-pacific

pacific: rgw/notification: fixing the "persistent=false" flag

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #40693 from singuliere/wip-50214-pacific

pacific: rgw: objectlock: improve client error messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/store_test: more testing for deferred writes

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 324ea6b425e5898eca6e86264fa4d34cb868a6ae)

os/bluestore: enforce one more non-inclusive comparision against prefer_deferred_size

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit cb30c9965854e7dbfe851ed0cfaf113b16bcab5e)

os/bluestore: Better handling of deferred write trigger

Now deferred write in _do_alloc_write does not depend on blob size,
but on size of extent allocated on disk.
It is now possible to set bluestore_prefer_deferred_size way larger than
bluestore_max_blob_size and still get desired behavior.
Example: for deferred=256K, blob=64K : when op write is 128K both blobs will be
written as deferred. When op write is 256K then all will go as regular write.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 06ac561a72a93bd24d05b6c2266d66fee49228b8)

os/bluestore: account for alignment with max_blob_size when splitting
write I/O into chunks.

Without the fix the following write seq:
0~4M
4096~4M

produces tons of deferred writes at the second stage.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit b06bcf99f50cf7db5856eb1f6cd56100a0d0bbbc)

os/bluestore: introduce l_bluestore_write_deferred_bytes perf counter.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f38f18c149b9347a26d845539eaeef29979e9006)

os/bluestore: use non-inclusive comparision against prefer_deferred_size

Fixes: https://tracker.ceph.com/issues/52089
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f1c4448fc549ed58bb58f920dfa16886e4910315)

os/bluestore: improve logging around deferred writes

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 89a20d63ec7e255d36363bd8977c3fca1d277877)

os/bluestore: fix missing 'l_bluestore_write_deferred' perf counter increment

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dc367c4bbc701b04a5bde562e30ab5b9c1705657)

os/bluestore: cosmetic cleanups

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 080531d6813b868eda562cf27babe18d8a974413)

Merge pull request #42679 from callithea/wip-51992-pacific

pacific: monitoring/grafana/cluster: use per-unit max and limit values

Merge pull request #42755 from guits/wip-50896-pacific

pacific: ceph-volume: Consider /dev/root as mounted

ceph-volume: consider /dev/root mounted

Fixes: https://tracker.ceph.com/issues/50604
Signed-off-by: David Caro <david@dcaro.es>
(cherry picked from commit c7b959c35ff463cfbffa2abf0f56a1c1a5f594c8)

Merge pull request #42493 from dsavineau/wip-51852-pacific

pacific: ceph-volume: lvm batch: fast_allocations(): avoid ZeroDivisionError

Merge pull request #42546 from dsavineau/wip-51958-pacific

pacific: ceph-volume/tests: retry when destroying osd

Merge pull request #42606 from s0nea/wip-dashboard-pacific-translations

mgr/dashboard: update translations for pacific

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #42298 from callithea/wip-51636-pacific

pacific: monitoring: fix Physical Device Latency unit

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: clwluvw <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #42642 from cfsnyder/wip-51696-pacific

pacific: rgw: fail as expected when set/delete-bucket-website attempted on a non-exis…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42566 from cfsnyder/wip-51514-pacific

pacific: rgw/notifications: support metadata filter in CompleteMultipartUploa…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41367 from cbodley/wip-50845

pacific: rgw: deprecate the civetweb frontend

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42637 from cfsnyder/wip-51779-pacific

pacific: rgw : add check for tenant provided in RGWCreateRole

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42635 from cfsnyder/wip-50645-pacific

pacific: rgw: allow rgw-orphan-list to process multiple data pools

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42634 from cfsnyder/wip-51014-pacific

pacific: rgw: remove quota soft threshold

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42633 from cfsnyder/wip-50679-pacific

pacific: rgw: fix segfault related to explicit object manifest handling

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42631 from cfsnyder/wip-50465-pacific

pacific: rgw/notifications: delete bucket notification object when empty

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42626 from cfsnyder/wip-52007-pacific

pacific: rgw: avoid occuring radosgw daemon crash when access a conditionally …

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42625 from cfsnyder/wip-51751-pacific

pacific: RGW Zipper - Make sure bucket list progresses

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42638 from cfsnyder/wip-50708-pacific

pacific: rgw: fix bucket object listing when marker matches prefix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42649 from cfsnyder/wip-50379-pacific

pacific: rgw/amqp/test: fix mock prototype for librabbitmq-0.11.0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42647 from cfsnyder/wip-49555-pacific

pacific: rgw/notification: add exception handling for persistent notification thread

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42646 from cfsnyder/wip-50463-pacific

pacific: rgw/multisite: return correct error code when op fails

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42645 from cfsnyder/wip-52051-pacific

pacific: rgw: when deleted obj removed in versioned bucket, extra del-marker added

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42644 from cfsnyder/wip-51804-pacific

pacific: rgw/http/notifications: support content type in HTTP POST messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42641 from cfsnyder/wip-50367-pacific

pacific: rgw: during reshard lock contention, adjust logging

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42640 from cfsnyder/wip-51801-pacific

pacific: rgw: radosgw-admin errors if marker not specified on data/mdlog trim

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #42639 from cfsnyder/wip-51781-pacific

pacific: rgw : modfiy error XML for deleterole

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42656 from cfsnyder/wip-51785-pacific

pacific: rgw multisite: metadata sync treats all errors as 'transient' for retry

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42655 from cfsnyder/wip-50803-pacific

pacific: radosgw-admin: skip GC init on read-only admin ops

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42654 from cfsnyder/wip-50731-pacific

pacific: rgw/rgw_file: Fix the return value of read() and readlink()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42653 from cfsnyder/wip-50728-pacific

pacific: rgw : add check empty for sync url

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42652 from cfsnyder/wip-50711-pacific

pacific: rgw: fix for mfa resync crash when supplied with only one totp_pin.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42651 from cfsnyder/wip-51771-pacific

pacific: test/rgw: fix use of poll() with timers in unittest_rgw_dmclock_scheduler

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42650 from cfsnyder/wip-50642-pacific

pacific: rgw/sts: read_obj_policy() consults iam_user_policies on ENOENT

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42648 from cfsnyder/wip-50094-pacific

pacific: librgw/notifications: initialize kafka and amqp

Reviewed-by: Casey Bodley <cbodley@redhat.com>

monitoring/grafana/cluster: use per-unit max and limit values

The value we get is a perunit, so the limits and the max value should
be over 1, not 100. Note that the value being shown was correct, it
was the gauge that was not showing the correct indicators.

Signed-off-by: David Caro <david@dcaro.es>
(cherry picked from commit c98129803935e720bb068fcf3700812c550f61b4)

qa/rgw: run multisite tests with metadata sync error injection

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit cff6bf37e65964d2d987231598659f08a84b9410)

rgw: limit concurrency of metadata sync

limit the number of concurrent RGWMetaSyncSingleEntryCRs that each mdlog
shard is allowed to spawn. use META_SYNC_SPAWN_WINDOW=20 to match data-
and bucket sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a07add0a5408f0480e93ea5920bc013119ba6401)

rgw: metadata sync treats all errors as 'transient'

collect_children() had a special case for EAGAIN that it treated as
a 'transient' error, which set the can_adjust_marker = false to bail out
of RGWMetaSyncShardCR and retry from the previous marker

but the http client doesn't return EAGAIN - rgw_http_error_to_errno()
defaults to EIO - so this retry logic based on can_adjust_marker never
runs. on any other error, RGWMetaSyncSingleEntryCR would not call
marker_tracker->finish() to advance the sync status marker, and
RGWMetaSyncShardCR would continue on with full- or incremental sync
without ever attempting to retry the failed entries

a detailed comment in collect_children() describes a different strategy
for handling 'permanent' errors, but that was never fully elaborated.
i also don't think there's a reasonable way to differentiate between
transient and permanent errors, so this treats all errors as transient
to be retried

if an error really is permanent for a given metadata key, metadata sync
will get stuck there and require manual intervention

Fixes: https://tracker.ceph.com/issues/39657
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 866d66b8749b28ec626a8d0adba3d14fdd8abead)

radosgw-admin: skip GC init on read-only admin ops

Fixes: https://tracker.ceph.com/issues/50520
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit 9ac1991fc798af7e0ba0fac18209b71b5ae3f02b)

Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rados.h
src/rgw/rgw_sal.cc
src/rgw/rgw_sal.h

Cherry-pick notes:
- Conflicts due to move from of RGWStoreManager in rgw_sal_rados.h to StoreManager in rgw_sal.h

rgw/rgw_file: Fix the return value of read() and readlink()

Fixes: https://tracker.ceph.com/issues/49189
Signed-off-by: Dai zhiwei <daizhiwei3@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>
(cherry picked from commit bfd83e8fa142873a0bdf09a4d1ad1b04127f5885)

rgw : add check empty for sync url

Fixes: https://tracker.ceph.com/issues/50103
Signed-off-by: caolei <halei15848934852@163.com>
(cherry picked from commit 3a4e0b79310b21eeee37043d5419887bb41c0cf6)

rgw: fix for mfa resync crash when supplied with only one totp_pin.

The fix returns an appropriate error message.

fixes: https://tracker.ceph.com/issues/50394

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit 1b943419c45a9b17be71668bd023aeb814171b2c)

test/rgw: use spawn library for test_rgw_dmclock_scheduler

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a8e3589a2c875b6fadc853c75f20cb9256f294ca)

test/rgw: fix use of poll() with timers in unittest_rgw_dmclock_scheduler

the AsyncScheduler uses an asio timer to dispatch work to its executor
with an optional delay. when no delay is requested, it waits on the
timer with an expiration time in the past (crimson::dmclock::TimeZero)

tests are failing here because poll() is returning without executing the
handlers of those expired timers

asio implements these timers with timerfd and epoll. debugging with
strace, i see that these timers armed with timerfd_settime() are not
always immediately ready according to epoll_wait():

  eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK)   = 3
  epoll_create1(EPOLL_CLOEXEC)            = 4
  timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC) = 5
  epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLERR|EPOLLET, data={u32=14164052, u64=14164052}}) = 0
  epoll_ctl(4, EPOLL_CTL_ADD, 5, {events=EPOLLIN|EPOLLERR, data={u32=14164064, u64=14164064}}) = 0
  timerfd_settime(5, TFD_TIMER_ABSTIME, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=1}}, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=0}}) = 0
  epoll_wait(4, [{events=EPOLLIN, data={u32=14164052, u64=14164052}}], 128, 0) = 1
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [{events=EPOLLIN, data={u32=14164064, u64=14164064}}], 128, 0) = 1

in this example, it took 6 calls to context.poll() before it was ready
to execute the timer's handler

to work around this, replace calls to context.poll() with calls to
context.run_for() with a very short duration

Fixes: https://tracker.ceph.com/issues/42788
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 21baed999e31c5e69c75f0cbb8757ef91585d917)

rgw: read_obj_policy() consults iam_user_policies on ENOENT

when the head object doesn't exist, read_obj_policy() has to decide
whether to return ENOENT or EACCES

when there's a bucket policy, we check whether it has s3ListBucket
permissions. when there's an assumed role, we also need to check
against the role's policies in s->iam_user_policies

Fixes: https://tracker.ceph.com/issues/49780
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 5dc9375fa1888242f388f8b502f445f3ddc891f7)

rgw/amqp/test: fix mock prototype for librabbitmq-0.11.0

also use extern C for to get compilation errors when
function prototype change

Fixes: https://tracker.ceph.com/issues/50291
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 2ba598ec4c294bd09d2df18ccd2096382e303d39)

librgw/notifications: initialize kafka and amqp

Fixes: https://tracker.ceph.com/issues/49738
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 178f6bdac97b57300bbe0956633cf686a7e3ccee)

rgw/notification: add exception handling for persistent notification thread

Fixes: https://tracker.ceph.com/issues/49322
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 915963ecb9effcb1f2d38f444c1bb9307f8ffbe1)

Conflicts:
src/rgw/rgw_notify.cc

rgw/multisite: return correct error code when op fails

when trying to disable/enbale sync on non-master zone

Fixes: https://tracker.ceph.com/issues/50201
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 83e89dfa3358fe91597d6714483f96b21a234ae6)

rgw: when deleted obj removed in versioned bucket, extra del-marker added

After initial checks are complete, this will read the OLH earlier than
previously to check the delete-marker flag and under the bug's
conditions will return -ENOENT rather than create a spurious delete
marker.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 69d7589fb1305b7d202ffd126c3c835e7cd0dda3)

rgw/http/notifications: support content type in HTTP POST messages

Fixes: https://tracker.ceph.com/issues/51530
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 6a1688b57d1e329cecd0c26c494cb08c9a4c3079)

Merge pull request #42629 from rhcs-dashboard/wip-52049-pacific

pacific: mgr/dashboard: show perf. counters for rgw svc. on Cluster > Hosts

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42628 from rhcs-dashboard/wip-52048-pacific

pacific: mgr/dashboard: fix ssl cert validation for rgw service creation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

rgw: fail as expected when set/delete-bucket-website attempted on a non-existent bucket, rgw should return HTTP 404 and NoSuchBucket.

Fixes: https://tracker.ceph.com/issues/51536
Signed-off-by: xiangrui meng <mengxr@chinatelecom.cn>
(cherry picked from commit c623aa45d35b269c6701a57e44ac05bb29a79dc8)

rgw: during reshard lock contention, adjust logging

When RGW fails to get a lock on a reshard log, we log it in such a way
that it looks like an error. Instead we'll make sure that the log
message is informational.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 6d3dee37791ad427a3435c493a1d7874ba075674)

rgw: remove quota soft threshold

Remove quota soft threshold, which causes expensive checks for sharded buckets

Fixes: 14eabd4aa7b8a2e2c0c43fe7f877ed2171277526
Signed-off-by: Zulai Wang <wangzl31@outlook.com>
(cherry picked from commit 32a39705765af0f87bec9101e5d337b797e05fea)

Conflicts:
src/common/options/rgw.yaml.in

Cherry-pick notes:
- Options defined in src/common/options.cc in Pacific vs src/common/options/rgw.yaml.in

rgw: radosgw-admin errors if marker not specified on data/mdlog trim

Check that a marker was specified and trim if we don't have one.

Also: In a world where we're parsing for generation, it doesn't really
make sense to have a 'no marker specified' as separate from a marker
that is just an empty string.

Also: Successful datalog trim returns zero, not
-ENODATA, and radosgw-admin should expect this.

Fixes: https://tracker.ceph.com/issues/51712
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 4cb6fcf7f4e9548a8a0dd017b49a6ea23bedffd6)

Conflicts:
src/rgw/rgw_admin.cc

Cherry-pick notes:
- static_cast for RadosStore not needed in Pacific

rgw : modfiy error XML for deleterole

Fixes: https://tracker.ceph.com/issues/51157
Signed-off-by: caolei <halei15848934852@163.com>
(cherry picked from commit c7ab6579c7655352d08a4c12fc3a6951217dbe6f)

rgw: fix bucket object listing when marker matches prefix

When an iniitial marker that ends with a delimiter is provided, it
prevents listing of that "subdirectory" due to new logic at the cls
level to make listing more efficient. The fix catches that situation.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 220ef4b22d1d1667eb4f2c300a0b788e87b9067d)

rgw : add check for tenant provided in RGWCreateRole

Fixes: https://tracker.ceph.com/issues/51206
Signed-off-by: caolei <halei15848934852@163.com>
(cherry picked from commit 3c99ac14080c9f5b1611c9bbe4a223a9fd2927a0)

Conflicts:
src/rgw/rgw_rest_role.cc

Cherry-pick notes:
- Pacific constructs role explicitly vs. using store->get_role(), and does not wrap in a unique_ptr

rgw: allow rgw-orphan-list to process multiple data pools

We now allow multiple data pools to be specified either on the
command-line or at the prompt. The list should be space-separated.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit bde1278a5de3e9363c0f136b6ebc913bbb5e269d)

rgw: fix segfault related to explicit object manifest handling

Fixes: https://tracker.ceph.com/issues/50467
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit ae00453515b17d25a5d42430e744855b020ccec6)

rgw/notifications: delete bucket notification object when empty

Fixes: https://tracker.ceph.com/issues/49650
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 08e9846fd5a040887e2ddbfca36da97777fc76f6)

Conflicts:
src/rgw/rgw_pubsub.cc

Cherry-pick notes:
- RGWPubSub methods take a DoutPrefixProvider as first argument in Pacific

mgr/dashboard: show perf. counters for rgw svc. on Cluster > Hosts

- Use service_map_id to retrieve perf. counters for rgw services.
- Move logic from controller to CephService.

Fixes: https://tracker.ceph.com/issues/52022
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 69a87a1dea6fe148f2a81b1d7775f5ade9626c09)

mgr/dashboard: Cluster > Hosts > Performance Counters: do not show context bar.

Do not show the context bar (Object Gateway selection).

Fixes: https://tracker.ceph.com/issues/52022
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 73acb9b60d23e79bdcdd03437b0a813487420e33)

mgr/dashboard: fix ssl cert validation for rgw service creation

Fixes: https://tracker.ceph.com/issues/51965
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 5f52c00e5291b53b4fe9ab7f9fae5d499017f91b)

rgw: avoid occuring radosgw daemon crash when access a conditionally redirected static website

Fixes: https://tracker.ceph.com/issues/51715
Signed-off-by: xiangrui meng <mengxr@chinatelecom.cn>
Signed-off-by: yupeng chen <chenyupeng@chinatelecom.cn>
(cherry picked from commit 865d13e9fa043954dce5110f4f8ae693af9604ce)

RGW Zipper - Make sure bucket list progresses

The bucket list wrapper lost the automatic advancement of the marker for
each iteration.  Some places manually set that marker, and it worked
fine, but some depended on it being automatically set.  Fix it so that
the marker advances automatically each iteration.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
(cherry picked from commit 9cd74053b8bd95d3dee6cac6723cb051768865c3)

Merge pull request #42585 from rhcs-dashboard/wip-51994-pacific

pacific: mgr/dashboard: cephadm-e2e job script: improvements

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: update translations for pacific

Note: it's not a backport because the change is not intended to land in
master. It's pacific related only.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: cephadm-e2e script: improvements

Improvements and some adaptations related to the jenkins job.

Fixes: https://tracker.ceph.com/issues/51612
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 65b75000b7694cb3cbe617bbec28c513a2522be8)

 rgw/notifications: support metadata filter in CompleteMultipartUpload events

Fixes: https://tracker.ceph.com/issues/51261
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 93b9f0fb77ca5ed0b5c89d45229732850e0a0c49)

Conflicts:
src/rgw/rgw_notify.h
src/rgw/rgw_op.cc
src/rgw/rgw_sal.h
src/rgw/rgw_sal_rados.cc
src/rgw/rgw_sal_rados.h
src/test/rgw/bucket_notification/test_bn.py

Cherry-pick notes:
- differences due to renaming of rgw::sal::RGWObject to rgw::sal::Object
- differences due to use of RadosNotification in master and reservation_t in Pacific

Merge pull request #42480 from smithfarm/wip-51836-pacific

pacific: rpm: drop use of $FIRST_ARG in ceph-immutable-object-cache

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #42482 from MrFreezeex/wip-51840-pacific

pacific: osd: log snaptrim message to dout

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/MDSMonitor: propose if FSMap struct_v is too old

To flush older versions which may still be an empty MDSMap (for clusters
that have never used CephFS), we need to force a proposal so older
versions of the struct are trimmed.

This is the main fix of this branch. We removed code which processed old
encodings of the MDSMap in the mon store via 60bc524. That broke old
ceph clusters which never used CephFS (see cited ticket below).  This is
because the initial epoch is an empty MDSMap (back in Infernalis/Hammer)
that is never updated. So, the fix here is to just do proposals
periodically until all of the old structs are automatically trimmed by
the mons.

Fixes: 60bc524827bac072658203e56b1fa3dede9641c5
Fixes: https://tracker.ceph.com/issues/51673
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 56c3fc802ee8848ba85da4300adcc2ee8bd95416)

mon/MDSMonitor: give a proper error message if FSMap struct_v is too old

Fixes: https://tracker.ceph.com/issues/51673
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 4298f9758138c30e08975a84ced25d81e1fd53b9)

mds/FSMap: use DECODE_OLDEST to gate FSMap version

This throws a proper exception which can be handled.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 47a8273faebf66737673af4fa04a58ac3628cdf3)

qa: add tests for fs dump of epoch and trimming

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5ddaa36d17c875a8ab7a6445f5ab0917d9a2dffd)

Conflicts:
qa/tasks/cephfs/test_admin.py: trivial conflict

qa: add file system support for dumping epoch

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit ee899d9a44f3ab73c8ebbb82720d89364b9bb600)

mon/MDSMonitor: return mon_mds_force_trim_to even if equal to current epoch

The PaxosService code already excludes the value returned by
PaxosService::get_trim_to as the upper bound of the range of epochs to
trim. Without this fix, you need to set mon_mds_force_trim_to to one
greater than the epoch you want to trim _and_ force the current epoch to
be one greater than that; the net result being that you can only force
trimming up to 2 epochs behind the current epoch.

This change is helpful for resolving issue 51673, but not strictly
necessary.

Related-to: https://tracker.ceph.com/issues/51673
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit d9dc2f11d56fb4341ba5823f8d17459d10f3b2c1)

Conflicts:
src/common/options/mon.yaml.in: doc change dropped