Merge pull request #45508 from phlogistonjohn/jjm-backport-pacific-pr-44845-44870-45211

pacific: mgr/nfs: support managing exports without orchestration enabled

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #45569 from mgfritch/backport-45420-pacific

pacific: cephadm: infer the default container image during pull

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45582 from idryomov/wip-rbd-nbd-cookie-pacific

pacific: backport nbd cookie support

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #45580 from idryomov/wip-fix-pids-limit-pacific

pacific: cephadm: Remove containers pids-limit

Reviewed-by: Adam King adking@redhat.com

Merge pull request #45477 from cfsnyder/wip-53637-pacific

pacific: test/rgw: disable cls_rgw_gc test cases with defer_gc()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45476 from cfsnyder/wip-53518-pacific

pacific: rgw: add object null point judging when listing pubsub  topics

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45484 from cfsnyder/wip-53835-pacific

pacific: rgw: document S3 bucket replication support

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45483 from cfsnyder/wip-53818-pacific

pacific: rgw: RadosBucket::get_bucket_info() updates RGWBucketEnt

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45480 from cfsnyder/wip-53654-pacific

pacific: rgw: init bucket index only if putting bucket instance info succeeds

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45478 from cfsnyder/wip-53639-pacific

pacific: rgw/amqp: add default case to silence compiler warning

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45463 from cfsnyder/wip-53254-pacific

pacific: rgw: fix rgw.none statistics

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45596 from neha-ojha/wip-45512-pacific

pacific: ceph/admin: s/master/main

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45532 from idryomov/wip-fix-schedule-status-pacific

pacific: mgr/rbd_support: cast pool_id from int to str when collecting LevelSpec

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

rgw: fix rgw.none statistics

Only unaccount for objects being removed if they exist.

Also, clean up some formatting.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 95b662fa176cd841e26b76341f91bf2e6c266003)

ceph/admin: s/master/main

This PR changes the name "master" to "main" so
that builds (and, I assume, a great many other
things) will not fail.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 6a1dd3a8a2f3dc9fe8615d402c9041273516ff89)

qa/workunits/rbd: test map/attach with --show-cookie/--cookie options

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit 2adfe55bcce08f8f46989e9ddbf46aa74e681992)

rbd-nbd: allow user to specify cookie at map

Allow user to specify cookie of choice at the time of map

$ rbd device attach rbd-pool/image --device /dev/nbd0 \
--cookie 6f85d970-10b2-456b-8baf-676aa4d782e4 --options try-netlink

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit 16404dede832f3908d48fc36c7714891bfcfde20)

rbd: add --show-cookie/--cookie option for map/attach commands

Example:
$ rbd device map rbd-pool/image --show-cookie --try-netlink --device-type nbd

$ rbd device attach rbd-pool/image --device /dev/nbd0 \
      --cookie 6f85d970-10b2-456b-8baf-676aa4d782e4 --device-type nbd

older Kernel versions can use --force to skip the cookie validation

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit 6df06c7a62bdd6e9ce61b26c738dcd1509e68273)

Conflicts:
src/tools/rbd/action/Device.cc [ commit e1b4811bc324 ("rbd: add
  missing switch arguments for recognition by get_command_spec()")
  already in pacific -- re-add show-cookie ]

rbd-nbd: allow attach without --cookie for old kernel versions

For backward compatibility allow attach without --cookie option:

[root@linux-vm1]# rbd-nbd attach rbd-pool/image0 --device /dev/nbd0
/dev/nbd0

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit 833e71282ad82023237c5eb407b9fa4630280c15)

rbd-nbd: show per device cookie with list command

[root@linux-vm1]# rbd-nbd list-mapped
id    pool      namespace  image   snap  device     cookie
8133  rbd-pool             image0  -     /dev/nbd0  c704cb91-c6cf-466e-a335-0e935c0d5e47

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit efeb255d107075516c95f719f9a2ec9375d5bb2f)

rbd-nbd: mandate device cookie for attach command

[root@linux-vm1]# rbd-nbd attach rbd-pool/image0 --device /dev/nbd0 \
                          --cookie c704cb91-c6cf-466e-a335-0e935c0d5e47
/dev/nbd0

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit 6ca70ee4f078aba86a6edd013bd8d95f690a46a1)

rbd-nbd: provide a flag to show device cookie with map command

[root@linux-vm1]# rbd-nbd map rbd-pool/image0 --try-netlink --show-cookie
/dev/nbd0 c704cb91-c6cf-466e-a335-0e935c0d5e47

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit 59c08eabf12fdcf155b54dc422fb0f67d6ad747e)

rbd-nbd: generate and send device cookie with netlink connect request

[root@linux-vm1]# rbd-nbd map rbd-pool/image0 --try-netlink
/dev/nbd0

[root@linux-vm1]# cat /sys/block/nbd0/backend
c704cb91-c6cf-466e-a335-0e935c0d5e47

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit eee8d8a32e3f39440ac417b2e9ea46d93d4a1d73)

qa/workunits/rbd: use rbd cli for device attach/detach commands

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit e2aa4128f1e451a85200441f70a13336e843dad6)

rbd: improve conditional compilation specific checks

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit 4d7db9e86a246bf77e8545d0d4e220d90f48ac29)

rbd: promote rbd-nbd attach and detach at rbd integrated cli

Example:
$ rbd device attach rbd-pool/image --device /dev/nbd0 --device-type nbd --force
$ rbd device detach rpool/image --device-type nbd

for now returning EOPNOTSUPP with krbd, ggate and wnbd

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
(cherry picked from commit 35e5352f719857bed99caa4c76ae19bb68918c8e)

Conflicts:
src/tools/rbd/action/Device.cc [ commit e1b4811bc324 ("rbd: add
  missing switch arguments for recognition by get_command_spec()")
  already in pacific ]
src/tools/rbd/action/Wnbd.cc [ ditto ]

qa/suites/orch/cephadm: restrict test_iscsi_pids_limit to CentOS

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit f0ade57458b93f8401de8670ae62bf2295a6c40c)

[ commit 1f714da81440 ("qa: fix or add missing .qa links") not in
  pacific -- added qa/suites/orch/cephadm/workunits/task/.qa ]

cephadm: remove containers pids-limit

The default pids-limit (docker 4096/podman 2048) prevent some
customization from working (http threads on RGW) or limits the number
of luns per iscsi target.

Fixes: https://tracker.ceph.com/issues/52898
Signed-off-by: Teoman ONAY <tonay@redhat.com>
(cherry picked from commit de8b3c2676e65eb61df54c65cfd3b3af1e68da56)

Merge pull request #45426 from cfsnyder/wip-51348-pacific

pacific: rgw/notification: send correct size in COPY events

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45236 from k0ste/wip-54451-pacific

pacific: mgr/prometheus: Added `avail_raw` field for Pools DF Prometheus mgr module

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #45397 from kamoltat/wip-ksirivad-pacific-backport-45078

pacific: mon/MonCommands.h: fix target_size_ratio range

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #45376 from ronen-fr/wip-rf-43303-pacific

pacific: osd: fix the truncation of an int by int division

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #45373 from smithfarm/wip-54276-pacific

pacific: common: fix FTBFS due to dout & need_dynamic on GCC-12

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #45323 from ljflores/wip-54466-pacific

pacific: osd: require osd_pg_max_concurrent_snap_trims > 0

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #45319 from ljflores/wip-52078-pacific

pacific: test: fix wrong alarm (HitSetWrite)

Reviewed-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #45152 from chrisphoffman/wip-53644-pacific

pacific: mon: Omit MANY_OBJECTS_PER_PG warning when autoscaler is on

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

Merge pull request #43698 from cfsnyder/wip-52636-pacific

pacific: mon: do not quickly mark mds laggy when MON_DOWN

Reviewed-by: Venky Shankar vshankar@redhat.com
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #45459 from cfsnyder/wip-53079-pacific

pacific: src/rgw: Fix for malformed url

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45457 from cfsnyder/wip-53036-pacific

pacific: rgw: cls_bucket_list_unordered() might return one redundent entry every time is_truncated is true

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45442 from cfsnyder/wip-52107-pacific

pacific: radosgw-admin: 'sync status' is not behind if there are no mdlog entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45440 from cfsnyder/wip-52071-pacific

pacific: rgw: add the condition of lock mode conversion to PutObjRentention

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45427 from cfsnyder/wip-51351-pacific

pacific: rgw/amqp: remove the explicit "disconnect()" interface

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45445 from cfsnyder/wip-52115-pacific

pacific: qa/rgw: update apache-maven mirror for rgw/hadoop-s3a

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45439 from cfsnyder/wip-52050-pacific

pacific: rgw: default ms_mon_client_mode = secure

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45434 from cfsnyder/wip-51777-pacific

pacific: rgw: forward request in multisite for RGWDeleteBucketPolicy and RGWDeleteBucketPublicAccessBlock

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45430 from cfsnyder/wip-51701-pacific

pacific: rgw: url_decode before parsing copysource in copyobject

Merge pull request #43956 from cfsnyder/wip-52782-pacific

pacific: rgw/sts: adding role name and role session to ops log.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: infer the default container image during pull

Fixes: https://tracker.ceph.com/issues/54588
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 736af2b9c29e375a184b863220641b7d254e7d04)

Merge pull request #45455 from cfsnyder/wip-52990-pacific

pacific: rgw: have "bucket check --fix" fix pool ids correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45461 from cfsnyder/wip-53158-pacific

pacific: rgw:When KMS encryption is used and the key does not exist, we should…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45453 from cfsnyder/wip-52988-pacific

pacific: rgw: document rgw_lc_debug_interval configuration option

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45450 from cfsnyder/wip-52611-pacific

pacific: rgw: clean-up logging of function entering to make thorough and consistent

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45448 from cfsnyder/wip-52348-pacific

pacific: rgw: change order of xml elements in ListRoles response

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45444 from cfsnyder/wip-52113-pacific

pacific: qa/rgw: barbican and pykmip tasks upgrade pip before installing pytz

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45438 from cfsnyder/wip-51810-pacific

pacific: RGW - Zipper - Make default args match in get_obj_state

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45432 from cfsnyder/wip-51703-pacific

pacific: rgw: fix md5 not match for RGWBulkUploadOp upload when enable rgw com…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45428 from cfsnyder/wip-51352-pacific

pacific: rgw: write meta of a MP part to a correct pool

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #45421 from cfsnyder/wip-54426-pacific

pacific: rgw: add OPT_BUCKET_SYNC_RUN to gc_ops_list, so that

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45413 from cfsnyder/wip-54035-pacific

pacific: qa: move certificates for kmip task into /etc/ceph

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45411 from cfsnyder/wip-54494-pacific

pacific: rgw: fix segfault in UserAsyncRefreshHandler::init_fetch

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44729 from cfsnyder/wip-53849-pacific

pacific: rgwlc:  remove lc entry on bucket delete

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #44885 from BenoitKnecht/wip-54118-pacific

pacific: mgr: Fix ceph_daemon label in ceph_rgw_* metrics

Reviewed-by: BenoitKnecht <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #45351 from nmshelke/wip-54478-pacific

pacific: ceph-fuse: perform cleanup if test_dentry_handling failed

Reviewed-by: Venky Shankar vshankar@redhat.com
Reviewed-by: Kotresh HR khiremat@redhat.com

Merge pull request #45227 from rkachach/backport_of_pr_44928

pacific: mgr/cephadm: validating service_id for MDS

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Adam King adking@redhat.com

Merge pull request #45205 from kotreshhr/pacific-mgr-volumes-backports-0

pacific: mgr/volumes: A few mgr volumes pacific backports

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Venky Shankar vshankar@redhat.com

qa/workunits/rbd/cli_generic.sh: relax trash purge schedule status assert

Commit 08df6e0fd006 ("qa/workunits/rbd: expand LevelSpec parsing
coverage") didn't account for images with a separate data pool.  This
was missed because of small-cache-pool.yaml breakage.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 90a90ad47dd3140c796ef4da7263c9633d34e841)

rbd: fix a typo in "rbd trash purge schedule ls" formatted output

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit beffb7f9c6a4a19b6c4cfb65d04386575af50961)

qa/workunits/rbd: expand LevelSpec parsing coverage

Invoke "rbd mirror snapshot schedule ls -R" and "rbd mirror snapshot
schedule status" commands on all levels, consistently.  In particular,
make sure that an image level schedule is listed for a recursive query
at the pool level both before and after the schedule kicks in:

  $ rbd create --size 1G --mirror-image-mode snapshot -p foo bar
  $ rbd mirror snapshot schedule add -p foo --image bar 1m
  $ rbd mirror snapshot schedule ls -p foo -R
  POOL  NAMESPACE  IMAGE  SCHEDULE
  foo              bar    every 1m
  <wait for schedule to become visible in status>
  $ rbd mirror snapshot schedule ls -p foo -R
  POOL  NAMESPACE  IMAGE  SCHEDULE
  foo              bar    every 1m

Also, make sure that pool and image level status queries work:

  $ rbd mirror snapshot schedule status -p foo
  SCHEDULE TIME        IMAGE
  2022-03-04 07:14:00  foo/bar
  $ rbd mirror snapshot schedule status -p foo --image bar
  SCHEDULE TIME        IMAGE
  2022-03-04 07:14:00  foo/bar

Both of these issues are fixed by the previous commit.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 08df6e0fd00689649b802e975df320efe3b07c83)

mgr/nfs: remove redundant check

Remove the extra check of the cluster id from _apply method. As _apply
is a "private" method that should be only called from other private
methods that have already validated the cluster_id. It also removes
a dependency on the orch-requiring func available_clusters.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit dd5a47f83349c8f2b539ba3881f58f5270024cbb)

mgr/nfs: fix unintentional recursion

The `exports` property of the ExportMgr exists to cache the exports
configuration found in the .nfs namespace. Using that property
within the property method is probably not intentional and is probably
only working due to the lucky construction of the _exports dict
immediately after the None check so that the _exports dict is returned
(and is a mutable type).

Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit daa455cd168d62cd8fbcaba4d7aa79b56e68ef0d)

mgr/nfs: add known_cluster_ids to generalize nfs cluster id fetching

The changes to the nfs module in 8c711afc are working but when I began
writing more test automation I found a few more places in the
export-configuration code path relying on the orchestration module
only. This change generalizes the logic to source nfs clusters from
orchestration when it's enabled but from the .nfs pool when
orchestration is disabled. It then uses that call when loading
the exports cache on the ExportMgr object.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit 4d09660dea5696e5085a75694968aafe9253f47a)

doc/mgr/nfs: document that nfs exports related mgr call requirements

A recent change in the mgr/nfs module should enable the functioning
of export management commands/API calls as long as the rados namespaces
and objects have been already established. Document this fact, noting
that now only the `ceph nfs cluster ...` calls *require* an
orchestration module.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit b5b3e0bcb5e2a27375f50f7717786a3928cba711)

mgr/nfs: support managing exports without orchestration enabled

This change allows the `ceph nfs export ...` commands to function
without the entire mgr/nfs subsystem requiring orchestration to be
enabled.  When there's no orchestration available, the code falls back
to examining the namespaces in the ".nfs" rados pool to determine what
cluster_id values are valid.

This change does not add support for creating the rados objects and
namespace needed to manage a nfs cluster. As discussed with the
orchestration group on 2022-01-22, rook does not need the mgr module to
establish the namespace. So, for now, we'll defer the work needed to
create the namespace/objects when orchestration is disabled.

Fixes: https://tracker.ceph.com/issues/54043
Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit 8c711afc4ab898942a2569b619eb8379ee02ffba)

mgr/nfs: fix typo in error message

Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit 56323a2625133d5a53bf1ee1662346daa1b4f09b)

mgr/nfs: add unit test for normalize_path

Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit ffa95fbc796aa5c00eaa32138291c0ef2a48949a)

mgr/nfs: change method format_path to function normalize_path

This function was not using self and thus has no need to be a method.
While we're at it, rename it to normalize_path because that's what
it is doing.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit f91dd1bf7bfab251d671a30d622bb544a4ce37d0)

mgr/nfs: clean up rados object naming code

The naming of rados objects used to store the nfs config was spread
all over the code, including inline f-strings, not-static methods,
etc.
This change unifies the naming by putting constant string prefixes
and name generating functions into the utils.py file.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit 88266144423e6876dc392bc6ea59e32393024323)

mgr/nfs: make _check_rados_notify a function

This was previously a staticmethod. This static method was only used by
NFSRados object. Staticmethods are nearly always better implemented as
functions, which is done so here.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit c51a6755b52954910512f804cde9e5255c1db9e7)

mgr/nfs: limit dependency of NFSRados object

Previously, the NFSRados object accepted the "Module" as the
first argument but only used the rados attribute (type rados.Rados).
It's better to limit the scope of types when reasonably possible
so we can see what the true dependencies are. So we restrict
NFSRados to accepting a rados.Rados as the argument.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit d94b63830d94f21ba276452844a46d21e084fb3f)

doc: rgw: document S3 bucket replication support

Support was added at Octopus.

Fixes: https://tracker.ceph.com/issues/48755
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 774a247b2b854538b679490581e6950372142797)

rgw: RadosBucket::get_bucket_info() updates RGWBucketEnt

if we construct a RadosBucket without knowing its instance id,
get_bucket_info() will look it up for us and store it in
info.bucket.bucket_id

but RadosBucket::update_container_stats() gets this instance id from
'RGWBucketEnt ent', which wasn't updated by get_bucket_info(). this
causes update_container_stats() to try to read a bucket instance with
empty instance id, which fails with -ENOENT, and the RGWListBucket op
returns uninitialized X-Container-* stats

Fixes: https://tracker.ceph.com/issues/47781
Fixes: https://tracker.ceph.com/issues/47861
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e391e56c7dda24b4a9d6981a5be31fe8b9defd9d)

rgw: init bucket index only if putting bucket instance info succeeds

Signed-off-by: Huber-ming <zhangsm01@inspur.com>
(cherry picked from commit 6e97f2a32df80f00d44ed3daceac381c46c17026)

rgw/amqp: add default case to silence compiler warning

Fixes: https://tracker.ceph.com/issues/53252
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a32ae4b61ac4af244533ce53388411ca50fc1148)

test/rgw: disable cls_rgw_gc test cases with defer_gc()

Fixes: https://tracker.ceph.com/issues/53325
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 29beb17e04bdaf12305e8dc5613d8e4891781828)

rgw: add object null point judging when listing pubsub  topics
Fixs: https://tracker.ceph.com/issues/53464

Signed-off-by: zhipeng li <qiuxinyidian@gmail.com>
(cherry picked from commit 9bec1829483f4f724c08afbe640e6d8383579348)

rgw:When KMS encryption is used and the key does not exist, we should not throw ERR_ INVALID_ ACCESS_ Key error code.

When kms encryption is used, the key_id is null or the actual_key size is wrong, we should not throw "ERR_INVALID_ACCESS_KEY " error code, instead of "EINVAL"error code, is used to indicate parameter error.

Signed-off-by: wangyingbin <wangyingbin@inspur.com>
(cherry picked from commit 40dbc29984d67a3f4946a0b30d53f3db19952bf0)

src/rgw: Fix for malformed url

This PR solves: https://tracker.ceph.com/issues/52738
It is solved by making changes to rgw_url.cc
A test is also added to check it's working.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>
(cherry picked from commit 2916f2439eb2f62bc08c3e283b13391302b3e497)

rgw: update last_added_entry when count == num_entries

RGWRados::cls_bucket_list_unordered() will produce one redundent entry
every time is_truncated is true.The issue could be easily reproduced
when a bucket is filled with amounts of incomplete multipart upload.
To be more specific, the number of incomplete multipart upload objects
should be greater than 1100.

Signed-off-by: Peng Zhang <zhangpeng@vclusters.com>
(cherry picked from commit 7511f9f675ea4e43992605dc03109bc5f356a5e1)

rgw: during "bucket check --fix" set index entry pool id correctly

The current code sets the pool id of bucket index entries to the
bucket index pool rather than the data pool. This fixes that.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 4c2ac79621c7d110d9b3664ef5ce2027d817560c)

Conflicts:
src/rgw/rgw_rados.cc

Cherry-pick notes:
- if condition under changes did not use braces in Pacific but does in later releases

rgw: document rgw_lc_debug_interval configuration option

Updates the yaml describing this config option with a "desc" and a
"long_desc".

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 9171d3626b5a0181456a68555d5742109abaabbc)

Conflicts:
src/common/options/rgw.yaml.in

Cherry-pick notes:
- options not defined in yaml in Pacific

rgw: clean-up logging of function entering to make thorough and consistent

This provides more thorough and consistent function tracing in CLS/RGW
when logging is set to 10 or higher.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 9982e152095b6e7f94ed59afb5fd976f04dccd34)

Conflicts:
src/cls/rgw/cls_rgw.cc

Cherry-pick notes:
- new lines added to logs

rgw: change order of xml elements in ListRoles response

one or more AWS sdks fail to parse our response because they expect
ListRolesResult to come first. not really an rgw bug, but it's easy
enough to fix

Fixes: https://tracker.ceph.com/issues/52027
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 5f1f4212a81a7b124d657d4653de39ee85051964)

qa/rgw: update apache-maven mirror for rgw/hadoop-s3a

Fixes: https://tracker.ceph.com/issues/52069
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9253733d0883d01988b163ee22cfc3481c01a52d)

qa/rgw: barbican and pykmip tasks upgrade pip before installing pytz

Downloading https://files.pythonhosted.org/packages/9b/77/461087a514d2e8ece1c975d8216bc03f7048e6090c5166bc34115afdaa53/cryptography-3.4.7.tar.gz (546kB)
  Complete output from command python setup.py egg_info:

          =============================DEBUG ASSISTANCE==========================
          If you are seeing an error here please try the following to
          successfully install cryptography:

          Upgrade to the latest pip and try again. This will fix errors for most
          users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
          =============================DEBUG ASSISTANCE==========================

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-7fhnk5us/cryptography/setup.py", line 14, in <module>
      from setuptools_rust import RustExtension
  ModuleNotFoundError: No module named 'setuptools_rust'

Fixes: https://tracker.ceph.com/issues/52070
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e5a5b4e3794baf3f6bc86ed0c02ecfc8cc0f99b4)

radosgw-admin: 'sync status' is not behind if there are no mdlog entries

if remote mdlogs are trimmed prematurely, sync status will report
that it's behind the remote's max-marker even if there are no mdlog
entries to sync

for each behind shard, we fetch the next mdlog entry from the remote. if
we get an empty listing, remove that shard from behind_shards. this
logic now has to run before we print "behind shards:" so that empty
shards aren't listed

Fixes: https://tracker.ceph.com/issues/52091
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 14d43f740d101c8d41a2ced4525bf8efd8c9d943)

rgw: add the condition of lock mode conversion to PutObjRentention

Signed-off-by: wangzhong <wangzhong@cmss.chinamobile.com>
(cherry picked from commit 585eae46a2bb7ed39ca2f3213801e0f77642c9d4)

rgw: default auth_client_required=cephx

This makes this warning go away:

2021-08-09T15:51:52.882+0000 7f2373837400 -1 warn_if_insecure(): WARNING: rgw is configured to optionally allow insecure connections to the monitors (auth_supported, ms_mon_client_mode), ssl certificates stored at the monitor configuration could leak

7e22d2a31d277ab3eecff47b0864b206a32e2332 only fixed half of the problem.

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 6010eae66cd12358880fff64e2229a4265a5cf0a)

rgw: default ms_mon_client_mode = secure

Require a secure connection to the monitor.  This is important because
we may store SSL certs on the mon.

Note that secure mode was introduce before Nautilus, so this can be
backported at least to Pacific.

Fixes: https://tracker.ceph.com/issues/52000
Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 7e22d2a31d277ab3eecff47b0864b206a32e2332)

RGW - Zipper - Make default args match in get_obj_state

The default arg for follow_olh was different between the base class and
the Rados class.  Fix it so they're the same.

Fixes: https://tracker.ceph.com/issues/51677
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
(cherry picked from commit 664fe75c2a88cbcc27575649a6900640f4fd90b5)

Conflicts:
src/rgw/rgw_sal.h

Cherry-pick notes:
- changes to other parts of get_obj_state method signature

rgw: forward request in multisite for RGWDeleteBucketPolicy
 and RGWDeleteBucketPublicAccessBlock

Fix: https://tracker.ceph.com/issues/45551

Signed-off-by: yuliyang_yewu <yuliyang_yewu@cmss.chinamobile.com>
(cherry picked from commit 1f9267edd81fd1bf05bd6440c5e112a6cc042145)