qa: fix test_nfs_export_with_invalid_path

It actually didn't test the invalid path but still ended with
ENOENT(which is expected in case path is invalid) as the test
didn't create a fs, and it failed saying "FS nfs-cephfs not found"
which too raises ENOENT and thus it always passed.

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit 5cc0857)

mgr/nfs: handle exceptions for cephfs_path_is_dir()
in create_cephfs_export()

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit d759fe4103174d412cc1ba317390383b299a7d35)

mgr/nfs/utils: changes to helper func to check cephfs path

- Renamed to cephfs_path_is_dir

- Removed exception handling to prevent redundant log statements like:
   "No such file or directory error in stat: b'/mnt/testdir_symlink': No such file or directory [Errno 2]"

  Exceptions handled inside caller eliminates this redundancy

- Set modifier flag AT_SYMLINK_NOFOLLOW

- Removed string "{path} is not a dir" when raising NotADirectoryError
  Rationale: will be handled in export.py

- change mock to cephfs_path_is_dir

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit f3d7370)

qa: fix issue with fn unable to fetch port and ip

_get_port_ip_info() fails to fetch port and ip due to empty 'backend' key:

2023-02-24T20:49:09.084 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph nfs cluster info test
2023-02-24T20:49:09.471 INFO:teuthology.orchestra.run.smithi042.stdout:{
2023-02-24T20:49:09.472 INFO:teuthology.orchestra.run.smithi042.stdout:  "test": {
2023-02-24T20:49:09.472 INFO:teuthology.orchestra.run.smithi042.stdout:    "backend": [],
2023-02-24T20:49:09.472 INFO:teuthology.orchestra.run.smithi042.stdout:    "virtual_ip": null
2023-02-24T20:49:09.472 INFO:teuthology.orchestra.run.smithi042.stdout:  }
2023-02-24T20:49:09.472 INFO:teuthology.orchestra.run.smithi042.stdout:}

it then raises:

2023-02-24T20:49:10.323 INFO:tasks.cephfs_test_runner:    info_output = json.loads(self._nfs_cmd('cluster', 'info', self.cluster_id))['test']['backend'][0]
2023-02-24T20:49:10.323 INFO:tasks.cephfs_test_runner:IndexError: list index out of range

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit 310286fa185194fdd27bfe7692d516703e18fa11)

qa: fix helper function _check_nfs_cluster_status()

Comment in the code says to wait for two minutes as cluster
creation takes time but actually it's waiting for thirteen
minutes, it's not required to wait this long, i think a minute
here is more than enough, also switched to using safe_while().

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit 503f8679ed8ecadf47defbf4bb76abdcab009846)

qa: fix testcase 'test_cluster_set_user_config_with_non_existing_clusterid'

Fixes: https://tracker.ceph.com/issues/58758
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit 3b3fa71edfad45a4f8b9085dd109b756d1f0778b)

qa: fix cluster creation failure in test_nfs.py

Also adds a function _nfs_complete_cmd() that returns process obj so that stdout/stderr
can be used for evaluation(_nfs_cmd() uses raw_cluster_cmd() that returns just stdout
and it became difficult to time cluster creation errors in _test_create_cluster()).

It takes sometime to update the cluster data, therefore running the command set
(check nfs server status -> nfs cluster create test -> check cluster status) in
a loop (max six iteration with sleep of 5 secs at each iteration) fixes the issue.

Fixes: https://tracker.ceph.com/issues/58744
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit 7cb3e58ecba037d21d76e417ee352a997fd028f0)

qa: test export creation at filepath and symlink

Fixes: https://tracker.ceph.com/issues/58228
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit afaf251d212044367d8abc4843de2ff98c86616c)

qa: added test case test_nfs_export_with_invalid_path

Fixes: https://tracker.ceph.com/issues/58228
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit b0bb25024d8fad2c06138571b830afcea4994be0)

mgr/nfs: disallow non-existent paths when creating export

Fixes: https://tracker.ceph.com/issues/58228
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit 992c686a3dbff3b917885603c017ef3831143296)

mgr/nfs/tests: mock check_cephfs_path

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit 6334e10fa5b99576ec4b307abdcaabc5a4eeb07b)

mgr/nfs/utils: add helper func to check cephfs path

this helper instantiates CephfsClient, however this was
initially planned in ExportMgr class in export.py but
due to make check failure where main python thread
experienced a dead lock which after several efforts
pointed at instantiation of CephfsClient in ExportMgr
was problematic, it was decided in order to achieve
singleton behavior, func has been added inside this
helper func that restricts instantiation using functool's
lru_cache.

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit 1356992c1c1362a994bc5e17d857d2dfc01791c5)

Merge pull request #50796 from zdover23/wip-doc-2023-03-31-backport-50674-to-reef

reef: doc/rados/ops: edit health-checks.rst (1 of x)

doc/rados/ops: edit health-checks.rst (1 of x)

Edit docs/rados/operations/health-checks.rst (1 of x).

https://tracker.ceph.com/issues/58485

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit cb1e20f879edb82ae619693a4f1f5e88fd0593dd)

Merge pull request #50769 from zdover23/wip-doc-2023-03-30-backport-50742-to-reef

reef: doc/start: update "notify us" section

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #50787 from zdover23/wip-doc-2023-03-30-backport-50764-to-reef

reef: doc/start: format procedure in documenting-ceph

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #50609 from rhcs-dashboard/wip-59094-reef

reef: mgr/dashboard: add button toggle to switch between landing pages

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

doc/start: format procedure in documenting-ceph

Organize three steps in a procedure into one step with three sub-steps.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6f3cd78f6116f3e19a8f6c5e0ebda816067a334e)

Merge pull request #50604 from neesingh-rh/wip-59912-reef

reef: cephfs-top: Handle METRIC_TYPE_NONE fields for sorting

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/start: update "notify us" section

Rewrite the "Notify Us" section in doc/start/documenting-ceph.rst so
that it is written in English sentences.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 613ece24777c4b222054fe9a6d82ba51949c45ea)

Merge pull request #50761 from zdover23/wip-doc-2023-03-30-backport-50744-to-reef

reef: doc/rados: add link to ops/health-checks.rst

doc/rados: add link to ops/health-checks.rst

Add an internal navigation link to
doc/rados/operations/health-checks.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f2f968f65bd1f1a6975fd340c66703cea335dcf6)
(cherry picked from commit 1ddc2b410258449e5d89a415a0494da2b8a79060)

Merge pull request #50751 from zdover23/wip-doc-2023-03-30-backport-50750-to-reef

reef: doc/rados/operations: Fix typo in erasure-code.rst

doc/rados/operations: Fix typo in erasure-code.rst

Signed-off-by: Sainithin Artham <sai.artham.19cse@bmu.edu.in>
(cherry picked from commit 3269fea2a209928137e55db16328c581988c048a)

Merge pull request #50739 from zdover23/wip-doc-2023-03-29-backport-50723-to-reef

reef: doc/start: documenting-ceph - add squash procedure

doc/start: documenting-ceph - add squash procedure

Add a procedure to doc/start/documenting-ceph.rst that explains how to
perform an interactive rebase to squash commits.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 71ee225d7b8bf54bd190a8b1f2422c6837601f16)

Merge pull request #50701 from zdover23/wip-doc-2023-03-28-backport-50697-to-reef

reef: doc/glossary: add "Scrubbing"

doc/glossary: add "Scrubbing"

Add "Scrubbing" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 4a66819da48cee430ca74830e1a6f63685b6b240)

Merge pull request #50484 from cbodley/wip-58932-reef

reef: rgw/admin: 'bucket stats' displays non-empty time

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #50688 from weirdwiz/wip-59175-reef

reef: rbd-mirror: switch to labeled perf counters

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rbd-mirror: add and rename perf counters for journal and snapshot mirroring

This commit renames the existing snapshot and journal based performance
counters in the rbd-mirror daemon to better reflect their purpose.
Additionally, new perf counters have been added to provide more detailed
information about the synchronization of snapshots between the source
and destination clusters.

This commit also switches to use labels instead of having the image spec
in the name of the counter.

Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>
(cherry picked from commit 1a1477b9fd7f3fcf7eb66c8a768476bea1322d1d)

Merge pull request #50677 from zdover23/wip-doc-2023-03-27-backport-50675-to-reef

reef: doc/rados: clean up ops/bluestore-migration.rst

doc/rados: clean up ops/bluestore-migration.rst

Clean up internal links, fix the numbering of a procedure, and implement
Anthony D'Atri's suggestions in
https://github.com/ceph/ceph/pull/50487 and
https://github.com/ceph/ceph/pull/50488.

https://tracker.ceph.com/issues/58485

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit b28be76d0f9a5d78943fe3c583a234d0b50356f1)

Merge pull request #50656 from petrutlucian94/wip-59161-reef

reef: rbd-wnbd: optionally handle wnbd adapter restart events

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50671 from zdover23/wip-doc-2023-03-26-backport-50654-to-reef

reef: doc/glossary: add "User"

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/glossary: add "User"

Add "User" to glossary.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit fd6bfaf3fe6337511dd7ab344c2936f22a5d43bf)

Merge pull request #50661 from zdover23/wip-doc-2023-03-25-backport-50660-to-reef

reef: doc/operations: fix prompt in bluestore-migration

doc/operations: fix prompt in bluestore-migration

Fix a single prompt in bluestore-migration.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 5e54641aec134188c8001e212ea66c1a15d5945c)

Merge pull request #50629 from mkogan1/wip-59126-reef

reef: rgw : fix python script using s3cmd with error code 403

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50603 from mkogan1/wip-59004-reef

reef: qa: d3n: fix test failure when "inline_data": false

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50577 from cbodley/wip-qa-reef-ubuntu22

reef: qa/distros: add ubuntu 22 as supported distro

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #50509 from ivancich/wip-59055-reef

reef: rgw: experimental support for restoring a lost bucket index

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

rbd-wnbd: consistently use negative error codes in rbd-wnbd

The rbd-wnbd iterators return positive errors, which is why
in certain cases we may end up with both positive and negative
error codes.

This change ensures that we'll consistently use negative
error codes.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit 98a7aff741eb95bcc98df625d6558d726711a5fb)

common, rbd-wnbd: bump Windows log level

We're increasing the log level for certain Windows operational log
messages.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit 3d8afc00215146fbc0226c2c6ccc7cb5dd3a2bb5)

rbd-wnbd: optionally handle wnbd adapter restart events

The WNBD adapter may be reset in certain situations (e.g. driver
upgrade, MS WHQL tests, etc).

We're going to monitor the WNBD adapter using WMI[1] events, restarting
the rbd-wnbd disk mappings whenever necessary. Adapter monitoring can be
enabled by passing the --adapter-monitoring-enabled flag to the service.

This feature is optional for the following reasons:

* it's mainly used during development / driver certification
* we had to use a relatively small polling interval, which might imply
  additional resource usage. WMI quotas also have to be considered.

While at it, we're updating two lambdas that are submitted to thread pools,
avoiding default reference capturing and explicitly specifying the variables
that get copied.

[1] https://learn.microsoft.com/en-us/windows/win32/wmisdk/wmi-start-page

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit 0c25ca656430b4d5280ab7d033495ad8ce4e776c)

common: add win32/wstring.h

Windows APIs heavily use wchar. ceph-dokan and rbd-wnbd
have some duplicated helpers that convert wstrings to/from
utf8 strings.

To avoid duplication and allow reusing those helpers, we're moving
them to common/win32/wstring.h.

We're using the "win32" subfolder because it's unlikely that this
will ever be used on other platforms.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit a47caa156a4e667d6b9ff7236cc9148ba37ea772)

Merge pull request #50640 from zdover23/wip-doc-2023-03-23-backport-50634-to-reef

reef: doc/rados: edit user-management.rst (1 of x)

doc/rados: edit user-management.rst (1 of x)

Edit doc/rados/operations/user-management.rst. (1 of x)

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 3f90958909815a287279eccb22193ba5d67cfc1d)

rgw : fix python script using s3cmd with error code 403 w/https

Fixes: https://tracker.ceph.com/issues/54104
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit 2cb94411801201b53789b3b9b509abccb63c4b1f)

rgw : fix python script using s3cmd with error code 403

Fixes: https://tracker.ceph.com/issues/54104
Signed-off-by: caolei <cao.leilc@inspur.com>
(cherry picked from commit a10baddb82b6febfb1765ccb3d3c2d007b1f2aee)

Merge pull request #50618 from zdover23/wip-doc-2023-03-22-backport-50606-to-reef

reef: doc/rados: line-edit erasure-code.rst

doc/rados: line-edit erasure-code.rst

Line-edit doc/rados/operations/erasure-code.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 2ff9965e02756636f8d33fd96b7c909f2fca7414)

mgr/dashboard: add button toggle to switch between new and old landing page

Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
(cherry picked from commit 21dc93ae11db02bc8e0a2175f21fa7dd5e423b00)

Merge pull request #50572 from petrutlucian94/wip-59098-reef

reef: common: avoid redefining clock type on Windows

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cephfs-top: Handle `METRIC_TYPE_NONE` fields for sorting

Currently, the sort feature in cephfs-top isn't applicable
to the fields which are of type `METRIC_TYPE_NONE` , named as
'ofiles', 'oicaps' and 'oinodes'. This commit will make the
cephfs-top sort feature to support the above mentioned fields too.
From this commit onwards, all the current fields displayed in
cephfs-top can be used for sorting.

Fixes: https://tracker.ceph.com/issues/58814
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>
(cherry picked from commit 4626cb5fb1141c38acd4b114ad4562bd14fa7aad)

reef: qa: d3n: fix test failure when "inline_data": false

backport tracker: https://tracker.ceph.com/issues/59004
backport of https://github.com/ceph/ceph/pull/50003
parent tracker: https://tracker.ceph.com/issues/58552

Fixes: https://tracker.ceph.com/issues/58552
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit f474c7fdbe937a07451ecef246e43e3f46b7010d)

Merge pull request #50589 from zdover23/wip-doc-2023-03-20-backport-50488-to-reef

reef: doc/rados: edit operations/bs-migration (2 of x)

Merge pull request #50586 from zdover23/wip-doc-2023-03-20-backport-50487-to-reef

reef: doc/rados: edit operations/bs-migration (1 of x)

doc/rados: edit operations/bs-migration (2 of x)

Disambiguate and improve the English language in
doc/rados/operations/bluestore-migration.rst up to but not including the
section called "Whole Host Replacement".

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ca803a24c64059023733e21d755edb9c6c973ecf)

doc/rados: edit operations/bs-migration (1 of x)

Disambiguate and improve the English language in
doc/rados/operations/bluestore-migration.rst up to but not including the
section called "Whole Host Replacement".

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 96166b463b1f8800e59f80faebfba1208248725e)

Merge pull request #50581 from zdover23/wip-doc-2023-03-19-backport-50579-to-reef

reef: doc/rados: line edit mon-lookup-dns top matter

doc/rados: line edit mon-lookup-dns top matter

Improve the syntax of the top matter of
doc/rados/operations/mon-lookup-dns.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ed5919964bd914c118f556baf3c517e5e35df277)

qa/workunits/rbd: use bionic version of qemu-iotests for jammy

Same as in commit 2de2146c300c ("qa/workunits/rbd: use bionic version
of qemu-iotests for focal").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 3b1610997a3f36c7ab4a7721cd266ec0f14bc107)

qa/distros/supported-random: update ubuntu_latest.yaml and add ubuntu_20.04.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 62e520cbad7a2beff130a4459abcb0fb7d762da0)

qa/distros/supported: update ubuntu_latest.yaml and add ubuntu_20.04.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 929172bd718bdc45f796ad52c64dd42c1bc70a9a)

qa/distros/all: add ubuntu_latest.yaml symlink to ubuntu_22.04.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 31846942edd4f10c5250fdedcf69c18d535e43f2)

Merge pull request #50565 from zdover23/wip-doc-2023-03-17-backport-50563-to-reef

reef: doc/rados: line-edit upmap.rst

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

common: avoid redefining clock type on Windows

mingw >= 8.0.1 defines CLOCK_REALTIME_COARSE, so we'll avoid
overriding it if already set.

Clock precision [1]:

mingw < 8.0.1:
  * CLOCK_REALTIME: ~10-55ms (GetSystemTimeAsFileTime)
mingw >= 8.0.1:
  * CLOCK_REALTIME: <1us (GetSystemTimePreciseAsFileTime)
  * CLOCK_REALTIME_COARSE: ~10-55ms (GetSystemTimeAsFileTime)

* CLOCK_MONOTONIC: <1us if TSC is usable, ~10-55ms otherwise
                   (QueryPerformanceCounter)

[1] https://github.com/mirror/mingw-w64/commit/dcd990ed423381cf35702df9495d44f1979ebe50

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit 489fd858479a79de75f6327d5cca40c289b53f35)

doc/rados: line-edit upmap.rst

Edit all of doc/rados/upmap.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit a4a681846499c8d0041a6632e7e31fa83e22ed0e)

Merge pull request #50541 from rhcs-dashboard/wip-59072-reef

reef: mgr/dashboard: fix prometheus api issues on landing page v3

Reviewed-by: Pegonzal <NOT@FOUND>

Merge pull request #50532 from zdover23/wip-doc-2023-03-15-backport-50390-to-reef

reef: doc/crimson: add SeaStore when using vstart.

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

mgr/dashboard: improve the logic on the alert viewer

only fetch the alerts when you toggle the alert viewer. don't fetch it
every 5 second cause its going to cause a heavy toll on the api.

Fixes: https://tracker.ceph.com/issues/58867
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 6cb0679aaf28b5f128ca3cae5f962cac50d084af)

mgr/dashboard: fix prometheus api error on landing page v3

When no prometheus is configured in the cluster, it gives out error
while polling to the prometheus endpoint. so a proper check needs to be
added there.

Also some improvements to the exisiting behaviour

Fixes: https://tracker.ceph.com/issues/58867
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 2d80e57dd4a60cbf52074d982ba3ebd4471a14ca)

Merge pull request #50525 from epuertat/wip-59050-reef

reef: mgr/dashboard: force TLS 1.3

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

doc/crimson: add SeaStore when using vstart.

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>
(cherry picked from commit 055bb2bd5364e66c6e7f87760b9c409f2b8c7935)

mgr/dashboard: force TLS 1.3

Fixes: https://tracker.ceph.com/issues/58942
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit cd8946631b222dfff1727083d75dfe2dd825d1a4)

Merge pull request #50460 from rhcs-dashboard/custom-image-kcli-bootstrap-reef

reef: mgr/dashboard: custom image for kcli bootstrap script

Reviewed-by: Pegonzal <NOT@FOUND>

Merge pull request #50517 from zdover23/wip-doc-2023-03-14-backport-50500-to-reef

reef: doc/dev/release-checklist: add another step for ceph-build

doc/dev/release-checklist: add another step for ceph-build

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 4fcad7161fa094af210229b6f15cded63781080e)

Merge pull request #50464 from ljflores/wip-tracker-58496-reef-backport

reef: osd/scrub: only call publish_stats_to_osd() where allowed and needed

rgw: add rgw-restore-bucket-index

Adds an experimental script that allows a bucket index of a
non-versioned bucket to be restored by applying `radosgw-admin object
reindex ...` to all objects in the specified bucket. The objects in
the bucket are determined by scanning the data pool for head objects
containing the bucket's marker.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 6ea11b1305c77f9141f2dd8093ac56b579f250be)

Merge pull request #50424 from ljflores/wip-tracker-58925-reef

reef: qa: suppress memory leak in rocksdb

rgw: add an "object reindex" subcommand to radosgw-admin

This subcommand will take a bucket and an object (or a list of objects
in a file) and add those objects to the bucket's index. It does so by
preparing the bucket index entry update and then allowing the
so-called "dir suggest" mechansim, which is triggered by listing the
bucket, to completing the bucket index entry. This mechanism is
triggered by listing the bucket.  Using this mechanism has the
advantage of doing this lazily that both allows the reindex operation
to run more quickly and distributes the workload over time.

This subcommand can help restore lost bucket index entries.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit be0583c3082b9f924fcdbe4c81d540ad08bcca24)

rgw/sysobj: read() can query size/mtime

rgw_get_system_obj() stopped calling stat() before read() in
90aec61298fc378e1733bc76f0d3f95ce8ec135a, which left the optional
'real_time *pmtime' argument uninitialized

when requested, read() will add a stat op to initialize size/mtime

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 20ab42758c8ba0cef867752be3b215064d44a6f2)

rgw/admin: 'bucket stats' displays non-empty mtime

Fixes: https://tracker.ceph.com/issues/58932
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e9720ddbe9ae033e14c2f3def51a06a4e204d4c8)

Merge pull request #50414 from zdover23/wip-doc-2023-03-07-backport-50384-to-reef

reef: doc/radosgw: Redd up s3select.rst as followup to #49411

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #50476 from zdover23/wip-doc-2023-03-11-backport-50470-to-reef

reef: doc/index: remove "uniquely" from landing page

doc/index: remove "uniquely" from landing page

Change "Ceph uniquely delivers[...]" to "Ceph delivers[...]"

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6e5be7ad66aa76cb798c52c87e34f8fe079cda5c)

Merge pull request #50452 from zdover23/wip-doc-2023-03-09-backport-50439-to-reef

reef: doc/rados/operations: Improve wording, capitalization, formatting

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

osd/scrub: only call publish_stats_to_osd() where allowed and needed

This patch removes several unnecessary calls to publish_stats_to_osd() in
the clear_pgscrub_state pathway used during interval changes. We didn't
actually have anything to publish and we can't validly call publish_stats_to_osd()
when we aren't an active primary anyway (which we can't be during an interval
change).

Fixes: https://tracker.ceph.com/issues/58496
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
(cherry picked from commit aeff19db99ab53fbaec7eb530e35d578e6254e4c)

mgr/dashboard: custom image for kcli bootstrap script

the stable branches like quincy pulls from the quay.io/ceph/ceph:v17 to
bootstrap the ceph cluster in test environments. This will cause issues
because the branches are changing constantly but the image is not. So
using the quay.ceph.io repo to bring the cluster in test environment.

Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit be8f1e7f4a81dbf04692446edcc429d93d59e09d)

Merge pull request #50387 from zdover23/wip-doc-2023-03-06-backport-50386-to-reef

reef: doc/rgw - fix grammar in table in s3.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados/operations: Improve wording, capitalizatiopn, formatting

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 5f2060c0829992b31a0d17e23d7b1d2fb2280ac0)

Merge pull request #50431 from zdover23/wip-doc-2023-03-08-backport-50430-to-reef

reef: doc/install: update index.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/install: update index.rst

Update index.rst by making minor grammar improvements. This file was
long overdue for a backport to Reef, Quincy, and Pacific, so this commit
was a good way to pass a human eyeball over the text before making those
backports.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 82d748ff590cfd31fefddc094b10622d090b054f)

Merge pull request #50417 from soumyakoduri/wip-skoduri-reef

[Reef] rgw/tests: Reset endpoint before creating additional zones

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

rgw/tests: Reset endpoint before creating additional zones

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Fixes: https://tracker.ceph.com/issues/58822

doc/radosgw: Redd up s3select.rst as followup to #49411

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 78b42ee0d27995a322cdd3998fb9726e490007a0)

Merge pull request #50399 from zdover23/wip-doc-2023-03-07-backport-50398-to-reef

reef: doc/dev: add full stop to sentence in basic-wo

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #50374 from cbodley/wip-rgw-reef-s3select-json

reef: json integration: identifying new data-source and execute json flow.

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>