test: Check for interuption of scrubs with nosrub/nodeep_scrub

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 365e48d6ec71cec7c85acbf918927a20856d55da)

Conflicts:
qa/standalone/scrub/osd-scrub-test.sh (The scrub/deep_scrub trigger
doesn't use ceph tell)

osd: Cancel in-progress scrubs (not user requested)

This change adds new scrubber.req_scrub to track user
requested scrubs, deep_scrub or repair.

Fixes: https://tracker.ceph.com/issues/46275
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 33749cc3c39131d7abed9b8c14064dbfaa87f3a2)

Conflicts:
PendingReleaseNotes (trivial)
src/osd/PG.cc (Due to code re-arrangement changes add manually)

osd: Arrange code so that it is clearer should not cause any change

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit ef9ab850d87a84ce42c2c8df1c9b502b2fb04989)

test: mon-last-epoch-clean.sh fixed to avoid shell globbing

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f272768802648f2cb4d519c4c790ab219ad43f9f)

Merge pull request #36186 from smithfarm/wip-45930-nautilus

nautilus: rgw: Add support wildcard subuser for bucket policy

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #36163 from ivancich/nautilus-bucket-listing-clear-results

nautilus: rgw: cls_bucket_list_(un)ordered should clear results collection

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35601 from smithfarm/wip-46012-nautilus

nautilus: qa/vstart_runner: update vstart_runner.LocalRemote.sh

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #35401 from smithfarm/wip-45854-nautilus

nautilus: tools/cephfs: don't bind to public_addr

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34461 from Vicente-Cheng/wip-44487-nautilus

nautilus: qa: add upgrade test for volume upgrade from legacy

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36166 from ShyamsundarR/wip-46527-nautilus

nautilus: mgr/volumes: Deprecate protect/unprotect CLI calls for subvolume snapshots

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36193 from smithfarm/wip-46476-nautilus

nautilus: rgw/iam: correcting the result of get role policy.

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #36192 from smithfarm/wip-46472-nautilus

nautilus: rgw: stop realm reloader before store shutdown

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #36191 from smithfarm/wip-46468-nautilus

nautilus: rgw: radoslist incomplete multipart uploads fix marker progression

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #36189 from smithfarm/wip-46004-nautilus

nautilus: rgw: add check for index entry's existing when adding bucket stats during bucket reshard.

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #36187 from smithfarm/wip-45932-nautilus

nautilus: rgw: Add subuser to OPA request

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #36185 from smithfarm/wip-45927-nautilus

nautilus: rgw: fix loop problem with swift stat on account.

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #36184 from smithfarm/wip-45925-nautilus

nautilus: rgw: add quota enforcement to CopyObj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #36165 from ivancich/wip-45923-nautilus

nautilus: rgw: bucket list/stats truncates for user w/ >1000 buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: cls_bucket_list_(un)ordered should clear results collection

Each call to cls_bucket_list_(un)ordered should have an empty
collection to populate with results. Rather than rely on the caller to
insure this, it's more reliable to have these functions do the clear.

Additionally in some cases, a reserve call was added to the collection
to pre-allocate the space needed for the expected number of
results. This will potentially result in fewer re-allocations plus
copies.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit c52039a9e8a49f3a29e2d2ee80e50d7b1b441842)

Conflicts:
    src/rgw/rgw_admin.cc
    src/rgw/rgw_rados.cc

mgr/volumes: Remove enum usage in code for Py2 support

SubvolumeFeatures uses an enum for the feature list, this is
replaced with a constant instead.

Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

rgw/iam: correcting the result of get role policy.

Fixes: https://tracker.ceph.com/issues/45671
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit 9e1ae9de9905587a227e50d0aff6533e290a8f87)

rgw: add quota enforcement to CopyObj

Fixes: https://tracker.ceph.com/issues/45382
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 70d397541cc8c0a68b3e420a78fec8f0f66f06b0)

Conflicts:
    src/rgw/rgw_op.cc
- nautilus uses plain "store->" where master has "store->getRados()->"
- nautilus store->get_object_state takes 6 arguments instead of 7 in
  master

rgw: stop realm reloader before store shutdown

otherwise the two could race to destroy the store

Fixes: https://tracker.ceph.com/issues/45969
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 7787dfb4926dcc2baa7530b620ae8cfccf90bc30)

Conflicts:
src/rgw/rgw_main.cc
- RGWRealmWatcher realm_watcher(...) line looks different in nautilus

rgw: radoslist incomplete multipart uploads fix marker progression

When the radosgw-admin radoslist was running, the portion where it
lists the incomplete multipart uploads did not handle the marker
correctly between calls to the ordered list op. This fixes those
issues and does some additional clean-up.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 653848da4ae67a45a5d72b0daece86de939f5e15)

rgw: add check for index entry's existing when we add  bucket stats during resharding.

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>
(cherry picked from commit ce42734773203bfd2d620e6974e428e1b5d3f480)

rgw: Add subuser to OPA request

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit 793aaaaed9029e032128b50767a5faf1bb7f6d81)

Conflicts:
src/rgw/rgw_opa.cc
- jf.dump_object("user_info", ...) line looks different in nautilus

rgw: Add support wildcard subuser for bucket policy

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit 0008f84cb03402b159001bf6b6c3a2ff48b2de9b)

rgw: fix loop problem with swift stat on account.

Loop logic in RGWStatAccount::execute was failing to change the
marker before repeating a call to rgw_read_user_buckets().  This resulted
that in the case that a user had more than 1000 buckets, this routine
would loop forever.

Fixes: https://tracker.ceph.com/issues/44671
Signed-off-by: Marcus Watts <mwatts@redhat.com>
(cherry picked from commit c9f3cf6ed6a649812d47dc1fa1357ae0749dbcbf)

Conflicts:
src/rgw/rgw_op.cc

mgr/volumes: Deprecate protect/unprotect CLI calls for subvolume snapshots

Subvolume snapshots required to be protected, prior to cloning the same.

Also, protected snapshots were not allowed to be unprotected or removed,
if there were in-flight clones, whose source was the snapshot being
removed.

The protection of snapshots explicitly is not required, as these can be
prevented from being removed based only on the in-flight clones checks.

This commit hence deprecates the additional protect/unprotect requirements
prior to cloning a snapshot.

In addition to deprecating the above, support to query a subvolume for
supported features, via the info command, is added. The feature list
is set to "clone" and "auto-protect", where the latter is useful to
decide if protect/unprotect commands are required or not.

Fixes: https://tracker.ceph.com/issues/45371
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>
(cherry picked from commit c52a7383d391864b7dce3ba09197a077a998dff7)

Conflicts:
src/pybind/mgr/volumes/fs/operations/versions/subvolume_v1.py
- required manual resolution during cherry-pick to delete unused functions
doc/cephfs/fs-volumes.rst
- added line space for correcting multi-line command rendering

Merge pull request #36104 from neha-ojha/wip-45228-nautilus

nautilus: mgr/alert: can't set inventory_cache_timeout/service_cache_timeout from CLI

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35918 from callithea/wip-46172-nautilus

nautilus: mgr/prometheus: improve Prometheus module cache

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

rgw: convert rgw_user_max_buckets to new config options

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 97bd1dca5c97c4e077de0b88f1280477e521bc43)

rgw: clean up code around buckets listing

This commit adds some const correctness, avoids some string/structure
copying, and reformats some code.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 1bef0e216a58402ce2a60e9b61d215c58ca6d765)

Conflicts:
src/rgw/rgw_bucket.cc -- small organizational differences

rgw: bucket list/stats truncates for user w/ >1000 buckets

The is_truncated flag was not being handled properly and therefore
limited the output on these radosgw-admin commands to 1000 buckets.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 11ab7c8234daa3d4c077710ea4c8b0437e57daf8)

Conflicts:
src/rgw/rgw_bucket.cc -- small organizational adjustments
src/rgw/rgw_sal.h -- does not exist in nautilus

Merge pull request #36075 from smithfarm/wip-46515-nautilus

nautilus: mgr/progress: Skip pg_summary update if _events dict is empty.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #36031 from smithfarm/wip-46017-nautilus

nautilus: remove ceph_test_rados_watch_notify

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35738 from neha-ojha/wip-46164-nautilus

nautilus: osd: make message cap option usable again

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge pull request #35959 from callithea/wip-46393-nautilus

nautilus: mon/PGMap: do not consider changing pg stuck

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #36086 from smithfarm/wip-46512-nautilus

nautilus:  rgw: lc: fix Segmentation Fault when the tag of the object was not found

Reviewed-by: Friedmann <ofriedma@redhat.com>

Merge pull request #34127 from ivancich/nautilus-rados-list

nautilus: rgw: add "rgw-orphan-list" tool and "radosgw-admin bucket radoslist ..."

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #35044 from smithfarm/wip-45480-nautilus

nautilus: doc/rgw: document 'rgw gc max concurrent io'

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #35672 from kotreshhr/wip-46002-nautilus

nautilus: mgr/volumes: Add snapshot info command

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35404 from smithfarm/wip-45684-nautilus

nautilus: core: os/bluestore: fix large (>2GB) writes when bluefs_buffered_io = true

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #35390 from smithfarm/wip-44506-nautilus

nautilus: mgr/telemetry: force --license when sending while opted-out

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35389 from smithfarm/wip-45890-nautilus

nautilus: osd/PeeringState.h: Fix pg stuck in WaitActingChange

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35388 from smithfarm/wip-45883-nautilus

nautilus: tests: qa/*/osd-scrub-repair.sh: Convert to python3 print syntax

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35386 from smithfarm/wip-45776-nautilus

nautilus: osd: make "missing incremental map" a debug log message

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35822 from bk201/wip-46198-nautilus

nautilus: mgr/dashboard: fix Source column i18n issue in RBD configuration tables

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #35998 from ricardoasmarques/wip-46435-nautilus

nautilus: mgr/dashboard: Allow to edit iSCSI target with active session

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #35563 from pcuzner/wip-45972-nautilus

nautilus: mgr/k8sevents: sanitise kubernetes events

rgw: orphan list teuthology test & fully-qualified domain issue

Sometimes when teuthology machines are provisioned, the command
`hostname --fqdn` does not provide a fully qualified domain name but
instead just the hostname (e.g., smithi149 instead of
smithi149.front.sepia.ceph.com). This prevents the teuthology test for
rgw-orphan-list from running successfully [for example, the hostname
was for some reason mis-interpreted as the bucket name in the
request].

This commit checks whether the hostname derived from `hostname --fqdn`
contains any '.'s and if it does not, it will append
".front.sepia.ceph.com" to the hostname. This is a hack, but until
teuthology machines are configured appropriately it seems to be a
reasonable work-around.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 3afc8ac25390493a2485ad1999e6370ddbe8b566)

rgw: rgw-orphan-list -- fix interaction, quoting, and percentage calc

The interactive mode wasn't working due to prompts going to stdout
instead of stderr. If a space should appear in temporary file, it will
generate a shell error, so quoting was added. Furthermore if there are
no objects found in a pool, a divide by zero error will be
generated. This commit addresses these issues.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 6d9d548096d2b5b27a2868562112d90486ab2c78)

rgw: orphan-list timestamp fix

When creating intermediate and output files, the rgw-orphan-list
script uses a timestamp using the `date` command. The hour was
inserted with "%k" but that padds with a space rather than a zero. So
that's changed to "%H".

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit b1daf30d61989d0813ea73ac19f36b4de6abf13d)

rgw: remove unused function obj_force_ns

Removing the function eliminates a compiler warning.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 9a1e89173d398bf6e39a431d4b4ef9ab1f20e2bb)

PendingReleaseNotes: `rgw-orphan-list` and deprecated `radosgw-admin orphans`

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit fe253c9d5f629cf7688b3dd1a3f8c3de89657a02)

qa/rgw: integration test for `rgw-orphan-list` & `radosgw-admin radoslist`

Add teuthology test for `rgw-orphan-list` in a new tool suite under
rgw. It only needs to be tested under one configuration. And the new
tool sub-suite can be used by other tooling int he
future. radosgw-admin `radoslist` is tested indirectly through
`rgw-orphan-list` and therefore does not need its own test.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 43c948a4abdf6b87c956d4ee00a19ecf9144c564)

qa/rgw: allow the rgw teuthology task to capture/set dns names

A teuthology workunit might want to use the rgw task, setting the
rgw-dns-name and/or rgw-dns-s3website-name configuration options to
the fully-qualified domain name. Existing code implies that setting
these configuration options to the empty string will do that. However
the current logic does not support that given it has Python
conditionals that treat the empty string as false. This fixes that.

Now the following teuthology tasks YAML will work as expected:

  tasks:
  - rgw:
      client.0:
        dns-name: ''
        dns-s3website-name: ''

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 89654d0477a25592fafbc438aacce12988dc23c1)

test/rgw: add static testing for `ceph-diff-sorted`

Tests are run during "make check".

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 3aff078700b63377876c8e146ebdb359ef7ae954)

doc/rgw: add docs for `rgw-orphan-list` and `ceph-diff-sorted`

Add man pages and documentation for both tools.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 9d5e9c3031ecf9bda537ef2e0a6cffa5c2109b2a)

rgw: add `rgw-orphan-list` tool & `radosgw-admin bucket radoslist ...`

Adds a radosgw-admin subcommand and possibly a bucket name (if not,
all buckets are assumed) and walks the bucket(s) listing(s) and the
manifest (if it exists) for each object in the listing to generate the
rados objects that represent the rgw objects in the bucket.

Also adds a tool named rgw-orphan-list that will produce a list in a
local file of what appear to be rgw orphans.

NOTE: This is not a cherry-pick from master because the feature was
originally written for an older version of Ceph and is being gradually
forward-ported to newer and newer versions.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

mgr/alerts: fix module fails to start without smtp_host

The option setting loop in config_notify raises an exception if
smtp_host option is not configured.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
(cherry picked from commit b6b6f046b8af1589816b7bff33b2642ab331824e)

mgr/alerts: fix type of interval option

The option needs to be in `secs` type to enable input checking and
type casting.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
(cherry picked from commit 368c810a4200a57a772b8122fc606a2b36de7413)

Merge pull request #35798 from sseshasa/wip-46228-nautilus

nautilus: mon/OSDMonitor: Reset grace period if failure interval exceeds a threshold.

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35712 from rhcs-dashboard/wip-46019-nautilus

nautilus: mgr/dashboard: work with v1 RBD images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35882 from ivancich/nautilus-rgw-ordered-listing-stuck

nautilus: rgw: ordered listing lcv not managed correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgwlc: fix Segmentation Fault related to tags.

It is found that rgw crashes when putting a tag beyond the rule to the object, which can be solved by adding the return value judgment.

Fixes: https://tracker.ceph.com/issues/46485
Signed-off-by: zhuo li <lizhuo@chinatelecom.cn>
Signed-off-by: yupeng chen <chenyupeng@chinatelecom.cn>
(cherry picked from commit 5994ec12a7039f21b8f9e1077f2f721003de8cb6)

Conflicts:
src/rgw/rgw_lc.cc
- automated cherry-pick failed due to post-nautilus refactoring:
  made the change manually

mgr/progress: Skip pg_summary update if _events dict is empty.

The get call of pg_stats and pg_ready are expensive and are not necessary in this case. This avoids blocking other modules like the prometheus module

Fixes: https://tracker.ceph.com/issues/46416
Signed-off-by: Manuel Lausch <manuel.lausch@1und1.de>
(cherry picked from commit 1031edd89d09d18812a3f11b7cbb2ce6759bad13)

remove ceph_test_rados_watch_notify

This is an old test, we have good watch/notify coverage in the newer
tests, and it is buggy.

Fixes: https://tracker.ceph.com/issues/43861
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 282932867ab4c3e3a395b86f4a456fb9aaf7e530)

Merge pull request #36014 from djgalloway/nautilus

qa: Target CentOS 7.8 notcmalloc for this job

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: Target CentOS 7.8 notcmalloc for these jobs

We don't build nautilus on centos8 notcmalloc

Signed-off-by: David Galloway <dgallowa@redhat.com>

mgr/dashboard: work with RBD images v1

Add support for RBD Image Format v1:
- This format lacks ID field, required for dashboard. Instead,
RBD image `block_name_prefix` is used as unique ID (together with pool
id and namespace)
- Additionally, `image_format` is now exposed.
- In the front-end side:
  - Copy action on a v1 image will cause the image to be copied to v2
format.
  - List doesn't allow Move to Trash on v1 images,
  - Details section now shows `image_format` for images,
  - Edit Form disables flags not supported for v1 (`deep-flatten`,
`layering`, `exclusive-lock`).
  - Protect does not work on v1 images or v2 images created from v1
ones.

Fixes: https://tracker.ceph.com/issues/36354
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit 119851ec89745026c273334efbcbf2d50128c99d)

 Conflicts:
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-details/rbd-details.component.html: add row
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form.component.ts: add imports
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-list/rbd-model.ts: add new fields
src/pybind/mgr/dashboard/services/ceph_service.py: just add get_pool_name_from_id
src/pybind/mgr/dashboard/services/rbd.py: discard changes
src/pybind/mgr/dashboard/controllers/rbd.py: bring and adapt code from services/rbd.py

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: Allow to edit iSCSI target with active session

Fixes: https://tracker.ceph.com/issues/46383
Signed-off-by: Ricardo Marques <rimarques@suse.com>
(cherry picked from commit df6163e1369703117ac683bcbd5056e1543f4d06)

 Conflicts:
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-list/iscsi-target-list.component.spec.ts

Merge pull request #35400 from smithfarm/wip-45852-nautilus

nautilus: mds: flag backtrace scrub failures for new files as okay

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35399 from smithfarm/wip-45847-nautilus

nautilus: tests: qa/cephfs: run modprobe beforehand so that /sys/fs/fuse/connections is never absent

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35398 from smithfarm/wip-45843-nautilus

nautilus: cephfs: ceph_fuse: add the '-d' option back for libfuse

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35397 from smithfarm/wip-45839-nautilus

nautilus: cephfs: mds: cleanup uncommitted fragments before mds goes to active

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35396 from smithfarm/wip-45774-nautilus

nautilus: cephfs: vstart_runner: set mounted to True at the end of mount()

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35394 from smithfarm/wip-45709-nautilus

nautilus: mds: preserve ESlaveUpdate logevent until receiving OP_FINISH

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35377 from thmour/nautilus

nautilus: cephfs: mds: add config to require forward to auth MDS

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35403 from smithfarm/wip-45887-nautilus

nautilus: tests: qa/tasks/cephfs/test_scrub.py: use umount_wait to avoid possible ceph-fuse daemon stuck

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34095 from rhcs-dashboard/wip-44696-nautilus

nautilus: mgr/dashboard: add popover list of Stand-by Managers & Metadata Servers (MDS) in landing page

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: add popover list of managers in landing page
Fixes: https://tracker.ceph.com/issues/42979
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit cdfeb1d196c7d47340baae2be5910b90c889e778)
Conflicts:
       src/pybind/mgr/dashboard/controllers/health.py
-removed few lines as those lines were removed in the master branch too
       src/pybind/mgr/dashboard/frontend/src/app/ceph/dashboard/health/health.component.html
-added the braces missing

mon/PGMap: do not consider changing pg stuck

there is chance that we have a PG just created but fails to peered
before a mgr module retrieves the health report from mgr. in that
case, the "last_peered" field is not set, as that pg has not peered. but
normally, the newly created PG will be active+clean in couple seconds
which is way under the default setting of mon_pg_stuck_threshold (60
seconds).

so in this change, if the "last_whatever" is not set, we also use the
"last_changed" as a reference to see if the PG is healthy, and only
consider PG stuck if the last_changed is also too old.

Fixes: https://tracker.ceph.com/issues/45717
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 34e1df66cdf9ac4aeea338a8f3d5b9a10fa5983a)

Merge pull request #35385 from smithfarm/wip-45800-nautilus

nautilus: rbd: librbd: Watcher should not attempt to re-watch after detecting blacklisting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35384 from smithfarm/wip-45778-nautilus

nautilus: rgw/url: fix amqp urls with vhosts

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #35884 from rhcs-dashboard/wip-46312-nautilus

nautilus: mgr/dashboard: Prometheus query error in the metrics of Pools, OSDs and RBD images

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

mgr/prometheus: improve Prometheus module cache

Improve cache by running requests in a thread and prevent multiple
requests to Ceph from multiple sources (e.g. Prometheus instances) which
increase load on the manager.

Fixes: https://tracker.ceph.com/issues/45554
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit fa69d8e1112d2688e2979ab10c303d1facb6bc76)

 Conflicts:
src/pybind/mgr/prometheus/module.py
- line 1096 if condition block changed
- second commit "mgr/prometheus: enable mypy type checking for prometheus module"
  is not required for the backport as discussed with Patrick
  who is the creator of the original PR.

qa/cephfs: don't crash if mountpoint dir is already deleted

Teuthology jobs crash due to this after the test and it's teardown has
run successfully.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit af2a8eb24b0dfd3aadfb2f60e5200d6fe4377327)

qa/cephfs: don't abort if mountpoint is already present

Instead, let the execution continue.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 39a15d091fd1e62c3dcc7e5c4c042f18cc65a37c)

Conflicts:
qa/tasks/cephfs/fuse_mount.py
  - save the parameter cwd in nautilus

qa/cephfs: allow specifying mountpoint for kernel mounts

Allow specifying the directory in CephFS that should be kernel mounted
at the given path.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit b7005a2ac2dacb7eb7b8116ca86068be82cc3807)

qa/vstart_runner.py: allow specifying mountpoint for local FUSE mounts

Allow specifying the directory in CephFS that should be FUSE mounted on
the given path when running tests locally.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 514b58e3a32835e10c36a16316e8102cad54f75e)

qa/mount.py: allow setting mountpoint

Add setter method so that mountpoint for CephFSMount object can be set
later.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit f1e521a43ba25ea1a764c2f012aebef5c8e0432d)

qa/cephfs: allow specifying mountpoints for FUSE mounts

Allow specifying the directory in CephFS that should be FUSE mounted at
the given path.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 48a95d00ba19e4ffa10a950444073135e83517bd)

qa: add upgrade test for volume upgrade from legacy

This tests that volumes created using the ceph_volume_client.py library
continue to be accessible/function via the Nautilus/Octopus ceph-mgr
volumes plugin.

Fixes: https://tracker.ceph.com/issues/42723
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 0c8899c985763826fb77fed599eeea5624263a35)

Conflicts:
qa/tasks/cephfs/fuse_mount.py
  - use the coding format with nautilius
  - use `require-osd-release nautilus` for this version

Merge pull request #35367 from Devp00l/wip-45558-nautilus

nautilus: mgr/dashboard: Prevent dashboard breakdown on bad pool selection

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #35215 from shyukri/wip-40407-nautilus

nautilus: mgr/dashboard: Add QA tests for RBD configuration

Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: Prometheus query error in the metrics of Pools, OSDs and RBD images
Fixes: https://tracker.ceph.com/issues/45068
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 47b515c09496da8fc326300bab6618250466effe)

rgw: clean up attempt loop in ordered listing

Convert a while loop into a for loop to better indicate how the
"attempt" lcv is managed.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 6ef83820564b07dbff91b3fc6909082537c0f815)

Merge pull request #35831 from guits/wip-46250-nautilus

nautilus: ceph-volume: add dmcrypt support in raw mode

Merge pull request #35778 from cbodley/wip-46119

nautilus: test/rgw: update hadoop versions

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>