mgr/mgr_module: metadata is fetched async

fetch metadata using a defaultdict

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit f06e60b591d7921b0afdd40219f3866773b00683)

Conflicts:
src/pybind/mgr/status/module.py

mgr/status: metadata is fetched async

fix timing window during mds rejoin where metadata is not available

Fixes: https://tracker.ceph.com/issues/45633
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 4006fff43856c851e7a07f7d8ff05364cbe03529)

Conflicts:
src/pybind/mgr/status/module.py

Merge pull request #37510 from rhcs-dashboard/wip-47717-nautilus

nautilus: mgr/dashboard: Fix pool renaming functionality

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #37241 from rhcs-dashboard/wip-47538-nautilus

nautilus: mgr/dashboard: Disabling the form inputs for the read_only modals

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #37177 from batrick/i47017

nautilus: mds: kcephfs parse dirfrag's ndist is always 0

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #37035 from smithfarm/wip-47317-nautilus

nautilus: mds: fix incorrect check for if dirfrag is being fragmented

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36967 from Vicente-Cheng/wip-46941-nautilus

nautilus: mds: place MDSGatherBuilder on the stack

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36966 from Vicente-Cheng/wip-46787-nautilus

nautilus: cephfs: client: fix extra open ref decrease

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36814 from smithfarm/wip-47081-nautilus

nautilus: messages,mds: Fix decoding of enum types on big-endian systems

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36168 from smithfarm/wip-46151-nautilus

nautilus: mds: do not defer incoming mgrmap when mds is laggy

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36968 from Vicente-Cheng/wip-46943-nautilus

nautilus: mds: resolve SIGSEGV in waiting for uncommitted fragments

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36963 from Vicente-Cheng/wip-46633-nautilus

nautilus: mds: fix mds forwarding request 'no_available_op_found'

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #37435 from Vicente-Cheng/wip-47413-nautilus

nautilus: rgw: urlencode bucket name when forwarding request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Fix pool renaming functionality

The problem was that, if a pool was renamed the routine to update tried
to work on the old pool name and didn't used the info that the pool got
renamed, therefore an error occurred in the UI saying that the pool
could not be found.

Fixes: https://tracker.ceph.com/issues/42776
Signed-off-by: Stephan Müller <smueller@suse.com>
(cherry picked from commit de933cd3c82b6a7d5c4ddfa77804509c9fcaa3db)

Conflicts:
      qa/tasks/mgr/dashboard/test_pool.py:
      - Ignore incoming quota test

mgr/dashboard: Fix pg_num counts in pool API tests

To prevent the not a number of two cluster health warning, all created
pools now have a pg_num that is a power of two.

Fixes: https://tracker.ceph.com/issues/42776
Signed-off-by: Stephan Müller <smueller@suse.com>
(cherry picked from commit ca46c9941bc6837f09aab7b4aaab799d8a9ebf48)

Conflicts:
      qa/tasks/mgr/dashboard/test_pool.py:
      - Ignore quota tests and accept 8 PG number
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: Refactor pool API tests

Renamed "__create_pool" to "__yield_pool" to make it's purpose clearer
and added a description.

Split up create pool test into three different tests.
Used "__yield_pool" for creation tests, too.

Fixes: https://tracker.ceph.com/issues/42776
Signed-off-by: Stephan Müller <smueller@suse.com>
(cherry picked from commit 96fd06b7db1d57b8384edc1af6cc39e4e532fb77)

Conflicts:
      qa/tasks/mgr/dashboard/test_pool.py:
      - Ignore incoming pool quota tests
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37413 from jan--f/wip-47650-nautilus

nautilus: [ceph-volume]: remove unneeded call to get_devices()

Merge pull request #37377 from shyukri/wip-47283-nautilus

nautilus: ceph-volume: fix journal size argument not work

Merge PR #37508 into nautilus

* refs/pull/37508/head:
nautilus: qa: stop using kclient testing branch builds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37478 from smithfarm/wip-47345-nautilus

nautilus: qa/*/mon/mon-last-epoch-clean.sh: mark osd out instead of down

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #37477 from smithfarm/wip-47250-nautilus

nautilus: tools/osdmaptool.cc: add ability to clean_temps

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #37476 from smithfarm/wip-46965-nautilus

nautilus: mgr: decrease pool stats if pg was removed

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #37475 from smithfarm/wip-46935-nautilus

nautilus: tools/rados: Set locator key when exporting or importing a pool

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #37474 from smithfarm/wip-46738-nautilus

nautilus: mon: fix the 'Error ERANGE' message when conf "osd_objectstore" is filestore

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #37473 from smithfarm/wip-46710-nautilus

nautilus: osd/PeeringState: prevent peer's num_objects going negative

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #37470 from smithfarm/wip-46262-nautilus

nautilus: common, osd: add sanity checks around osd_scrub_max_preemptions

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #37471 from smithfarm/wip-46461-nautilus

nautilus: pybind/mgr/balancer: use "==" and "!=" for comparing str

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37447 from badone/wip-nautilus-ca-ansible-to-2.9

nautilus: qa/ceph-ansible: Bump required ansible to 2.9

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

nautilus: qa: stop using kclient testing branch builds

... in kcephfs and multimds suites.

This is a nautilus only fix. In master and octopus we still use
kclient testing branch builds.

Fixes: https://tracker.ceph.com/issues/47642
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #37469 from smithfarm/wip-47459-nautilus

nautilus: qa/workunits/mon: fixed excessively large pool PG count

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37468 from smithfarm/wip-47417-nautilus

nautilus: rbd: include RADOS namespace in krbd symlinks

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #37467 from smithfarm/wip-47322-nautilus

nautilus: rgw: replace '+' with "%20" in canonical query string for s3 v4 auth.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37465 from smithfarm/wip-47318-nautilus

nautilus: rgw: Expiration days can't be zero and  transition days can be zero

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37464 from smithfarm/wip-47315-nautilus

nautilus: rgw: radosgw-admin: period pull command is not always a raw_storage_op

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37463 from smithfarm/wip-46956-nautilus

nautilus: rgw: fix shutdown crash in RGWAsyncReadMDLogEntries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37462 from smithfarm/wip-46950-nautilus

nautilus: rgw/cls: preserve olh entry's name on last unlink

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37461 from smithfarm/wip-46930-nautilus

nautilus: rgw: Empty reqs_change_state queue before unregistered_reqs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37460 from smithfarm/wip-46594-nautilus

nautilus: rgw: add negative cache to the system object

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37459 from smithfarm/wip-47320-nautilus

nautilus: rgw: RGWObjVersionTracker tracks version over increments

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37438 from Vicente-Cheng/wip-47347-nautilus

nautilus: rgw: Swift API anonymous access should 401

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37472 from smithfarm/wip-46587-nautilus

nautilus: doc/rados: Fix osd_scrub_during_recovery default value

Reviewed-by: Neha Ojha <nojha@redhat.com>

mon: fix the 'Error ERANGE' message when conf "osd_objectstore" is filestore

Fixes: https://tracker.ceph.com/issues/37532
Signed-off-by: wangyunqing <wangyunqing@inspur.com>
(cherry picked from commit 4155a79f76b177ada79af746de4448773e07584a)

Conflicts:
        src/mon/OSDMonitor.cc
- in nautilus, "cmd_getval()" needs cct as first argument

qa/*/mon/mon-last-epoch-clean.sh: mark osd out instead of down

The test should mark the OSD out to check if only "in" OSDs are considered by
the osdmap trimming logic.

Fixes: https://tracker.ceph.com/issues/47309
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 21c08f0be2e048edd2d3ce7ca803f94a6d32f97c)

test/cli/osdmaptool/help.t: add clean-temps

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 762966a08ab31234148fe181eb2c107ae5266906)

doc: add clean-temps to osdmaptool.rst

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 78224c21d24d68d471eb266a37624d00d1483ed8)

tools/osdmaptool.cc: add ability to clean_temps

This is particularly useful for debugging purposes when clean_temps()
takes abnormally high amount of time due to flaws in crush rules etc.

Fixes: https://tracker.ceph.com/issues/47159
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit ab54d3821a61a4ff1ea9973c5f31ee86868b3009)

radosgw-admin: period pull command is not always a raw_storage_op

if a --url is given, 'period pull' does not depend on any zone/period
configuration and can be a raw_storage_op. if we get a --remote instead,
we do need to initialize the zone/period configuration to find the
correct endpoint/access keys

Fixes: https://tracker.ceph.com/issues/47217
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2b44a9d060d33dca9768c758e1908365488aac2a)

Conflicts:
src/rgw/rgw_admin.cc
- nautilus has different (but unrelatedly so) raw_storage_ops_list

mgr: decrease pool stats if pg was removed

After merge of placement groups resulting pg contains
objects from itself and merged one.
PGMap::apply_incremental treat this growth as pool stats delta,
but forget to decrease stats for removed pg.

Fixes: https://tracker.ceph.com/issues/44815
Signed-off-by: Aleksei Gutikov <aleksey.gutikov@synesis.ru>
(cherry picked from commit 6090acdae4495e11f117df2330b579744eeada2a)

tools/rados: Set locator key when exporting or importing a pool

Fixes the following error when exporting a pool that contains objects
with a locator key set:

        error getting xattr set [object name]: (2) No such file or directory
        error from export: (2) No such file or directory

Fixes: https://tracker.ceph.com/issues/46824
Signed-off-by: Iain Buclaw <iain.buclaw@dunnhumby.com>
(cherry picked from commit ecb2df9177f30213b3ff48e76e6fe31fcbb430a7)

osd/PeeringState: prevent peer's num_objects going negative

Saw it in a teuthology run:

 -5645> 2020-07-20 04:34:32.067 7f351e329700  5 osd.5 pg_epoch: 667 ... exit Started/Primary/Active/Backfilling
 -5642> 2020-07-20 04:34:32.067 7f351e329700  5 osd.5 pg_epoch: 667 ... enter Started/Primary/Active/Recovered
 -5633> 2020-07-20 04:34:32.067 7f351e329700 20 osd.5 pg_epoch: 667 ... _update_calc_stats shard 5 primary objects 0 missing 0
 -5632> 2020-07-20 04:34:32.067 7f351e329700 20 osd.5 pg_epoch: 667 ... _update_calc_stats shard 3 objects -1 missing 1
 -5631> 2020-07-20 04:34:32.067 7f351e329700 20 osd.5 pg_epoch: 667 ... _update_calc_stats shard 6 objects 0 missing 0

This will crash the choose_acting() procedure as it will mistakenly
think that peer 3 should continue to perform asynchronous recovery
(e.g., due to num_objects_missing = 1) in contrast to fully
backfill-recovered.

While I did not dig into the real cause, there are a couple of
possible explanations of how num_objects can be off. I think that
if a roll forward or log replay could delete something twice, maybe
there would be an undercount. Or maybe something as simple as a
corruption.

Since _update_calc_stats() is going to fix num_objects_missing
for that peer anyway, let's make sure it always starts with a
clean state.

Fixes: https://tracker.ceph.com/issues/46705
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 10eff2567971ca57b1e821f704de490add021c8e)

Conflicts:
src/osd/PeeringState.cc
- file does not exist in nautilus; made the change manually in
  src/osd/PG.cc instead

doc/rados: Fix osd_scrub_during_recovery default value

Since 8dca17c, `osd_scrub_during_recovery` defaults to `false`, but the
documentation was still stating that its default value is `true`.

Fixes: https://tracker.ceph.com/issues/46531
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
(cherry picked from commit 535b103d1848f8b5322af0815e1bf163267d7f2a)

pybind/mgr/balancer: use "==" and "!=" for comparing str

we cannot assume that two values with the same value share the same
identity in Python.

also silences warnings like:

balancer/module.py:473: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if pool_ids is '':

Fixes: https://tracker.ceph.com/issues/46406
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 65dc464977b4c2ea5ab19b8d9d0904ab31f08b94)

common, osd: add sanity checks around osd_scrub_max_preemptions

to limit maximum preempt_divisor we can use when backing off the
chunky-scrub range on preempting.

Otherwise large osd_scrub_max_preemptions values (i.e., >= 32)
would cause preempt_divisor overflow, hence the dreaded
“divide by zero error”.

Fixes: https://tracker.ceph.com/issues/46024
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit ae05de3e9b2e9868216e5168e50dfcb5074684cb)

Conflicts:
src/osd/PG.cc
- git got confused about where the change was/is

qa/workunits/mon: fixed excessively large pool PG count

Fixes: https://tracker.ceph.com/issues/47405
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2fa9442dcc7a0448ab7e3588f82f93ca2e55d686)

qa: add test for krbd symlinks created by udev

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 7ccd2c0dcee175e4c5a03985f43e9259a7e4dbd4)

rbd: include RADOS namespace in krbd symlinks

Fixes: https://tracker.ceph.com/issues/40247
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 0b5c11ff30dbb79690e47d5285f197f677e11bf7)

rgw: replace '+' with "%20" in canonical query string for s3
 v4 auth.

fix https://tracker.ceph.com/issues/45983

Signed-off-by: yuliyang_yewu <yuliyang_yewu@cmss.chinamobile.com>
(cherry picked from commit 9002be34aa8524816708db4f3429bfe8634b776a)

rgw: transitio days can be zero in transition check

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>
(cherry picked from commit 191fc25a97cb748f1105a463e9772194ba724d97)

rgw: Expiration days can't be zero and  transition days can be zero

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>
(cherry picked from commit b471fd07aa7d9ca7868572688baf89ca5a295e6f)

rgw: fix shutdown crash in RGWAsyncReadMDLogEntries

RGWAsyncReadMDLogEntries must not store pointers into coroutine memory,
because it's not guaranteed to outlive our call. store these by-value
instead, and have RGWReadMDLogEntriesCR::request_complete() copy/move
them back on completion

Fixes: https://tracker.ceph.com/issues/45771
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 13bf06dbe961132ca99f470ac026674e45fecc38)

Conflicts:
src/rgw/rgw_sync.cc
- difference in RGWAsyncReadMDLogEntries() argument list: in nautilus,
  "rgw::sal::RGWRadosStore *_store" becomes just plain "RGWRados
  *_store"

cls/rgw: preserve olh entry's name on last unlink

When rgw_bucket_unlink_instance removes the last instance of a name, it
also clears the value of rgw_bucket_olh_entry.key. However, bucket index
resharding uses this key when choosing its shard placement, so an empty
key causes all of these olh entries to be misplaced in shard 0. After
reshard, all of the olh recovery/cleanup logic would be sent to the
correct shard, and these misplaced olh entries would never be cleaned
up.

Preserving the key's name on last unlink allows the olh entry to be
resharded correctly and cleaned up normally.

Fixes: https://tracker.ceph.com/issues/46456
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit acf8f3cad9f55e34c703fdaef684853a3fb3b369)

rgw: Empty reqs_change_state queue before unregistered_reqs

In RGWHTTPManager::manage_pending_request(), before unregistering
or unlinking the http requests, empty the reqs_change_state list
to avoid use after free.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit b210437e5f28d53e770dd6938ce3c4be443da055)

rgw: ObjectCache::put() clears stale objv

if an existing object is cached with an object version, but it's
mutated without updating that version number, clear the OBJV flag so
that later cache reads asking for an object version result in a miss and
re-read the version from the osd

Fixes: https://tracker.ceph.com/issues/47306
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit cf531cdd5e655a033f47e04b7dda81435a90271d)

rgw: system object cache tracks version over increments

instead of checking write_version before the write (which doesn't take
cls_version_inc() into account), check read_version after apply_write()
has been called. only cache the result if we got a read_version != 0

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit ad326ffc3fba865d8c426de4be0193172b7688b7)

Conflicts:
src/rgw/services/svc_sys_obj_cache.cc
- in nautilus, the functions do not take an optional_yield argument

rgw: RGWObjVersionTracker tracks read version over increments

when no write_version is given, cls_version_inc() is used to increment
the version so other writers can use cls_version_check() to detect races

however, apply_write() will clear its cached read_version, which means
that later writes can no longer use cls_version_check() to detect other
racing writers

in cases where cls_version_inc() is used AND we know the previous version,
we can increment the cached read_version and preserve the ability to use
cls_version_check(). we know the previous version if we provided a valid
read_version to cls_version_check() and it succeeded

Fixes: https://tracker.ceph.com/issues/46849
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 298d721c1d3d8d430bfbc9e0ef5db7b44cdbe017)

Conflicts:
src/rgw/rgw_rados.cc
- nautilus does not have "RGWObjState::RGWObjState()"

rgw: add negative cache to the system object

add negative cache to the system object

Signed-off-by: Or Friedmann <ofriedma@redhat.com>
Fixes: https://tracker.ceph.com/issues/45816
(cherry picked from commit 0900bd8cf90babd6fafcb398854a4ddb071a27ee)

Conflicts:
src/rgw/services/svc_sys_obj_cache.cc
- RGWSI_SysObj_Core::raw_stat() takes different arguments in nautilus

Merge pull request #37181 from callithea/wip-47411-nautilus

nautilus: mgr: don't update pending service map epoch on receiving map from mon

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

nautilus: qa/ceph-ansible: Bump required ansible to 2.9

https://github.com/ceph/ceph-ansible/commit/fd0b9491b60303a5d27f79226f30d51c438de2ff
requires us to move to 2.9.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #37379 from shyukri/wip-46939-nautilus

nautilus: qa/tasks/ragweed: always set ragweed_repo

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #37280 from ceph/nautilus-bucket-list-perf

nautilus: mgr/dashboard: fix perf. issue when listing large amounts of buckets

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36909 from ifed01/wip-ifed-fix-bluefs-sel-nautilus

nautilus: bluestore/bluefs: make accounting resiliant to unlock()

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #37378 from shyukri/wip-47244-nautilus

nautilus: rgw: Add bucket name to bucket stats error logging

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: Swift API anonymous access should 401

There was a previous patch to fix this but turns out that only fixed it
for the Swift V1 auth. And it actaully broke keystone because it didn't
take into account the idiosyncrasies of multi tenancy. Which resulted in
the incorect behaviour for keystone. Worse, because it didn't take
tenants properly into account keystone ACLs where broken.

This patch reworks, and simplifies the original patch to work for both
auths. It even extends the ThirdPartyAccountApplier to check for an ANON
user and properly scope it to a tenant.

Fixes: https://tracker.ceph.com/issues/46295
Signed-off-by: Matthew Oliver <moliver@suse.com>
(cherry picked from commit 67081098dc2dddd80d52d5acd166e68954cae618)

Conflicts:
src/rgw/rgw_swift_auth.h
  - only need to modify the user related code to rgw_user construct

Merge pull request #36704 from ShyamsundarR/wip-47013-nautilus

nautilus: mon: store mon updates in ceph context for future MonMap instantiation

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

rgw : add URL encode toward bucket name in forward request

Signed-off-by: caolei <halei15848934852@163.com>
(cherry picked from commit 1e4ec14dd0df4e9053ef630c831c93d392508417)

Merge pull request #37306 from p-se/wip-47546-nautilus

nautilus: mgr/dashboard: Fix many-to-many issue in host-details Grafana dashboard

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: Custom Directive for verifying if user is read_only

This commit introduces two new Custom Directives: `FormInputDisableDirective` and `FormScopeDirective`.
By using the `cdFormScope` attribute of the directive we can customize the behaviour of the modal components.

Fixes: https://tracker.ceph.com/issues/43527
Signed-off-by: Nizamudeen <nia@redhat.com>
(cherry picked from commit f6e1ed657125e0f394e80da5dfeb51617304d091)

Conflicts:
      src/pybind/mgr/dashboard/frontend/src/app/shared/directives/directives.module.ts
 - Import the FormInputDisableDirective and FormScopeDirective to the directive.module imports and exports
      src/pybind/mgr/dashboard/frontend/src/app/shared/directives/form-input-disable.directive.spec.ts
 - Changed TestBed.inject(...) to TestBed.get(...) to match the syntax for the nautilus TS version
      src/pybind/mgr/dashboard/frontend/src/app/shared/directives/form-input-disable.directive.ts
 - Changed some of the operators to match the nautilus TS version

Merge pull request #37307 from rhcs-dashboard/wip-47303-nautilus

nautilus: mgr/dashboard: REST API returns 500 when no Content-Type is specified

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37407 from idryomov/wip-krbd-read-only-override-nautilus

nautilus: rbd: make common options override krbd-specific options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/msg_types: entity_addrvec_t: fix decode on big-endian hosts

When decoding an entity_addrvec_t with marker 1, we just have
a single (non-legacy) entity_addr_t.  This should be decoded
exactly the same as done by entity_addr_t::decode, but it
currently is not.  Specifically, the sa_family member of
the sockaddr is not converted from the on-wire little-endian
format to host byte order (as done by entity_addr_t::decode).

Fixed by using the same code as in entity_addr_t::decode.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit 31da17378b712542e915adbf4084e0212b8bb615)

messages,mds: Fix decoding of enum types on big-endian systems

When a struct member that has enum type needs to be encoded or
decoded, we need to use an explicit integer type, since there
are no encode routines for the enum type.  (This is probably
to avoid introducing dependencies on implementation-defined
choices by the compiler to use a particular underlying type.)

This leads to code sequences along the lines of:
  encode((int32_t)state, bl);
and
  decode((int32_t&)(state), bl);

The encode line is actually fine, but the decode line is
incorrect on big-endian systems if the underlying type of
the enum differs from the explicitly chosen integer type.

This is because this performs in effect a pointer cast,
and will write the decoded int32_t value into the memory
backing the "state" member variable.  If the sizes differ,
the value is written into the wrong bytes of "state" on
big-endian systems.

This patch fixes the problem by decoding into an intermediate
variable of the integer type first, and then casting the result
while assigning to the struct member of enum type.

This bug showed up initially as invalid health-status values
causing Ceph daemon aborts on s390x.  I've tried to find and
fix all other instances of the same enum decode pattern as well.

Fixes: https://tracker.ceph.com/issues/47015
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit 7ed716823fd02d84ea53cb61350bf14f248ebb8b)

Conflicts:
src/mds/PurgeQueue.cc
- nautilus has "p.advance(pad_size)", instead of "p += pad_size",
  in the line immediately preceding the first change

ceph-volume: remove unneeded call to get_devices()

there is no need to probe the device to generate the argparse help
message.

also removing the test on the function as it's not there anymore.

Signed-off-by: Marc Gariepy <gariepy.marc@gmail.com>
Fixes: https://tracker.ceph.com/issues/47502
(cherry picked from commit 5c6f66166a7afad87627032cafdc5c4f11f94eac)

rbd: make common options override krbd-specific options

ceph-csi has added support for passing custom map and unmap options via
mapOptions and unmapOptions storage class parameters.  However, it also
uses --read-only for implementing ROX (ReadOnlyMany) PVs.  If the user
supplies "mapOptions: rw", they will get around the intended read-only
restriction (at least on the block device).

ceph-csi could be patched to use "-o ro", but it actually makes sense
for common options to win over device type-specific equivalents.

Fixes: https://tracker.ceph.com/issues/47625
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit a107c47360ecdb8c09768ca9eab2341100245711)

Conflicts:
src/tools/rbd/action/Kernel.cc [ snapshot quiesce support and
  commit 34f539d8af33 ("rbd: delay parsing of default kernel map
  options") not in nautilus ]

nautilus: mgr/dashboard: fix perf. issue when listing large amounts of buckets

NOTE: Due to base code divergence between master (pacific) & nautilus,
This is a dedicated fix for nautilus.

Fixes: https://tracker.ceph.com/issues/47618
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #37254 from bstillwell/wip-47425-nautilus

nautilus: compressor: Add a config option to specify Zstd compression level

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37269 from badone/wip-nautilus-enable-mgr-client-debug

nautilus: tests/qa: Enable debug_client for mgr tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37318 from yaarith/nautilus-fix-dev-id-split

nautilus: mgr/telemetry: fix device id splitting when anonymizing serial

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37288 from yaarith/nautilus-add-smartctl-nvme-dependencies

nautilus: ceph.spec.in, debian/control: add smartmontools and nvme-cli dependen…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37324 from aaSharma14/wip-47570-nautilus

nautilus: mgr/dashboard: table detail rows overflow

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37309 from rhcs-dashboard/wip-47579-nautilus

nautilus: mgr/dashboard: fix pool usage calculation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #37226 from smithfarm/wip-47532-nautilus

nautilus: ceph.in: ignore failures to flush stdout

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36984 from k0ste/wip-47281-nautilus

nautilus: prometheus: Properly split the port off IPv6 addresses

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37209 from smithfarm/wip-46983-nautilus

nautilus: test/rbd-mirror: pool watcher registration error might result in race

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37165 from dillaman/wip-47100-nautilus

nautilus:  librbd: using migration abort can result in the loss of data

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #37157 from smithfarm/wip-46519-nautilus

nautilus: rgw: fix boost::asio::async_write() does not return error...

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37040 from trociny/wip-46720-nautilus

nautilus: librbd: don't resend async_complete if watcher is unregistered

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36880 from smithfarm/wip-47186-nautilus

nautilus: rgw: dump transitions in RGWLifecycleConfiguration::dump()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #36843 from Huber-ming/nautilus-loglevel

nautilus: rgw: log resharding events at level 1 (formerly 20)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37195 from guits/wip-47504-nautilus

nautilus: ceph-volume: fix simple activate when legacy osd

Merge pull request #37295 from rhcs-dashboard/wip-47573-nautilus

nautilus: mgr/dashboard: cpu stats incorrectly displayed

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>