rgw: add rgw-gap-list-comparator tool

The rgw-gap-list tool can produce a number of false positives when the
cluster is being used during its run. One technique to minimize the
number of false positives is to run the tool twice and look for the
objects that appear in both lists. The rgw-gap-list-comparator tool is
designed to do this comparison.

Signed-off-by: Michael Kidd <linuxkidd@gmail.com>
(cherry picked from commit 3c5154f010a34f9845b40e49a7870771e7f5372e)

rgw: add rgw-gap-list tool

Due to a prior bug (pr: 38228) tail rados objects of some RGW objects
could have been incorrectly deleted. This tool is designed to look for
such cases. It essentially does the opposite of rgw-orphan-list,
looking for rados objects that RGW expects to be there, but which are
not to be found.

IMPORTANT: This is very experimental at this point in time, and any
"results" produced should be verified by other means.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Signed-off-by: Michael Kidd <linuxkidd@gmail.com>
(cherry picked from commit 07b42195fbbcd27e330cb1daa35e77e0952f8a3c)

rgw: add radosgw-admin bucket radoslist bucket/obj mapping

A new command-line option "--rgw-obj-fs" is added to
radosgw-admin. When used with the "bucket radoslist" subcommand, will
output lines with a rados object, bucket name, and object name,
separated by the field separator specified. Without this command-line
option, only the rados object is output, which is the previous
behavior.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 81d6260c1a42400bfa8c677c419e2c3f76d7a920)

Merge pull request #39103 from k0ste/wip-49022-octopus

octopus: mgr/dashboard: trigger alert if some nodes have a MTU different than the median value

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #39027 from rhcs-dashboard/wip-48960-octopus

octopus: mgr/dashboard: Fix for incorrect validation in rgw user form

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39306 from rhcs-dashboard/wip-48829-octopus

octopus: mgr/dashboard/monitoring: upgrade Grafana version due to CVE-2020-13379

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: p-se <NOT@FOUND>

Merge pull request #39271 from jan--f/wip-49141-octopus

octopus: ceph-volume: add some flexibility to bytes_to_extents

Merge pull request #39534 from ideepika/wip-backport-blocklist-ignorelist-qa

octopus: whitelist -> ignorelist for qa/* only

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39235 from k0ste/wip-49107-octopus

octopus: mgr/zabbix: format ceph.[{#POOL},percent_used as float

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39277 from smithfarm/wip-49131-octopus

octopus: rgw: fix trailing null in object names of multipart reuploads

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39583 from liewegas/pr-39561-octopus

octopus: qa: use normal build for valgrind

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

15.2.9

qa/valgrind.supp: suppress free[] warnings

This is apparently not going to get fixed any time soon.

https://github.com/gperftools/gperftools/issues/792

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 28fe149ea276693b47cc4d600780971f723672e4)

qa/suites: do not use notcmalloc flavor

teuthology now knows how to run valgrind against a tcmalloc binary

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit dc64ccf063f1ddf843fae4d2621ec25187f56135)

# Conflicts:
# qa/suites/rgw/sts/tasks/0-install.yaml
  - test not present in octopus
# qa/suites/rgw/verify/validater/valgrind.yaml

Merge pull request #39289 from kamoltat/wip-ksirivad-octopus-turn-off-backports

octopus: mgr/progress: introduce turn off/on feature

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39256 from ifed01/wip-ifed-fsck-spanning-blobs-octopus

octopus: os/bluestore: detect and fix "zombie" spanning blobs using fsck.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39253 from ifed01/wip-ifed-fix-huge-bluefs-reads-oct

octopus: os/bluestore: fix huge (>4GB) bluefs reads

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #39560 from neha-ojha/wip-revert-39248-octopus

octopus: Revert "mgr/pg_autoscaler: avoid scale-down until there is pressure"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

Revert "PendingReleaseNotes: mgr/pg_autoscaler"

This reverts commit 1177acb6de32331c67d707032a2ecaeff6bf8365.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "mgr/pg_autoscaler: avoid scale-down until there is pressure"

This reverts commit 49dba6edaa807f2c1047c40ce96a76fc5f409b82.

Reason: Needs to bake more given that it may cause pg budget issues
when multiple pools are created at the same time.

More details in https://tracker.ceph.com/issues/49364.

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash: log-whitelist -> log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit c0a358670a8cda2d9e319f0528d6cb9d74b6c605)

qa/suites/krbd/thrash: log-whitelist -> log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 01fb7e7f7b1f5c6d2c8371560ccf295567c0eb07)

qa/suites/multimds: log-whitelist -> log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit a8c7efab2d51a249e716dad710e2ca3d01482f16)

qa: log-whitelist -> log-ignorelist

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 2ee9365d0bc220523632753f02c61eb5c2f4fbdd)

Merge pull request #39540 from tchaikov/octopus-pr-35352

octopus: qa/tasks/vstart_runner: do not teardown test_path if "create-cluster-only"

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa/tasks/vstart_runner: do not teardown test_path if "create-cluster-only"

otherwise we could be removing a "None" directory when tearing down the cluster,
and have following failure:

Exception ignored in: <bound method LocalContext.__del__ of <__main__.LocalContext object at 0x7f99fd4a6cc0>>
Traceback (most recent call last):
  File "../qa/tasks/vstart_runner.py", line 1189, in __del__
    shutil.rmtree(self.teuthology_config['test_path'])
  File "/tmp/tmp.mmM2ugspuR/venv/lib/python3.6/shutil.py", line 477, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/tmp/tmp.mmM2ugspuR/venv/lib/python3.6/shutil.py", line 475, in rmtree
    orig_st = os.lstat(path)
TypeError: lstat: path should be string, bytes or os.PathLike, not NoneType

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 954e9a6fa67cce8e3eb8105ee858340b60b84b15)

Merge pull request #39532 from liewegas/pr-39496-octopus

octopus: mgr/cephadm: fix host refresh

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/cephadm: fix host refresh

Fixes: 01f60cf4e0a751c314120c02956d4ff941eb71b4
Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 9df5a6d73ed21b394c01afe6c9800b6e50737c90)

Merge pull request #39393 from kamoltat/wip-ksirivad-octopus-release-notes

octopus: PendingReleaseNotes: mgr/pg_autoscaler

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39230 from ifed01/wip-ifed-fix-pin-octopus

octopus: os/bluestore: fixing onode pinning and more

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #39004 from Vicente-Cheng/wip-48568-octopus

octopus: qa/tasks/cephfs/nfs: Check if host ip is in cluster info output

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

PendingReleaseNotes: mgr/pg_autoscaler

Added details describing the changes that
occured in after merging #39248 into octopus
upstream.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #39000 from Vicente-Cheng/wip-48521-octopus

octopus: cephfs: client: add ceph.{cluster_fsid/client_id} vxattrs suppport

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38949 from Vicente-Cheng/wip-48644-octopus

octopus: cephfs: client: ensure we take Fs caps when fetching directory link count from cached inode

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38947 from Vicente-Cheng/wip-48642-octopus

octopus: cephfs: client: set CEPH_STAT_RSTAT mask for dir in readdir_r_cb

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38612 from ShyamsundarR/wip-47158-octopus

octopus: mgr/volumes: Add a per subvolume trash

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #38466 from Vicente-Cheng/wip-48458-octopus

octopus: cephfs: client: do not use g_conf().get_val<>() in libcephfs

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #39161 from smithfarm/wip-48496-octopus

octopus: mon: paxos: Delete logger in destructor

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39120 from rhcs-dashboard/wip-48739-octopus

octopus: mgr/dashboard: Use secure cookies to store JWT Token

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #39248 from kamoltat/wip-ksirivad-octopus-backports

octopus: mgr/pg_autoscaler: avoid scale-down until there is pressure

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39122 from smithfarm/wip-48692-octopus

octopus: librbd: clear implicitly enabled feature bits when creating images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #38474 from ifed01/wip-ifed-fix-avl-octopus

octopus: os/bluestore: fix inappropriate ENOSPC from avl/hybrid allocator

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38040 from kshtsk/wip-octopuse-cephadm-bootstrap-remote

octopus: tests: qa/task/cephadm: run cephadm only on bootstrap_remote

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #38184 from smithfarm/wip-48101-octopus

octopus: rgw/rgw-admin: fixes BucketInfo for missing buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39169 from sebastian-philipp/octopus-backport-38978

octopus: mgr/cephadm: raise HEALTH_WARN when cephadm daemon in 'error' state

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39321 from idryomov/wip-krbd-stable-writes-attr-octopus

octopus: qa: krbd_stable_pages_required.sh: move to stable_writes attribute

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38422 from smithfarm/wip-48285-octopus

octopus: qa: ignore evicted client warnings

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39297 from sebastian-philipp/octopus-backport-39106

octopus: cephadm: use `apt-get` for package install/update

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39300 from sebastian-philipp/octopus-backport-38998-38927

octopus: mgr/cephadm: try again calling ceph-volume without --filter-for-batch

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa: krbd_stable_pages_required.sh: move to stable_writes attribute

bdi/stable_pages_required attribute was deprecated in 5.10 and now
always returns 0.  The replacement is queue/stable_writes.  (It is
also writeable, so we can simplify these test cases somewhat in the
future.)

Fixes: https://tracker.ceph.com/issues/48232
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 5adfc15b873bc16d698e7398d3ef2c2a46e8a9df)

Merge pull request #39203 from idryomov/wip-krbd-msgr2-octopus

octopus: krbd: add support for msgr2 (kernel 5.11)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38893 from smithfarm/wip-48519-octopus

octopus: pybind/cephfs: fix missing terminating NULL char in readlink()'s C string

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38424 from smithfarm/wip-48375-octopus

octopus: cephfs: client: check rdonly file handle on truncate

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38352 from smithfarm/wip-48370-octopus

octopus: cephfs: mds: dir->mark_new() should together with dir->mark_dirty()

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38349 from smithfarm/wip-48129-octopus

octopus: cephfs: release client dentry_lease before send caps release to mds

Reviewed-by: Ramana Raja <rraja@redhat.com>

mgr/dashboard/monitoring: upgrade Grafana version due to CVE-2020-13379

Fixes: https://tracker.ceph.com/issues/48685
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 9441fda4dc9f0257c3b686de811a01f047ef9f53)

 Conflicts:
monitoring/grafana/build/Makefile
          - Remove file as it does not exist on octopus.
src/pybind/mgr/cephadm/module.py
          - Resolve conflicts.

Merge pull request #39296 from sebastian-philipp/octopus-backport-39113

octopus: python-common: fix test_datetime_to_str_2 on non-UTC hosts

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: try again calling ceph-volume without --filter-for-batch

Fixes: https://tracker.ceph.com/issues/48870
This deals with a cephadm upgrade issue:

1. user calls `ceph orch upgrade`
2. mgr/cephadm calls `ceph orch config set mgr.x container_image <new-container>`
3. standby mgr gets upgraded
4. mgr failover to new mgr
5. mgr/cephadm calls `_refresh_host_devices`
6. `_refresh_host_devices` calls` ceph orch config get osd container_image`.
  But this returns the old image
7. `_refresh_host_devices` calls `ceph-volume ... --filter-for-batch`
  with an image that doesn't support `filter-for-batch`

The idea is to simply retiry calling ceph-volume inventory without `--filter-for-batch`

(also removed `out` being used without being declared)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit ede3d6d147dd7b99d37aee5c5fb9340f2878db18)

Conflicts:
  src/pybind/mgr/cephadm/tests/test_cephadm.py

mgr/cephadm: Properly handle JSON Decode error

Fixes 6d759fb5deac0c52b3c738a2e695738228749420

I.e. don't use `out`, until it is acutally defined

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit eb1a40c3ea8e19259d8ab68a6eeb16d27e4cdbda)

Conflicts:
src/pybind/mgr/cephadm/serve.py

cephadm: use `apt-get` for package install/update

avoids errors during prepare-host:
```
apt: stderr WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
```

Fixes: https://tracker.ceph.com/issues/49032
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit fa4706039cfece36815df46cd4452dc25448c340)

python-common: fix test_datetime_to_str_2 on non-UTC hosts

The old test parsed to a datetime without a tz, which was interpreted as
the local time zone when rendering back to a string.  Specify that it's a
UTC datetime so that behavior is consistent regardless of the test host
timezone.

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 78aca4db249c409d0cd5a24bfae81e55cf930bc3)

Merge pull request #39170 from sebastian-philipp/octopus-backport-38910

octopus: cephadm: fix rgw osd cap tag

Merge pull request #39166 from sebastian-philipp/octopus-backport-38804-39003

octopus: cephadm: silence "Failed to evict container" log msg

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39167 from sebastian-philipp/octopus-backport-38850

octopus: mgr/cephadm: tolerate old host inventory without 'hostname' key

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39168 from sebastian-philipp/octopus-backport-38945

octopus: qa/cephadm: Add yaml output to smoke test

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39171 from sebastian-philipp/octopus-backport-39083

octopus: python-common/drivegroups: avoid dropping "rotational: 0" from Device Selection

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/progress: introduce turn off/on feature

progress module can be turned off/on by using
the commands: 'progress off' and 'progress on'

As well as refractoring teuthology test suite
to prevent future bugs that can possibly occur

fixes: https://tracker.ceph.com/issues/47238

Signed-off-by: kamoltat <ksirivad@redhat.com>
(cherry picked from commit 993bb02b30cf73a1c1c70da1ef266be8373d56dd)

Conflicts:
PendingReleaseNotes - update relesae notes
qa/tasks/mgr/test_progress.py - trivial fix

rgw: fix trailing null in object names of multipart reuploads

the overload being called had a comment saying:
/* size should be the required string size + 1 */

this calls a different overload that returns std::string and handles
this null termination correctly

Fixes: https://tracker.ceph.com/issues/48874
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2ede30782b00fe28ac714047f55b11b76ae1593f)

ceph-volume: mock VGs with string properties in tests

The real things uses string properties as well.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit c4bbb1b999b08bd84d793537f2602fc8e4c93bae)

ceph-volume: add some flexibility to bytes_to_extents

For batch we want to report the projected sizes before creating any LVM
structures, we use the device size for that. This means we pass this
projected size to lvm/api.py::create_lv. This could result in situations
where we request more extents for the new LV than are actually
available, as LVM takes up some available space on the block device
(plus alignment).
We fix this by adjusting the extent count downwards if its less than 1% off the
requested value.

Fixes: https://tracker.ceph.com/issues/47758
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit ece6e7eb9a92cb944b2f201a05a5acfb877c2dba)

os/bluestore: fix huge (>4GB) bluefs reads

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit fe558c4dd12b25b2d523987f22fde5b5898cfb83)

 Conflicts:
 (trivial) src/os/bluestore/BlueFS.h
 (trivial) src/os/bluestore/BlueRocksEnv.cc
 (trivial) src/test/objectstore/test_bluefs.cc

mgr/pg_autoscaler: avoid scale-down until there is pressure

The autoscaler will start out with scaling each
pools to have a full complements of pgs from the start
and will only decrease it when pools need more due to
increased usage.

Introduced a unit test that tests only the
function get_final_pg_target_and_ratio() which
deals with the distrubtion of pgs amongst the
pools

Edited workunit script to reflect the change
of how pgs are calculated and distrubted.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit daeb6f6ac0c8f77ae07147f9d1e2ed18d6d8e4cc)

Conflicts:
src/pybind/mgr/pg_autoscaler/module.py - trivial fix

mgr/zabbix: format ceph.[{#POOL},percent_used as float

also, multiply it with 100, as we are using 1.0 for 100% here.

for more details regarding the mapping from value_type to its values.
see
https://www.zabbix.com/documentation/4.0/manual/api/reference/item/object

Fixes: https://tracker.ceph.com/issues/48825
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c8e5a051a016fa29a621ef021f942d9dac97205a)

Merge pull request #38430 from smithfarm/wip-48281-octopus

octopus: osd: fix bluestore bitmap allocator calculate wrong last_pos with hint

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38427 from smithfarm/wip-48283-octopus

octopus: rpm,deb: change sudoers file mode to 440

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38333 from b-ranto/wip-prom-fixes-octopus

octopus: mgr/prometheus: Sync and backport prometheus fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: detect and fix "zombie" spanning blobs using fsck.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 18d120dc2c15c0cf8d3b27ddba3c7f2057856555)

 Conflicts:
 (trivial)      src/os/bluestore/BlueStore.cc
 (trivial)      src/os/bluestore/BlueStore.h

Merge PR #38425 into octopus

* refs/pull/38425/head:
lvm/create.py: fix a typo in the help message

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

os/bluestore: fix a bug causing unexpected Onode's unpinned state.

There could be a race for Onodes put() and get() methods:

put()(pinned, nref=3)
  int n = --nref; (nref = 2)
  if (n == 2) {
    ..
    std::lock_guard l(ocs->lock);
    ...
    pinned = pinned && nref > 2; (= false)
    ...                                     get()
    if (r) {                                ++nref; (=3)
      n = --nref; (nref = 2)                return;
    }
    ...
    return

As a result nref = 2, pinned = false which is wrong

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit ea0fc57ef57eddc1fb9610850ae766c9fd582bae)

os/bluestore: Prevented erasure of element from onode_map during iteration

When onode.exists == false getting reference and then releasing it might delete it from container.
It must not happen during iteration.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 6c8e8a757485d27bfa93d344f3e04aaf29c68cc4)

os/bluestore: Purge onode when it does exist

Added logic for erasing onode from onode_map it is last reference and exists==false.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit aaebfe0abe78bf46c52ef6c4481c517353edf9ac)

 Conflicts:
   (trivial) src/os/bluestore/BlueStore.h

os/bluestore: Refactor pin() to get more control over its logic

Got rid of OnodeCacheShard pin() and unpin() functions.
Moved their validator logic right into Onode put and get functions.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit eaf1b2366aa7701b10eca4ed3e53d51909e8011b)

os/bluestore: Only pass that decremented nref to 0 deletes object

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit b0e2964ac8a8f3a273046ab8e87a62c1bc4db55c)

rgw : modify error message to NoSuchBucket when bucket doesn't exist in bucket info API

Fixes: https://tracker.ceph.com/issues/48073
Signed-off-by: caolei <halei15848934852@163.com>
(cherry picked from commit bc5ef5c9cf0ea89fc028332c39766eb8e7e1bd0b)

rgw: fixes BucketInfo for missing buckets

The admin api BucketInfo endpoint should now return 404 for buckets that
are not found where only the bucket name is passed as a parameter.

Fixes: https://tracker.ceph.com/issues/45193
Signed-off-by: Nick Janus <njanus@digitalocean.com>
(cherry picked from commit d70ca81502d25bd7a76dd2ed2a538bf5e6584822)

Merge pull request #38971 from smithfarm/wip-48743-octopus

octopus: rgw: distribute cache for exclusive put

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38970 from smithfarm/wip-48693-octopus

octopus: rgw: adding user related web token claims to ops log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38826 from smithfarm/wip-48725-octopus

octopus: rgw: fix bucket limit check fill_status warnings

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38821 from smithfarm/wip-48804-octopus

octopus: rgw: cls/user: set from_index for reset stats calls

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39046 from smithfarm/wip-48968-octopus

octopus: ocf: add support for mapping images within an RBD namespace

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38981 from smithfarm/wip-48864-octopus

octopus: rgw/multisite: Verify if the synced object is identical to source

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38829 from smithfarm/wip-48544-octopus

octopus: rgw_file: return common_prefixes in lexical order

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #38824 from smithfarm/wip-48546-octopus

octopus: rgw: lc: correctly dimension lc shard index vector

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #39059 from votdev/issue_48068_tz_octopus

octopus: cephadm: Various properties like 'last_refresh' do not contain timezone

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/suites/krbd: add msgr2 modes to most subsuites

basic, rbd and rbd-nomount subsuites are expanded to run with each
of ms_mode=legacy, ms_mode=crc and ms_mode=secure.  This increases
the total number of jobs in the suite from 100 to 220.

fsx, singleton and thrash subsuites choose ms_mode at random (from
the above plus ms_mode=prefer-crc).

unmap and wac subsuites remain msgr1-only.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 65948736a41f424d8152b208d013419f8d6038a4)

doc: deprecate [no]cephx_require_signatures map options

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit fd5f5722a29964bb33b305a381bcf9a48cdcbb47)

krbd: add support for msgr2

Recognize ms_mode map option and filter initial monitor addresses
accordingly: if ms_mode is not given or ms_mode=legacy, discard v2
addresses, otherwise discard v1 addresses.

Note that nothing was discarded (i.e. v2 addresses were passed to
the kernel) previously.  The intent was to preserve that behaviour
in case ms_mode is not given, allowing to change the kernel default
in the future.  However, it turns out that mount.ceph helper has
been misguidedly discarding v2 addresses since commit eae01275134e
("mount.ceph: fork a child to get info from local configuration"),
so that ship has sailed.

Fixes: https://tracker.ceph.com/issues/48976
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 08f714964b7fe5024504818f01328a41acc24965)

Conflicts:
src/tools/rbd/action/Kernel.cc [ commit 34f539d8af33 ("rbd:
  delay parsing of default kernel map options") not in octopus ]

Merge pull request #38336 from votdev/wip-48398-octopus

octopus: mgr/dashboard: display placement column in service table

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

python-common/drivegroups: avoid dropping "rotational: 0" from DeviceSelection

False is a legitimate value for the rotational setting and should be included in the JSON output, only None should be ignored.

Fixes: http://tracker.ceph.com/issues/49014
Fixes: cd6a488ab2ca036dd4fb36751b938f605e97e1c8
Signed-off-by: Lukas Stockner <lstockner@genesiscloud.com>
(cherry picked from commit c32f6f5448e51d3196f7a2644ea97ecd22a04f92)

cephadm: fix rgw osd cap tag

The syntax is "allow rwx tag rgw *=*'.

Sorry, I thought this would have gotten caught in testing :(

Fixes: 373cc847cf0f8b4ec7aefbfe64c01c3f18a4e021
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit de1efbd62b9258630c2a2d55bfd12034cc8b603f)