mgr/rbd_support: make mirror_snapshot_schedule rescan only updated pools

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 69259c8d3722830c13da1e33f61fb6a72151b803)

pybind/rbd: add async mirror image get mode and info methods

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 13ce488d7aaf60f85429c2e4c036362d61e5f763)

librbd: add missing declaration of rbd_aio_mirror_image_get_mode

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit fd5e642cd74c02e8ef00b7018baaa3093167a01f)

pybind/rbd: add image async open and close methods

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit b27db87c0391231b39ecd4b5aacf8ac24193c1cf)

pybind/rbd: async API for creating mirror snapshots

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit d1571ed2743f97280a2f3a2a70fff59e0cafc0b9)

Conflicts:
src/pybind/rbd/rbd.pyx: trival resolution re: quiesce flags

librbd: async API for creating mirror snapshots

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit d8468ec7e49d93860c3b8b06ff36bde539d10a94)

Conflicts:
src/librbd/api/Mirror.h/cc: no quiesce flags
src/test/rbd_mirror/test_ImageReplayer.cc: no quiesce flags

pybind/rados: don't try to close watch in dealoc if rados not connected

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit eeee5578877b2c96aa46d5ee427ea5c369a42501)

pybind/rados: add watch object bindings

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit dfef23d6d3fdf162f48f909b90c10577f03116bc)

pybind/rados: add Ioctx dup method

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 0ec9cfcf13a1346a628962d315ed19a37e398eba)

pybind/rados: correct the declarations of C APIs

so they are consistent with the ones declared in include/rados/librados.h

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 31db9ce5dc21c769d78e615c2335bd2dbd94f321)

Conflicts:
src/pybind/rados/rados.pyx: trivial resolution

Merge pull request #39161 from smithfarm/wip-48496-octopus

octopus: mon: paxos: Delete logger in destructor

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39120 from rhcs-dashboard/wip-48739-octopus

octopus: mgr/dashboard: Use secure cookies to store JWT Token

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #39248 from kamoltat/wip-ksirivad-octopus-backports

octopus: mgr/pg_autoscaler: avoid scale-down until there is pressure

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39122 from smithfarm/wip-48692-octopus

octopus: librbd: clear implicitly enabled feature bits when creating images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #38474 from ifed01/wip-ifed-fix-avl-octopus

octopus: os/bluestore: fix inappropriate ENOSPC from avl/hybrid allocator

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38040 from kshtsk/wip-octopuse-cephadm-bootstrap-remote

octopus: tests: qa/task/cephadm: run cephadm only on bootstrap_remote

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #38184 from smithfarm/wip-48101-octopus

octopus: rgw/rgw-admin: fixes BucketInfo for missing buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39169 from sebastian-philipp/octopus-backport-38978

octopus: mgr/cephadm: raise HEALTH_WARN when cephadm daemon in 'error' state

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39321 from idryomov/wip-krbd-stable-writes-attr-octopus

octopus: qa: krbd_stable_pages_required.sh: move to stable_writes attribute

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38422 from smithfarm/wip-48285-octopus

octopus: qa: ignore evicted client warnings

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39297 from sebastian-philipp/octopus-backport-39106

octopus: cephadm: use `apt-get` for package install/update

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39300 from sebastian-philipp/octopus-backport-38998-38927

octopus: mgr/cephadm: try again calling ceph-volume without --filter-for-batch

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa: krbd_stable_pages_required.sh: move to stable_writes attribute

bdi/stable_pages_required attribute was deprecated in 5.10 and now
always returns 0.  The replacement is queue/stable_writes.  (It is
also writeable, so we can simplify these test cases somewhat in the
future.)

Fixes: https://tracker.ceph.com/issues/48232
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 5adfc15b873bc16d698e7398d3ef2c2a46e8a9df)

Merge pull request #39203 from idryomov/wip-krbd-msgr2-octopus

octopus: krbd: add support for msgr2 (kernel 5.11)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38893 from smithfarm/wip-48519-octopus

octopus: pybind/cephfs: fix missing terminating NULL char in readlink()'s C string

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38424 from smithfarm/wip-48375-octopus

octopus: cephfs: client: check rdonly file handle on truncate

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38352 from smithfarm/wip-48370-octopus

octopus: cephfs: mds: dir->mark_new() should together with dir->mark_dirty()

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38349 from smithfarm/wip-48129-octopus

octopus: cephfs: release client dentry_lease before send caps release to mds

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #39296 from sebastian-philipp/octopus-backport-39113

octopus: python-common: fix test_datetime_to_str_2 on non-UTC hosts

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: try again calling ceph-volume without --filter-for-batch

Fixes: https://tracker.ceph.com/issues/48870
This deals with a cephadm upgrade issue:

1. user calls `ceph orch upgrade`
2. mgr/cephadm calls `ceph orch config set mgr.x container_image <new-container>`
3. standby mgr gets upgraded
4. mgr failover to new mgr
5. mgr/cephadm calls `_refresh_host_devices`
6. `_refresh_host_devices` calls` ceph orch config get osd container_image`.
  But this returns the old image
7. `_refresh_host_devices` calls `ceph-volume ... --filter-for-batch`
  with an image that doesn't support `filter-for-batch`

The idea is to simply retiry calling ceph-volume inventory without `--filter-for-batch`

(also removed `out` being used without being declared)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit ede3d6d147dd7b99d37aee5c5fb9340f2878db18)

Conflicts:
  src/pybind/mgr/cephadm/tests/test_cephadm.py

mgr/cephadm: Properly handle JSON Decode error

Fixes 6d759fb5deac0c52b3c738a2e695738228749420

I.e. don't use `out`, until it is acutally defined

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit eb1a40c3ea8e19259d8ab68a6eeb16d27e4cdbda)

Conflicts:
src/pybind/mgr/cephadm/serve.py

cephadm: use `apt-get` for package install/update

avoids errors during prepare-host:
```
apt: stderr WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
```

Fixes: https://tracker.ceph.com/issues/49032
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit fa4706039cfece36815df46cd4452dc25448c340)

python-common: fix test_datetime_to_str_2 on non-UTC hosts

The old test parsed to a datetime without a tz, which was interpreted as
the local time zone when rendering back to a string.  Specify that it's a
UTC datetime so that behavior is consistent regardless of the test host
timezone.

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 78aca4db249c409d0cd5a24bfae81e55cf930bc3)

Merge pull request #39170 from sebastian-philipp/octopus-backport-38910

octopus: cephadm: fix rgw osd cap tag

Merge pull request #39166 from sebastian-philipp/octopus-backport-38804-39003

octopus: cephadm: silence "Failed to evict container" log msg

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39167 from sebastian-philipp/octopus-backport-38850

octopus: mgr/cephadm: tolerate old host inventory without 'hostname' key

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39168 from sebastian-philipp/octopus-backport-38945

octopus: qa/cephadm: Add yaml output to smoke test

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39171 from sebastian-philipp/octopus-backport-39083

octopus: python-common/drivegroups: avoid dropping "rotational: 0" from Device Selection

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/pg_autoscaler: avoid scale-down until there is pressure

The autoscaler will start out with scaling each
pools to have a full complements of pgs from the start
and will only decrease it when pools need more due to
increased usage.

Introduced a unit test that tests only the
function get_final_pg_target_and_ratio() which
deals with the distrubtion of pgs amongst the
pools

Edited workunit script to reflect the change
of how pgs are calculated and distrubted.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit daeb6f6ac0c8f77ae07147f9d1e2ed18d6d8e4cc)

Conflicts:
src/pybind/mgr/pg_autoscaler/module.py - trivial fix

Merge pull request #38430 from smithfarm/wip-48281-octopus

octopus: osd: fix bluestore bitmap allocator calculate wrong last_pos with hint

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38427 from smithfarm/wip-48283-octopus

octopus: rpm,deb: change sudoers file mode to 440

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38333 from b-ranto/wip-prom-fixes-octopus

octopus: mgr/prometheus: Sync and backport prometheus fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38425 into octopus

* refs/pull/38425/head:
lvm/create.py: fix a typo in the help message

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

rgw : modify error message to NoSuchBucket when bucket doesn't exist in bucket info API

Fixes: https://tracker.ceph.com/issues/48073
Signed-off-by: caolei <halei15848934852@163.com>
(cherry picked from commit bc5ef5c9cf0ea89fc028332c39766eb8e7e1bd0b)

rgw: fixes BucketInfo for missing buckets

The admin api BucketInfo endpoint should now return 404 for buckets that
are not found where only the bucket name is passed as a parameter.

Fixes: https://tracker.ceph.com/issues/45193
Signed-off-by: Nick Janus <njanus@digitalocean.com>
(cherry picked from commit d70ca81502d25bd7a76dd2ed2a538bf5e6584822)

Merge pull request #38971 from smithfarm/wip-48743-octopus

octopus: rgw: distribute cache for exclusive put

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38970 from smithfarm/wip-48693-octopus

octopus: rgw: adding user related web token claims to ops log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38826 from smithfarm/wip-48725-octopus

octopus: rgw: fix bucket limit check fill_status warnings

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38821 from smithfarm/wip-48804-octopus

octopus: rgw: cls/user: set from_index for reset stats calls

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39046 from smithfarm/wip-48968-octopus

octopus: ocf: add support for mapping images within an RBD namespace

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38981 from smithfarm/wip-48864-octopus

octopus: rgw/multisite: Verify if the synced object is identical to source

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38829 from smithfarm/wip-48544-octopus

octopus: rgw_file: return common_prefixes in lexical order

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #38824 from smithfarm/wip-48546-octopus

octopus: rgw: lc: correctly dimension lc shard index vector

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #39059 from votdev/issue_48068_tz_octopus

octopus: cephadm: Various properties like 'last_refresh' do not contain timezone

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/suites/krbd: add msgr2 modes to most subsuites

basic, rbd and rbd-nomount subsuites are expanded to run with each
of ms_mode=legacy, ms_mode=crc and ms_mode=secure.  This increases
the total number of jobs in the suite from 100 to 220.

fsx, singleton and thrash subsuites choose ms_mode at random (from
the above plus ms_mode=prefer-crc).

unmap and wac subsuites remain msgr1-only.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 65948736a41f424d8152b208d013419f8d6038a4)

doc: deprecate [no]cephx_require_signatures map options

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit fd5f5722a29964bb33b305a381bcf9a48cdcbb47)

krbd: add support for msgr2

Recognize ms_mode map option and filter initial monitor addresses
accordingly: if ms_mode is not given or ms_mode=legacy, discard v2
addresses, otherwise discard v1 addresses.

Note that nothing was discarded (i.e. v2 addresses were passed to
the kernel) previously.  The intent was to preserve that behaviour
in case ms_mode is not given, allowing to change the kernel default
in the future.  However, it turns out that mount.ceph helper has
been misguidedly discarding v2 addresses since commit eae01275134e
("mount.ceph: fork a child to get info from local configuration"),
so that ship has sailed.

Fixes: https://tracker.ceph.com/issues/48976
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 08f714964b7fe5024504818f01328a41acc24965)

Conflicts:
src/tools/rbd/action/Kernel.cc [ commit 34f539d8af33 ("rbd:
  delay parsing of default kernel map options") not in octopus ]

Merge pull request #38336 from votdev/wip-48398-octopus

octopus: mgr/dashboard: display placement column in service table

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

python-common/drivegroups: avoid dropping "rotational: 0" from DeviceSelection

False is a legitimate value for the rotational setting and should be included in the JSON output, only None should be ignored.

Fixes: http://tracker.ceph.com/issues/49014
Fixes: cd6a488ab2ca036dd4fb36751b938f605e97e1c8
Signed-off-by: Lukas Stockner <lstockner@genesiscloud.com>
(cherry picked from commit c32f6f5448e51d3196f7a2644ea97ecd22a04f92)

cephadm: fix rgw osd cap tag

The syntax is "allow rwx tag rgw *=*'.

Sorry, I thought this would have gotten caught in testing :(

Fixes: 373cc847cf0f8b4ec7aefbfe64c01c3f18a4e021
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit de1efbd62b9258630c2a2d55bfd12034cc8b603f)

mgr/cephadm: raise HEALTH_WARN when cephadm daemon in 'error' state

If cephadm daemons are not happy we should raise a warning.  Aside from
being an important part of the user experience, this will also help us
catch teuthology test errors.

Fixes: https://tracker.ceph.com/issues/45628
Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 01f60cf4e0a751c314120c02956d4ff941eb71b4)

qa/cephadm: Add yaml output to smoke test

this will provide a more detailed output, like

```yaml
...snip...
service_type: node-exporter
service_name: node-exporter
placement:
  host_pattern: '*'
status:
  created: '2021-01-18T11:21:56.024810Z'
  last_refresh: '2021-01-18T11:23:24.477672Z'
  running: 0
  size: 1
events:
- "2021-01-18T11:23:09.602644Z service:node-exporter [ERROR] \"Failed while placing\
  \ node-exporter.ubuntuon ubuntu: cephadm exited with an error code: 1, stderr:Deploy\
  \ daemon node-exporter.ubuntu ...\nVerifying port 9100 ...\nTraceback (most recent\
  \ call last):\n  File \"<stdin>\", line 7274, in <module>\n  File \"<stdin>\", line\
  \ 1563, in _default_image\n  File \"<stdin>\", line 3698, in command_deploy\n  File\
  \ \"<stdin>\", line 2338, in deploy_daemon\n  File \"<stdin>\", line 1961, in create_daemon_dirs\n\
  AssertionError\""
...snip...
```

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 88c6c34e2ba32e38c7fca93722737c3b4d31fe6c)

mgr/cephadm: tolerate old host inventory without 'hostname' key

Older cephadm clusters lack the 'hostname' key in the host spec.  e.g.,

 "cpach": {"addr": "cpach", "labels": ["mon"]}, "eutow": {"addr": "eutow", "labels": ["mon"]}, "stud": {"addr": "stud", "labels": ["mon"]}}

Populate hostname from the dict key if necessary for compatibility.

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit ad4ea787d063ba65269f38f185ff18a327cb7bbb)

cephadm: fix 2> syntax in unit.run

We need a space between the command (which ends with a container name)
and the 2> or else the 2 is considered part of the command.  E.g.,

! /usr/bin/podman rm -f ceph-a9a8c7ee-5b72-11eb-8f93-001a4aab830c-mon.a2> /dev/null

Fixes: 1bed46e4b0094863a119df59c6ae5f254c2e211d
Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit ce4743f72e6c7eea9514b8c9b6c20309fea5d455)

Conflicts:
src/cephadm/cephadm

cephadm: silence "Failed to evict container" log msg

Right now, we're printing some evil looking messages in the log:

```
systemd[1]: Starting Ceph mgr.node2.ankmgz for ...
podman[32354]: Error: no container with name or ID ceph-... found: no such container
bash[32363]: Error: Failed to evict container: "": Failed to find container "ceph-..." in state: no container with name or ID ceph-... found: no such container
bash[32363]: Error: no container with ID or name "ceph-..." found: no such container
````

Also, the unit.run command already removes the container. No need
for ExecStartPre to do the same.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 1bed46e4b0094863a119df59c6ae5f254c2e211d)

cephadm: Various properties like 'last_refresh' do not contain timezone

Fixes: https://tracker.ceph.com/issues/48068
Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 3fe715201c8c07cf4ea86b590f9682422eeccf33)

mon: paxos: Delete logger in destructor

reset() can race with shutdown() leading to a use-after-free on the
'logger' object.

Fixes: https://tracker.ceph.com/issues/48386
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit cc295d1c87552eb76b9188d88d7e6ab2f3108149)

Conflicts:
src/mon/Paxos.h
- stable branch has slightly different Paxos and get_name function declarations

mgr/dashboard: Use secure cookies to store JWT Token

This PR intends to store the jwt token in secure cookies instead of local storage

Fixes: https://tracker.ceph.com/issues/44591
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 36703c63381e6723fff57266235f8230e6af1d92)
(cherry picked from commit 3c72dc309936b23e413dc1aee8ca49c795c48a0f)

 Conflicts:
qa/tasks/mgr/dashboard/helper.py
qa/tasks/mgr/dashboard/test_auth.py
src/pybind/mgr/dashboard/controllers/__init__.py
src/pybind/mgr/dashboard/controllers/auth.py
src/pybind/mgr/dashboard/controllers/saml2.py
src/pybind/mgr/dashboard/frontend/cypress/integration/orchestrator/01-hosts.e2e-spec.ts
src/pybind/mgr/dashboard/frontend/cypress/integration/orchestrator/02-hosts-inventory.e2e-spec.ts
src/pybind/mgr/dashboard/frontend/cypress/integration/orchestrator/03-inventory.e2e-spec.ts
src/pybind/mgr/dashboard/frontend/cypress/integration/orchestrator/04-osds.e2e-spec.ts
src/pybind/mgr/dashboard/frontend/cypress/integration/ui/language.e2e-spec.ts
src/pybind/mgr/dashboard/frontend/cypress/integration/ui/navigation.e2e-spec.ts
src/pybind/mgr/dashboard/frontend/package-lock.json
src/pybind/mgr/dashboard/frontend/package.json
src/pybind/mgr/dashboard/frontend/src/app/app.module.ts
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/dashboard-help/dashboard-help.component.ts
     - Adopting the changes from the master branch, ignoring few e2e changes
       as few files doesn't exist in octopus.

Merge pull request #39018 from sebastian-philipp/octopus-backport-38766

octopus: cephadm: make "ceph orch {restart|...}" asynchronous

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39019 from sebastian-philipp/octopus-backport-38815

octopus: mgr/cephadm: lock multithreaded access to OSDRemovalQueue

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39020 from sebastian-philipp/octopus-backport-38904

octopus: cephadm: Don't make sysctl spam the log file

Reviewed-by: Michael Fritch <mfritch@suse.com>

rgw: use static_ptr for etag verifiers

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 45a060612ec376110b84c2a2b7783c8a6aee191c)

rgw: add factory function create_etag_verifier()

move all of the etag verifier initialization into a helper function.
none of the errors there should be fatal and fail the download, they
should just turn etag verification off

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8fa8974bbd63fbc8be9cdf929a875910e2147d65)

rgw: move etag verifiers to namespace rgw::putobj

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6ad2e3eef3f9bbf41de471ef5bad9502023e113c)

rgw: simplify out SourceObjType

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 21cc9034410bb6e675b1b04888e1b85e3eb5d71f)

rgw: rgw_sync_obj_etag_verify accounts for compressed multipart uploads

the etag verifier for multipart uploads uses the manifest to get the
logical offsets for each part. but when compression is enabled, those
are offsets into the compressed data. use the source object's compression
info to translate those compressed part offsets back to their original
offsets

Fixes: https://tracker.ceph.com/issues/45992
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 51f79fad8398d35e07f01fa45704124e16fadeec)

rgw: ETagVerifier_MPU takes existing offset vector

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2c1934bd3746697249b95b3f79f4c05425d7b40e)

rgw: add helper to decode compression info from single attr

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a5520be135156c867a6502845603e2afdbb0a44a)

RGW:Multisite: Check rgw_sync_obj_etag_verify option only once

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit f92cfaf74f1eb8809653867b8c67a3ace37619f2)

RGW:Multisite: Convert is_mpu_obj into an enum SourceObjType

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit fa5422597837032d97f9afceff8b5a22fad0cda7)

RGW:Multisite: Rename rgw_sync_obj_integrity to rgw_sync_obj_etag_verify

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit 6c4262bfa7d54563ea8bf616154a6b3491d59347)

RGW:Multisite: Rename rgw_copy_verify_object to rgw_sync_obj_integrity

This patch renames the option rgw_copy_verify_object to
rgw_sync_obj_integrity and incorporates more changes suggested through
code-review comments.

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit 31e944fced60e47139973361cbb753aeaeb3c863)

RGW:Multisite: Create a new filter for ETag Verifier

This patch re-writes the ETag verifier into a filter that peeks into the
incoming stream of data and calculates MD5 checksum.

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit 2677c4b88806d4af6d525157e7006c1b0ca1b964)

[RGW][Multisite] Add multisite verifier support for MPU objects

The Etag for MPU objects is calculated using a method different from how
it is done for atomic objects. This patch makes use of the RGWObjManifest
to determine the parts in the source cluster and re-computes the ETag in
a similar fashion at the destination cluster during multisite sync for
verification.

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit 6f085bb9cb6cffddcf2eed18a7295a3dd59f755f)

RGW:Multisite: Verify if the synced object is identical to source

Introduce an option 'rgw_copy_verify_object' which allows the object
copied from remote cluster through multisite sync is identical to the
source object. This is done by generating the MD5 checksum of the data
being copied and compared to the ETAG stored as part of the object's
attribute.

Signed-off-by: Prasad Krishnan <prasad.krishnan@flipkart.com>
(cherry picked from commit 23fb5d7b786f1034a02c991c6a648e455b5b9c30)

mgr/prometheus: don't store exception as e

Python's logging module's exception() method will log the full exception
and stack trace for us, so we do not need to store the exception in the
"e" variable here.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit a17c603effd3367dc64c87a1d6c53d6d3d794fc7)

mgr/prometheus: use threading.Event instead of sleep

This allows us to avoid waiting for the sleep to finish when waiting for
the thread to finish.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit dd5886c3c006e388283df50cc87addeffb3b2b52)

mgr/prometheus: Log collection issues

Log any issues encountered during the data collection and continue to
collect the data anyway (after a sleep).

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 28a5c13bf993679e3098d73df27ded249f34dc99)

mgr/prometheus: Use mgr.release_name for always on modules

The host_version is not populated properly in the early stages of ceph
mgr start up process. We can use mgr.release_name instead. It is more
stable and it provides the data even if mgr_map does not contain the
versions, yet.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit aa0650092da3cbf1a73151999874001352cfb9ef)

mgr/prometheus: Clean up collection thread

We need to clean up the metrics collection thread.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 03fcaccafc877d10a894b1c39af5547f172c1ed3)

pybind/mgr/prometheus: use get_module_option() wrapper

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 5700419b272786b5a5e2e7b8d3d5e9d9917a9469)

mgr/prometheus: Add healthcheck metric for SLOW_OPS

SLOW_OPS is triggered by op tracker, and generates a health
alert but healthchecks do not create metrics for prometheus to
use as alert triggers. This change adds SLOW_OPS metric, and
provides a simple means to extend to other relevant health
checks in the future

If the extract of the value from the health check message fails
we log an error and remove the metric from the metric set. In
addition the metric description has changed to better reflect
the scenarios where SLOW_OPS can be triggered.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
(cherry picked from commit 2010432b5045c8f1dd0c052def497e590a1bf2fe)

Merge pull request #39071 from smithfarm/wip-49003-octopus

octopus: test/run-cli-tests: use cram from github

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: drop explicit masking of implicit feature bits

Now that the create image state machine is handling the masking
of implicit features, all callers to the state machine can skip
the need to perform the masking themselves.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e1dd69391900a236fbda5e4cee58e9bdc20824fc)

Conflicts:
src/librbd/api/Migration.cc
- in octopus, this file does not mention RBD_FEATURES_IMPLICIT_ENABLE

librbd/image: mask out all implicit features when creating an image

This will ensure that all paths to the create image state machine
properly handle this condition. Previously, it was up to the callers
of the state machine to clear the implicit feature bits.

Fixes: https://tracker.ceph.com/issues/48647
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f52f78caca6f9743e75c8289771375f5f582300a)

librbd/image: use internal feature flag mask when creating images

When validating the provided features for an image, utilize the
RBD_FEATURES_INTERNAL flag instead of just RBD_FEATURE_OPERATION.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7dd77814402f0710f9283084b45960a054078897)

librbd/api: do not explicitly add the MIGRATION feature bit

This feature bit is automatically added to the source and destination
images via the cls method on the OSD.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 5380615037e1f78369e58eac9847b465e9e8bf19)

Conflicts:
src/librbd/api/Migration.cc
- no Migration<I>::prepare_import template in octopus

cephadm: Don't make sysctl spam the log file

`sysctl -a` really spams the log file in Teuthology

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 70722a2efb5207f7d6fc7e8ecf4aab71e2a1a39d)

Conflicts:
  src/cephadm/cephadm

test/downloads: drop cram-0.5 tar ball

since we install cram from github, drop the tarball from repo

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d8cbe4eec11ef7f9fbbb61666b9ec7a262c1a85c)

test/run-cli-tests: use cram from github

to address the failure of

  WARNING: Built wheel for cram is invalid: Metadata 1.2 mandates PEP 440 version, but '0.5.0ceph.2011-01-14' is not
Failed to build cram

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c74e614813d69ef38139e5e745873dc2a31b1c59)