librbd/migration: make ImageDispatch handle encryption for non-native formats

With NativeFormat now being handled via dispatch, handling encryption
for non-native formats (i.e. mapping to raw image extents and performing
decryption/mapping back on completion) in the migration layer is really
straightforward.

Note that alignment doesn't need to be performed in the migration layer
because it happens on the destination image -- the "align and resubmit"
logic in C_UnalignedObjectReadRequest should kick in before the call to
read_parent().

Fixes: https://tracker.ceph.com/issues/53674
Co-authored-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 0000c3447407772039121bb4499f243df1c889da)

Conflicts:
qa/workunits/rbd/luks-encryption.sh [
          - commit 9892ead7fcd9 ("librbd/crypto: allow loading luks
            format without specifying version") not in quincy -- pass
            through $format to test_migration_read_and_copyup() and
            test_migration_native_with_snaps()
          - commit 1d3de19c4005 ("tools/rbd: add encryption format
            support for cloned image") not in quincy -- drop
            test_migration_clone() ]
src/librbd/migration/ImageDispatch.cc [ ImageArea support and
  commit 20aee5bbbcb5 ("neorados: Make IOContext getters/setters
  less weird") not in quincy ]

librbd: don't make an extra copy of image_extents in C_ImageReadRequest ctor

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit b20a897061feabc4e22c339c4e7a8aa5155151e8)

qa/workunits/rbd: perform cleanup in test_clone_encryption()

... so that RAW_DEV can be unmapped and future tests can reuse testimg
and other image names without bumping into watchers and older snapshots.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 82d26909cb12b62d88f69f90eb8af692e497bddb)

 Conflicts:
qa/workunits/rbd/luks-encryption.sh [ commit 1d3de19c4005
  ("tools/rbd: add encryption format support for cloned image")
  not in quincy ]

qa/workunits/rbd: no need to chmod in luks-encryption.sh

Most workunits expect the user to be a member of "disk" group, so we
can pretty much rely on that being the case at this point.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 830cbee7a5f875af04f335266b02ad96e4cd71c4)

Conflicts:
qa/workunits/rbd/luks-encryption.sh [ commit 1d3de19c4005
  ("tools/rbd: add encryption format support for cloned image")
  not in quincy ]

librbd/migration: make FormatInterface::read() void again

Now that NativeFormat is handled via dispatch, FormatInterface::read()
can be void again for consistency with FormatInterface::list_snaps().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit b6c7f69b8044f9206aa982c2aba6466c49fd2bea)

Conflicts:
src/librbd/migration/ImageDispatch.cc [ commit 20aee5bbbcb5
  ("neorados: Make IOContext getters/setters less weird") not
  in quincy ]

librbd/migration: close source image in OpenSourceImageRequest

Currently, on errors in FormatInterface::open(), RawFormat disposes
of src_image_ctx, but QCOWFormat doesn't, which is a leak.  Rather than
having each format do it internally, do it in OpenSourceImageRequest.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 63159d6b431470f5edc4b110cebf46865c550689)

librbd/migration: don't instantiate NativeFormat, handle it via dispatch

Trying to shoehorn NativeFormat under FormatInterface doesn't really
work.  It fundamentally doesn't fit in:

- Unlike for RawFormat and QCOWFormat, src_image_ctx for NativeFormat
  is not dummy -- it's an ImageCtx for a real RBD image.  Pre-creating
  it in OpenSourceImageRequest with the expectation that placeholder
  values would be overridden later forces NativeFormat to reach into
  ImageCtx guts, duplicating the logic in the constructor.  This also
  necessitates calling snap_set() in a separate step, since snap_id
  isn't known at the time ImageCtx is created.

- Unlike for RawFormat and QCOWFormat, get_image_size() and
  get_snapshots() implementations for NativeFormat are dummy.

- read() and list_snaps() implementations for NativeFormat are
  inconsistent: read() passes through io::ImageDispatch layer, but
  list_snaps() doesn't.  Both can be passing through, meaning that in
  essence these are also dummy.

All of this is with today's code.  Additional complications arise with
planned support for migrating from external clusters where src_image_ctx
would require more invasive patching to "move" to an IoCtx belonging to
an external cluster's CephContext and also with other work.

With the above in mind, NativeFormat actually consists of:

1. Code that parses the "type: native" source spec
2. Code that patches ImageCtx, working around the fact that it's
   pre-created in OpenSourceImageRequest
3. A bunch of dummy implementations for FormatInterface

With this change, (1) is wrapped into a static method that also creates
ImageCtx after all required parameters are known and (2) and (3) go away
entirely.  NativeFormat no longer implements FormatInterface and doesn't
get instantiated at all.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit cacf7ca941876f64f9a04867ffc6cdcb484d89b9)

Conflicts:
src/librbd/migration/NativeFormat.cc [ ImageArea support not in
  quincy ]

librbd/migration/NativeFormat: refactor source spec parsing

In preparation for not instantiating NativeFormat and losing a copy of
the source spec JSON object in m_json_object, refactor the parsing code
to use only const methods (which std::map's operator[] isn't) and local
variables where possible.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 3bbf1f5ddbaa4a8c252d70a384e23852f0c537c1)

librbd/migration/NativeFormat: do pool lookup instead of creating io_ctx

A Rados instance is sufficient to map the pool name to the pool ID,
no need to involve an IoCtx instance as well.  While at it, report
distinctive errors for a non-existing pool and an invalid JSON value
for pool_name key cases.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 1ba9a32598f50073b574b4649736d76b678a1c58)

librbd/migration: make SourceSpecBuilder::parse_source_spec() static

In preparation for divorcing NativeFormat from FormatInterface and
changing when/how src_image_ctx is created, make parse_source_spec()
independent of src_image_ctx.  The "invalid source-spec JSON" error is
duplicated by the "failed to parse migration source-spec" error, so
just get rid of the former to spare having to pass CephContext to
parse_source_spec().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit f172fb97be9a6129be7cdbaa87346dc6c8e8ccb1)

librbd/migration/OpenSourceImageRequest: rename io_ctx -> dst_io_ctx

For now, this is just slightly clearer.  The distinction would become
important with planned support for migrating from external clusters.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit c14356b1f9eea0988e071f40dc0df005f70edd4d)

librbd/migration: massage some error messages

Add missing spaces, don't use the word stream when reporting errors
on POSIX file operations (open() and lseek64()) and fix a cut-and-paste
typo in RawSnapshot.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 789df2ce38f35ffb3e86974e02868e5fff71e72c)

librbd/api: clean up leftovers in Migration::prepare_import()

Dead code after return and an unused variable.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit b92ad49a35536fff87d840ffbf171aee61b05424)

Merge pull request #51136 from RaminNietzsche/wip-cloud-sync-fixes-to-the-quincy

cloud sync: fix crash due to objs on cr stack

Merge pull request #56166 from vshankar/wip-64249-quincy

quincy: qa: correct usage of DEBUGFS_META_DIR in dedent

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53626 from lxbsz/wip-62513

quincy: qa/cephfs: switch to python3 for centos stream 9

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53629 from lxbsz/wip-62867

quincy: qa: increase the http postBuffer size and disable sslVerify

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #55392 from dparmar18/wip-64067-quincy

quincy: qa: fix error reporting string in assert_cluster_log

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #55745 from lxbsz/wip-64221

quincy: qa/tasks/cephfs/test_misc: switch duration to timeout

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #56404 from batrick/wip-65062-quincy

quincy: qa/cephfs: ignorelist clog of MDS_UP_LESS_THAN_MAX

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #59101 from zdover23/wip-doc-2024-08-09-backport-59807-to-quincy

quincy: docs/rados/operations/stretch-mode: warn device class is not supported

docs/rados/operations/stretch-mode: warn device class is not supported

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>
(cherry picked from commit aa1d8cf4fa321e24e850bd5f687a6ddad3ce05e3)

Merge pull request #54015 from k0ste/wip-59362-quincy

quincy: rgw: fix rgw cache invalidation after unregister_watch() error

Merge pull request #53471 from k0ste/wip-57636-quincy

quincy: rgw: Drain async_processor request queue during shutdown

Merge pull request #53462 from k0ste/wip-55678-quincy

quincy: rgw: remove potentially conficting definition of dout_subsys

Merge pull request #53438 from k0ste/wip-62824-quincy

quincy: RadosGW API: incorrect bucket quota in response to HEAD /{bucket}/?usage

Merge pull request #53374 from jzhu116-bloomberg/wip-59694-quincy

quincy: rgw/notification: remove non x-amz-meta-* attributes from bucket notifications

Merge pull request #52995 from k0ste/wip-59025-quincy

quincy: rgw: use unique_ptr for flat_map emplace in BucketTrimWatche

Merge pull request #52937 from k0ste/wip-58904-quincy

quincy: rgw: Fix Browser POST content-length-range min value

Merge pull request #52935 from k0ste/wip-54610-quincy

quincy: RGW - Get quota on OPs with a bucket

Merge pull request #52606 from cbodley/wip-62139-quincy

quincy: rgw: s3 object lock avoids overflow in retention date

Merge pull request #52054 from trociny/wip-61431-quincy

quincy: rgw: multisite data log flag not used

Merge pull request #51446 from k0ste/wip-59730-quincy

quincy: RGW: Solving the issue of not populating etag in Multipart upload result

Merge pull request #51257 from k0ste/wip-55062-quincy

quincy: radosgw-admin: fix segfault on pipe modify without source/dest zone specified

Merge pull request #50539 from k0ste/wip-55151-quincy

quincy: rgw: Update "CEPH_RGW_DIR_SUGGEST_LOG_OP" for remove entries

Merge pull request #50520 from k0ste/wip-54227-quincy

quincy: rgw: Erase old storage class attr when the object is rewrited using r…

Merge pull request #50519 from k0ste/wip-54201-quincy

quincy: rgw: CopyObject works with x-amz-copy-source-if-* headers

Merge pull request #48082 from pritha-srivastava/wip-57509-quincy

quincy: rgw/sts: modify max_session_duration using update role REST API/ radosgw-admin command.

Merge pull request #57488 from ljflores/wip-66040-quincy

quincy: qa/suites/rados/singleton: add POOL_APP_NOT_ENABLED to ignorelist

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #50551 from rzarzynski/wip-59084-quincy

quincy: mon: add proxy to cache tier options

nReviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #58753 from adk3798/quincy-only-bind-to-network

quincy: mgr/cephadm: add ability for haproxy, prometheus, grafana to bind on specific ip

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59080 from zdover23/wip-doc-2024-08-07-backport-59040-to-quincy

quincy: doc: documenting the feature that scrub clear the entries from damage…

Merge pull request #58762 from adk3798/wip-66477-quincy

quincy: qa/suites: add "mon down" log variations to ignorelist

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58761 from adk3798/wip-66957-quincy

quincy: cephadm: turn off cgroups_split setting  when bootstrapping with --no-cgroups-split

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58756 from adk3798/wip-65726-quincy

quincy: python-common: handle "anonymous_access: false" in to_json of Grafana spec

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56308 from adk3798/quincy-timemaster

quincy: cephadm: add timemaster to timesync services list

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57442 from joscollin/wip-65984-quincy

quincy: mgr/stats: initialize mx_last_updated in FSPerfStats

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #54712 from mchangir/wip-63589-quincy

quincy: qa: run kernel_untar_build with newer tarball

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #56393 from batrick/wip-65059-quincy

quincy: qa/suites/fs/nfs: use standard health ignorelist

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #56417 from batrick/wip-65089-quincy

quincy: qa/cephfs: add mgr debugging

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #56667 from batrick/wip-65107-quincy

quincy: qa/cephfs: add probabilistic ignorelist for pg_health

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52986 from leonid-s-usov/backport/strip-nofail/quincy

quincy: nofail option in fstab not supported

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #55238 from joscollin/wip-64074-quincy

quincy: pybind/mgr/mirroring: drop mon_host from peer_list

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #56168 from vshankar/wip-64046-quincy

quincy: qa: set mds config with `config set` for a particular test

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #58316 from cbodley/wip-66725

quincy: build: Make boost_url a list

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #56815 from Matan-B/wip-65307-quincy

quincy: osd/SnapMapper: fix _lookup_purged_snap

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55868 from trociny/wip-64649-quincy

quincy: mon/OSDMonitor: fix get_min_last_epoch_clean()

Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

doc: documenting the feature that scrub clear the entries from damage table on repair

Fixes: https://tracker.ceph.com/issues/67337
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>
(cherry picked from commit 140e77638dc95fefcf62f01fe7e45bc0729439be)

Merge pull request #58760 from adk3798/wip-66790-quincy

quincy: cephadm: disable ms_bind_ipv4 if we will enable ms_bind_ipv6

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58759 from adk3798/wip-65967-quincy

quincy: mgr/cephadm: make setting --cgroups=split configurable for adopted daemons

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58757 from adk3798/wip-65950-quincy

quincy: mgr/cephadm: cleanup iscsi keyring upon daemon removal

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58755 from adk3798/wip-65419-quincy

quincy: cephadm: fix host-maintenance command always exiting with a failure

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58754 from adk3798/wip-65380-quincy

quincy: mgr/cephadm: make client-keyring deploying ceph.conf optional

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56930 from adk3798/wip-65513-quincy

quincy: mgr/dashboard: fix duplicate grafana panels when on mgr failover

Reviewed-by: Avan Thakkar <athakkar@redhat.com>

cephadm.py: add timemaster to timesync services list

On debian/ubuntu, if you need PTP, it's possible to use the linuxptp package for time-synchonization.
In that case the systemd service is called timemaster and is a wrapper for chrony/ntpd/phc2sys/ptp4l.

Signed-off-by: Florent Carli <florent.carli@rte-france.com>
(cherry picked from commit 4ee1c761e50e3c3feed89ccc5c450ac96681b778)

Conflicts:
src/cephadm/cephadm.py

Merge pull request #56837 from rishabh-d-dave/wip-65441-quincy

quincy: qa/cephfs: CephFSTestCase.create_client() must keyring

Merge pull request #56753 from dparmar18/quincy-backport-fix-test-cluster-enonent

quincy: qa/test_nfs: fix test failure when cluster does not exist

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #57445 from mchangir/wip-65900-quincy

quincy: mgr/snap_schedule: restore yearly spec to lowercase y

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #58818 from Svelar/wip-67136-quincy

quincy: bluestore/bluestore_types: avoid heap-buffer-overflow in another way to keep code uniformity

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #57404 from k0ste/wip-65914-quincy

quincy: kv/RocksDBStore: Configure compact-on-deletion for all CFs

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #57196 from k0ste/wip-64592-quincy

quincy: os/bluestore: fix the problem of l_bluefs_log_compactions double recording

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #59064 from zdover23/wip-doc-2024-08-07-backport-59060-to-quincy

quincy: doc/dev/peering: Change acting set num

doc/dev/peering: Change acting set num

This commit updates the peering doc page with the changed [2,1,2] to [3,1,2]

Signed-off-by: qn2060 <qn2060@users.noreply.github.com>
(cherry picked from commit 103d65ea6787ab16d1824f71578264e0abb9820c)

rgw/sts: code changes to replace modify role with
modify role trust policy, to better reflect
what the command does.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit 875da7f9041b54ee507bff00f40b65f6810b939a)

Merge pull request #50538 from k0ste/wip-55502-quincy

quincy: rgw: address crash and race in RGWIndexCompletionManager

Merge pull request #55357 from cbodley/wip-64227-quincy

quincy: rgw/rest: fix url decode of post params for iam/sts/sns

Merge pull request #58313 from cbodley/wip-66711-quincy

quincy: rgw/notifications/test: fix rabbitmq and kafka issues in centos9

Merge pull request #57907 from adamemerson/wip-62292-quincy

quincy: rgw: modify string match_wildcards with fnmatch

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #57405 from ivancich/wip-65891-quincy

quincy: rgw: make incomplete multipart upload part of bucket check efficient

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #56119 from mkogan1/wip-64767-quincy

quincy: rgw/beast: Enable SSL session-id reuse speedup mechanism

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #51806 from soumyakoduri/wip-skoduri-quincy

[quincy] RGW backports

Reviewed-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #51797 from yuvalif/wip-61478-quincy

quincy: rgw/kafka: remove potential race condition between creation and deletion of endpoint

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59033 from zdover23/wip-doc-2024-08-05-backport-57905-to-quincy

quincy: doc/mgr/dashboard: fix TLS typo

doc/mgr/dashboard: fix TLS typo

Change "SSL/TSL" to "SSL/TLS"

Signed-off-by: Mindy Preston <mcpreston@icecube.wisc.edu>
(cherry picked from commit e63b967aeb00a667ab5c91b9b526c7a19a9539ec)

Merge pull request #59009 from zdover23/wip-doc-2024-08-04-backport-59006-to-quincy

quincy: doc/glossary: add "ceph-ansible"

doc/glossary: add "ceph-ansible"

Add a "ceph-ansible" entry to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 484e9d2b9d81b6de498ac08125c9a458730dd444)

Merge pull request #58852 from idryomov/wip-67051-quincy

quincy: qa/workunits/rbd: avoid caching effects in luks-encryption.sh

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #58565 from idryomov/wip-rbd-c9-on-quincy

quincy: centos 9 related backports for RBD

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58208 from idryomov/wip-66300-quincy

quincy: librbd: disallow group snap rollback if memberships don't match

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58979 from zdover23/wip-doc-2024-08-01-backport-58899-to-quincy

quincy: doc/glossary: add "Prometheus"

Merge pull request #58975 from zdover23/wip-doc-2024-08-01-backport-58972-to-quincy

quincy: doc: Improve doc/radosgw/placement.rst

doc/glossary: add "Prometheus"

Add a glossary entry for "Prometheus".

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 1a76464999cb2d2cc748d69e1bb0567544a0bb8b)

doc: Improve doc/radosgw/placement.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 5890a156524b8d1fe3a51ea1c76d800e5903a01c)

Merge pull request #58949 from zdover23/wip-doc-2024-07-31-backport-58946-to-quincy

quincy: doc/rados/operations: remove vanity cluster name reference from crush…

doc/rados/operations: remove vanity cluster name reference from crush-map.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 62562ec65e3c1cc040a57ab77fc219e0aad25651)

Merge pull request #58942 from zdover23/wip-doc-2024-07-31-backport-58922-to-quincy

quincy: doc/cephadm: edit "Using Custom Images"

Merge pull request #58945 from zdover23/wip-doc-2024-07-31-backport-58908-to-quincy

quincy: doc/glossary: add "ceph-fuse" entry

Merge pull request #58939 from zdover23/wip-doc-2024-07-31-backport-58918-to-quincy

quincy: doc/dev: improve basic-workflow.rst

doc/glossary: add "ceph-fuse" entry

Add a "ceph-fuse" entry to the glossary.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d1c049d24befedc396221dafcc66db8c40acf5ab)

doc/cephadm: edit "Using Custom Images"

Edit the section "Using Custom Images" in
doc/cephadm/services/monitoring.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit e99fc5c82fd698c222a489a1ebc26c8fcbca321d)