librbd/migration: add external clusters support

This commit extends NativeFormat (aka migration where the migration
source is an RBD image) to support external Ceph clusters, limited to
import-only mode.

Co-authored-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/migration: move away from util::create_ioctx() in NativeFormat

This is another step towards supporting migration from external
clusters, where creating an IoCtx from a Rados instance that has
nothing to do with dst_io_ctx would be needed.  It also allows to
get rid of a pool lookup in the middle of parsing code.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

common/config: export CEPH_CONF_FILE_DEFAULT

It used to be exported until commit 318c62f8ae16 ("common/config:
cleanup remove some unused macros").  Having CEPH_CONF_FILE_DEFAULT
avaialable is handy to prevent parse_config_files() from picking up
CEPH_CONF environment variable.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: RefreshParentRequest::m_parent_snap_id is unused

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: assert on parent in ImageCtx destructor

... and switch to in-class initializers while at it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/qemu: remove hard-coding of cluster name

This commit allows running the qemu task on an arbitrary cluster name.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/rbd: support non-default ceph clusters

This commit allows running the rbd task on an arbitrary cluster name.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/migration: don't clone when flattening

When the flatten flag is set, instead of creating the
destination image by cloning, create it independently,
as the parent relation is unnecessary in this case.
This will be particularly useful when the migration source
is located in an external Ceph cluster, which will soon be
supported.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #44366 from orozery/rbd-crypto-migration

librbd/crypto: fix issue when live-migrating from encrypted export

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58002 from nbalacha/wip-nbalacha-sorted-snaps

cls/rbd: add group_snap_list_order method to enable sorting snapshots in creation order

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #59073 from anoopcs9/cephadm-docs-smb-spec

doc/cephadm/smb: Add missing user_sources option to service spec

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56233 from kamoltat/wip-ksirivad-fix-64802

RADOS: Generalize stretch mode pg temp handling to be usable without stretch mode
Samuel Just <sjust@redhat.com>

Merge pull request #57381 from kamoltat/wip-ksirivad-fix-65371

PeeringState.cc: Only populate want_acting when num_osds < bucket_max
Reviewed by: Samuel Just <sjust@redhat.com>

Merge pull request #58585 from anoopcs9/cephadm-update-smbd-version

cephadm/smb: Determine samba version within container

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52489 from ifed01/wip-ifed-alloc2

os/bluestore: introduce hybrid_btree2 allocator

librbd/migration: make ImageDispatch handle encryption for non-native formats

With NativeFormat now being handled via dispatch, handling encryption
for non-native formats (i.e. mapping to raw image extents and performing
decryption/mapping back on completion) in the migration layer is really
straightforward.

Note that alignment doesn't need to be performed in the migration layer
because it happens on the destination image -- the "align and resubmit"
logic in C_UnalignedObjectReadRequest should kick in before the call to
read_parent().

Fixes: https://tracker.ceph.com/issues/53674
Co-authored-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: don't make an extra copy of image_extents in C_ImageReadRequest ctor

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd: perform cleanup in test_clone_encryption()

... so that RAW_DEV can be unmapped and future tests can reuse testimg
and other image names without bumping into watchers and older snapshots.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd: no need to chmod in luks-encryption.sh

Most workunits expect the user to be a member of "disk" group, so we
can pretty much rely on that being the case at this point.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58907 from adam-lj/rbd-throttle-perf

librbd: Reduce use of atomics in librbd throttling

Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58993 from pereman2/unused-func

os/bluestore: Unused func

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #58693 from xxhdx1985126/wip-crimson-clear-temp-objects-startup

crimson/osd: clear ondisk temp objects on startup

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57562 from perezjosibm/wip-perezjos-cpu-map

script/: add cpu-map.sh to aid manual selection of CPU cores for threads for profiling

Merge pull request #55374 from aclamk/wip-aclamk-bs-multi-label

os/bluestore: Multiple bdev labels on main block device

Merge PR #59040 into main

* refs/pull/59040/head:
doc: documenting the feature that scrub clear the entries from damage table on repair

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/cephadm/smb: Add missing user_sources option to service spec

Signed-off-by: Anoop C S <anoopcs@cryptolab.net>

Merge pull request #56782 from joscollin/wip-B65171-support-replication-start-end-notifications

cephfs_mirror: provide metrics for last successful snapshot sync

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57458 from lxbsz/wip-session-evict

qa/cephfs: add test_session_evict_non_blocklisted test case

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57987 from joscollin/wip-B64752-valgrind-report-leaks

qa/suites: drop --show-reachable=yes from fs:valgrind tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58097 from gardran/wip-gardran-kill-count

client: eliminate duplicate lookups wherever possible

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: documenting the feature that scrub clear the entries from damage table on repair

Fixes: https://tracker.ceph.com/issues/67337
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #59060 from qn2060/patch-3

doc/dev/peering: Change acting set num

Merge PR #52939 into main

* refs/pull/52939/head:
mon/MonClient: handle ms_handle_fast_authentication return

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #58437 into main

* refs/pull/58437/head:
mds: remove the misleading and redundant logs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #58582 into main

* refs/pull/58582/head:
test: add a test for `fs set down` on online cluster
mon: fix `fs set down` to adjust max_mds only when cluster is not down

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58511 from pereman2/ch-umount

os/bluestore: assert CollectionRef count is 1 on umount

Merge pull request #57572 from pereman2/refactors

os/bluestore: Improve Readability of Extent Map Sharding

Merge pull request #59011 from xxhdx1985126/wip-67327

crimson/osd: send empty transactions to backfill targets that haven't backfilled the objects yet

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58708 from xxhdx1985126/wip-67068

crimson/osd/osd_operation/client_request: requeue client requests from a temporary queue other than ClientRequest::Orderer::list

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58148 from xxhdx1985126/wip-65696

crimson/osd/osd_operations: hang requests if the objects are unfound

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59051 from dang/wip-dang-67321

RGW - Zipper: pull in upstream fix for zpp_bits

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/dev/peering: Change acting set num

This commit updates the peering doc page with the changed [2,1,2] to [3,1,2]

Signed-off-by: qn2060 <qn2060@users.noreply.github.com>

Merge pull request #58971 from cbodley/wip-rm-boost-redis-again-again

submodule: remove the boost_redis submodule again (again)

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #58037 from Suyashd999/refractor-PrimaryLogPG

osd/PrimaryLogPG.cc bl length would be stored before moving

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

RGW - Zipper: pull in upstream fix for zpp_bits

zpp_bits included code that required a newere compiler.  Pull in the
upstream fix from:

https://github.com/eyalz800/zpp_bits/pull/170

Fixes: https://tracker.ceph.com/issues/67321
Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

Merge pull request #59024 from zdover23/wip-doc-2024-08-05-README-building-ceph

doc/README.md - add ordered list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58282 from NitzanMordhai/wip-nitzan-daemonwatchdog-should-terminate-thrasher-when-bark

qa/tasks: watchdog should terminate thrasher

Merge pull request #58664 from aclamk/wip-aclamk-qa-less-bluestore-debug

qa/suites/rados: Reduced BlueStore log levels

script/: add cpu-map.sh to aid manual selection of CPU cores for threads for profiling

Signed-off-by: Jose J Palacios-Perez <perezjos@uk.ibm.com>

thrashers: standardize stop and join method names

Thrashers that do not inherit from ThrasherGreenlet previously used a
method called do_join, which combined stop and join functionality. To
ensure consistency and clarity, we want all thrashers to use separate
stop, join, and stop_and_join methods.

This commit renames methods and implements missing stop and stop_and_join
methods in thrashers that did not inherit from ThrasherGreenlet.

Fixes: https://tracker.ceph.com/issues/66698
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

qa/tasks: watchdog terminate thrasher

If a thrasher exception occurs, the do_dump_ops thread will continue
looping until the Teuthology timeout is reached.
The watchdog should terminate the thrasher to free up resources.

Fixes: https://tracker.ceph.com/issues/66698
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge PR #58500 into main

* refs/pull/58500/head:
ptl-tool: Fix token file name and variable names

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58963 from rhcs-dashboard/fix-67009-main

mgr/dashboard: Dashboard not showing Object/Overview correctly.

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #58746 from rhcs-dashboard/fix-67069-main

mgr/dashboard: make multi-cluster context switcher scrollable if we have many entries in it

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

cls/rbd: add functions to get group snap orders

Added functions to get the group snap order keys.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #57905 from mcpreston/docs-dashboard-typo

doc/mgr/dashboard: fix TLS typo

Merge pull request #58767 from kshtsk/wip-python-invalid-escape-sequence

mgr: fix invalid escape sequence

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58373 from ceph/wip-tasks-ceph-mkfs-has-no-f

qa: ceph.py mkfs.ext4 has no -f

os/bluestore: BluestoreRepairTest ch cleanup

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

cls/rbd: save max group snap order

Save the last used group snap order value to speed up
snapshot creation.

Signed-off-by: N Balachandran <nibalach@redhat.com>

cls/rbd: save group snapshot creation order in a new key

In order to be able to list group snapshots in creation order, a
new snapshot order key is stored on snapshot creation.

[nbalacha: removed the group_snap_unsorted function,
reverted the group_snap_list function changes]

Signed-off-by: Mykola Golub <mgolub@suse.com>
Signed-off-by: N Balachandran <nibalach@redhat.com>

doc/README.md - add ordered list

Add ordered-list formatting to the "Building Ceph" section of README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58891 from zdover23/wip-doc-2024-07-28-cephfs-file-layouts-layout-fields

doc/cephfs: edit "Layout Fields" text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

PeeringState.cc: Only populate want_acting when num_osds < bucket_max

Currently, in `calc_replicated_acting_stretch` we would
call `pop_ancester` before checking whether the next
OSD we populate in the want_acting set would have
the ancestor count that is >= bucket_max.

Fixes: https://tracker.ceph.com/issues/65371
Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #58115 from ronen-fr/wip-rf-memmodel

common: MemoryModel: performance improvements and API changes

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

librbd: Reduce use of atomics in librbd throttling

Signed-off-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>

Merge pull request #57786 from Svelar/asan_cli_test

crush: avoid out-of-bound access and simplify enlarging buckets

Merge pull request #58718 from wanglinke521/wlk-kv-stats

kv/RocksDBStore: add kv_stats function to print all CF info

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #58470 from mohit84/issue_66867

mon: Remove any pg_upmap_primary mapping during remove a pool

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Josh Salomon <josh.salomon@gmail.com>

Merge pull request #58302 from jianwei1216/feat_add_mclock_perfcounter_main

osd/scheduler: add mclock queue length perfcounter

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

crimson/osd: send empty transactions to backfill targets that haven't
backfilled the objects yet

Fixes: https://tracker.ceph.com/issues/67327
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #59006 from zdover23/wip-doc-2024-08-03-glossary-ceph-ansible

doc/glossary: add "ceph-ansible"

doc/glossary: add "ceph-ansible"

Add a "ceph-ansible" entry to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58969 from gbregman/main

mgr/cephadm: Add SPDK log level to nvmeof configuration file

Merge pull request #58402 from rkachach/fix_issue_mtls_support

mgr/cephadm: adding mTLS for ceph mgmt-gateway and backend services communication

Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: add SPDK log level to nvmeof configuration
Fixes https://tracker.ceph.com/issues/67258

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

python-common/ceph/deployment: add SPDK log level to nvmeof configuration
Fixes https://tracker.ceph.com/issues/67258

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #58883 from idryomov/wip-backport-create-resourceattrerror

script/backport-create-issue: handle ResourceAttrError when getting CF_TAGS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58882 from idryomov/wip-native-format-dispatch

librbd/migration: don't instantiate NativeFormat, handle it via dispatch

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: N Balachandran <nibalach@redhat.com>

Merge pull request #58862 from cbodley/wip-rgw-posix-bucket-cache-lock

rgw/posix: name the lock_guard in BucketCacheEntry::reclaim()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

os/bluestore: remove unused functions

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #58765 from yuvalif/wip-yuval-64184

rgw/async/notifications: use common async waiter in pubsub push

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58672 from YiteGu/code-cleanup-osdmon

mon/osdmonitor: cleanup the code for preprocess_mark_me_dead

Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #58635 from badone/wip-tracker-50371-rados_api_test-timeout-failures

qa: Restrict rados api tests to large clusters and increase timeout

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #58631 from cbodley/wip-66336

msg: insert PriorityDispatchers in sorted position

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57863 from NitzanMordhai/wip-nitzan-thrash-erasure-code-crush-4-nodes-8-6-overrides

suites/ec-rados-plugin=jerasure-k=8-m=6-crush: roles set with overrides

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #58747 from AliMasarweh/wip-alimasa-bn-test-fix-http

RGW|BN: fix lifecycle test issue

Reviewed-by: yuvalif<ylifshit@redhat.com>

mgr/cephadm: fixing Grafana domain handling

it seems that with Grafana 10.4.0 the domain parameter is taken into
account while building the final url (earlier versions didn't seem to
behave the same way). This change sets the domain to the hostname where
Grafana daemon is running instead of '*.lab'. serve_from_sub_path is
removed as it's no needed and when add it causes some undesirable
redirections that could break monitoring HA.

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/dashboard: Dashboard not showing Object/Overview correctly.

Add undefined checks wherever necessary to the updateChartData method in
dashboard-area-chart component

Fixes: https://tracker.ceph.com/issues/67009
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge PR #51332 into main

* refs/pull/51332/head:
qa: add test for ceph tell with unknown cephtype
pybind/ceph_argparse: fixing error message for ceph tell command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: make multi-cluster context switcher scrollable if we have many entries in it

Fixes: https://tracker.ceph.com/issues/67069
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

doc/cephfs: edit "Layout Fields" text

Improve the English in the "Layout Fields" section of
doc/cephfs/file-layouts.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58899 from zdover23/wip-doc-2024-07-29-glossary-prometheus

doc/glossary: add "Prometheus"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58972 from anthonyeleven/glacier-melting

doc: Improve doc/radosgw/placement.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc: Improve doc/radosgw/placement.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

submodule: remove the boost_redis submodule again (again)

5843c6b04bacf9a7c981fca5d874ab3400f855db accidentally added back to the
boost_redis submodule that was last removed in 924f2e9f87f0ee940eff363c26a714a309639d26

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/cephadm: add SSL support to ceph-exporter

This commit adds SSL support to the ceph-exporter deployment
made by cephadm. When `secure_monitoring_stack` is set to `True`,
the `ceph-exporter` container is restarted with SSL enabled.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding mTLS support

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #56792 from cbodley/wip-qa-rgw-notifications-valgrind

qa/rgw: enable valgrind in rgw/notifications suite

Merge pull request #54671 from baum/ceph-nvmeof-mon

mon: add NVMe-oF gateway monitor and HA

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>