doc/mgr: document new external ceph cluster support for smb

Document the new feature that allows smb clusters on one Ceph cluster to
make use of CephFS running on a different external Ceph cluster.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: update the handler to support external ceph cluster type

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: update staging funcs to include ext ceph cluster resource

Update the functions in the staging.py support file to include support
for the newly added external ceph cluster type.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add new external ceph cluster to internal store mechs

Extend the internal store mechanisms to support the newly added
external ceph cluster resource type.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add external ceph cluster to sqlite store

Starting to get silly with the boilerplate but it should do the trick.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add new external ceph cluster resource type

Add a new resource type that will allow an smb cluster to use
cephfs from a ceph cluster *other* than the one hosting the smb
instance.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add new enum for external ceph cluster resource type

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: remove largely unused nuisance function

Remove a function that existed largely for debugging that was frequently
an attractive nuisance when reading or looking over the code path that
actually creates the configuration and deploys smb resources.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: move resource types to class level

Move resource types to class level to make these more obvious
for future modifications.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: avoid using thandler.generate_config function

This function was already "deprecated" and will be removed in a future
commit.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: add external cluster config file contents

Add support for generating ceph conf and keyring file contents based on
the external cluster config objects in the smb service spec.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add support for custom ceph conf/keyring files

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: let smb sidecars be extra-ified

Add code needed to add extra custom configuration files to sidecar
containers.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: fix naming issue when using sidecars

When extending a container with extra settings a sidecar or init
container should take the custom settings from that of the primary
container. So use the newly added parent_identity function (when
available) to determine the correct identity to use.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add parent_identity function

Add a new parent_identity function that a sub-identity type can use to
reconstruct a precise parent identity.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: allow sidecar containers to make use custom configs

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/deployment: add class to defining smb external ceph clusters

Add a new class that defines an external ceph cluster to be used by smb
clusters that need to connect to CephFS file systems that are external
to the ceph cluster hosting the smb services.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/deployment: allow smb external store to use memory

This is needed to fix a test failure, users probably don't even know
mem: exists and the risk that they'd use it is low.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/deployment: fix typo in comment in smb class

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #67218 from jamiepryde/jerasure-unit-test-big-endian-fix

test/erasure-code: Use memory comparisons that are safe on big endian arch

Merge pull request #67240 from guits/cv-raw-list-perf

ceph-volume: single lvs call to speed up exclude_lvm_osd_devices

Merge pull request #66646 from ujjawal-ibm/ceph-volume-format-device

Add NVMe preformat support to skip redundant mkfs discard

Merge PR #66183 into main

* refs/pull/66183/head:

Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Shachar Sharon <ssharon@redhat.com>

Merge pull request #66606 from shraddhaag/add-logs

PeeringState::needs_recovery(): log first missing item

Merge pull request #66352 from tchaikov/debian-invoke-rc.d

debian: remove invoke-rc.d calls from postrm scripts

Merge pull request #66591 from NitzanMordhai/wip-nitzan-pickup-object-corpus-20.2.0-380-gdbcbbd3f281

ceph-object-corpus: pick up 20.2.0-380-gdbcbbd3f281

Merge pull request #66553 from smanjara/wip-fix-object-sync-enoent

rgw: retry ENOENT if bucket reshard races with index operation

Merge pull request #66721 from swjtu-zhanglei/main-wip

rgw: refact by encapsulating io_context_pool to enforce lazy initialization

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #66567 from mertsunacoglu/wip-prerequest-auth

RGW: Change prerequest hook to run after authorization process

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #66509 from morphes1995/rgw_qos_fix

rgw: ops large budget caused by rate limiter was enabled and max_ops …

Reviewed-by: Mark Kogan <mkogan@ibm.com>

ceph-volume: skip redundant NVMe mkfs discards

- Avoid redundant discard during mkfs when discard is disabled
- Reduces mkfs time on large NVMe devices by skipping long running discard operations

Fixes: https://tracker.ceph.com/issues/74908
Signed-off-by: Ujjawal Anand <ujjawal.anand@ibm.com>

Merge pull request #67264 from guits/fix-74818

ceph-volume: pass osd_type value to ceph-volume

Merge pull request #66594 from cbodley/wip-qa-rgw-account-migration

qa/rgw: add test_account_migration workunit

Reviewed-by: Krunal Chheda <kchheda3@bloomberg.net>

Merge PR #66657 into main

* refs/pull/66657/head:
developer: Add name to maps

Reviewed-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #67309 into main

* refs/pull/67309/head:
doc: correct spelling

Reviewed-by: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #67078 into main

* refs/pull/67078/head:
doc: Fixes a spelling error on the RATIO for PG scaling recommendation.

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #66526 from gardran/wip-gardran-show-sharding

os/bluestore: show RocksDB sharding information

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge PR #67320 into main

* refs/pull/67320/head:
.github/workflows: use main branch for config diff

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Naveen Naidu <naveen.naidu@ibm.com>

Merge pull request #66821 from rhcs-dashboard/cephfs-mirroring-filesystem

mgr/dashboard: Cephfs mirroring Filesystem Selection

Reviewed-by: Naman Munet <nmunet@redhat.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #67044 from Hezko/nvme-submodule-to-1-6-5

mgr/dashboard: bump nvmeof submodule to 1.6.5

Merge pull request #66929 from mkogan1/wip-t74245

ceph-volume: fix build on RHEL 10/python 3.12

Reviewed-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #66906 from rhcs-dashboard/gateway-resources

mgr/dashboard: NVme-gateway-resource

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Naman Munet <nmunet@redhat.com>

Merge PR #67316 into main

* refs/pull/67316/head:
doc/dev/release-checklist: add release to redmine-upkeep trigger
.github/workflows/redmine-upkeep: run for release branches

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

.github/workflows: use main branch for config diff

This script does not exist for the "head" reference of backport PRs.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

qa/rgw: add test_account_migration workunit

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #67307 from leonidc/fix-tests-after-beacon-diff

 qa: fixing the qa test in mon/misc.sh after adding beacon-diff feature

doc/dev/release-checklist: add release to redmine-upkeep trigger

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

.github/workflows/redmine-upkeep: run for release branches

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #67011 into main

* refs/pull/67011/head:
qa/multisite: use boto3's ClientError in place of assert_raises from tools.py.
qa/multisite: test fixes
qa/multisite: boto3 in tests.py
qa/multisite: zone files use boto3 resource api
qa/multisite: switch to boto3 in multisite test libraries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: correct spelling

I saw a typo so I fixed it

Signed-off-by: Amnay <amnay.m@gmail.com>

libcephfs_proxy: log more information about the negotiation

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: implement client support for batched readdir

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: create internal ceph_dir_result structure

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: add the number of supported operations to negotiation

The v0 negotiation structure has been modified to hold the total number
of operations and callbacks supported by the peer. The changes are done in
a way that it's completely transparent and harmless for a peer expecting
the previous definition.

This will be useful to quickly check if the daemon supports some
operation, or the client supports some callback before sending them.

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: implement daemon support for batched readdir

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: define protocol for batched readdir

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: cleanup unneded variable

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: fix include dependency

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

 qa: fixing the qa test in mon/misc.sh after adding beacon-diff feature

Signed-off-by: Leonid Chernin <leonidc@il.ibm.com>

mgr/dashboard: NVme-gateway-resource

Fixes: https://tracker.ceph.com/issues/74334
Signed-off-by: Sagar Gopale <sagar.gopale@ibm.com>

Merge pull request #67290 from shraddhaag/wip-shraddhaag-cephadm-crimson-tests

cephadm, ceph-volume: add tests for crimson OSD support

Merge pull request #66255 from rkachach/fix_issue_73853

mgr/cephadm: don't remove TLS certs if svc still has daemons on host

Reviewed-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>

mgr/dashboard: Cephfs Mirroring - Filesystem Selection

Fixes: https://tracker.ceph.com/issues/74280
Signed-off-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

Merge PR #65538 into main

* refs/pull/65538/head:

Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Shachar Sharon <ssharon@redhat.com>

Merge pull request #67182 from Kh9705/add-s3select-limit-examples

doc/radosgw/s3select: added SQL LIMIT operator examples

Merge pull request #67104 from rhcs-dashboard/header-page-component

mgr/dashboard: add page header component

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

Merge pull request #67059 from bluikko/wip-src-script-backport-fixes

ceph-backport.sh: clean function argument use and update GitHub help link

Merge pull request #67202 from tchaikov/wip-ec-isa-cleanup

erasure-code/isa: eliminate redundant map lookups

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #67128 from afreen23/subsystem-step-2

mgr/dashboard: Step two subsystem creation form

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: pujaoshahu <pshahu@redhat.com>

mgr/dashboard: bump nvmeof submodule to 1.6.5

update proto files and gateway submodule
Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>
tomer

Merge pull request #67174 from bluikko/wip-fix-74721-trailing-space

mgr/dashboard: remove trailing space from directory name

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: pujaoshahu <pshahu@redhat.com>

Merge pull request #66092 from kshtsk/wip-fix-lua-kernel-check

qa/cephfs: lua to respect missing kernel in yaml

cephadm, ceph-volume: add tests for crimson OSD support

This commit adds tests for the crimson OSD support in cephadm and ceph-volume.
The following tests are added for the same:

1. cephadm: DriveGroupSpec validation checks for osd_type.
2. cephadm: entrypoint verification in runfile.
3. cephadm to ceph-volume: command verification when osd_type is specified in spec.
4. ceph-volume: binary selection verification for mkfs cmd.

Fixes: https://tracker.ceph.com/issues/74851
Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>

Merge pull request #66815 from bluikko/wip-doc-man-fix-directives

doc/man: fix invalid directive missing a second colon

mgr/dashboard: Add step two of subsystem create form

- add steps to add initiators
- can add by input field
- added right influencer (right panel) in tearsheet component
- added unit tests
- includes api updates

Fixes https://tracker.ceph.com/issues/74096

Signed-off-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: add page header component

Adds page header component and applies it to the CephFS Mirroring list

Fixes: https://tracker.ceph.com/issues/74626
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@ibm.com>

Merge PR #64743 into main

* refs/pull/64743/head:

Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #66979 from idryomov/wip-73824

librbd: introduce RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #64819 from aclamk/aclamk-rocksdb-binned-cache-perf-counters

kv/RocksDB: Add instrumentation to BinnedLRUCache

Reviewed-by: Jaya Prakash <jayaprakash@ibm.com>

Merge pull request #67170 from afreen23/subsystem-step-3

mgr/dashboard: Step three subsystem creation form

Reviewed-by: pujaoshahu <pshahu@redhat.com>

Merge pull request #67217 from aliakseimakarau/main-s390x-fixexecstack

Mark s390x CRC assembly code as not requiring ELF execstack

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #67261 from Matan-B/wip-matanb-crimson-66488-revert

Revert "crimson/os/seastore/cache: TRIM_DIRTY/CLEANER_* transactions won't invalidate other transactions anymore"

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #67266 from gbregman/main

mgr/cephadm: Change the default value of omap_file_lock_duration in  NVMEoF spec file

Merge PR #67251 into main

* refs/pull/67251/head:
qa: set column for insertion
qa: bail sqlite3 on any error
qa: use actual sqlite3 blob instead of string
test: use json_extract instead of awkward json_tree

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #66552 from Jayaprakash-ibm/wip-add-scrub-flag

osd: Add flag to Identify Scrub IO

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>
Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

ceph-volume: pass osd_type value to ceph-volume

Use OSDType.value when building the ceph-volume lvm batch command
so that the CLI receives "classic" or "crimson" instead of the enum
repr ("OSDType.classic").

Fixes: https://tracker.ceph.com/issues/74818
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #67024 from amathuria/wip-amat-fix-73791

crimson/os/seastore: Add progress logging to transaction processing

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #65814 from amathuria/wip-amat-crimson-add-pg-num-testing

qa/suites/crimson-rados: Add pg splitting tests to crimson thrash tes…

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados/thrash: Modify osd_target_transaction_size

We observed in Seastore, deletion of a large batch (default osd_target_transaction_size=30)
can take a significant amount of time.

Because this happens inside the peering_pp.process stage, it blocks the PG's peering pipeline.
During this block, any incoming OSDMap updates (PGAdvanceMap) are stalled behind the deletion work.
This eventually causes a global OSD-wide map progression hang because
the OSD cannot advance past an epoch until all PGs have processed
it.

To fix this, we are reducing osd_target_transaction_size to 5 to lower
conflict rates and allow deletion transactions to complete.

Fixes: https://tracker.ceph.com/issues/73791
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>

mgr/cephadm: Change the default value of omap_file_lock_duration in NVMEoF spec file

Fixes: https://tracker.ceph.com/issues/74822
Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Revert "crimson/os/seastore/lba_mapping: don't allow classes above"

This reverts commit 45726996d65ec07b2c35a420c3ee0e42f0bf6cb4.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/os/seastore/btree_types: BtreeCursors don't hold local copies of"

This reverts commit 5a24cac63a676f0a4641257286f1d1f4f7377ce3.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/os/seastore/async_cleaner: avoid its header dependence on"

This reverts commit 1aae6f34b68faf15a8691f599420f23d5a84943e.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/os/seastore/lba_manager: make sure alloc_extents return viewable"

This reverts commit e6fd2e87a18ed25845e5ba5f6d03ebb20818e583.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/os/seastore/async_cleaner: renew backref cursors when they are"

This reverts commit a9c17a2c0912064f51492c2b012cd5608c6a57c1.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/os/seastore/seastore_types: define rewriting transactions"

This reverts commit 1696011379dfa5f1558b3119d4936ef4943ecedf.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/os/seastore/cache: add facilities to synchronize data and states"

This reverts commit 2fc047c51f37cbdfd3952f616cc07f99cf2483d7.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/os/seastore/cache: unlink mutated extents from the stable"

This reverts commit bfc1231e3716f7b780fb1362a76e13078e767f4c.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/os/seastore/cache: since extent committer will also set"

This reverts commit 7e72a79af75d24119c094c533ec4101358aa258d.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/os/seastore/cache: drop unused last_commit"

This reverts commit 33e63bc85fafc057974142071318a32121a62a59.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>