mgr/cephadm: Add KMIP server support for NVMeoF gateway

Fixes: https://tracker.ceph.com/issues/75739
Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #68023 from rzarzynski/wip-doc-check-umbrella-initial-tag

doc/dev/release-checklist: check v21.0.0 tag

Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>

Merge pull request #68001 from cbodley/wip-qa-rgw-upgrade-distros-latest

qa/rgw/upgrade: symlinks are explicit about distro versions

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #68003 from yuvalif/wip-yuval-75713

test/rgw/notification: fix the cloudevents package version

Merge pull request #67777 from pritha-srivastava/wip-rgw-d4n-cleaning

rgw/d4n: erasing dirty object entry for delete markers

doc/dev/release-checklist: check v21.0.0 tag

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #66524 from aainscow/umbrella-release

Umbrella checklist kickoff!

Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #67997 from rhcs-dashboard/abount-window

mgr/Dashboard : [Dashboard] Not able to open "About" window

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #66318 from aclamk/aclamk-fcm-plugin

FCM plugin for block device

extblkdev/fcm: Add plugin identification for fcm

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

extblkdev/fcm: Do not enable set_keepcaps by default

When one runs FCM enabled hardware config.set_keepcaps must be set.
However, having it as default is inadvised.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

extblkdev/fcm: Change douts/derrs

Remove printing fcm values to dout(1).
Upgraded inability to access FCM log from dout to derr.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

extblkdev/fcm: Create perf counters with fcm stats

Created "extblkdev" section.
Each time plugin is asked to `get_stats()` also write to perf counters.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

fixup warning of fcm plugin core

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

extblkdev/fcm: derr when FCM and no block device discard

Add derr message when FCM is in use but discard is not enabled.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

extblkdev/fcm: Refuse to operate on multimedia lvm block devices

BlueStore is selecting were data is put to the device.
Merging 2 FCM devices together means that BlueStore will see free space
on one of the devices, but not know the other is full and asking to put
data there. It will cause -ENOSPC while free space is reported.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

extblkdev/fcm: Adapt FCM plugin to RHCS9

Device /sys content is different from what original plugin expected.
Adapted to it.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

added FCM plugin

Signed-off-by: Martin Ohmacht <mohmacht@us.ibm.com>

Merge pull request #67289 from aclamk/aclamk-extblkdev-mandatory

os/bluestore: Make extblkdev mandatory

Merge pull request #65847 from pritha-srivastava/wip-rgw-sts-expired-creds

rgw/sts: correct error code to 400 (from 403)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/rgw/notification: fix the cloudevents package version

Fixes: https://tracker.ceph.com/issues/75713
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #64641 from phlogistonjohn/jjm-smb-keybridge

smb: add keybridge support and fscrypt configuration

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

docs: reorder mon and mds for future new releases

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

doc: Check redmine-upkeep.yml to add umbrella.

Previous PR already added umbrella, therefore adding redmin-upkeep is a no-op.
Any future developer copying this needs to look at PR 67316

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

script/ceph-release-notes: add umbrella

Just add umbrella.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

osd: Remove unused osd_perf_counter "scrbcnt_write_blocked"

This counter is no longer needed in umbrella.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

qa: qa tasks from the release checklist for Umbrella

All the qa tasks in the umbrella release checklist.
This includes removing X-3 releases and adding X-1 releases
(where appropriate).

Also updated release-checklist.rst

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

include: Remove incorrect "available" comment for feature bit. (cleanup)

This flag was not available, so I have deleted it.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

qa: add X-1 release (tentacle) to fs upgrade suite

Add files for the tentacle release (X-1) to the qa/suites/fs/upgrade suites

Signed-off-by: Chris Harris <harriscr@uk.ibm.com>

qa: remove reef code from fs

Remove the reef code from the fs upgrade suites as we no longer want to test X-3 releases

Signed-off-by: Chris Harris <harriscr@uk.ibm.com>

osd: Remove octopus-depending code in Fast EC

We are leaving alone legacy EC

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

osd: Update remove after squid to remove after umbrella comment

Following consulting with author, we don't yet want to remove.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

qa Upgrade to Umbrella

All the qa tasks in the umbrella release checklist

Signed-off-by: Chris Harris <harriscr@uk.ibm.com>
Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
# Conflicts:
# qa/suites/upgrade/telemetry-upgrade/tentacle-x/1-tasks.yaml
# qa/workunits/test_telemetry_tentacle.sh
# qa/workunits/test_telemetry_tentacle_x.sh

doc: Confirm DEPRECATED checklist item and clean up pg_scrubber

No DEPRECATED features.

Remove some code labeled as "Remove in Umbrella"

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

qa: update require-osd-release to umbrella

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

tools/monmaptool: bump new cluster version to X

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

osd/OSDMap: update to umbrella

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

qa/workunits/cephtool/test: update to umbrella

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

mon/OSDMonitor: update to umbrella

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

common/options/global.yaml.in: update for umbrella

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

mon/MgrMonitor: update for umbrella

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

qa/standalone/mon/misc: update for umbrella

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

doc: update compatset for umbrella

The unknown bit shown in commit was picked as being "far in the future".
We have now used it and now need to move the bit one release at a time.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

include/ceph_features: add SERVER_UMBRELLA feature bit

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

cephadm,ceph-volume: update to umbrella

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

doc/dev/release-checklist: add backport-create-issue

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

*: add constants and release names

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

ceph_release: update to umbrella

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

librbd: bump version

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

CMakeLists.txt: update VERSION

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

doc: reset for umbrella

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

qa/rgw/upgrade: symlinks are explicit about distro versions

avoid relying on "ubuntu_latest" and "rpm_latest" symlinks, which change
over time on main. be explicit about the distro versions supported by
the initial release

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: [Dashboard] Not able to open "About" window

Fixes: https://tracker.ceph.com/issues/75702
Signed-off-by: pujaoshahu <pshahu@redhat.com>

mgr/smb: update fs.py to request all caps for the cephfs volume

Update the authorizer class in fs.py to request all caps (*) for our
volume. This is necessary in order to make use of the fscrypt feature
from smbd.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add pid file directory option to cfg watch sidecar

This will enable the config watch sidecar to signal processes
with a SIGHUP to tell them to reload configuration when config
watch has detected a configuration change. Currently only used
by keybridge.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add pidfile option to smb keybridge sidecar

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: fix unit test assertion

The unit test was looking for the wrong string (but the right
condition). This fixes the string to check.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc: add documentation for keybridge and fscrypt options

Add docs for the keybrige configuration and cephfs fscrypt options
added to the smb mgr module resource definitions.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add some keybridge related unit test cases

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add support for generating keybridge configuration

Add support for generating the sambacc configuration section for
keybridge. Add support for configuring smb shares for keybridge access.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add cross-check validation for keybridge scopes

Validate that scope names are not re-used, etc. Check on things that
can't be done in single object validation.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add new cephfs parameter for getting fscrypt keys

Add a new field to the cephfs configuration section for shares. This
section selects the keybridge scope and key name to use when acquiring
the key to use for fscrypt.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add keybridge configuration to cluster resource

Add keybridge service configuration classes and parameters to the
resources module. This supports enabling the keybridge, setting up
scopes for the keybridge and it's access control.

A helper class is added that parses and helps validate the scope names.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add enums that will be used for configuring keybridge

Add a pair of enum types that will be used for configuring the
keybridge. The scope type identifies what kind of scope is being
used. The peer policy can be used to allow a dev or other user
more access to the keybridge api for development purposes.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add raw data methods to MemConfigStore

Add the set_data/get_data methods to the MemConfigStore so that future
test updates will not fail to save tls credential objects.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: fix a resource error unpacking str instead of list

Add special handling for the case where a string is passed instead of a
list. Without this fix a string will be converted into a list of single
letter items, something pretty much no one ever wants. Raise an
exception instead.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add keybridge sidecar to smb daemon module

The keybridge uses the sambacc configuration but can also be passed
CLI options. Since cephadm writes the cert files, cephadm must also
pass the file names to use to the container args.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: enable setting up SSL/TLS files for keybridge sidecar

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/deployment: add keybridge feature to smb service spec

The keybridge sidecar is enabled by the keybridge feature flag.
This sidecar will be used to help fetch keys over various protocols
for the ceph module to use to set up fs encryption.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #67314 from aainscow/block_non_4k_chunks

mon: Deny EC optimizations (fast EC) for non-4k-aligned chunk-sizes.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>

Merge pull request #67723 from perezjosibm/wip-perezjos-tracker75202

qa/workunits: add crimson/test_crimson_dump_metrics.sh

Merge pull request #67936 from shraddhaag/wip-shraddhaag-fix-crimson-qa

qa/distros/crimson-supported-all-distro: reinstate centos

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

os/bluestore: Make extblkdev missing a warning

When mounting BlueStore for OSD problem with extblkdev meta are fatal error.
When mounting BlueStore in other cases extblkdev are just derr messages.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

extblkdev: Make preload one-time function

Subsequent calls to extblkdev::preload() do not do anything.
It is assumed that first call already loads all plugins that could be
loaded.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add config bluestore_use_ebd

When EBD(extblkdev) plugin is in use usually it needs to present all the time.
For bluestore deployed with EBD plugin it makes it an error if bluestore tries
to mount and EBD plugin is not present.

Preload of extblkdev plugins was misplaced.
Moved loading plugins into BlueStore.
This way both OSD and tools can load plugins.

Plugins are now loaded only:
- before mkfs
- when extblkdev plugin is signalled in label meta

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

extblkdev: Expand interface, add plugin id retrieval

Plugin identification is a tool to implement two features:
1) Check the plugin is initialized
2) Verify that the plugin initialzied is the proper one

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

rgw/sts: correct error code to 400 (from 403)
for expires sts credentials.

Fixes: https://tracker.ceph.com/issues/73441
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #57395 from rzarzynski/wip-wip-crimson-ec-with-recovery

crimson/osd: bring support for EC, including recovery

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

qa/distros/crimson-supported-all-distro: reinstate centos

Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>

Revert "crimson/osd, test: turn the EC tests on for crimson"

This reverts commit 36fc26b2b803fcf425de674a578c4aba20a5d9ea.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: fixes after crimson::os::with_store<>()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: Avoid pwlc spanning intervals

This is a crimson port of `dc0a195b937eb175cf663d48a856f486435e987f`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: improve !is_omap() logging

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimon/osd: drop present_shards from PGLog handler's partial_write()

In correspondence with classical's 880a17e39626d99a0b6cc8259523daa83c72802c.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: improve debugs in ECTransaction::Generate::Generate()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimon/osd: implement PG::PGLogEntryHandler::partial_write() for FastEC

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: port changes from classial FastEC

This commit ports changes from c116b8615d68a3926dc78a4965cc0a28ff85d4f2.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tests: LibRadosIoEC allows for overwrites / unalligned writes on crimson

1. crimson-osd does support solely the FastEC flavor of EC pools;
   SlowEC isn't now and won't be implemented.
2. FastEC imposes the EC overwrites support.
3. When run against crimson's EC implementation, the tests shall
   not enforce e.g. ENOTSUPP on overwrite.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement handling of RADOS' extended attirbutes in FastEC

Support for `attr_cache` in replicated backend is planned for later.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop unused variable from PGBackend::getxattr()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop unneccessary variant of PGBackend::getxattr()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: don't decode EC only attrs are needed

This a crimson coutnerpart of 417fb71c9b5628726d3217909ba1b6d3e7bf251a.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rmw_pipeline in RMWPipeline::Op

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

fixup: crimson/osd: settle ECRecoveryBackend -- local MOSDPGPush

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

fixup: crimson/osd: settle RMWPipeline and RMWPipeline within ECBackend

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

fixup: crimson/osd: wire MOSDECSubOpWriteReply up with ECBackend

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

fixup: crimson/osd: route EC messages over PG

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: fix unused variable warnings pg.cc and seastore

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ECCommon: fixups for crimson's FastEC support

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

fixup: crimson/osd: bring ECRecoveryBackend, phase 0.1

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>