cephadm: add support to remove sidecar/init-ctr systemd units

Add support to the command_rm_daemon function to remove systemd services
and corresponding unit files for any sidecars or init-containers
associated with a primary daemon.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: replace some calls to rm with fs function calls

This should be a little cleaner, perhaps a tiny performance improvement,
and most importantly work better with the virtual-fs provided for the
unit tests and make this function more testable in the future.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: minor reorgs to command_rm_daemon

Make a few small chanes to command_rm_daemon to clean it up in
preparation for future changes. Move the construction of a
DaemonIdentity earlier and use the service name derived directly from it
if possible. I didn't remove the old approach of looking up a service
name in the listing because I don't know if doing so would break
old/adopted services and there's no unit test coverage for that.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add a terminate_service systemd function

Add a function that encapsulates the common 3-step process found in
cephadm of stopping, resetting errors, and disabling a service.
This will be used to handle new services to stop as well as cleaning
up some places where the three steps are performed inline.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add sidecars_from_dropin function

Add a sidecars_from_dropin function that takes a minimal PathInfo object
and returns a PathInfo populated with sidecars information based on
reading the content of the dropin file. This will be useful when we need
information on configured sidecars but are not on the deploy path.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add method for parsing a sub identity from a service name

This will become handy later when we need DaemonSubIdentity objects and
are not on the deploy path.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add unlink_file function

Add the unlink_file function to file_utils. This fills in a gap between
python 3.6 and features provided in pathlib.Path in later versions of
python. Adds an option to ignore all errors for good measure.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: rename {get,lookup}_unit_name_by_daemon_name

Rename get_unit_name_by_daemon_name to lookup__unit_name_by_daemon_name
to emphasize that the function does a full lookup (via the daemon list)
to determine the systemd service name for the given values and that
there's a potential performance implication to doing that.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: avoid errors when calling sub-ident method

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: convert iscsi daemon form to use sidecars

Convert the iscsi container daemon form class to produce a sidecar
container for the tcmu runner and reduce the amount of special casing
for iscsi.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cepahdm: add support for sidecars to container daemon forms

Add support for producing sidecar containers to the container daemon
form type and pass the sidecars through to the unit/script management
function.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add function for writing sidecar script

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add template for sidecar run script

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: switch to new style init containers

Convert the call that writes out files for a daemon to use the new
dedicated systemd-unit support for init containers.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: black format systemd_unit.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add support for systemd unit files for init/sidecar ctrs

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add templates for new systemd unit files

Add templates that will be used for advanced systemd services: sidecars,
init containers, and drop-in files to glue them all together.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cepahdm: avoid subclass related issues for data_dir

Do not have data dir depend on the daemon_name property. This property
is overriden by the subclass(es) but we want DaemonIdentities and
DaemonSubIdentities to share the same data_dir.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add a daemon identity method for getting sidecar script path

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add a utilty method for type checking DaemonSubIdentity types

Add a `must` method that ensures a DaemonIdentity is a DaemonSubIdentity
in a single call.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add SidecarContainer type

Add a new type representing sidecar containers.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add a container copying method

Add a method for duplicating the container properties into a new object.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add daemon identity methods for upcoming sidecars/init services

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: use service_name method of daemon identity

Use the new service_name method of the DaemonIdentity type in some basic
cases where string concatenation was being used.
I also converted the name of the agent's helper method to match and made
it private to avoid future confusion.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add systemd name generation to daemon identity

There are a bunch of places in the code that wants the full systemd
service name but the code generally is just doing string concatenation.
Add a method to avoid stringing it all the time. This will help as there
will be more systemd work in the future.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55043 from zdover23/wip-doc-2024-01-02-rados-troubleshooting-log-and-debug-title

doc/rados: improve subsection title in log-and-debug.rst

doc/rados: improve subsection title in log-and-debug.rst

Change the title of "Logging Settings" to "Logging and Debugging
Settings" in doc/rados/troubleshooting/log-and-debug.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54876 from ifed01/wip-ifed-fix-63795

test/store_test: fix deferred writing test cases

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>
Reviewed-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #55038 from dparmar18/minor-doc-update-disaster-recovery-experts

doc/cephfs: Update disaster-recovery-experts.rst to mention Slack

doc/cephfs: Update disaster-recovery-experts.rst to mention Slack

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #55031 from zdover23/wip-doc-2024-01-01-mgr-nfs-mounting-windows

doc/mgr: document lack of MSWin NFS 4.x support

doc/mgr: document lack of MSWin NFS 4.x support

Document that no version of Microsoft Windows supports mounting an NFS
v4.x export natively, as of the time of this commit.

This commit has been made in response to an anonymous note left on
https://pad.ceph.com/p/Report_Documentation_Bugs.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55030 from zdover23/wip-doc-2023-12-31-radosgw-admin-remove-user-subuser-rewrite

doc/radosgw: edit "remove a subuser"

doc/radosgw: edit "remove a subuser"

Edit the English language in the section "Remove a Subuser" in
doc/radosgw/admin.rst. This commit is made in response to Matt
Benjamin's request for improvement of this section
(https://github.com/ceph/ceph/pull/55028#discussion_r1438599833).

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55023 from zdover23/wip-doc-2023-12-29-radosgw-admin-remove-user-subuser

doc/radosgw: edit sections

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit sections

Edit the following sections in doc/radosgw/admin.rst:

* Remove a User
* Remove a Subuser

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54982 from ronen-fr/wip-rf-be-error

osd/scrub: unhandled scrub backend errors should cause an abort

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54907 from ljflores/wip-balancer-incremental

osd: fix Incremental decode for new/old_pg_upmap_primary

Reviewed-by: Josh Salomon <josh.salomon@gmail.com>

Merge pull request #54627 from maydaycc/wip-63615

mgr: set MMgrBeacon high priority

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #54194 from ifed01/wip-ifed-refresh-perf-counters

os: introduce ObjectStore::refresh_perf_counters() method

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #54047 from lxbsz/wip-62081

osd: clean up the sparse-read related code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54644 from idryomov/wip-63607

librados: make querying pools for selfmanaged snaps reliable

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #55011 from zdover23/wip-doc-2023-12-27-radosgw-admin-create-a-subuser

doc/radosgw: admin.rst - edit "Create a Subuser"

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #54998 from ronen-fr/wip-rf-lmap-event

osd/scrub: remove unused PGScrubGotLocalMap message

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

Merge pull request #55016 from zdover23/wip-doc-2023-12-27-radosgw-user-info-suspend-enable

doc/radosgw: admin.rst - edit sections

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: admin.rst - edit sections

Edit the following sections in doc/radosgw/admin.rst:

- Get User Info
- Modify User Info
- User Suspend/Enable (I split this into two sections)

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54835 from rhcs-dashboard/directories-fixes

mgr/dashboard: small fixes to directories view

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

doc/radosgw: admin.rst - edit "Create a Subuser"

Edit the section "Create a Subuser" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54663 from Matan-B/wip-matanb-cot-pgmeta-attrs

tools/ceph_objectstore_tool: action_on_all_objects_in_pg to skip pgmeta

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54597 from sseshasa/wip-mclk-snaptrim-cost

osd: Tune snap trim item cost to reflect a PGs' average object size for mClock scheduler

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54518 from NitzanMordhai/wip-nitzan-cleanomap-rados-optional

Tools/rados: Improve Error Messaging for Object Name Resolution

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #54498 from Matan-B/wip-matanb-send-inc-map-refactor

osd/OSD: rewrite send_incremental_map()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55003 from zdover23/wip-doc-2023-12-26-radosgw-admin-create-a-user

doc/radosgw: admin.rst - edit "Create a User"

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

doc/radosgw: admin.rst - edit "Create a User"

Edit the section "Create a User" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54997 from zdover23/wip-doc-2023-12-24-radosgw-admin-main

doc/radosgw: edit admin.rst 1 of x

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit admin.rst 1 of x

Edit doc/radosgw/admin.rst to the end of the section "User Management".
Correct grammar (mostly awkward prepositions) and tidy up weird clauses.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

osd/scrub: remove unused PGScrubGotLocalMap message

The relevant event (IntLocalMapDone) is processed directly within the
scrubber, instead of being queued to the OSD events queue.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: unhandled scrub backend errors should cause an abort

... as we do not have any mechanism to handle them.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54801 from WangWNico/balancer_project

Improve Ceph Balancer Status Output

Merge pull request #53191 from rzarzynski/wip-all-kickoff-s-2

kickoff v19 squid

Merge pull request #54989 from zdover23/wip-doc-2023-12-22-radosgw-compression-supported-plugins

doc/radosgw: list supported plugins-compression.rst

doc/radosgw: list supported plugins-compression.rst

List supported compression plugins in doc/radosgw/compression.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54970 from zdover23/wip-doc-2023-12-20-radosgw-compression-grammar

doc/radosgw: edit compression.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit compression.rst

Improve the grammar and simplify the sentence structure of
doc/radosgw/compression.rst. This commit is made in anticipation of a
near-future commit that will list the compression algorithms available
to users of Ceph.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54918 from tchaikov/cmake-build-rockdb-with-cxxflags

cmake/modules/BuildRocksDB.cmake: inherit parent's CMAKE_CXX_FLAGS

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54936 from ronen-fr/wip-rf-pcnt3

osd/scrub: initial set of performance counters

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

osd/scrub: collect replica reservations performance counters

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

test/osd: log scrub performance counters in osd-scrub-test.sh

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: provide selected counters set to Scrubber & FSM

also - move most time keeping from the Scrubber to the FSM.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: add a basic set of performance counters

Add a labeled set of performance counters, with the labels selecting
one of four copies of the counters - one per each scrub level and
pool type combination.

Inside the Scrubber, the relevant set of counters is selected
when the scrub is initialized.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54828 from ronen-fr/wip-rf-dec-fixes-1

osd/scrub: don't block high-priority scrubs on local resources

Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #54023 from Matan-B/wip-matanb-objectstore-tool-superblock

tools/ceph_objectstore_tool: Support get/set/superblock

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54195 from ifed01/wip-ifed-silent-warns

core: silent a couple of compiler warnings on uninitialized vars.

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #52871 from NitzanMordhai/wip-nitzan-ceph-dencoder-extend-osd-type-available

ceph-dencoder: OSD - Add missing types

ceph-dencoder: osd - Add missing types

Currently, ceph-dencoder lacks certain osd types, preventing us from accurately checking the ceph corpus for encode-decode mismatches.
This pull request aims to address this issue by adding the missing types to ceph-dencoder.

To successfully incorporate these types into ceph-dencoder, we need to introduce the necessary `dump` and `generate_test_instances`
functions that was missing in some types. These functions are essential for proper encode and decode of the added types.

This PR will enhance the functionality of ceph-dencoder by including the missing types, enabling a comprehensive analysis of encode-decode consistency.
With the addition of these types, we can ensure the robustness and correctness of the ceph corpus.

This update will significantly contribute to improving the overall reliability and accuracy of ceph-dencoder.
It allows for a more comprehensive assessment of the encode-decode behavior, leading to enhanced data integrity and stability within the ceph ecosystem.

Fixes: https://tracker.ceph.com/issues/61788
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #54916 from tchaikov/cmake-boost-workaround

cmake: only apply workaround for boost < 1.84

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #54919 from tchaikov/cmake-boost-1.84

cmake: add 1.82 and 1.83 to known versions

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #53591 from EdwardVitor/fix-makecheck-warning

kv: resolve three compilation warnings

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #53524 from sseshasa/wip-fix-osd-op-queue-rndmize-62171

osd: Apply randomly selected scheduler type across all OSD shards

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53228 from rzarzynski/wip-bug-62645

 common/weighted_shuffle: don't feed std::discrete_distribution with all-zero weights

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #54953 from ljflores/wip-fix-18.2.1-doc

doc/releases: specify dashboard improvements

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases: specify dashboard improvements

...and address other nits

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #54935 from yuvalif/wip-yuval-return-lua-devel

build/rgw/lua: return lua-devel runtime dependency

reviewed-by: cbodley

Merge pull request #54710 from kalaspuffar/replace_jwt_with_pure_python

mgr/dashboard: Simplify authentication protocol

Reviewed-by: Fabian-Gruenbichler <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54897 from aclamk/wip-aclamk-bs-fix-esb-multichunk-alloc

os/bluestore: Fix assert() caused by ESB

Merge pull request #54609 from xxhdx1985126/wip-crimson-rollback-fixes

crimson/osd/pg_backend: fix rollback on whiteouts

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54960 from zdover23/wip-doc-2023-12-19-install-clone-source-submodule-update

doc/install: update "update submodules"

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/install: update "update submodules"

Remove misleading material that would give readers the wrong idea about
when stale submodules are present. This commit is made in response to
information given to me by Ilya Dryomov here: https://github.com/ceph/ceph/pull/54929#issuecomment-1859237986.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: small fixes to directories view

Increased the precedence for Directories
Expand the node by default and select them as well

Fixes: https://tracker.ceph.com/issues/63754
Signed-off-by: Nizamudeen A <nia@redhat.com>

crimson/osd/pg_backend: whiteout objects if they will have clones when
rolling back

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #54917 from tchaikov/cmake-cephfs-sans-libcephfs

cmake: differentiate WITH_CEPHFS from WITH_LIBCEPHFS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #54931 from athanatos/sjust/wip-crimson-2023-12-13-fixes

crimson: misc fixes

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54506 from ceph/wip-yuriw-release-18.2.1-main

doc: Add Reef 18.2.1 release notes

Merge pull request #52495 from adamemerson/wip-neorados-learning-from-experience

neorados: Learning from experience

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd/OSD: rewrite build_incremental_map_msg()

* use lambda to maintain size limits.
* avoid copying by std::move the bl into the message.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #53227 from Matan-B/wip-matanb-cleanup-stale-osdmap

osd/OSD: introduce trim_stale_maps

Reviewed-by: Samuel Just <sjust@redhat.com>

osd/OSD: send_incremental_map fix log message

from:
```
send_incremental_map 88 -> 88 to 0xab1c880 v2:172.21.5.151:6820/252144826
send_incremental_map 89 -> 89 to 0xab1a900 v2:172.21.5.151:6812/1097023676
```

to:
```
send_incremental_map epoch range: (75, 75] to osd.1 v2:172.21.5.151:6812/3634942927
send_incremental_map epoch range: (76, 77] to osd.1 v2:172.21.5.151:6812/3634942927
send_incremental_map epoch range: (78, 78] to osd.1 v2:172.21.5.151:6812/3634942927
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/OSD: send_incremental_map pass correct send_from

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/OSD: remove unused send_map()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/OSD: further cleanup send_incremental_map()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>