Merge PR #54825 into main

* refs/pull/54825/head:
mds: change priority of mds rss perf counter to useful

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #54607 from athanatos/sjust/wip-async-recovery-62338-release-note

PendingReleaseNotes: add release note for 62338

Reviewed-by: Neha Ojha <nojha@redhat.com>

PendingReleaseNotes: add release note for 62338

See https://tracker.ceph.com/issues/62338 and
2fc5486e.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #54024 from Matan-B/wip-matanb-snap_seq_inc

osd/osd_types: don't increment snap_seq on removal

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55043 from zdover23/wip-doc-2024-01-02-rados-troubleshooting-log-and-debug-title

doc/rados: improve subsection title in log-and-debug.rst

doc/rados: improve subsection title in log-and-debug.rst

Change the title of "Logging Settings" to "Logging and Debugging
Settings" in doc/rados/troubleshooting/log-and-debug.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54876 from ifed01/wip-ifed-fix-63795

test/store_test: fix deferred writing test cases

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>
Reviewed-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #55038 from dparmar18/minor-doc-update-disaster-recovery-experts

doc/cephfs: Update disaster-recovery-experts.rst to mention Slack

doc/cephfs: Update disaster-recovery-experts.rst to mention Slack

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #55031 from zdover23/wip-doc-2024-01-01-mgr-nfs-mounting-windows

doc/mgr: document lack of MSWin NFS 4.x support

doc/mgr: document lack of MSWin NFS 4.x support

Document that no version of Microsoft Windows supports mounting an NFS
v4.x export natively, as of the time of this commit.

This commit has been made in response to an anonymous note left on
https://pad.ceph.com/p/Report_Documentation_Bugs.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55030 from zdover23/wip-doc-2023-12-31-radosgw-admin-remove-user-subuser-rewrite

doc/radosgw: edit "remove a subuser"

doc/radosgw: edit "remove a subuser"

Edit the English language in the section "Remove a Subuser" in
doc/radosgw/admin.rst. This commit is made in response to Matt
Benjamin's request for improvement of this section
(https://github.com/ceph/ceph/pull/55028#discussion_r1438599833).

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55023 from zdover23/wip-doc-2023-12-29-radosgw-admin-remove-user-subuser

doc/radosgw: edit sections

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit sections

Edit the following sections in doc/radosgw/admin.rst:

* Remove a User
* Remove a Subuser

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54982 from ronen-fr/wip-rf-be-error

osd/scrub: unhandled scrub backend errors should cause an abort

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54907 from ljflores/wip-balancer-incremental

osd: fix Incremental decode for new/old_pg_upmap_primary

Reviewed-by: Josh Salomon <josh.salomon@gmail.com>

Merge pull request #54627 from maydaycc/wip-63615

mgr: set MMgrBeacon high priority

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #54194 from ifed01/wip-ifed-refresh-perf-counters

os: introduce ObjectStore::refresh_perf_counters() method

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #54047 from lxbsz/wip-62081

osd: clean up the sparse-read related code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54644 from idryomov/wip-63607

librados: make querying pools for selfmanaged snaps reliable

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #55011 from zdover23/wip-doc-2023-12-27-radosgw-admin-create-a-subuser

doc/radosgw: admin.rst - edit "Create a Subuser"

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #54998 from ronen-fr/wip-rf-lmap-event

osd/scrub: remove unused PGScrubGotLocalMap message

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

Merge pull request #55016 from zdover23/wip-doc-2023-12-27-radosgw-user-info-suspend-enable

doc/radosgw: admin.rst - edit sections

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: admin.rst - edit sections

Edit the following sections in doc/radosgw/admin.rst:

- Get User Info
- Modify User Info
- User Suspend/Enable (I split this into two sections)

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54835 from rhcs-dashboard/directories-fixes

mgr/dashboard: small fixes to directories view

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

doc/radosgw: admin.rst - edit "Create a Subuser"

Edit the section "Create a Subuser" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54663 from Matan-B/wip-matanb-cot-pgmeta-attrs

tools/ceph_objectstore_tool: action_on_all_objects_in_pg to skip pgmeta

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54597 from sseshasa/wip-mclk-snaptrim-cost

osd: Tune snap trim item cost to reflect a PGs' average object size for mClock scheduler

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54518 from NitzanMordhai/wip-nitzan-cleanomap-rados-optional

Tools/rados: Improve Error Messaging for Object Name Resolution

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #54498 from Matan-B/wip-matanb-send-inc-map-refactor

osd/OSD: rewrite send_incremental_map()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55003 from zdover23/wip-doc-2023-12-26-radosgw-admin-create-a-user

doc/radosgw: admin.rst - edit "Create a User"

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

doc/radosgw: admin.rst - edit "Create a User"

Edit the section "Create a User" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54997 from zdover23/wip-doc-2023-12-24-radosgw-admin-main

doc/radosgw: edit admin.rst 1 of x

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit admin.rst 1 of x

Edit doc/radosgw/admin.rst to the end of the section "User Management".
Correct grammar (mostly awkward prepositions) and tidy up weird clauses.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

osd/scrub: remove unused PGScrubGotLocalMap message

The relevant event (IntLocalMapDone) is processed directly within the
scrubber, instead of being queued to the OSD events queue.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: unhandled scrub backend errors should cause an abort

... as we do not have any mechanism to handle them.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54801 from WangWNico/balancer_project

Improve Ceph Balancer Status Output

Merge pull request #53191 from rzarzynski/wip-all-kickoff-s-2

kickoff v19 squid

Merge pull request #54989 from zdover23/wip-doc-2023-12-22-radosgw-compression-supported-plugins

doc/radosgw: list supported plugins-compression.rst

doc/radosgw: list supported plugins-compression.rst

List supported compression plugins in doc/radosgw/compression.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54970 from zdover23/wip-doc-2023-12-20-radosgw-compression-grammar

doc/radosgw: edit compression.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit compression.rst

Improve the grammar and simplify the sentence structure of
doc/radosgw/compression.rst. This commit is made in anticipation of a
near-future commit that will list the compression algorithms available
to users of Ceph.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54918 from tchaikov/cmake-build-rockdb-with-cxxflags

cmake/modules/BuildRocksDB.cmake: inherit parent's CMAKE_CXX_FLAGS

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54936 from ronen-fr/wip-rf-pcnt3

osd/scrub: initial set of performance counters

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

osd/scrub: collect replica reservations performance counters

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

test/osd: log scrub performance counters in osd-scrub-test.sh

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: provide selected counters set to Scrubber & FSM

also - move most time keeping from the Scrubber to the FSM.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: add a basic set of performance counters

Add a labeled set of performance counters, with the labels selecting
one of four copies of the counters - one per each scrub level and
pool type combination.

Inside the Scrubber, the relevant set of counters is selected
when the scrub is initialized.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54828 from ronen-fr/wip-rf-dec-fixes-1

osd/scrub: don't block high-priority scrubs on local resources

Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #54023 from Matan-B/wip-matanb-objectstore-tool-superblock

tools/ceph_objectstore_tool: Support get/set/superblock

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54195 from ifed01/wip-ifed-silent-warns

core: silent a couple of compiler warnings on uninitialized vars.

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #52871 from NitzanMordhai/wip-nitzan-ceph-dencoder-extend-osd-type-available

ceph-dencoder: OSD - Add missing types

ceph-dencoder: osd - Add missing types

Currently, ceph-dencoder lacks certain osd types, preventing us from accurately checking the ceph corpus for encode-decode mismatches.
This pull request aims to address this issue by adding the missing types to ceph-dencoder.

To successfully incorporate these types into ceph-dencoder, we need to introduce the necessary `dump` and `generate_test_instances`
functions that was missing in some types. These functions are essential for proper encode and decode of the added types.

This PR will enhance the functionality of ceph-dencoder by including the missing types, enabling a comprehensive analysis of encode-decode consistency.
With the addition of these types, we can ensure the robustness and correctness of the ceph corpus.

This update will significantly contribute to improving the overall reliability and accuracy of ceph-dencoder.
It allows for a more comprehensive assessment of the encode-decode behavior, leading to enhanced data integrity and stability within the ceph ecosystem.

Fixes: https://tracker.ceph.com/issues/61788
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #54916 from tchaikov/cmake-boost-workaround

cmake: only apply workaround for boost < 1.84

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #54919 from tchaikov/cmake-boost-1.84

cmake: add 1.82 and 1.83 to known versions

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #53591 from EdwardVitor/fix-makecheck-warning

kv: resolve three compilation warnings

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #53524 from sseshasa/wip-fix-osd-op-queue-rndmize-62171

osd: Apply randomly selected scheduler type across all OSD shards

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53228 from rzarzynski/wip-bug-62645

 common/weighted_shuffle: don't feed std::discrete_distribution with all-zero weights

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #54953 from ljflores/wip-fix-18.2.1-doc

doc/releases: specify dashboard improvements

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases: specify dashboard improvements

...and address other nits

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #54935 from yuvalif/wip-yuval-return-lua-devel

build/rgw/lua: return lua-devel runtime dependency

reviewed-by: cbodley

Merge pull request #54710 from kalaspuffar/replace_jwt_with_pure_python

mgr/dashboard: Simplify authentication protocol

Reviewed-by: Fabian-Gruenbichler <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54897 from aclamk/wip-aclamk-bs-fix-esb-multichunk-alloc

os/bluestore: Fix assert() caused by ESB

Merge pull request #54609 from xxhdx1985126/wip-crimson-rollback-fixes

crimson/osd/pg_backend: fix rollback on whiteouts

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54960 from zdover23/wip-doc-2023-12-19-install-clone-source-submodule-update

doc/install: update "update submodules"

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/install: update "update submodules"

Remove misleading material that would give readers the wrong idea about
when stale submodules are present. This commit is made in response to
information given to me by Ilya Dryomov here: https://github.com/ceph/ceph/pull/54929#issuecomment-1859237986.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: small fixes to directories view

Increased the precedence for Directories
Expand the node by default and select them as well

Fixes: https://tracker.ceph.com/issues/63754
Signed-off-by: Nizamudeen A <nia@redhat.com>

crimson/osd/pg_backend: whiteout objects if they will have clones when
rolling back

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #54917 from tchaikov/cmake-cephfs-sans-libcephfs

cmake: differentiate WITH_CEPHFS from WITH_LIBCEPHFS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #54931 from athanatos/sjust/wip-crimson-2023-12-13-fixes

crimson: misc fixes

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54506 from ceph/wip-yuriw-release-18.2.1-main

doc: Add Reef 18.2.1 release notes

Merge pull request #52495 from adamemerson/wip-neorados-learning-from-experience

neorados: Learning from experience

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd/OSD: rewrite build_incremental_map_msg()

* use lambda to maintain size limits.
* avoid copying by std::move the bl into the message.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #53227 from Matan-B/wip-matanb-cleanup-stale-osdmap

osd/OSD: introduce trim_stale_maps

Reviewed-by: Samuel Just <sjust@redhat.com>

osd/OSD: send_incremental_map fix log message

from:
```
send_incremental_map 88 -> 88 to 0xab1c880 v2:172.21.5.151:6820/252144826
send_incremental_map 89 -> 89 to 0xab1a900 v2:172.21.5.151:6812/1097023676
```

to:
```
send_incremental_map epoch range: (75, 75] to osd.1 v2:172.21.5.151:6812/3634942927
send_incremental_map epoch range: (76, 77] to osd.1 v2:172.21.5.151:6812/3634942927
send_incremental_map epoch range: (78, 78] to osd.1 v2:172.21.5.151:6812/3634942927
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/OSD: send_incremental_map pass correct send_from

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/OSD: remove unused send_map()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/OSD: further cleanup send_incremental_map()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/OSD: optimize send_incremental_map

Previosuly, if `since` was earlier than the oldest_map,
only the lastest full osdmap would be sent.
However, we could actually send inc/full maps from oldest_map
up to the current epoch.

Moreover, build_incremental_map_msg is suited to handle this case as well.
if `since` is earlier than `cluster_osdmap_trim_lower_bound` it
will be adjusted accordingly.
At any point, the cluster_osdmap_trim_lower_bound is later (or equal) to
the superblock's oldest_map. See: past_intervals.rst (OSDSuperblock::maps)
Hence, if since < oldest_map, and oldest_map <= clutser_lower_bound
then since < cluster_lower_bound.

Note: oldest_map stands as a mark to indicate up until which epoch
      the current OSD trimmed its osdmaps. See OSD::trim_maps().
      Meaning, the OSDMaps in the range of [oldest_map, trim_lower_bound]
      may be at hand to be shared. Nevertheless, we should actually use
      the cluster_trim_lower_bound to set the correct range.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/OSD: cleanup send_incremental_map

See: 9fba69a11aa940ed36339bb24b05cb92165db516

build_incremental_map_msg() no longer returns NULL on failures.
while(!m) if-case is removed.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/tasks/ceph_manager: thrash - add trim stale osdmaps

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/OSD: introduce trim_stale_maps

```
  /**
   * trim_stale_maps
   *
   * trim_maps had a possible (rare) leak which resulted in stale osdmaps.
   * This method will cleanup any existing osdmaps from the store
   * with an epoch earlier than the superblock's oldest_map epoch.
   * See: https://tracker.ceph.com/issues/61962
   */
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

cmake: differentiate WITH_CEPHFS from WITH_LIBCEPHFS

WITH_CEPHFS is used for enabling the server side of cephfs, i.e., mds,
while WITH_LIBCEPHFS is for the client side of it, namely libcephfs
and its python bindings. since we have these two different options,
in theory, user is allowed to build with WITH_CEPHFS=ON and
WITH_LIBCEPHFS=OFF, and in that case, `vstart` should not depend on
the non-existent target of cephfs and cython_cephfs. the same applies to
the build of cython_cephfs. this is an unusual combination, but
it is a valid one.

in this change, we

* build the python binding only if WITH_LIBCEPHFS is ON
* add cephfs and cython_cephfs as dependencies of vstart only if
  WITH_LIBCEPHFS is ON

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cmake: add deps to vstart and vstart-base with loop

so we don't have to repeat the settings enabling a given target
when adding the deps to vstart and vstart-base

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #54929 from dann1/main

doc: Improve documentation regarding OpenNebula integration with ceph

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54937 from gruenich/feature/cmake-drop-redundant-policies

cmake: Do not set CMake policy to new that are set anyway

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

docs: Add information about OpenNebula integration

- Exclude doc build output from git
- Fix missing doc build dependency
- Also includes some involuntary automatically persistent linting by vscode

Co-authored-by: Ilya Dryomov <idryomov@redhat.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Zac Dover <zac.dover@proton.me>
Signed-off-by: Daniel Clavijo <dclavijo@opennebula.io>

cmake: Do not set CMake policy to new that are set anyway

CMP0097 and below are all implicitly set to new because
the minimum required CMake version is 3.16 and these
policies are older.

Signed-off-by: Christoph Grüninger <foss@grueninger.de>

Merge pull request #52474 from vedanshbhartia/coverity_1515850

rgw: Fix potential null dereference in rgw/driver/dbstore/sqlite/statement.cc

reviewed-by: thotz, yuvalif

Merge pull request #52276 from vedanshbhartia/coverity_1523402

rgw: Add a wrapper for librados::AioCompletion to prevent memory leaks

reviewed-by: cbodley, yuvalif

Merge pull request #53902 from vedanshbhartia/coverity_except_main

rgw: Add coverity annotations for uncaught exceptions in standalone binaries

reviewed-by: yuvalif

build/rgw/lua: return lua-devel runtime dependency

reverting: b0a77a53ab86b9b24a2cfb46049957d40a8c80aa
after sepia lab issue is resolved

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #54547 from idryomov/wip-53897

librbd: don't report HOLE_UPDATED when diffing against a hole

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #54569 from AliMasarweh/wip-alimasa-multi-pubsub-etag

RGW: pubsub publish commit with etag populated

revieweed-by: cbodley, yuvalif

Merge pull request #52354 from bluikko/patch-13

doc/radosgw/admin.rst: use underscores in config var names

Merge pull request #54888 from cbodley/wip-rgw-multisite-forward-empty-buffer

rgw/multisite: forwarded requests always pass a bufferlist

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #54739 from cbodley/wip-qa-rgw-upgrade-distros

qa/rgw/upgrade: stop testing upgrade from pacific

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

crimson/.../client_request_common: skip non-existent oids in recover_missings

See comment.

Introduced: 38cc750f
Fixes: https://tracker.ceph.com/issues/63821
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../client_request_common: simplify recover_missings

- Modify ClientRequest::snaps_need_to_recover() to return all
  relevant snaps including the operation target.
- Update ClientRequest::snaps_need_to_recover() and
  CommonClientRequest::recover_missings to use set::set
  for snaps.
- Remove special handling for soid from
  CommonClientRequest::recover_missings
- Simplify CommonClientRequest::recover_missings.

Signed-off-by: Samuel Just <sjust@redhat.com>