rgw/auth: Fix the return code returned by AuthStrategy,
in case of LocalEngine authentication, when the previous
auth engine is Not Applicable. The error code now returned
is the one returned by LocalEngine.

fixes: https://tracker.ceph.com/issues/61916

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #52838 from zztaki/fix-backport-resolve-issue

script/backport-resolve-issue: fix script to handle RC versions

Merge pull request #52490 from zdover23/wip-doc-2023-07-18-add-reef-18-2-0-release-notes

doc: Add Reef 18.2.0 release notes

doc: Add Reef 18.2.0 release notes

Add release notes for the first stable release of Ceph Reef (18.2.0).

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>
Co-authored-by: Laura Flores <lflores@redhat.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52736 from ljflores/wip-update-stale

.github/workflows: pin the stale bot to the latest version

Merge pull request #52494 from alimaredia/wip-rgw-crypt-keystone-fix

qa: fix keystone in rgw/crypt/barbican.yaml

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #52860 from Matan-B/wip-matanb-monstore-osd_snap-docs

tools/ceph_monstore_tool: add docs

doc/man/8/ceph-monstore-tool: add documentation

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52720 from debjyoti-pandit/api-docs-routes

mgr/dashboard: exposed the open api documentations to details card

Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: exposed the open api documentations to details card

Fixes: https://tracker.ceph.com/issues/62296
Signed-off-by: dpandit <dpandit@li-6f946d4c-287f-11b2-a85c-b0cb07ed936d.ibm.com>

Merge pull request #52395 from rhcs-dashboard/upgrade-ui-enhancements

mgr/dashboard: add logs and daemon version to upgrade UI

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #52647 from rhcs-dashboard/rbd-typing

mgr/dashboard: rbd type status

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52703 from zdover23/wip-doc-2023-07-31-documenting-ceph-linking-customs-2

doc/start: update linking conventions

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #52827 from zdover23/wip-doc-2023-08-05-troubleshooting-troubleshooting-mon-2-of-x

doc/rados: edit troubleshooting-mon.rst (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

script/backport-resolve-issue: fix script to handle RC versions

Fixes: https://tracker.ceph.com/issues/61746
Signed-off-by: Zhelong Zhao <zztaki@outlook.com>
Signed-off-by: Zhelong Zhao <50295684+zztaki@users.noreply.github.com>

Merge pull request #52785 from Matan-B/wip-matanb-crimson-add-asan-req

ceph.spec.in: add libasan to seastar requires

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/rados: edit troubleshooting-mon.rst (2 of x)

Edit the second part of
doc/rados/troubleshooting/troubleshooting-mon.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52824 from bluikko/bluikko-doc-radosgw-config-language

doc/radosgw: Improve language and formatting in config-ref.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/radosgw: Improve language and formatting in config-ref.rst

The "Topic persistency settings" section includes convoluted
and incomplete sentences so attempt to improve the
language. The section lacks context so add a link to the
relevant documentation about Notifications.

The "Garbage Collection settings" note includes a sentence
that looks incomplete so try to improve the language.

Don't capitalize "Ops" in "RGW Ops" especially when the
same sentence talks about "op queue" with lower case "o".

Also improve formatting: indentation of a note and wrap
several very long lines.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

ceph.spec.in: add sanitizers to seastar requires

```
CMake Error at src/seastar/CMakeLists.txt:782 (message):
  Sanitizers not found!
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52784 from Matan-B/wip-matanb-crimson-only-centos-9

qa/suites/crimson-rados: add centos9 to supported distros

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52778 from zdover23/wip-doc-2023-08-03-rados-operations-add-or-rm-mons-2-of-x

doc/rados: edit operations/add-or-rm-mons (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit operations/add-or-rm-mons (2 of x)

Edit the second part of doc/rados/operations/add-or-rm-mons.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52802 from dvanders/support_the_foundation

doc: Add a Ceph Foundation footer to docs.ceph.com

Merge pull request #52814 from cbodley/wip-doc-rgw-zonegroup-feature-default

doc/rgw: correct statement about default zone features

Merge pull request #52787 from rhcs-dashboard/fix-rgw-average-object-size

mgr/dashboard: fix average object size calculation in rgw overview dashboard

Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: update the upgrade layout by introducing new items

Shows Cluster Health
Shows the count of mgr daemons
Shows a table with all the daemon versions
Display the cluster logs

Fixes: https://tracker.ceph.com/issues/62016
Signed-off-by: Nizamudeen A <nia@redhat.com>

doc/rgw: correct statement about default zone features

this sentence wasn't updated when we added the Default column to the
table below

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: pin rgw/crypt suite temporarily to ubuntu_latest

barbican tests are only passing on ubunutu latest,
this commit allows us to have a green crypt suite.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

qa: change admin_host to admin_url in barbican.py

Signed-off-by: Ali Maredia <amaredia@redhat.com>

qa: bump keystone version in barbican.yaml

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #52248 from cbodley/wip-46062

rgw: fetch_remote_obj() preserves original part lengths for BlockDecrypt

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

mgr/dashboard: fix average object size calculation in rgw overview
dashboard

Fixes: https://tracker.ceph.com/issues/62318
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #52274 from cbodley/wip-61862

qa/rgw: specify cluster name in 'radosgw-admin gc process'

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #52800 from cbodley/wip-62312

doc/rgw: several response headers are supported

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc: Add a Ceph Foundation footer to docs.ceph.com

Add an info box to the footer of every docs page inviting visitors
to support the Ceph Foundation.

Signed-off-by: Dan van der Ster <dan.vanderster@clyso.com>

doc/rgw: several response headers are supported

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ReleaseNotes: document recovery of encrypted multipart objects

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add command to resync encrypted multipart objects

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRados::set_attrs() takes optional mtime

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: expose part id in RGWObjManifest iterator

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #51786 from cbodley/wip-61473

rgw/crypt: apply rgw_crypt_default_encryption_key by default

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

mgr/dashboard: make the logs component configurable and reusable

This will make the logs component easier to reuse for now in upgrade
component and later, wherever we want

Fixes: https://tracker.ceph.com/issues/62016
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52447 from liumiaomiaoIntel/qatzip_update

common/compressor: update the interfaces and data structures

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52634 from AliMasarweh/wip-alimasa-persistant-q-enhance

RGW: control the persistency of the notification

Merge pull request #52405 from rhcs-dashboard/rgw-overview-dashboard-graphs

mgr/dashboard: add graphs to rgw overview dashboard

Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52711 from cbodley/wip-62250

rgw: retry metadata cache notifications with INVALIDATE_OBJ

Reviewed-by: Adam Emerson <aemerson@redhat.com>

rgw: preserve RGW_ATTR_CRYPT_PARTS of already-replicated objects

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fetch_remote_obj() preserves original part lengths for BlockDecrypt

because multisite replicates multipart objects as a single part, we lose
information about the part sizes from the original manifest that is
necessary to correctly decrypt across those part boundaries

on replication, parse the part lengths out of the source object's
manifest, and store them in a separate RGW_ATTR_CRYPT_PARTS for use on
decryption

Fixes: https://tracker.ceph.com/issues/46062
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BlockDecrypt filter parses manifest parts before construction

users now call a static read_manifest_parts() function, and pass the
resulting vector into the BlockDecrypt constructor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52657 from rhcs-dashboard/fs-subvolumes

mgr/dashboard: cephfs subvolume list

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: cephfs subvolume list

Fixes: https://tracker.ceph.com/issues/62182
Signed-off-by: Nizamudeen A <nia@redhat.com>

qa/suites/crimson-rados: use crimson-supported-all-distro

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/distros: add crimson-supported-all-distro

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

add graphs to rgw overview dashboard

Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit a63026348cb4fcb287046c6dc5432d5709d5dff8)

Merge pull request #52757 from aisakaki/wip-retire-cp

crimson/os/seastore: retire_extent_addr clean up

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #52775 from dvanders/kplus1

 doc/rados/operations: Correct EC min_size recommendation to K+1 in erasure-code.rst

Merge pull request #52719 from cbodley/wip-rgw-get_ws_listing_op

rgw/swift: remove redundant moves of object name

Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>

Merge pull request #52714 from cbodley/wip-ec-clay-unit

ec: initialize lost_chunk in ErasureCodeClay

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: for EC we recommend K+1

Update the doc to match the reality in the code. I don't know where
the recommendation to have min_size = k+2 came from, but for awhile
now we've defaulted to K+1. See PR #8008.

Signed-off-by: Dan van der Ster <dan.vanderster@clyso.com>

Merge pull request #52007 from tcoldrick-bb/fix-invalid-json

rgw/rgw_admin.cc: Handle pg_ver and source_zone explicitly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #48732 into main

* refs/pull/48732/head:
doc: add MDS treatise on segments
pybind/mgr/dashboard: bump teuthology version
qa/tasks/vstart_runner: stop overriding _run_python
qa: stop overriding ceph_w prefix in vstart_runner
qa/tasks/vstart_runner: update teuthology helper tool paths
qa/tasks/vstart_runner: allow writing to command's stdin
vstart.sh: always add CEPH_CONF to vstart_environment.sh
qa: use stdin-killer for python3 command
qa: add killpoint testing for mds shutdown
qa: fix background exit condition
qa: add filesystem helper for setting transient config
qa: add helper for waiting for a rank to fail
mds: add incompat feature for minor log segments
mds: introduce ELid event to create/close log
mds: change EResetJournal to major segment boundary
mds: add killpoints for MDS shutdown
qa: add numerous subtree test
mds: track larger log events in perf dump
mds: add minor LogSegment boundaries
mds: obviate MDLog::start_entry
mds: retype to properly sized unsigned ints
mds: use unsigned type for event count
mds: use base Context class for generalization
mds: optimize segment lookup
mds: add stream dump for LogSegment
mds: handle conf changes in mdlog
mds: remove redundant comment
mds: remove unused method
mds: set a reasonable minimum number of segments
mds: sort configs

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #52754 into main

* refs/pull/52754/head:
test: explicitly link to ceph-common for some libcephfs tests

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #52761 from bluikko/patch-18

doc/rados/configuration: Avoid repeating "support" in msgr2.rst

Merge pull request #52709 from rishabh-d-dave/cephfs-test_snapshots

qa/cephfs: fix test_disallow_monitor_managed_snaps_for_fs_pools

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52762 from yuvalif/wip-yuval-62264

rgw/amqp: skip idleness tests since it needs to sleep longer than 30s

Reviewed-by: Casey Bodley <cbodley@redhat.com>

RGW: control the persistency of the notification
via adding expiry by number of retries and time
and controling the frequency of retries by sleep duration (in the options of vstart)

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

Merge pull request #52559 from adamemerson/wip-62097

build: Remove ceph-libboost* packages in install-deps

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rgw/amqp: skip idleness tests since it needs to sleep longer than 30s

current idle timeout is 30s, so, making the test sleep for 30s may not
be enough. setting sleep time to be longer, and skippign the test so it
won't take too long.

Fixes: https://tracker.ceph.com/issues/62264
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #52698 from rhcs-dashboard/fix-instance-id-exporter

exporter: fix instance_id labeld value for rgw in exporter

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #51960 from rzarzynski/wip-doc-encoding

doc: improve doc/dev/encoding.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Neha Ojha <nohja@redhat.com>

test: explicitly link to ceph-common for some libcephfs tests

Fixes: http://tracker.ceph.com/issues/57206
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge PR #52569 into main

* refs/pull/52569/head:
test/libcephfs: update delegation test assertions

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/rados/configuration: Avoid repeating "support" in msgr2.rst

Instead of saying "[...] support is not supported"
change the first "support" to "operation".

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

crimson/os/seastore: retire_extent_addr clean up

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #47573 from phlogistonjohn/jjm-cpatch-py

cpatch.py:  a python based cpatch script

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #52732 from cbodley/wip-qa-cephfs-flake8-random

qa/cephfs: redefinition of unused 'random' from line 7

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

.github/workflows: pin the stale bot to the latest version

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge PR #52675 into main

* refs/pull/52675/head:
test/TestOSDMap: don't use the deprecated std::random_shuffle method

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs: redefinition of unused 'random' from line 7

seeing this run-tox-qa failure about tasks/cephfs/test_client_recovery.py:

246/285 Test #264: run-tox-qa ................................***Failed   58.54 sec
Requirement already satisfied: tox in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (4.6.4)
Requirement already satisfied: cachetools>=5.3.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (5.3.1)
Requirement already satisfied: chardet>=5.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (5.1.0)
Requirement already satisfied: colorama>=0.4.6 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (0.4.6)
Requirement already satisfied: filelock>=3.12.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (3.12.2)
Requirement already satisfied: packaging>=23.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (23.1)
Requirement already satisfied: platformdirs>=3.8 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (3.10.0)
Requirement already satisfied: pluggy>=1.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (1.2.0)
Requirement already satisfied: pyproject-api>=1.5.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (1.5.3)
Requirement already satisfied: tomli>=2.0.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (2.0.1)
Requirement already satisfied: virtualenv>=20.23.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (20.24.2)
Requirement already satisfied: distlib<1,>=0.3.7 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from virtualenv>=20.23.1->tox) (0.3.7)
flake8: install_deps /home/jenkins-build/build/workspace/ceph-pull-requests/qa> python -I -m pip install flake8
flake8: freeze /home/jenkins-build/build/workspace/ceph-pull-requests/qa> python -m pip freeze --all
flake8: flake8==6.1.0,mccabe==0.7.0,pip==22.3.1,pycodestyle==2.11.0,pyflakes==3.1.0,setuptools==65.6.3,wheel==0.38.4
flake8: commands[0] /home/jenkins-build/build/workspace/ceph-pull-requests/qa> flake8 --select=F,E9 --exclude=venv,.tox
./tasks/cephfs/test_client_recovery.py:12:1: F811 redefinition of unused 'random' from line 7
flake8: exit 1 (3.72 seconds) /home/jenkins-build/build/workspace/ceph-pull-requests/qa> flake8 --select=F,E9 --exclude=venv,.tox pid=706315
flake8: FAIL ✖ in 15.42 seconds

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc: add MDS treatise on segments

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/dashboard: bump teuthology version

So stdin-killer and other utilities are installed in the bin directory.
vstart_runner.py now relies on their presence.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/vstart_runner: stop overriding _run_python

Now that the teuthology tools can be run in vstart_runner, there's no
reason to override this method.

Importantly, this enables the use of the new stdin-killer tool [1].

[1] https://github.com/ceph/teuthology/pull/1846

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: stop overriding ceph_w prefix in vstart_runner

These tools are now available in the $PATH so it's no longer necessary
to remove them.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/vstart_runner: update teuthology helper tool paths

With [1], these tools are now installed in the teuthology virtualenv.
Update the path in the command arguments so these tools can be run via
sudo.

[1] https://github.com/ceph/teuthology/pull/1846

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/vstart_runner: allow writing to command's stdin

There's no technical reason to disallow this. The original intent was to
avoid deadlocks but this possibility is already present when interacting
with a teuthology RemoteProcess. Avoiding it only for local processes
does not make sense.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

vstart.sh: always add CEPH_CONF to vstart_environment.sh

This makes sourcing this for e.g. vstart_runner.py actually useful.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use stdin-killer for python3 command

This relies on the new stdin-killer [1] teuthology helper that allows
interacting with the command's stdin.

[1] https://github.com/ceph/teuthology/pull/1846

Fixes: 8bb77ed9e18d134e2e9964e950cbcbfc65e1b97d
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add killpoint testing for mds shutdown

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix background exit condition

This change causes the program to exit gracefully when stdin is closed
rather than with a Python exception.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add filesystem helper for setting transient config

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add helper for waiting for a rank to fail

For killpoint testing.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add incompat feature for minor log segments

To prevent old MDS from joining a file system with the new log events.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: introduce ELid event to create/close log

Prior to this set of commits, the MDS would write the ESubtreeMap to the
journal, trim everything up to that segment, then finally force the
trimming of that last segment (`MDLog::trim(0)`). This is awkward in the
new code which preserves a major segment boundary at the beginning of
the journal during trimming. Instead of writing a special case for this
situation, it seems more natural to just use a new "lid" or "cap" event
to mark the beginning of the journal when no subtree map can yet be
written but we need sequence numbers to tie in other MDS tables.

Like ESegment, ELid doesn't actually contain any state. It's just a
marker for the beginning the log after rank deactivation or rank
creation. It can appear in the middle of the log if the shutdown
sequence is interrupted while writing the event but the MDS will skip it
during replay in that case.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: change EResetJournal to major segment boundary

When the MDS journal is wiped, EResetJournal is a major segment boundary
as it necessarily begins the journal.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add killpoints for MDS shutdown

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add numerous subtree test

When the ESubtreeMap is very large (~5k+ subtrees), the MDS will
end up logging only a few events (as bad as 1) per segment as the
subtree map dominates the segment size.

This test simply creates an artificially large subtree and confirms that
other file system activity completes in a timely manner. This is now
taking advantage of the minor segments which allows for a normal set of
events per log segment (and fewer subtree maps). The test fails on the
current main HEAD.

Historical note: when I first observed this abberant behavior, the
vstart cluster was actually using mds_debug_subtrees = True (the default
for every vstart cluster). This caused the MDS to write out the subtree
map (for debugging reasons) with every event. When testing the MDS with
large subtrees (distributed ephemeral pinning), this caused the MDS to
slow to a trickle of operations per second. Despite this unintentional
misconfiguration, the problem still exists but the number of auth
subtrees must be large for a particlar rank to replicate the behavior.

On main HEAD, the creation of 10k files (workload stage) takes ~110
seconds. On this branch, it takes ~30 seconds.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: track larger log events in perf dump

Fixes: https://tracker.ceph.com/issues/58550
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add minor LogSegment boundaries

This commit adds a new ESegment event type which can delineate
LogSegments. This event can be used as an alternative to the heavy
weight ESubtreeMap which can be very expensive to generate when the MDS
has a large subtree map.

Fixes: https://tracker.ceph.com/issues/58154
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: obviate MDLog::start_entry

The major problem here is that the MDLog::_start_entry method puts the
current event sequence number in the EMetaBlob of the event (if
present). Because of this, no other event can be submitted as this would
invalidate the event sequence. Instead, fixup the event sequence during
submission and simplify related logic that uses it during EMetaBlob
construction.

Secondarily, for the purposes of this commit series, _start_entry
introduced recursive locks when generating the ESubtreeMap within
MDLog::_segment_upkeep. So, this commit is a necessary cleanup.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>