Merge pull request #53737 from adk3798/rm-host-crushmap

mgr/cephadm: support for removing host entry from crush map during host removal

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52492 from phlogistonjohn/jjm-cephadm-jinja

Add templating, via jinja2, to cephadm

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #53842 from BBoozmen/oozmen_sts_max_duration

RGW/STS: honor configured limits when updating max session duration

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #53997 from Inode1/rgw-subuser-user-policy-check

rgw: add subuser to user policy condition check

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #54050 from shreyanshjain7174/fix_62646

RadosGW: RGW Swift info not providing max_deletes_per_request in bulk_delete

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54174 from cbodley/wip-63306

rgw: fetch_remote_obj() uses uncompressed size for encrypted objects

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #54213 from cbodley/wip-63333

rgw-check-policy: remove check for nonempty tenant

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #50680 from linuxbox2/wip-notify-lctrans

rgwlc: dispatch S3 notifications on transition and mpu abort

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #54307 from rkachach/fix_issue_63326

mgr/dashboard: fix secure_monitoring_stack check

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54352 from zdover23/wip-doc-2023-11-06-rados-troubleshooting-mon-common-issues-formatting

doc/rados: format Q&A list in tshooting-mon.rst

Merge pull request #54222 from nbalacha/deep_copy_logs

librbd: additional logs for deep-copy debugging

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

librbd: additional logs for debugging

Added image and object information to the deep_copy class constructors
debugging easier.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #54351 from cyx1231st/wip-seastore-cleanup-is-stable-cleanup-only

crimson/os/seastore: cleanup, be explicit about stable and stable_written

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #54347 from yuvalif/wip-yuval-coverity-y2k38

rgw: adding y2k38 annotations to supress coverity issues

reviewed-by: vedanshbhartia, idryomov

Merge pull request #54286 from jnsaff/patch-1

doc: Update mount-using-fuse.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #54336 from adk3798/cephadm-dashboard-e2e-fedora-link

mgr/dashboard: update fedora link for dashboard-cephadm-e2e test

Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/rados: format Q&A list in tshooting-mon.rst

Bold the questions in a Q&A list in the "Most Common Monitor Issues"
section of doc/rados/troubleshooting/troubleshooting-mon.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore: cleanup, be explicit about stable and stable_written

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #54325 from cyx1231st/wip-seastore-fix-seastore-pr-54036

crimson/os/seastore/transaction_manager: add the missing UNIT_TESTS_BUILT guard

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #54344 from zdover23/wip-doc-2023-11-05-rados-troubleshooting-mon-clock-skews-edit

doc/rados: edit t-mon.rst text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: adding y2k38 annotations to supress coverity issues

the supression comments must be directly above the line
where the issue is.

this is a followup on these commits:
edfb5b9049ee897779863d780bfbaa130a7fbc40
aca2668c88a048ab556c53411dcf2ccd43099d50

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

doc/rados: edit t-mon.rst text

Clarify the text in the "Clock Skew" section of
doc/rados/troubleshooting/troubleshooting-mon.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54343 from zdover23/wip-doc-2023-11-05-rados-troubleshooting-troubleshooting-mon-clock-skews-formatting

doc/rados: format Q&A list in t-mon.rst

doc/rados: format Q&A list in t-mon.rst

Format a Q&A list in doc/rados/troubleshooting/troubleshooting.rst, in
the "Clock Skews" section.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54104 from phlogistonjohn/jjm-cephadm-move-extract

cephadm: move extract_uid_gid function

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54056 from rkachach/fix_issue_ci_for_rook

mgr/rook: adding e2e testing for rook orchestrator

reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54339 from aclamk/fix-aclamk-bs-reshard-doc-v2

doc/rados/configuration/bluestore-config-ref: Fix lowcase typo

Merge pull request #51784 from muahdib69/rsachere_cache_ratios

osd: make _set_cache_sizes ratio aware of cache_kv_onode_ratio

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

cephadm: switch to using template files

Switch off of the embedded template strings to using the recently
added template files.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: copy template strings into template files

Copy the current template strings into files under the `templates` dir.
Add a enum for holding the names of known template files.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: expand the templating module

Expand the templating module so that templates can be source from the
python package. Add (more) convenient to use methods.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: workaround jinja2 package loader issue

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: update agent unit file func to use jinja template

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: update get_unit_file to use jinja template

The somewhat complex string assembly of the main systemd unit file
for cephadm services can benefit from using a standard templating
approach.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add wrapper func for jinja templating

Add a simple wrapper function for templating from a string to a string.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add new requirements file to tox.ini envs

Add `-rzipapp-reqs.txt` to the unit tests and mypy environments in
tox.ini, enabling the use of dependencies outside the stdlib.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add zipapp-reqs.txt, mirroring cephadm's zipapp dependencies

Add Jinja2 and MarkupSafe dependencies to a requirements.txt style file.
This file tracks the dependencies needed to run the cephadm libs
in the unit test framework. The actual dependencies that get added
to the ziapp are managed by build.py but mirrored here.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add more thorough test coverage to unit file generation

Add tests that check the generation of the standard systemd unit
for cephadm services. This test ensures that non trivial changes
to the content of these files are noticed.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add additional test cases for unit file contents

Asserts more about the unit files.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc/rados/configuration/bluestore-config-ref: Fix lowcase typo

Letter "O" got lowcased to "o".
Reverted that.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

cephadm: move unit file related tests to dedicated file

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add tests for build.py script

Add tests that cover the four main distros that ceph is built on (in
the ceph infra). These tests should not be run by automation as they
are slow and have special requirements like a working podman.
Instead, these are provided to be run by a dev when build.py is updated.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: update the build.py script to work on multiple distros

Unfortunately, a single simple call to pip does not work on all the
distributions that ceph is built on. In particular, Ubuntu 20.04 and
Ubuntu 22.04 come with pip versions that can not correctly handle
disabling wheels and installing Jinja2 (it tries to use the markupsafe
dependency before it is installed). This can be worked around by using a
virtual env and updating pip before proceeding. However, this is not
enough because CentOS/RHEL 8 uses python 3.6 and there is no version of
pip that supports 3.6 that we can update to that is new enough to  fix
the issue with disabling wheels. The workaround in this case is to
install each dependency one at a time through multiple calls to pip.

Because of this extra complexity is it simpler to eschew the use of a
requirements.txt file in build.py entirely. Thus the zipapp is built
using build.py only. Requirements files for cephadm are for setting up
the tox environments *only*.

For completeness a new option is added that gives the caller control
over when build.py uses a virtualenv or not. Thus the build.py script
requires at least one of: a working pip that handles disabling wheels;
or, a virtualenv (venv) and the ability to update to a working version
of pip. If the list of distros ceph supports (and the python versions
they use) ever becomes simpler/newer some of this complexity could be
removed from the build.py script.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: disable wheels and C compilers when building cephadm zipapp

We can not rely on any particular python version (py 3.6+ is supported)
and can not assume any particular architecture. So using wheels
based on the build system is pointless. Installing binary .so files
compiled from C/C++ similarly so. Attempt to block the behaviors
when adding dependencies to the zipapp.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54077 from adk3798/event-update-timestamp

mgr/cephadm: update timestamp on repeat daemon/service events

Reviewed-by: John Mulligan <jmulligan@redhat.com>

mgr/dashboard: update fedora link for dashboard-cephadm-e2e test

The old link points to a directory that now only contains a
README saying

---

ATTENTION
======================================
The contents of this directory have been moved to our archives available at:

http://archives.fedoraproject.org/pub/archive/fedora/

If you are having troubles finding something there please stop by #fedora-admin on irc.freenode.net

---

so I think we either need to start using a newer fedora or point
to the archive link. This commit is currently doing the latter.

Signed-off-by: Adam King <adking@redhat.com>

qa/cephadm: test --rm-crush-entry host rm flag in host drain test

Signed-off-by: Adam King <adking@redhat.com>

qa/cephadm: add teuthology test for host draining

This was a gap in our testing in general, but I'm
adding it here right now specifically to use it
to test the "--rm-crush-entry" flag in a follow
up commit

Signed-off-by: Adam King <adking@redhat.com>

doc/cephadm: document --rm-crush-entry host rm flag

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: add --rm-crush-entry flag to host removal

This will tell cephadm to try and remove the
crush bucket for the host at the end of the host
removal process. If this fails, we still consider the
host as having been successfully remove from
cephadm's POV, but the user will get back an error
message telling them we failed to remove the
host from the crush map

Fixes: https://tracker.ceph.com/issues/63031
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #54331 from aclamk/fix-aclamk-bs-reshard-doc

Fixing example of BlueStore resharding.

Merge pull request #53747 from rkachach/fix_issue_12858

mgr/rook: Adding support to automatically discover storage classes on Rook cluster

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@ibm.com>

doc/rados/configuration/bluestore-config-ref: Fix lowcase typo

Letters "L P" got lowcased to "l p".
Reverted that.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

rgwlc: fix notification test imports

Found by Yuval Lifshitz review.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: send version_id with notifications

Also fix return value from handle_multipart_expiration bucket-entry
lambda.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: treat persistent notify reservation as failure as fatal

(Because that's consistent with what we have been doing for expiration.)

This is possibly a controversial choice, and it might be better to allow
setting a site policy, in future.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: fix remove expired for non-versioned case

Plus, reformat some adjacent code.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: split versioned and non-versioned transition notifications

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: constify notification event_type

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: remove extra handles to sal bucket and object

I missed the addition of those to lc_op_ctx.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: log error on failure to publish_commit

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: dispatch S3 notifications on transition and mpu abort

Fixes: https://tracker.ceph.com/issues/58641
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cephadm: update extract_uid_gid function annotations & format

Update the type annotations of extract_uid_gid to the type annotations
form used in more recent python versions. Reformat the function.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move extract_uid_gid func to container_types module

While extract_uid_gid isn't a perfect fit for container_types it is a
fairly fundamental function for working with containers in cephadm and
doesn't require anything beyond types in containers_types and that
module's existing imports.  Moving extract_uid_gid should allow us to
more easily move other functions in the future.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54081 from phlogistonjohn/jjm-cephadm-podman-props

cephadm: move some podman specific logic to Podman methods

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53987 from diabolocom/main

cephadm: add option to install systemd unit only

Reviewed-by: Adam King <adking@redhat.com>

mgr/dashboard: fix secure_monitoring_stack check
Fixes: https://tracker.ceph.com/issues/63326
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #54274 from aliakseimakarau/patch-1

doc: Update dynamicresharding.rst

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>

doc: Update dynamicresharding.rst

"3, After resharding completed:" should be
"3. After resharding completed:"

Signed-off-by: Aliaksei Makarau <aliaksei.makarau@ibm.com>

mgr/rook: adding behave as testing python dependency

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #53129 from rishabh-d-dave/xfstests-dev-deps

qa/cephfs: update dependencies for xfstests-dev

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #48508 from pkalever/rbd-tests

qa/workunits/rbd: merge journal and snapshot test scripts

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50615 from batrick/i59119

mds: respawn standby-replay if read_pos is <= expire_pos

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53611 from Patty8122/adding-useful-print

mds: Include more information in print statement in MDSRank.cc

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53126 from tengjie5/origin/tmp2

cephfs-shell: getxattr fail while the xattr's length > 256

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

mgr/rook: adding e2e version based on minikube

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

crimson/os/seastore/transaction_manager: add the missing UNIT_TESTS_BUILT guard

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #54036 from xxhdx1985126/wip-seastore-clone-remap-ut

crimson/os/seastore: add unittests for transaction manager's clone/remap/read_pin and fix bugs

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #53132 from benhanokh/maintain_prefix_itr_2

osd/SnapMapper:maintain the prefix_itr between calls to SnapMapper::get_next_objects_to_trim()

Merge pull request #54322 from bluikko/patch-22

doc/cephadm/services: remove excess rendered indentation in osd.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephadm/services: remove excess rendered indentation in osd.rst

Start bash command blocks at the left margin, removing
excessive padding/indentation that would render the
block too much towards the right.

At the same time ident the source consistently:
- Two spaces for command blocks and output blocks.
- Four spaces for notes, code blocks.

There seems to be no uniform style for this, sometimes
commands are indented with three spaces but it would
seem two spaces is common. In the end it all renders
the same I guess.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #54048 from rhcs-dashboard/rgw-bucket-policy

mgr/dashboard: get object bucket policies for a bucket

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #54303 from zdover23/wip-doc-2023-11-02-cephadm-troubleshooting-2-of-x

doc/cephadm: edit troubleshooting.rst (2 of x)

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54309 from linuxbox2/wip-fix-lockguard

rgwfile: fix lock_guard decl

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #47121 into main

* refs/pull/47121/head:
mds: ensure next replay is queued on req drop

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #53883 into main

* refs/pull/53883/head:
mon: fix mds metadata lost in one case.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #53636 into main

* refs/pull/53636/head:
mds: do not simplify fragset

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd/SnapMapper: maintain the prefix_itr between calls to SnapMapper::get_next_objects_to_trim()
Maintain the prefix_itr between calls to SnapMapper::get_next_objects_to_trim() to prevent searching depleted prefixes.
We got 8 distinct hash prefixes used for searching objects owned by a given PG.
On each call to SnapMapper::get_next_objects_to_trim() we start from the first prefix even after all objects mapped to it were depleted.
This means that we will be searching for 1 non-existing prefix after the first prefix was depleted, 2 after the first two prefixes were depleted... and so on until we will search 7 non-existing prefixes after the first 7 prefixes were depleted.

This is a performance improvement PR only!
It maintains the existing behavior and does not try to fix/change any of the TRIM logic.
I added an extra step after the last object is trimmed doing a full scan of the DB and only if no object was found it will return ENOENT.
This should make the new code no-worse than existing code which returns ENOENT after a full scan found no object.
It should not impact performance in real life snaps as it should only happen once per-snap.

added snap-mapper tests to rados-test-suite
disabled osd_debug_trim_objects when running (SnapMapperTest, prefix_itr) to prevent asserts(as this code does illegal inserts into DELETED snaps)
Code beautifing

Disabled the assert as there is a corner case when we retrieve the last valid object/s in a snap
The prefix_itr is advanced past the last valid value (as we completed a full scan)
If the OSD will call get_next_objects_to_trim() before the retrieved object/s was processed and removed from the SnapMapper DB it won't be found by the next call (as the prefix_itr is invalid).
The object will be found in the second-pass which will seems as if it was added after the trim was started (which is illegal) and will trigger an ASSERT

Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com>

Merge pull request #54299 from ionutbalutoiu/fix-rgw-cloud-sync-multi-tenancy

rgw: fix cloud-sync multi-tenancy scenario

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgwfile: fix lock_guard decl

Reported by Leonid Usov <leonid.usov@ibm.com>.

Fixes: https://tracker.ceph.com/issues/63409
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #53828 from smanjara/wip-sync-error-trim

rgw/multisite: fix sync_error_trim command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/cephadm: edit troubleshooting.rst (2 of x)

Edit doc/cephadm/troubleshooting.rst. This commit and the PR of which it
is a part was raised in response to https://github.com/ceph/ceph/pull/53976.

Co-authored-by: John Mulligan <jmulligan@redhat.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

qa/workunits/rbd: merge journal and snapshot test scripts

The idea is to avoid the maintenance of duplicate code in both the journal
and snapshot test scripts.

Usage:
   RBD_MIRROR_MODE=journal rbd_mirror.sh

Use environment variable RBD_MIRROR_MODE to set the mode
Available modes: snapshot | journal

Fixes: https://tracker.ceph.com/issues/54312
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

Merge pull request #49486 from ifed01/wip-ifed-fix-rados-pybind

pybind/rados: fix missed changes for PEP484 style type annotations

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

doc/cephfs: edit mount-using-fuse.rst

Fixes missing dash that lead to unnecessary debugging.

Signed-off-by: Jaanus Torp <jaanus@saun.ee>

Merge pull request #53843 from ronen-fr/wip-rf-1b1

osd/scrub: reserve replicas in OSD order, one by one

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: get object bucket policies for a bucket

Getting the bucket details will also fetch the bucket policy if its set.

Fixes: https://tracker.ceph.com/issues/63221
Signed-off-by: Nizamudeen A <nia@redhat.com>

crimson/os/seastore/lba_manager: rename clone_extent to clone_mapping

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/lba_manager: hide lba mapping ref count update away
from users of TransactionManager

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

test/crimson/seastore: add unittests for clone/remap/read_pin

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>