Merge pull request #52988 from ceph/gh-add-to-project-fix-perm-issue

.github: fix add to dashboard failing

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #52410 from phlogistonjohn/jjm-issue61852-haproxy-ips

cephadm: fix issue with limited haproxy_hosts IPs list

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51899 from adk3798/cephadm-signed-keys

cephadm: support for CA signed keys

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #51170 from kotreshhr/doc-mds-cap-acquisition-throttle

doc: Fix doc for mds cap acquisition throttle

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #53018 from rhcs-dashboard/rm-cephfs-subvolume-group

mgr/dashboard: cephfs remove subvolume group

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #52416 into main

* refs/pull/52416/head:
qa/cephfs: check if re-installing this mpich fixes the issue

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: Fix doc for mds cap acquisition throttle

The config values picks the definition from the
description. Removed the duplicate description
in the doc.

Signed-off-by: Kotresh HR <khiremat@redhat.com>

mgr/dashboard: cephfs remove subvolume group

Fixes: https://tracker.ceph.com/issues/62452
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #47560 from pdvian/wip-pool-app

mon, qa: issue pool application warning even if pool is empty

Merge pull request #52674 from asm0deuz/add_networks_cli

mgr/cephadm: Add "networks" parameter to orch apply rgw

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Francesco Pantano <fpantano@redhat.com>
Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #52691 from adk3798/nvme-of-cleanup

mgr/cephadm: nvme-of follow up work

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52678 from phlogistonjohn/jjm-fix-test_valid_addr

python-common: fix valid_addr on python 3.11

Reviewed-by: Adam King <adking@redhat.com>

qa/cephadm: add test for ca signed keys

Test that bootstraps with a CA signed key using
the use_ca_signed_key cephadm override. Then follows
up by doing a check-host on each host which verifies
the cephadm mgr module can reach and authenticate with
the nodes using the new key setup.

This probably should really be a workunit, but
I didn't want to create a full new section for
this test and I needed a section that didn't
already run the cephadm task for every test. I could
see this being moved into some sort of
"test_special_deployment_scenarios" section in the future

Signed-off-by: Adam King <adking@redhat.com>

qa/cephadm: add ca signed key to cephadm task

To allow bootstrapping a cluster using a CA signed
key instead of the standard pubkey authentication.
Will allow explicit testing of this as we add support
for it

Signed-off-by: Adam King <adking@redhat.com>

doc/cephadm: document setting up CA signed keys in running cluster

Signed-off-by: Adam King <adking@redhat.com>

doc/cephadm: document bootstrapping with CA signed keys

Signed-off-by: Adam King <adking@redhat.com>

doc/cephadm: document how to pass self made SSH key pairs to bootstrap

This didn't seem to exist in the install section of
the cephadm docs. Wanted to add it in before adding
documentation for bootstrapping with CA signed keys.

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: add support for CA signed SSH keys setups

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #51785 from muahdib69/rsachere_iscsi_fixes

cephadm: Fix extra_container_args for iSCSI

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #52776 from ljflores/wip-read-balancer-documentation

doc: clarify upmap balancer documentation

Merge pull request #52919 from rhcs-dashboard/cluster-upgrade-progress

mgr/dashboard: cluster upgrade progress UI

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52991 from zdover23/wip-doc-2023-08-15-README-md-submodule

doc: improve submodule update command - README.md

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52989 from phlogistonjohn/jjm-installing-pkgs-cephadm

doc: do not recommend pulling cephadm from git

Reviewed-by: Adam King <adking@redhat.com>

doc: improve submodule update command - README.md

Improve the portion of README.md that directs the reader to update the
submodules. Previously, the text assumed that the reader knew that they
should cd into the ceph directory. Now the text is explicit about the
necessity of this action.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52898 from rhcs-dashboard/cephfs-subvolumegroup-edit

mgr/dashboard: Cephfs subvolumegroup edit

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52926 from kchheda3/wip_notif_s3user

rgw/notification: Store user info while creating notification topics

reviewed-by: yuvalif

Merge pull request #52857 from cyx1231st/wip-crimson-cleanup-osd-sharddispatcher

crimson/osd: cleanup and drop OSD::ShardDispatcher

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc: minor improvements of the cephadm section of the packages doc

* Don't call cephadm a script, it could be misleading
* Use the current release in the example for copy-pasters
* Improve grammar (I think)

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc: do not recommend pulling cephadm from git

First off, these instructions are wrong because the file has been
renamed to cephadm.py, and will not succeed as written. Amusingly, due
to curl's default behavior (and my lack of access to a time machine) the
initial command will seem to have succeeded until the user discovers
that HTML delivered as part of the body of a 404 response is not an
executable script.

Furthermore, in preparation for Reef we updated the documentation in the
cephadm subsection to use the results of a build, rather than source
files, because reef now contains a "compiled" cephadm. Currently, the
contents of the compilation is essentially the same-old cephadm script
but that will no longer be the case in the very near future (see
7c6a2cad96405616719b85c740aabd98558c93ac). We missed this general
section of the Ceph docs that repeats the instructions to copy a raw
python file out of the Ceph tree on GitHub.  Update the section to
contain a simplified version of the instruction found in
doc/cephadm/install.rst.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

.github: fix add to dashboard failing

```
Error: Request failed due to following response errors:
 - Could not resolve to a ProjectV2 with the number 2.
```

On further investigation, it looks like it could be a permission issue.
I followed the recommendation in their documentation to fix it: https://github.com/actions/add-to-project#creating-a-pat-and-adding-it-to-your-repository

Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: edit cephfs subvolumegroup

Fixes: https://tracker.ceph.com/issues/62370
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #52869 from rhcs-dashboard/fs-subvolumes-rm

mgr/dashboard: cephfs subvolume remove

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>

mgr/dashboard: cephfs subvolume remove

Fixes: https://tracker.ceph.com/issues/62349
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: cluster upgrade progress UI

Fixes: https://tracker.ceph.com/issues/62343
Signed-off-by: avanthakkar <avanjohn@gmail.com>
Co-Authored-By: Nizamudeen A <nia@redhat.com>

Merge PR #52547 into main

* refs/pull/52547/head:
qa: add test cases for vanilla ops commands
mds: dump locks when printing mutation ops
common/TrackedOp: support overriding the _dump method
mds: remove op field obsoleted by more usable "reqid"
mds: dump metareq_t instead of full op
mds: add lock type to formatter dump of SimpleLock
mds: mark print methods const
mds: drop MDRequestImpl::msg_lock
mds: lock TrackedOp when dumping
mds: avoid recursive locks dumping state
common/TrackedOp: fix race updating description with proper lock
common/Formatter: add support for dumping null
common/Formatter: refactor generating xml name

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #52577 into main

* refs/pull/52577/head:
mds: adjust cap acquisition throttles

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #52834 into main

* refs/pull/52834/head:
mount.fuse.ceph: strip the 'nofail' option from those that fuse will see

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #48705 from theanalyst/cmake-fmt-versionbump

cmake: ensure fmt lib is at least 8.1.1

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #52865 from cloudbehl/status-popover

mgr/dashboard: open popup with hover for cluster status

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52861 from rhcs-dashboard/fs-subvolumes-edit

mgr/dashboard: provide ability to edit cephfs subvolume

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #52966 from zdover23/wip-doc-2023-08-14-src-tools-rados

doc: update rados.cc

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51141 from xxhdx1985126/wip-seastore-clone

crimson/os/seastore: OP_CLONE in seastore

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

doc: update rados.cc

Clarify a line in ceph/src/tools/rados/rados.cc.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52963 from zdover23/wip-doc-2023-08-13-rados-troubleshooting-cpu-profiling

doc/troubleshooting: edit cpu-profiling.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/troubleshooting: edit cpu-profiling.rst

Improve the language in the "oprofile" section of the RADOS
Troubleshooting Guide.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52957 from zdover23/wip-doc-2023-08-12-rados-ops-monitoring-osd

doc/rados: update monitoring-osd-pg.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

mgr/dashboard: provide ability to edit cephfs subvolume

Fixes: https://tracker.ceph.com/issues/62347
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52645 from rhcs-dashboard/cephfs-vol-rm

mgr/dashboard: cephfs volume rm and rename

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: cephfs volume rm and rename

Fixes: https://tracker.ceph.com/issues/62408
Signed-off-by: avanthakkar <avanjohn@gmail.com>

Merge pull request #52932 from adk3798/cephadm-reef-latest-stable-release

cephadm: bump LATEST_STABLE_RELEASE to "reef"

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52510 from pritha-srivastava/wip-rgw-61916-test-failures

rgw/auth: Fix the return code returned by AuthStrategy,

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rados: update monitoring-osd-pg.rst

Ingest Anthony D'Atri's notes from
https://github.com/ceph/ceph/pull/50856#discussion_r1289532902
which should have been included earlier.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50856 from zdover23/wip-doc-2023-04-04-rados-operations-monitoring-osd-pg-question-1

doc/rados/ops: question about down and in OSDS

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52907 from cloudbehl/add-telemetry

mgr/dashboard: Add telemetry link in dashboard

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge pull request #52903 from cloudbehl/rgw-details

mgr/dashboard: RGW Details card cleanup

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/rados/ops: question about down and in OSDS

Clear up a confusing sentence that purports to explain OSDS that are
both "down" and "in".

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52945 from kotreshhr/doc-mgr-deadlock

doc: Add a note on possible deadlock on volume deletion

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc: Add a note on possible deadlock on volume deletion

Fixes: https://tracker.ceph.com/issues/62407
Signed-off-by: Kotresh HR <khiremat@redhat.com>

mgr/dashboard: Add telemetry link in dashboard

Fixes: https://tracker.ceph.com/issues/62380
Signed-off-by: cloudbehl <cloudbehl@gmail.com>

qa/cephfs: check if re-installing this mpich fixes the issue

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #52765 into main

* refs/pull/52765/head:
mgr/volumes: Fix pending_subvolume_deletions in volume info
qa: Add testcase for pending_subvolume_deletions count

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

cephadm: bump LATEST_STABLE_RELEASE to "reef"

Now that reef is out, we should tell users
that it's the latest stable release, rather
than quincy

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #52886 from rhcs-dashboard/cephfs-sbuvolumegroup-create

mgr/dashboard: create subvolume groups

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

rgw/notification: Store user info while creating notification topic

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>

Merge pull request #52178 from phlogistonjohn/jjm-cephadm-init-ctrs

cephadm: add basic init containers support

Reviewed-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: Add "networks" parameter to orch apply rgw

This parameter is available in specs but not available as a parameter.
Having it will ease its use in cephadm-adopt playbook in ceph-ansible.

fixes: https://tracker.ceph.com/issues/62185

Signed-off-by: Teoman ONAY <tonay@ibm.com>

Merge pull request #52876 from kchheda3/wip-noti-logs

rgw: enrich the failed debug log line for sync notification

reviwed-by: BBoozmen, khu102-bloomberg, yuvalif, thotz

mgr/dashboard: create subvolumegroup

Fixes: https://tracker.ceph.com/issues/62363
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

crimson/os/seastore: debug outputs

Signed-off-by: xuxuehan <xuxuehan@qianxin.com>

crimson/os/seastore/object_data_handler: take indirect offset into
account when reading object data

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: don't remove intermediate
mapping when remapping pins

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: remap indirect mappings

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: remove extents for physical lba mappings if necessary
when removing indirect lba mappings

Signed-off-by: xuxuehan <xuxuehan@qianxin.com>

crimson/os/seastore/lba_manager: decrease refcount of the physical lba
mappings when removing indirect lba mappings

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/onode_manager: erase onodes from the OnodeTree at the moment of erase_onode

Signed-off-by: xuxuehan <xuxuehan@qianxin.com>

crimson/os/seastore/object_data_handler: don't reserve lba space when truncating objects to zero size

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

test/crimson/seastore: add OP_CLONE unittest case

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/lba_manager: take lba entry indirection into account when getting lba mappings

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: implement OP_CLONE

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #52635 from rhcs-dashboard/list-subvolume-groups

mgr/dashboard: List subvolume groups

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #52702 from zdover23/wip-doc-2023-07-31-documenting-ceph-linking-customs

doc/start: update linking conventions

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #52715 from zdover23/wip-doc-2023-08-01-radosgw-admin-rate-limit-management

doc/radosgw: update rate limit management

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #52225 from yuvalif/wip-yuval-58412

rgw/lua: give tables fully qualified names

Merge pull request #52864 from adk3798/cephadm-debian-recommends

debian/control: add docker-ce as recommends for cephadm package

Reviewed-by: John Mulligan <jmulligan@redhat.com>

qa/cephadm: add test case to verify custom container init_containers

Add a 2nd custom container yaml to the test_extra_daemon_features.yaml
file that uses two init containers. Add verification that the init
containers (and primary) ran successfully.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc/cephadm: document init_containers params for custom containers

Add documentation covering the new init_containers parameter and the
parameters within each init container's configuration.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/cephadm/tests: add test_deploy_custom_container_with_init_ctrs

This adds a test case to explicitly check custom containers with init
containers.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/cephadm: add test_deploy_custom_container test case

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/cephadm: reformat a long input line

Use the parenthesized format with each import on a line for
readability.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/cephadm: add init_containers field to CephadmDaemonDeploySpec

This will permit the transfer of init_containers from a spec to the
structure that actually generates the deploy JSON.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/cephadm: add support for init_containers in deploy

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/cephadm: add init_containers field to exchange module

This will permit serializing init_container information for sending
to cephadm binary.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common: add init_containers field to CustomContainerSpec

This allows custom containers to run init containers before the
primary container starts.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common: reformat service_spec import line

Use the "black" style of multi-line import with parens.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common: add InitContainerSpec type

The InitContainerSpec type will be used to define explicit "init
containers" - containers that are expected to run and then exit
and are executed prior to running a primary container.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add a test case to cover generating init containers

Add a test case that covers the act of setting init_containers in
the deploy config on a custom container instance. This test executes
custom containers that both specify custom volume_mounts and not,
which are expected to "inherit" the volume_mounts of the primary
container's config.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: use an f-string for readability

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: support deploying with explicit init containers

Complete the deployment functions so that any custom containers defined
with "init_containers" will add said init containers to the commands
to be executed.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add get_deployment_init_containers

Add get_deployment_init_containers to extract any init containers from
the deploy configuration context. Currently, all init containers are to
be explicitly defined, not inferred from the service type.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add function to write init container commands

Add functions that write commands to clean up and execute init
containers. These init containers will be executed before the
main container as part of unit.run files (for now).

Signed-off-by: John Mulligan <jmulligan@redhat.com>