mds/Server: mark a cap acquisition throttle event in the request

Fixes: https://tracker.ceph.com/issues/59067
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #53035 from zdover23/wip-doc-2023-08-17-README-md-contributing-code

doc: edit README.md - contributing code

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53023 from xxhdx1985126/wip-seastore-daggling-ref

crimson/os/seastore: fix daggling reference of oid in SeaStore::Shard::stat()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc: edit README.md - contributing code

Edit the "Contributing Code" section of the "ceph/README.md" file.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53022 from rhcs-dashboard/upgrade-start-enhancement

mgr/dashboard: support cluster upgrade even if the check for upgrade fails

Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53030 from petrutlucian94/rbd_tests

test/librbd/fsx: fix wnbd check

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd/fsx: fix wnbd check

When using WNBD, the IO size must be a multiple of the sector size
(WNBD only supports 512 at the moment).

We're currently checking the "wnbd_operations" variable, however
it's only defined on Windows, leading to a compilation failure
on other platforms.

We'll switch back to a separate boolean variable called "wnbd_disk".

Fixes: https://tracker.ceph.com/issues/62475
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

mgr/dashboard: support cluster upgrade even if the check for upgrade fails

Fixes: https://tracker.ceph.com/issues/62457
Signed-off-by: avanthakkar <avanjohn@gmail.com>

Merge pull request #51561 from rhcs-dashboard/hosts-no-data

mgr/dashboard: add not avaialable with unpresent host facts

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51227 from jsoref/spelling-dashboard

dashboard: Fix spelling

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #51126 from rhcs-dashboard/jeje-log

mgr/prometheus: remove jeje log

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #53021 from rhcs-dashboard/buckets-e2e-fix

mgr/dashboard: fix rgw buckets e2e

Reviewed-by: cloudbehl <NOT@FOUND>

Merge pull request #52740 from adk3798/ingress-vip-networks

mgr/cephadm: pick correct IPs for ingress service based on VIP

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #51881 from adk3798/logrotate-tcmu-runner

cephadm: add tcmu-runner to logrotate config

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52651 from galsalomon66/trino_tpcds_qe_fixes

rgw/s3select: s3select fixes related to Trino/TPCDS benchmark and QE tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50141 from petrutlucian94/rbd_tests

Enable librbd tests on Windows

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52988 from ceph/gh-add-to-project-fix-perm-issue

.github: fix add to dashboard failing

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #52410 from phlogistonjohn/jjm-issue61852-haproxy-ips

cephadm: fix issue with limited haproxy_hosts IPs list

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51899 from adk3798/cephadm-signed-keys

cephadm: support for CA signed keys

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #51170 from kotreshhr/doc-mds-cap-acquisition-throttle

doc: Fix doc for mds cap acquisition throttle

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os/seastore: fix daggling reference of oid in
SeaStore::Shard::stat()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

mgr/dashboard: fix rgw buckets e2e

Fixes: https://tracker.ceph.com/issues/62458
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53018 from rhcs-dashboard/rm-cephfs-subvolume-group

mgr/dashboard: cephfs remove subvolume group

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

test/librbd/fsx: add rbd-wnbd tests

rbd-wnbd can be used to map rbd images as Windows block devices.

This change will allow exercising rbd-wnbd and the wnbd driver
as opposed to using librbd directly.

One thing to note here is that read and write operations must be
512B aligned. Also, wnbd does not currently support "compare
and write".

For this reason, we'll stick to the basic operations for the time
being.

Note that we're renaming krbd_read/krbd_write to generic_pread
and generic_pwrite, reusing those functions for wnbd operations.
We're also removing the duplicated ggate read/write operations,
using the generic functions instead.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge PR #52416 into main

* refs/pull/52416/head:
qa/cephfs: check if re-installing this mpich fixes the issue

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: Fix doc for mds cap acquisition throttle

The config values picks the definition from the
description. Removed the duplicate description
in the doc.

Signed-off-by: Kotresh HR <khiremat@redhat.com>

mgr/dashboard: cephfs remove subvolume group

Fixes: https://tracker.ceph.com/issues/62452
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

cephadm: add tcmu-runner to logrotate config

This process could be used to set up the tcmu-runner
to log to a file much like other ceph daemons

- create /etc/tcmu directory
- create /etc/tcmu/tcmu.conf directory with default options
- change dir to /var/log
- change log level to 4
- add -v /etc/tcmu:/etc/tcmu to tcmu-runner container podman line in unit.run

In order to support this (mostly for debugging) we should
add tcmu-runner to the logrotate config

Fixes: https://tracker.ceph.com/issues/61571
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #47560 from pdvian/wip-pool-app

mon, qa: issue pool application warning even if pool is empty

Merge pull request #52674 from asm0deuz/add_networks_cli

mgr/cephadm: Add "networks" parameter to orch apply rgw

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Francesco Pantano <fpantano@redhat.com>
Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #52691 from adk3798/nvme-of-cleanup

mgr/cephadm: nvme-of follow up work

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52678 from phlogistonjohn/jjm-fix-test_valid_addr

python-common: fix valid_addr on python 3.11

Reviewed-by: Adam King <adking@redhat.com>

qa/cephadm: add test for ca signed keys

Test that bootstraps with a CA signed key using
the use_ca_signed_key cephadm override. Then follows
up by doing a check-host on each host which verifies
the cephadm mgr module can reach and authenticate with
the nodes using the new key setup.

This probably should really be a workunit, but
I didn't want to create a full new section for
this test and I needed a section that didn't
already run the cephadm task for every test. I could
see this being moved into some sort of
"test_special_deployment_scenarios" section in the future

Signed-off-by: Adam King <adking@redhat.com>

qa/cephadm: add ca signed key to cephadm task

To allow bootstrapping a cluster using a CA signed
key instead of the standard pubkey authentication.
Will allow explicit testing of this as we add support
for it

Signed-off-by: Adam King <adking@redhat.com>

doc/cephadm: document setting up CA signed keys in running cluster

Signed-off-by: Adam King <adking@redhat.com>

doc/cephadm: document bootstrapping with CA signed keys

Signed-off-by: Adam King <adking@redhat.com>

doc/cephadm: document how to pass self made SSH key pairs to bootstrap

This didn't seem to exist in the install section of
the cephadm docs. Wanted to add it in before adding
documentation for bootstrapping with CA signed keys.

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: add support for CA signed SSH keys setups

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #51785 from muahdib69/rsachere_iscsi_fixes

cephadm: Fix extra_container_args for iSCSI

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #52776 from ljflores/wip-read-balancer-documentation

doc: clarify upmap balancer documentation

Merge pull request #52919 from rhcs-dashboard/cluster-upgrade-progress

mgr/dashboard: cluster upgrade progress UI

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52991 from zdover23/wip-doc-2023-08-15-README-md-submodule

doc: improve submodule update command - README.md

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

- s3select fixes, related to Trino/TPCDS benchmark
- s3select submodule
- add log-message for better tracing the flow timeline
- fix valgrind issue
- new s3select version. fix crash upon wrong initialization on JSON flow
- s3select module. a fix for a constant number of columns(128).
- s3select module. QE fixes

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

Merge pull request #52989 from phlogistonjohn/jjm-installing-pkgs-cephadm

doc: do not recommend pulling cephadm from git

Reviewed-by: Adam King <adking@redhat.com>

doc: improve submodule update command - README.md

Improve the portion of README.md that directs the reader to update the
submodules. Previously, the text assumed that the reader knew that they
should cd into the ceph directory. Now the text is explicit about the
necessity of this action.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52898 from rhcs-dashboard/cephfs-subvolumegroup-edit

mgr/dashboard: Cephfs subvolumegroup edit

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52926 from kchheda3/wip_notif_s3user

rgw/notification: Store user info while creating notification topics

reviewed-by: yuvalif

Merge pull request #52857 from cyx1231st/wip-crimson-cleanup-osd-sharddispatcher

crimson/osd: cleanup and drop OSD::ShardDispatcher

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc: minor improvements of the cephadm section of the packages doc

* Don't call cephadm a script, it could be misleading
* Use the current release in the example for copy-pasters
* Improve grammar (I think)

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc: do not recommend pulling cephadm from git

First off, these instructions are wrong because the file has been
renamed to cephadm.py, and will not succeed as written. Amusingly, due
to curl's default behavior (and my lack of access to a time machine) the
initial command will seem to have succeeded until the user discovers
that HTML delivered as part of the body of a 404 response is not an
executable script.

Furthermore, in preparation for Reef we updated the documentation in the
cephadm subsection to use the results of a build, rather than source
files, because reef now contains a "compiled" cephadm. Currently, the
contents of the compilation is essentially the same-old cephadm script
but that will no longer be the case in the very near future (see
7c6a2cad96405616719b85c740aabd98558c93ac). We missed this general
section of the Ceph docs that repeats the instructions to copy a raw
python file out of the Ceph tree on GitHub.  Update the section to
contain a simplified version of the instruction found in
doc/cephadm/install.rst.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

.github: fix add to dashboard failing

```
Error: Request failed due to following response errors:
 - Could not resolve to a ProjectV2 with the number 2.
```

On further investigation, it looks like it could be a permission issue.
I followed the recommendation in their documentation to fix it: https://github.com/actions/add-to-project#creating-a-pat-and-adding-it-to-your-repository

Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: edit cephfs subvolumegroup

Fixes: https://tracker.ceph.com/issues/62370
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #52869 from rhcs-dashboard/fs-subvolumes-rm

mgr/dashboard: cephfs subvolume remove

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>

mgr/dashboard: cephfs subvolume remove

Fixes: https://tracker.ceph.com/issues/62349
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: cluster upgrade progress UI

Fixes: https://tracker.ceph.com/issues/62343
Signed-off-by: avanthakkar <avanjohn@gmail.com>
Co-Authored-By: Nizamudeen A <nia@redhat.com>

Merge PR #52547 into main

* refs/pull/52547/head:
qa: add test cases for vanilla ops commands
mds: dump locks when printing mutation ops
common/TrackedOp: support overriding the _dump method
mds: remove op field obsoleted by more usable "reqid"
mds: dump metareq_t instead of full op
mds: add lock type to formatter dump of SimpleLock
mds: mark print methods const
mds: drop MDRequestImpl::msg_lock
mds: lock TrackedOp when dumping
mds: avoid recursive locks dumping state
common/TrackedOp: fix race updating description with proper lock
common/Formatter: add support for dumping null
common/Formatter: refactor generating xml name

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #52577 into main

* refs/pull/52577/head:
mds: adjust cap acquisition throttles

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #52834 into main

* refs/pull/52834/head:
mount.fuse.ceph: strip the 'nofail' option from those that fuse will see

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #48705 from theanalyst/cmake-fmt-versionbump

cmake: ensure fmt lib is at least 8.1.1

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #52865 from cloudbehl/status-popover

mgr/dashboard: open popup with hover for cluster status

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52861 from rhcs-dashboard/fs-subvolumes-edit

mgr/dashboard: provide ability to edit cephfs subvolume

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #52966 from zdover23/wip-doc-2023-08-14-src-tools-rados

doc: update rados.cc

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51141 from xxhdx1985126/wip-seastore-clone

crimson/os/seastore: OP_CLONE in seastore

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

doc: update rados.cc

Clarify a line in ceph/src/tools/rados/rados.cc.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52963 from zdover23/wip-doc-2023-08-13-rados-troubleshooting-cpu-profiling

doc/troubleshooting: edit cpu-profiling.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/troubleshooting: edit cpu-profiling.rst

Improve the language in the "oprofile" section of the RADOS
Troubleshooting Guide.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52957 from zdover23/wip-doc-2023-08-12-rados-ops-monitoring-osd

doc/rados: update monitoring-osd-pg.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

mgr/dashboard: provide ability to edit cephfs subvolume

Fixes: https://tracker.ceph.com/issues/62347
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52645 from rhcs-dashboard/cephfs-vol-rm

mgr/dashboard: cephfs volume rm and rename

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: cephfs volume rm and rename

Fixes: https://tracker.ceph.com/issues/62408
Signed-off-by: avanthakkar <avanjohn@gmail.com>

Merge pull request #52932 from adk3798/cephadm-reef-latest-stable-release

cephadm: bump LATEST_STABLE_RELEASE to "reef"

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52510 from pritha-srivastava/wip-rgw-61916-test-failures

rgw/auth: Fix the return code returned by AuthStrategy,

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rados: update monitoring-osd-pg.rst

Ingest Anthony D'Atri's notes from
https://github.com/ceph/ceph/pull/50856#discussion_r1289532902
which should have been included earlier.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50856 from zdover23/wip-doc-2023-04-04-rados-operations-monitoring-osd-pg-question-1

doc/rados/ops: question about down and in OSDS

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52907 from cloudbehl/add-telemetry

mgr/dashboard: Add telemetry link in dashboard

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge pull request #52903 from cloudbehl/rgw-details

mgr/dashboard: RGW Details card cleanup

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/rados/ops: question about down and in OSDS

Clear up a confusing sentence that purports to explain OSDS that are
both "down" and "in".

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52945 from kotreshhr/doc-mgr-deadlock

doc: Add a note on possible deadlock on volume deletion

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc: Add a note on possible deadlock on volume deletion

Fixes: https://tracker.ceph.com/issues/62407
Signed-off-by: Kotresh HR <khiremat@redhat.com>

test/lirbd/fsx: fix Windows issues

There are a few more issues that need to be addressed in order to
run this test on Windows:

* aligned_free (compat.h wrapper) needs to be used after allocating
  memory using posix_memalign, "free" causes a crash on Windows
* long (4B) can't store x64 pointers on Windows, we'll use
  uintptr_t instead.
* use O_BINARY when opening files, otherwise certain characters
  get translated (e.g. LF to CRLF).

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

test/librbd: enable the functional tests on Windows

Right now, the librbd tests are skipped on Windows.

This change enables the functional tests, which can run with
just a few modifications.

We'll continue skipping the unit tests however, which require
too much OSD code that hasn't been ported (e.g. cls, libosd, etc).

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

mgr/dashboard: Add telemetry link in dashboard

Fixes: https://tracker.ceph.com/issues/62380
Signed-off-by: cloudbehl <cloudbehl@gmail.com>

qa/cephfs: check if re-installing this mpich fixes the issue

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #52765 into main

* refs/pull/52765/head:
mgr/volumes: Fix pending_subvolume_deletions in volume info
qa: Add testcase for pending_subvolume_deletions count

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

cephadm: bump LATEST_STABLE_RELEASE to "reef"

Now that reef is out, we should tell users
that it's the latest stable release, rather
than quincy

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #52886 from rhcs-dashboard/cephfs-sbuvolumegroup-create

mgr/dashboard: create subvolume groups

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

rgw/notification: Store user info while creating notification topic

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>

Merge pull request #52178 from phlogistonjohn/jjm-cephadm-init-ctrs

cephadm: add basic init containers support

Reviewed-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: Add "networks" parameter to orch apply rgw

This parameter is available in specs but not available as a parameter.
Having it will ease its use in cephadm-adopt playbook in ceph-ansible.

fixes: https://tracker.ceph.com/issues/62185

Signed-off-by: Teoman ONAY <tonay@ibm.com>

Merge pull request #52876 from kchheda3/wip-noti-logs

rgw: enrich the failed debug log line for sync notification

reviwed-by: BBoozmen, khu102-bloomberg, yuvalif, thotz

mgr/dashboard: create subvolumegroup

Fixes: https://tracker.ceph.com/issues/62363
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

crimson/os/seastore: debug outputs

Signed-off-by: xuxuehan <xuxuehan@qianxin.com>

crimson/os/seastore/object_data_handler: take indirect offset into
account when reading object data

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: don't remove intermediate
mapping when remapping pins

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: remap indirect mappings

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: remove extents for physical lba mappings if necessary
when removing indirect lba mappings

Signed-off-by: xuxuehan <xuxuehan@qianxin.com>

crimson/os/seastore/lba_manager: decrease refcount of the physical lba
mappings when removing indirect lba mappings

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/onode_manager: erase onodes from the OnodeTree at the moment of erase_onode

Signed-off-by: xuxuehan <xuxuehan@qianxin.com>

crimson/os/seastore/object_data_handler: don't reserve lba space when truncating objects to zero size

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

test/crimson/seastore: add OP_CLONE unittest case

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>