mds: quiescelock as local lock + cap masking

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

qa: run quiesce unit tests in fs:functional

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/63664

qa: add quiesce protocol unit tests

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/63664

qa: detect partial migrations during large config of dist epin

This method would wrongly "succeed" when looking for setup of distributed
ephemerally pinned directory fragments. If the migrator splits a subtree during
the course of migration (to reduce the migration size) then the operation may
not actually be complete.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use stdin-killer to timeout run_shell_payload

- simplify argument processing / forwarding
- use stdin-killer to kill all sub-processes of the shell
- do not needlessly use run_shell to execute the command as it adds a timeout
  to the stdout/stderr processing
- provide a stdin (PIPE) by default otherwise teuthology's code closes stdin
  and triggers stdin-killer to timeout the shell.
- use a 15 minute timeout by default

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: simplify run_shell argument processing

No functional changes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: add dev docs for quiesce protocol

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: block quiesce for critical .meta file

This file must remain accessible during quiesce to allow snapshots to be taken
on a subvolume.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add vxattr to block quiesce on an inode

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: convert encoded ephemeral dist pin to flags

To use space efficiently, convert this field to a 8 bit flags field that is
backward compatible.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add counter to throttle quiesce

So a storm of quiesce operations do not affect normal MDS operations.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add quiesce set feature flag

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/63664

mds: skip non-head inodes for quiesce

These are accessible only via snapshots.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add quiesce op

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/63664

mds: print all SimpleLock flags in debug output

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: pretty print mutation when dumping lock

Now looks like:

    ... (iquiesce xlock x=1 by request(mds.0:2785 nref=6)) ...

instead of:

    ... (iquiesce xlock x=1 by 0xdeadbeef)) ...

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add new inode quiescelock

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use 128 bits for waiters on MDSCacheObject

Adding a new inode lock will overflow inode wait bits into the MDSCacheObject
wait bits. Make space for the quiescelock.

This includes a minor refactor to no longer attempt scoping the set of masks we
test in MDSCacheObject::waiting when calling MDSCacheObject::is_waiter_for.
This optimization wasn't worth the overhead and would be awkard to keep as
std::bitset cannot be used as a key for a std::multimap (easily).  Instead, we
use the sequence number as a key which helps us to avoid allocating another map
whenever we call MDSCacheObject::take_waiting.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: provide mechanism to authpin while freezing

When a subtree is freezing, it's no longer possible to acquire new authpins.
This is a problem when a compound request like quiescing a subtree is trying to
acquire authpins for each sub-op. This creates a situation where some quiesce
sub-ops complete with authpins (thereby preventing the tree from becoming
"frozen") and new sub-ops cannot acquire authpins (because the tree is
"freezing"). To circumvent this, allow some authpin requests to proceed if
FLAG_BYPASSFREEZING is set.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add command to get specific op

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: finish request before completing internal req

Importantly, this marks the request dead so it cannot be killed.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: complete internal op if killed

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: avoid killing dead requests

This is important when you have:

* sub-op A finishes and resumes parent
* parent kills all sub-ops (including sub-op A)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add command to kill request

Fixes: https://tracker.ceph.com/issues/63374
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add path argument to `ops` and `dump tree` to stream result to local file

This file can be collected during testing to avoid sending a large JSON result
over the `ceph tell`/asok interface.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: print internal_request filepaths if present

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add more information to debug message

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove redundant parenthesis

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: implement Mutation::dump method

So we can dump it from asok commands.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: make LockType fields const

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: annotate mdr with try_rdlock_snap_layout failure

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: refactor if into switch

To make it easier to read and match other constructs in this code.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: call Locker method using this

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: simplify assert

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: dump locks passed to Locker::acquire_locks

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add LockOp::print method for debugging

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use new insert template via print

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add request result to mutation for analysis by tests

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add comment on locking order rules

From a past commit.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: allow specifying rdlock position

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove dead method

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: provide a template for object dumps

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: support long running ops without slow warnings

A quiesce op in the MDS may run for an indeterminate amount of time. Warnings
should not be generated.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: simplify loop

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add JSONFormatterFile class

To stream JSON to a file for collection (for testing).  The idea here that
trying to send a very large JSON result over `ceph tell` is best to be avoided.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: use more efficient vector for stack

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

include: use larger int for large gathers

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #56266 from cbodley/wip-qa-rgw-lua-pool-application

qa/rgw: ignore POOL_APP_NOT_ENABLED in rgw/lua subsuite

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #56220 from linuxbox2/wip-nfs-rename

rgw_file: fix mv/rename cases broken by zipper integration

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #56225 from linuxbox2/wip-cors-fix

rgw: Add missing empty checks to the split string in is_string_in_set().

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56044 from pr0ton11/rgw-disable-signature-url

RGW: allow user disabling presigned urls in rgw configuration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55795 from kchheda3/wip-rename-notif-events

rgw/notification: Make the LifeCycle & Replication events AWS Compatible and Support s3:Replication:Delete event

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #56333 from molpako/patch-3

doc/cephfs: fix architecture link to correct relative path

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55595 from VallariAg/wip-nvmeof-test-v3

qa/suite/rbd/nvmeof: Deploy multiple gateways and namespaces

Reviewed-by: Barak Davidov <barakda@il.ibm.com>
Reviewed-by: Aviv Caro <Aviv.Caro@ibm.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/cephfs: fix architecture link to correct relative path

Signed-off-by: molpako <35787116+molpako@users.noreply.github.com>

Merge pull request #55864 from rosinL/wip-speedup-check-generated

test/encoding/check-generated: parallel testing by type

Merge pull request #56203 from athanatos/sjust/wip-protoc-dependencies

crimson: add crimson-osd rpm and deb runtime dependencies for protobuf inherited from seastar

Merge pull request #56070 from phlogistonjohn/jjm-cephadm-sudo-cmd-auditing

cephadm: sudo cmd auditing

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56178 from adk3798/cephadm-pub-net-config-check-out-of-date

mgr/cephadm: refresh public_network for config checks before checking

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56175 from asm0deuz/bootstrap_no_container

cephadm: bootstrap fails if no container engine installed

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56147 from adk3798/stopped-daemon-shell

cephadm: fix `cephadm shell --name <daemon-name>` for stopped/failed daemon

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56143 from insatomcat/timemaster

cephadm.py: add timemaster to timesync services list

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56075 from phlogistonjohn/jjm-enable-black-cephadm

cephadm: enforce black formatting for cephadmlib

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #55883 from thegreenbear/ingress-haproxy-bind-ipv4v6

Ingress service: make HAProxy to listen on IPv4 and IPv6 …

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #55874 from adk3798/only-bind-to-network

mgr/cephadm: allow grafana and prometheus to only bind to specific network

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55645 from ronen-fr/wip-rf-nextobj

osd: improved get_objects_by_prefixes() ergonomics

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

qa/suite/rbd/nvmeof: Deploy multiple gateways and namespaces

1. Deploy 2 gateways on different nodes, then check for multi-path.
    To add another gateway, only "roles" need to be changed in job yaml.
2. Create "n" nvmeof namespaces, configured by 'namespaces_count'
3. Rename qa/suites/rbd/nvmeof/cluster/fixed-3.yaml to fixed-4.yaml
    which contains 2 gateways and 2 initiators.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #56109 from anthonyeleven/improve-monitoring

doc/cephadm/services: Improve monitoring.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55877 from adk3798/cephadm-pull-usage

cephadm: improve cephadm pull usage message

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55615 from rhcs-dashboard/grafana-mount-dashboards

mgr/cephadm: mount grafana dashboards from the path

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #52760 from thotz/add-healthchecker-timeout-haproxy

cephadm: adjust the ingress ha proxy health check interval

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56189 from rhcs-dashboard/fix-64913-main

mgr/dashboard: Allow DELETE method api request, X-TOTAL-COUNT header in CORS config in dashboard

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56250 from myoungwon/wip-laod-metadata-during-gc

crimson/os/seastore: cache metadata during trimming to prevent from disk read

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #56204 from zmc/do-cmake-args

do_cmake.sh: Use value of ARGS at runtime

osd/SnapMapper: add common logs

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit f302be018ebef365d8876d14185c4bdb246f22d8)

osd: improved get_objects_by_prefixes() ergonomics

Improved call signatures for get_next_objects_to_trim() &
get_objects_by_prefixes().

Also: as as get_next_objects_to_trim() has only a single
failure mode, we should not try to handle two distinct failures
in its callers' code.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55846 from athanatos/sjust/for-review/wip-crimson-coroutine-support

crimson: add coroutine support for errorated and interruptible futures

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #56034 from tchaikov/wip-msgr-dont-panic-at-hot-unplug

msg: do not abort if driver->del_event() returns -ENOENT

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55896 from ronen-fr/wip-rf-compactref

osd: avoid calling a virtual function in a ctor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #55872 from aclamk/wip-aclamk-bs-fix-unused

os/bluestore: Mute warnings

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #54492 from Matan-B/wip-matanb-op-thread-timeout

qa/suites/rados/verify/validater/valgrind: increase op thread timeout

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56240 from zdover23/wip-doc-2024-03-17-rados-config-common

doc/rados: update common.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

qa/rgw: ignore POOL_APP_NOT_ENABLED in rgw/lua subsuite

> cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/cephadm/services: Improve monitoring.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #56248 from nbalacha/wip-nbalacha-rbd-doc-man

doc/rbd: minor changes to the rbd man page

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/rbd: minor changes to the rbd man page

Fixes typos and grammar for some commands. Adds
additional details for some commandds.

Signed-off-by: N Balachandran <nibalach@redhat.com>

crimson/os/seastore: cache metadata during trimming to prevent from disk read

I encountered continous disk reads during trimming even though there are sufficient
cache available, in 4K random write test with RBM (RBD).
This is because metadata is note cached if its source is background transaction
within touch_extent(). So, seastore, including the trimming process,  needs to
constantly retrieve metadata (e.g., BACKREF_LEAF).

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #56223 from guits/tracker_64951

node-proxy: support more Location value formats

Merge pull request #56247 from nbalacha/wip-nbalacha-rbd-doc-2

docs/rbd: fix typos in the rbd basic cmds documentation

Merge pull request #56198 from ivoalmeida/fix-nfs-attach-cmd

mgr/dashboard: fixed nfs attach command

Reviewed-by: Nizamudeen A <nia@redhat.com>

docs/rbd: typo in "retrieving image information"

Replace "for the image" with "of the image".

Signed-off-by: N Balachandran <nibalach@redhat.com>

docs/rbd: replace introspect with inspect

Replace "introspect" with "inspect" in the rbd basic commands
description.

Signed-off-by: N Balachandran <nibalach@redhat.com>

docs/rbd: fix typo in arg name

Replace "{image-}" with "{image-id}" in the "rbd trash rm"
command description.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #56217 from xxhdx1985126/wip-seastore-remap-bug-fixes

test/crimson/sesastore/test_block: add constructor to construct

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/cephdam: document limiting passwordless sudo commands

Based on the previous commits making the remote executables auditable
and explicit, document the admin's ability to restrict password-less
sudo access to only the set of commands cephadm actually uses.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: add a simple unit test for RemoteCommand class

Converting a remote command to something that other libs uses requires
converting the enum to a string. Python behavior in the area varies
across versions so add a unit test that verifies the conversion
behaves as intended.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: add test to ensure the list of remote commands is known

Add a test file to help ensure the audit of remote commands is kept up
to date.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: make remote command execution auditable

Update ssh.py and other code using it to only allow commands wrapped
in particular python types as executables on the remote hosts.
By using a specific type for remote executables we make the code more
auditable, avoiding the possibility of executing arbitrary strings
as commands with sudo. This is all enforced by mypy's type checking.

The result is a list of commands that the cephadm mgr module may
execute on a remote host using sudo:
```
$ git ls-files -z | xargs -0 grep 'RemoteExecutable(' -d skip -h | grep
-v '(str)' | sed -e 's/.*RemoteExecutable(//' -e 's/)//' -e 's/,$//'
'which'
'/usr/bin/cephadm'
python
'chmod'
'ls'
'sysctl'
'chown'
'mkdir'
'mv'
'touch'
'rm'
'true'
```
Note that *python* is special as it is based on the output of which and
may vary from OS to OS. The quoted items are used exactly as named.
Only the binary at `/usr/bin/cephadm` _or_ the dynamically discovered
python3 binary will be used. This depends on a configuration option for
the cephadm module.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc/rados: update common.rst

Clarify some overwritten paragraphs in
doc/rados/configuration/common.rst. Cf.
https://github.com/ceph/ceph/pull/50795/files#r1153897231.

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56182 from zdover23/wip-doc-2024-03-14-glossary-librados

doc/glossary: add "librados" entry