client: skip unexpected command replies

Instead of marking the message as handled, give another component (or Client) a
chance to process.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

mgr: indicate map message is acked instead of unhandled

This avoids messages like:

    2025-02-18T05:31:17.738+0000 7f5206546640  0 ms_deliver_dispatch: unhandled message 0x5632d05f0700 fsmap(e 9) from mon.0 v2:172.21.3.230:40412/0

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

osdc/Objecter: convert to ms_dispatch2 for ack

Convert ms_dispatch to ms_dispatch2 to enable indicating that a map message is
acknowledged and instead of processed (or deliberately not processed).

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: indicate maps are acked not processed

Ancillary change: do not do client upkeep after map processing.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

msg: add alternate statuses for ms_dispatch2 handling

Many dispatchers return false to allow other dispatchers also common messages
like MOSDMap or MFSMap. They implicitly depend on some dispatcher which is
always at the "tail" of the dispatcher queue to return "true" indicating the
msg was processed to avoid messages like:

    2025-02-18T05:31:17.738+0000 7f5206546640  0 ms_deliver_dispatch: unhandled message 0x5632d05f0700 fsmap(e 9) from mon.0 v2:172.21.3.230:40412/0

but this cannot always happen when some libraries like the RadosClient used standalone.

So, add a variant for encapsulating other indications for how the message was
processed by dispatch2.  For example, a message may be "acknowledged" but
explicitly allow other dispatchers to try processing the message.

Note: we're using a variant to avoid updating all of the ms_dispatch code to
use the sentinel classes.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

tools/cephfs_mirror: do not process maps with fast dispatch

This acquires a lock which is a no-no in the messenger.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

doc: add docs for volumes interface for charmap

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

qa: add tests for subvolume charmap settings

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

pybind/mgr/volumes: wire up charmap for subvol/subvolgroup

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

pybind/mgr: send MDS commands through cephfs client

To avoid linking to the CephFS client statically, use the dynamically-linked
`cephfs` module to send commands to the MDS.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

pybind/cephfs: wire up mds_command2

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

mgr: add module method to send notifications

For use by MgrModule.send_command to signal MDS command completion.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

libcephfs: add mds_command2 for asynchronous commands

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

mgr: excise CephFS client from mgr C++ base

Linking to the client causes two copies of the Client library to be linked in
the ceph-mgr when modules also dynamically link to libcephfs via the "cephfs"
python library. This creates problems with duplicate boost::locale.

Instead, modules should just use the "cephfs" library to send commands to the
MDS.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

mgr: use std namespace

This C++ code was relying on the Client.h header to bring in these names from
the std:: namespace. A subsequent commit plans to remove that header so add
namespace qualifier now.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

doc: add docs for CephFS charmap config

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

qa: add charmap tests

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

qa: add helpful exceptions for attr changes

It's not possible to extract the error message from stderr by the caller.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

qa: ignore libicu leaks

These are likely spurious and not interesting.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: add wrappings for charmap manipuluation of dentry names

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: add dir_result_t::dentry::print

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

win32: add libicu Windows build

cephfs introduces a new dependency: libicu. We need to update the
Windows build scripts to build and link against libicu.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

CMakeLists: add boost::locale dependency for client

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

install-deps: unconditionally install boost libraries

If we add a new Boost component, the install-deps.sh script will not install it
because other Boost libraries are installed with the desired version. Just
unconditionally install. The package manager is smart enough to skip already
installed packages.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

test/libcephfs: update root operation return values

Switch from EBUSY to reasonable and normal errors for operations on the root
directory. Use the Client::path_walk machinery changes from the previous commit
to catch most of them, otherwise return EINVAL (which is normal for other POSIX
fs).

Fixes: 6ed7f2364ae5507bab14c60b582929aa7b0ba400
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: refactor all path traversals through path_walk

This is groundwork to ensure that all code paths properly apply dentry name
transformations during traversal, specifically for casefolding.

Note some ancillary changes:

* readlink now correctly does lookup permission checks during path walk (previously a TODO).

* There has been some tweaking of the method signatures to accept an `const
  InodeRef&`. This was convenient when passing walk_dentry_result::diri around
  but the cascade in changes prompted me to leave it partially done to avoid
  blowing up this changeset.

It's worth noting that this change is somewhat "half done". I had made an
effort to have all operations (whether fuse or libcephfs) call path_walk once.
This was done for Client::_mkdir and a few others. See for example:

  - do_mkdirat -> _mkdir
  - mksnap -> _mkdir
  - ll_mkdir -> _mkdir
  - mkdirs -> _mkdir (this had other significant simplification/cleanup);
    however: note that we still have two calls to path_walk for any given
    directory to be created. This is not a problem because:

Client::path_walk is now more tolerant of being called in a openat-style way.
The method now accepts an anchor directory and relative path (which may be a
single dentry name for ll_* calls). It is also tolerant of the directory inode
in fact referencing the target inode with relative path == "". That is a useful
property for many openat-style APIs (although largely unofficially).  It also
means that if we resolve a path then we can pass the resolved Inode (directory)
/ string (Dentry name) pair to another method which may also call path_walk
with minimal replication of path walk work. This is done a few times in this
changeset.

Fixes: https://tracker.ceph.com/issues/66373
Fixes: https://tracker.ceph.com/issues/70100
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

test/libcephfs: test parallel creates

That the cap_shared_gen value does not result in unexpected ENOENT.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

test/libcephfs: add test for lookup failure after readdir

That we do not get unexpected ENOENT following readdirs (particularly thinking
of cap_shared_gen differences).

Test-case-for: https://tracker.ceph.com/issues/70100
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: init dentry shared_gen with invalid value

Directories and dentries are initialized with value 0 which makes detecting a
null  (or placeholder) dentry created via Client::get_or_create difficult.  We
already do checks to see if a dentry is invalid when the directory's shared_gen
changes so use an invalid value for these synthetic dentries until they can be
appropriately updated from Client::update_dentry_lease.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: add _lookup debugging

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: remove redundant check

This condition:

    (!dn->inode || dn->inode->caps_issued_mask(mask, true))

is already checked in the surrounding `if`.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: dump InodeStat from mds

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: encode optmetadata in InodeStat sent to clients

Deliberately do not dump the entire struct. Some metadata may not be
appropriate to share to clients.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: check client features for charmap

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: add client feature bit for charmap

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: wire up vxattr for changing charmap

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: inherit charmap on mkdir

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds,include: add charmap optmetadata

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/66373

mds,include: add inode_t optional metadata

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/66373

client: hide alternate_name from API

The dentry alternate_name was exposed primarily to facilitate testing. Instead,
put these methods in the TestClient scaffolding to allow reading/manipulating
alternate_name.

Because we will be using alternate_name to handle encrypted names and the
normalized / case folded name, we do not want the application to change the
meaning of the metadata out-of-band.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: move alternate_name once

Client::update_dentry_lease also moves alternate_name from the lease
which causes it to zero out alternate_name on the second move.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/66373

client: optimize alternate_name passing to helper

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/66373

client: relocate definition

It doesn't need to be public; the API does not expose alternate_name.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/66373

client: print dentry with alternate_name on dump

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: move inode dump to print method

There is no functional change.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: add debugging for encoding lease stat

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: make encode_lease a proper method

So we can print debugging.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: add fscrypt metadata for inode stat size

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: use DentryRef for ref counting in MetaRequest

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: add DentryRef

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: add helper for determining if a perm check is necessary

To be used in subsequent commits.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: cache client_permissions config

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: add debugging for conf changes

So logs show when they are modified.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: sort configs

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client/UserPerm: add print method

For debug prints.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: note mount parameters in debug log

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: print stat mode in octal

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

common: add missing op string

This gets converted in messenger douts.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

include/filepath: add empty path check

When passing an empty string to filepath, it would test if the first character
is '/' which is an invalid access through std::string_view.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

Merge PR #61991 into main

* refs/pull/61991/head:
qa: remove unreacheable/redundant code from test_data_scan.py

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #62019 from ceph/template-updates

Links to Jenkins jobs in PR comment commands / Remove deprecated commands

Merge pull request #61863 from spuiuk/ctdb_ports

cephadm/smb: add ctdb ports to smb daemon endpoint

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61925 from JoshuaGabriel/joshua_70058

mgr/cephadm: Check if a daemon is ok-to-stop before scheduling a restart

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #61897 from cbodley/wip-70077

qa/rgw: configure 'iam root' accounts outside of rgw/verify

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #61920 from adk3798/cephadm-rgw-ingress-test-timeouts

qa/cephadm: add timeouts and debug commands to rgw-ingress test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61710 from adk3798/mgr-nfs-upgrade-start-reef

qa/cephadm: start mgr-nfs-upgrade test from reef

Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #61013 from Kushal-deb/fix_issue-2047153

cephadm: orch upgrade status to return output according to the format specified

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61504 from kchheda3/wip-fix-healthchecks

rgw:lua: Skip the healthchecks and system requests from going to backend storage

Merge pull request #61509 from kchheda3/wip-config-lua

rgw/lua: Add rgw config to enable/disable lua

Merge pull request #61560 from oshrey16/fix-lua-runtime-background

rgw/lua: Fix Lua background thread execution timing and config updates

Merge pull request #61938 from ceph/retrigger-rtd

doc: Support to retrigger docs/readthedocs.org:ceph PR build

doc: PR Template - Remove non-functional trigger phrases

Signed-off-by: David Galloway <david.galloway@ibm.com>

doc: PR Template - Add Jenkins job URLs to commands

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #61822 from rhcs-dashboard/smb-edit-cluster

mgr/dashboard: SMB - Edit Cluster

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #61815 from ljflores/wip-tracker-69456

qa/suites/upgrade: ignore when pg is stuck peering

Merge pull request #61703 from MaxKellermann/kstore_includes

os/kstore: add missing include

Merge pull request #61674 from connorfawcett/wip-ec-default-fixes

erasure-code: Set reed_sol_van to be the default technique for Jerasure if none is specified.

Merge pull request #61015 from nbalacha/wip-nbalacha-rbd-mirror-namespace-2

rbd-mirror: default <-> non-default namespace remapping

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #61269 from cbodley/wip-rgw-createbucket-layout

rgw/s3: CreateBucket extension for layout type and shard count

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #61982 from anthonyeleven/nvme-proxy

doc/rbd: Improve nvmeof-requirements.rst with proxy and other info

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60524 from thotz/multisitie-fix-cloud-restore

rgw/cloudrestore: sync cloud restore objects accoridngly

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #61849 from yuvalif/wip-yuval-69944

rgw: be explicit on moving rados op in APIs

Merge pull request #61376 from athanatos/sjust/wip-crimson-gcc-13

src/CMakeLists: require gcc13 for crimson, clarify error messages

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61846 from rhcs-dashboard/tox-generate-proto

mgr/dashboard: tox command to generate the nvmeof proto files

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #61903 from ronen-fr/wip-rf-crimson-conf

crimson/osd: replace obsolete get_tracked_conf_keys()

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62001 from cybozu/fix-stale-delete

doc: fix incorrect radosgw-admin subcommand

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60871 from leonidc/leonidc-epoch-filter

Epoch filtering

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Aviv Caro <Aviv.Caro@ibm.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

doc: fix incorrect radosgw-admin subcommand

Signed-off-by: Toshikuni Fukaya <toshikuni-fukaya@cybozu.co.jp>

Merge PR #61889 into main

* refs/pull/61889/head:
msg/async/Stack: include cleanup
msg/Message: include cleanup
msg/async/crypto_onwire: include cleanup
msg/async/ProtocolV1: add missing includes
msg/async/frames_v2: add missing includes
msg/msg_types: add missing includes

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #61888 into main

* refs/pull/61888/head:
os/memstore: add missing includes

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #61876 into main

* refs/pull/61876/head:
qa/cephfs: ignore warning that pg is stuck peering for upgrade jobs

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #61802 into main

* refs/pull/61802/head:
perfglue: add missing include

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #61801 into main

* refs/pull/61801/head:
kv: add missing includes

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #61799 into main

* refs/pull/61799/head:
erasure-code: add missing include

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #61798 into main

* refs/pull/61798/head:
crush: include cleanup

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #61955 from idryomov/wip-70075-test

qa/workunits/rbd: add a test for force promote with a user snapshot

Reviewed-by: N Balachandran <nibalach@redhat.com>

rgw: reject PutBucketLifecycleConfiguration on indexless buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: improved error when listing indexless bucket

without this change, ListObjects would attempt to list index shard
objects that don't exist and fail with:

> ERROR: S3 error: 404 (NoSuchKey)

after:

> ERROR: S3 error: 405 (MethodNotAllowed): Indexless buckets cannot be listed

Signed-off-by: Casey Bodley <cbodley@redhat.com>

examples/rgw: register boto extension for BucketIndex on CreateBucket

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/s3: CreateBucket extension for layout type and shard count

extend s3's CreateBucketConfiguration [1] with a custom BucketIndex element
that can override rgw's default bucket index type and shard count

to create an indexless bucket:
  <CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <BucketIndex>
      <Type>Indexless</Type>
    </BucketIndex>
  </CreateBucketConfiguration>

to create a normal pre-sharded bucket:
  <CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <BucketIndex>
      <Type>Normal</Type>
      <NumShards>1023</NumShards>
    </BucketIndex>
  </CreateBucketConfiguration>

[1] https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html#API_CreateBucket_RequestSyntax

Signed-off-by: Casey Bodley <cbodley@redhat.com>