pybind/mgr/volumes: wire up charmap for subvol/subvolgroup

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

pybind/mgr: send MDS commands through cephfs client

To avoid linking to the CephFS client statically, use the dynamically-linked
`cephfs` module to send commands to the MDS.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

pybind/cephfs: wire up mds_command2

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

mgr: add module method to send notifications

For use by MgrModule.send_command to signal MDS command completion.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

libcephfs: add mds_command2 for asynchronous commands

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

mgr: excise CephFS client from mgr C++ base

Linking to the client causes two copies of the Client library to be linked in
the ceph-mgr when modules also dynamically link to libcephfs via the "cephfs"
python library. This creates problems with duplicate boost::locale.

Instead, modules should just use the "cephfs" library to send commands to the
MDS.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

mgr: use std namespace

This C++ code was relying on the Client.h header to bring in these names from
the std:: namespace. A subsequent commit plans to remove that header so add
namespace qualifier now.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

doc: add docs for CephFS charmap config

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

qa: add charmap tests

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

qa: add helpful exceptions for attr changes

It's not possible to extract the error message from stderr by the caller.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

qa: ignore libicu leaks

These are likely spurious and not interesting.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: add wrappings for charmap manipuluation of dentry names

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: add dir_result_t::dentry::print

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

win32: add libicu Windows build

cephfs introduces a new dependency: libicu. We need to update the
Windows build scripts to build and link against libicu.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

CMakeLists: add boost::locale dependency for client

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

install-deps: unconditionally install boost libraries

If we add a new Boost component, the install-deps.sh script will not install it
because other Boost libraries are installed with the desired version. Just
unconditionally install. The package manager is smart enough to skip already
installed packages.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

test/libcephfs: update root operation return values

Switch from EBUSY to reasonable and normal errors for operations on the root
directory. Use the Client::path_walk machinery changes from the previous commit
to catch most of them, otherwise return EINVAL (which is normal for other POSIX
fs).

Fixes: 6ed7f2364ae5507bab14c60b582929aa7b0ba400
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: refactor all path traversals through path_walk

This is groundwork to ensure that all code paths properly apply dentry name
transformations during traversal, specifically for casefolding.

Note some ancillary changes:

* readlink now correctly does lookup permission checks during path walk (previously a TODO).

* There has been some tweaking of the method signatures to accept an `const
  InodeRef&`. This was convenient when passing walk_dentry_result::diri around
  but the cascade in changes prompted me to leave it partially done to avoid
  blowing up this changeset.

It's worth noting that this change is somewhat "half done". I had made an
effort to have all operations (whether fuse or libcephfs) call path_walk once.
This was done for Client::_mkdir and a few others. See for example:

  - do_mkdirat -> _mkdir
  - mksnap -> _mkdir
  - ll_mkdir -> _mkdir
  - mkdirs -> _mkdir (this had other significant simplification/cleanup);
    however: note that we still have two calls to path_walk for any given
    directory to be created. This is not a problem because:

Client::path_walk is now more tolerant of being called in a openat-style way.
The method now accepts an anchor directory and relative path (which may be a
single dentry name for ll_* calls). It is also tolerant of the directory inode
in fact referencing the target inode with relative path == "". That is a useful
property for many openat-style APIs (although largely unofficially).  It also
means that if we resolve a path then we can pass the resolved Inode (directory)
/ string (Dentry name) pair to another method which may also call path_walk
with minimal replication of path walk work. This is done a few times in this
changeset.

Fixes: https://tracker.ceph.com/issues/66373
Fixes: https://tracker.ceph.com/issues/70100
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

test/libcephfs: test parallel creates

That the cap_shared_gen value does not result in unexpected ENOENT.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

test/libcephfs: add test for lookup failure after readdir

That we do not get unexpected ENOENT following readdirs (particularly thinking
of cap_shared_gen differences).

Test-case-for: https://tracker.ceph.com/issues/70100
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: init dentry shared_gen with invalid value

Directories and dentries are initialized with value 0 which makes detecting a
null  (or placeholder) dentry created via Client::get_or_create difficult.  We
already do checks to see if a dentry is invalid when the directory's shared_gen
changes so use an invalid value for these synthetic dentries until they can be
appropriately updated from Client::update_dentry_lease.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: add _lookup debugging

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: remove redundant check

This condition:

    (!dn->inode || dn->inode->caps_issued_mask(mask, true))

is already checked in the surrounding `if`.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: dump InodeStat from mds

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: encode optmetadata in InodeStat sent to clients

Deliberately do not dump the entire struct. Some metadata may not be
appropriate to share to clients.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: check client features for charmap

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: add client feature bit for charmap

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: wire up vxattr for changing charmap

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: inherit charmap on mkdir

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds,include: add charmap optmetadata

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/66373

mds,include: add inode_t optional metadata

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/66373

client: hide alternate_name from API

The dentry alternate_name was exposed primarily to facilitate testing. Instead,
put these methods in the TestClient scaffolding to allow reading/manipulating
alternate_name.

Because we will be using alternate_name to handle encrypted names and the
normalized / case folded name, we do not want the application to change the
meaning of the metadata out-of-band.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: move alternate_name once

Client::update_dentry_lease also moves alternate_name from the lease
which causes it to zero out alternate_name on the second move.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/66373

client: optimize alternate_name passing to helper

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/66373

client: relocate definition

It doesn't need to be public; the API does not expose alternate_name.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/66373

client: print dentry with alternate_name on dump

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: move inode dump to print method

There is no functional change.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: add debugging for encoding lease stat

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: make encode_lease a proper method

So we can print debugging.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

mds: add fscrypt metadata for inode stat size

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: use DentryRef for ref counting in MetaRequest

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: add DentryRef

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: add helper for determining if a perm check is necessary

To be used in subsequent commits.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: cache client_permissions config

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: add debugging for conf changes

So logs show when they are modified.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: sort configs

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client/UserPerm: add print method

For debug prints.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

client: note mount parameters in debug log

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

client: print stat mode in octal

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

common: add missing op string

This gets converted in messenger douts.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

include/filepath: add empty path check

When passing an empty string to filepath, it would test if the first character
is '/' which is an invalid access through std::string_view.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Fixes: https://tracker.ceph.com/issues/66373

Merge PR #61991 into main

* refs/pull/61991/head:
qa: remove unreacheable/redundant code from test_data_scan.py

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #62019 from ceph/template-updates

Links to Jenkins jobs in PR comment commands / Remove deprecated commands

Merge pull request #61863 from spuiuk/ctdb_ports

cephadm/smb: add ctdb ports to smb daemon endpoint

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61925 from JoshuaGabriel/joshua_70058

mgr/cephadm: Check if a daemon is ok-to-stop before scheduling a restart

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #61897 from cbodley/wip-70077

qa/rgw: configure 'iam root' accounts outside of rgw/verify

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #61920 from adk3798/cephadm-rgw-ingress-test-timeouts

qa/cephadm: add timeouts and debug commands to rgw-ingress test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61710 from adk3798/mgr-nfs-upgrade-start-reef

qa/cephadm: start mgr-nfs-upgrade test from reef

Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #61013 from Kushal-deb/fix_issue-2047153

cephadm: orch upgrade status to return output according to the format specified

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61504 from kchheda3/wip-fix-healthchecks

rgw:lua: Skip the healthchecks and system requests from going to backend storage

Merge pull request #61509 from kchheda3/wip-config-lua

rgw/lua: Add rgw config to enable/disable lua

Merge pull request #61560 from oshrey16/fix-lua-runtime-background

rgw/lua: Fix Lua background thread execution timing and config updates

Merge pull request #61938 from ceph/retrigger-rtd

doc: Support to retrigger docs/readthedocs.org:ceph PR build

doc: PR Template - Remove non-functional trigger phrases

Signed-off-by: David Galloway <david.galloway@ibm.com>

doc: PR Template - Add Jenkins job URLs to commands

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #61822 from rhcs-dashboard/smb-edit-cluster

mgr/dashboard: SMB - Edit Cluster

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #61815 from ljflores/wip-tracker-69456

qa/suites/upgrade: ignore when pg is stuck peering

Merge pull request #61703 from MaxKellermann/kstore_includes

os/kstore: add missing include

Merge pull request #61674 from connorfawcett/wip-ec-default-fixes

erasure-code: Set reed_sol_van to be the default technique for Jerasure if none is specified.

Merge pull request #61015 from nbalacha/wip-nbalacha-rbd-mirror-namespace-2

rbd-mirror: default <-> non-default namespace remapping

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #61269 from cbodley/wip-rgw-createbucket-layout

rgw/s3: CreateBucket extension for layout type and shard count

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #61982 from anthonyeleven/nvme-proxy

doc/rbd: Improve nvmeof-requirements.rst with proxy and other info

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60524 from thotz/multisitie-fix-cloud-restore

rgw/cloudrestore: sync cloud restore objects accoridngly

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #61849 from yuvalif/wip-yuval-69944

rgw: be explicit on moving rados op in APIs

Merge pull request #61376 from athanatos/sjust/wip-crimson-gcc-13

src/CMakeLists: require gcc13 for crimson, clarify error messages

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61846 from rhcs-dashboard/tox-generate-proto

mgr/dashboard: tox command to generate the nvmeof proto files

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #61903 from ronen-fr/wip-rf-crimson-conf

crimson/osd: replace obsolete get_tracked_conf_keys()

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62001 from cybozu/fix-stale-delete

doc: fix incorrect radosgw-admin subcommand

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60871 from leonidc/leonidc-epoch-filter

Epoch filtering

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Aviv Caro <Aviv.Caro@ibm.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

doc: fix incorrect radosgw-admin subcommand

Signed-off-by: Toshikuni Fukaya <toshikuni-fukaya@cybozu.co.jp>

Merge PR #61889 into main

* refs/pull/61889/head:
msg/async/Stack: include cleanup
msg/Message: include cleanup
msg/async/crypto_onwire: include cleanup
msg/async/ProtocolV1: add missing includes
msg/async/frames_v2: add missing includes
msg/msg_types: add missing includes

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #61888 into main

* refs/pull/61888/head:
os/memstore: add missing includes

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #61876 into main

* refs/pull/61876/head:
qa/cephfs: ignore warning that pg is stuck peering for upgrade jobs

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #61802 into main

* refs/pull/61802/head:
perfglue: add missing include

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #61801 into main

* refs/pull/61801/head:
kv: add missing includes

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #61799 into main

* refs/pull/61799/head:
erasure-code: add missing include

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #61798 into main

* refs/pull/61798/head:
crush: include cleanup

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #61955 from idryomov/wip-70075-test

qa/workunits/rbd: add a test for force promote with a user snapshot

Reviewed-by: N Balachandran <nibalach@redhat.com>

rgw: reject PutBucketLifecycleConfiguration on indexless buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: improved error when listing indexless bucket

without this change, ListObjects would attempt to list index shard
objects that don't exist and fail with:

> ERROR: S3 error: 404 (NoSuchKey)

after:

> ERROR: S3 error: 405 (MethodNotAllowed): Indexless buckets cannot be listed

Signed-off-by: Casey Bodley <cbodley@redhat.com>

examples/rgw: register boto extension for BucketIndex on CreateBucket

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/s3: CreateBucket extension for layout type and shard count

extend s3's CreateBucketConfiguration [1] with a custom BucketIndex element
that can override rgw's default bucket index type and shard count

to create an indexless bucket:
  <CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <BucketIndex>
      <Type>Indexless</Type>
    </BucketIndex>
  </CreateBucketConfiguration>

to create a normal pre-sharded bucket:
  <CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <BucketIndex>
      <Type>Normal</Type>
      <NumShards>1023</NumShards>
    </BucketIndex>
  </CreateBucketConfiguration>

[1] https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html#API_CreateBucket_RequestSyntax

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: create_bucket() can override index type and shards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: indexless buckets skip init_index/clean_index

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/s3: CreateBucket's LocationConstraint is optional

clean up the parsing and make the LocationConstraint optional

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61740 from Matan-B/wip-matanb-clang-16

script/run-make: clang 14->19

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #61319 from rhcs-dashboard/gklm-kmip

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #61619 from ronen-fr/wip-rf-more-keys

rgw: replace obsolete get_tracked_conf_keys()
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/dashboard: SMB - Edit Cluster
Fixes: https://tracker.ceph.com/issues/69964
Signed-off-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

qa: remove unreacheable/redundant code from test_data_scan.py

call to self.fs.journal_tool(["journal", "reset", "--force", "--yes-i-really-really-mean-it"], 0)
is already made below it, the same call under if False is unreachable.

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>