If we add a new Boost component, the install-deps.sh script will not install it
because other Boost libraries are installed with the desired version. Just
unconditionally install. The package manager is smart enough to skip already
installed packages.
Switch from EBUSY to reasonable and normal errors for operations on the root
directory. Use the Client::path_walk machinery changes from the previous commit
to catch most of them, otherwise return EINVAL (which is normal for other POSIX
fs).
Fixes: 6ed7f2364ae5507bab14c60b582929aa7b0ba400 Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit 292f5ac45b8190223e97b3f9ae5904605de7a888)
Patrick Donnelly [Thu, 14 Nov 2024 18:28:20 +0000 (13:28 -0500)]
client: refactor all path traversals through path_walk
This is groundwork to ensure that all code paths properly apply dentry name
transformations during traversal, specifically for casefolding.
Note some ancillary changes:
* readlink now correctly does lookup permission checks during path walk (previously a TODO).
* There has been some tweaking of the method signatures to accept an `const
InodeRef&`. This was convenient when passing walk_dentry_result::diri around
but the cascade in changes prompted me to leave it partially done to avoid
blowing up this changeset.
It's worth noting that this change is somewhat "half done". I had made an
effort to have all operations (whether fuse or libcephfs) call path_walk once.
This was done for Client::_mkdir and a few others. See for example:
- do_mkdirat -> _mkdir
- mksnap -> _mkdir
- ll_mkdir -> _mkdir
- mkdirs -> _mkdir (this had other significant simplification/cleanup);
however: note that we still have two calls to path_walk for any given
directory to be created. This is not a problem because:
Client::path_walk is now more tolerant of being called in a openat-style way.
The method now accepts an anchor directory and relative path (which may be a
single dentry name for ll_* calls). It is also tolerant of the directory inode
in fact referencing the target inode with relative path == "". That is a useful
property for many openat-style APIs (although largely unofficially). It also
means that if we resolve a path then we can pass the resolved Inode (directory)
/ string (Dentry name) pair to another method which may also call path_walk
with minimal replication of path walk work. This is done a few times in this
changeset.
Fixes: https://tracker.ceph.com/issues/66373 Fixes: https://tracker.ceph.com/issues/70100 Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit b8662fdb03373ed6d29c47efa94cd391a7c9e740)
Conflicts:
src/client/Client.cc: minor code differences
Patrick Donnelly [Wed, 19 Feb 2025 17:49:09 +0000 (12:49 -0500)]
client: init dentry shared_gen with invalid value
Directories and dentries are initialized with value 0 which makes detecting a
null (or placeholder) dentry created via Client::get_or_create difficult. We
already do checks to see if a dentry is invalid when the directory's shared_gen
changes so use an invalid value for these synthetic dentries until they can be
appropriately updated from Client::update_dentry_lease.
Patrick Donnelly [Tue, 10 Sep 2024 20:47:39 +0000 (16:47 -0400)]
client: hide alternate_name from API
The dentry alternate_name was exposed primarily to facilitate testing. Instead,
put these methods in the TestClient scaffolding to allow reading/manipulating
alternate_name.
Because we will be using alternate_name to handle encrypted names and the
normalized / case folded name, we do not want the application to change the
meaning of the metadata out-of-band.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> Fixes: https://tracker.ceph.com/issues/66373
(cherry picked from commit 1e34963af11210ebd940e20219728b91031e720f)
John Mulligan [Thu, 19 Sep 2024 00:44:39 +0000 (17:44 -0700)]
python-common: add a utils function to replace distutils.util.strtobool
As distutils is removed from python 3.12 ceph can no longer use the
simple conversion function once located in that module. Add our own
trivial replacement function.
Patrick Donnelly [Mon, 17 Mar 2025 19:20:20 +0000 (15:20 -0400)]
Merge PR #61994 into squid
* refs/pull/61994/head:
release notes: removed lines added by automatic merge rule using 'union'
mds: use system error codes instead of CEPHFS_EXXX codes
mds,client: change result field handling in the MClientReply message
mds: prepare infra to replace CEPHFS_EXXX codes to system codes
pybind/cephfs: switch CEPHFS_E error codes to system error codes
PendingReleaseNotes: add note on client/mds error codes
Patrick Donnelly [Thu, 13 Mar 2025 16:32:36 +0000 (12:32 -0400)]
Merge PR #57675 into squid
* refs/pull/57675/head:
doc: document new --output-file switch
test/cli: ignore tmp_file_template
qa/workunits: add --output-file test in cephtool workunit
common,ceph: add output file switch to dump json to
common/options: add configs for temporary files made by daemons
common/Formatter: write the pending string on flush
This test deals with enabling/disabling the modules. The assumption I
have is after enabling the
module test will wait for an active mgr but its not able to find it in
time and it fails. so taking inspiration from https://github.com/ceph/ceph/pull/58995/commits/6c7253be6f6fbfa6faed7a539cb78847fec04580 adding retries and logs to see if that's the case
Aashish Sharma [Thu, 28 Nov 2024 05:58:59 +0000 (11:28 +0530)]
mgr/dashboard: Add ceph_daemon filter to rgw overview grafana panel
queries
Currently rgw_servers filtering is not working in RGW Overview garfana graphs.
It is showing data of all the RGW services, even though filter set to single service.
This PR intends to solve this issue
Hannes Baum [Wed, 6 Nov 2024 08:46:09 +0000 (09:46 +0100)]
mgr: fix subuser creation via dashboard
Subusers couldn't be created through the dashboard, because the get call was overwritten with Python magic due to it being the function under the HTTP call.
The get function was therefore split into an "external" and "internal" function, whereas one
can be used by functions without triggering the magic. Since the user object was then returned correctly, json.loads could be removed.
ceph-volume: allow zapping partitions on multipath devices
ceph-volume refuses to zap a device if it is a partition on a multipath
device due to an overly strict condition. This change ensures that only
full mapper devices (excluding partitions) are blocked from being zapped,
allowing partitions on multipath devices to be processed correctly.
John Mulligan [Fri, 14 Feb 2025 19:51:03 +0000 (14:51 -0500)]
doc: document the new container build tool and link to it in README
Add a new markdown file in the root of the tree, ContainerBuild.md, that
can serve as a basic introduction to the new container build tools
recently merged to ceph.
Add a small 'breadcrumb' section to the project README.md to help find
this new document.
John Mulligan [Thu, 20 Feb 2025 00:17:30 +0000 (19:17 -0500)]
script/build-with-container: add support for overlay dir
The source dir (aka homedir, default /ceph) is mounted in the container
read-write. This is needed as the various ceph build scripts expect to
write things into the tree - often this is in the build directory - but
not always. This can lead to small messes and/or situations that are
confusing to debug, especially if one is jumping between distros often.
Add an option to use an overlay volume for the homedir - by default we
enable a persistent overlay with a supplied "upper dir" where files that
were written will appear. One can also enable a temporary overlay that
forgets the writes when the container exits - maybe useful when doing
experiments in 'interactive' mode.
To use this option run the command with the `--overlay=<dir>` option.
For example: `./src/script/build-with-container.py -b build.inner
--overlay-dir build.ovr`. This will create a directory
`build.ovr/content` automatically and all new files will appear there.
For example the build directory will appear at
`build.ovr/content/build.inner`.
To use the temporary overlay use a `-` as the directory name. For
example: `./src/script/build-with-container.py -b build.inner
--overlay-dir -`
John Mulligan [Thu, 20 Feb 2025 14:50:49 +0000 (09:50 -0500)]
script/build-with-container: skip dnf cache dir volume mounts on docker
When using docker the --volume option is not available during build
(docker [buildx] build), unlike podman. Since passing these volumes must
be conditional on them being set up I see no way to handle this short of
just disabling the option on docker. Log the fact that it's being
skipped - the only other issue is that we pointlessly set up some dirs
and the build may be a bit slower.
John Mulligan [Wed, 19 Feb 2025 18:20:36 +0000 (13:20 -0500)]
script/build-with-container: remove default --volume arg from ctr build
On the original github pr #59841 user fayak kindly informed us that the
--volume option was not supported by docker build. Since this section
was a leftover from a previous way of constructing the builder image and
was no longer needed we simply removed it.
John Mulligan [Wed, 19 Feb 2025 18:20:01 +0000 (13:20 -0500)]
script/build-with-container.py: build builder image with --pull=always
Construct the builder image using the --pull=always flag to initiate a
pull of the base image (centos, ubuntu, etc) in order to avoid using a
stale base image. Since the script automatically (by default) avoids
building if a matching tag is in local container storage it is handy to
use a fresh base when it *is* time to build something. Otherwise, you
end up in a situation like I sometimes do - using a months old base
unintentionally.
John Mulligan [Fri, 14 Feb 2025 19:50:42 +0000 (14:50 -0500)]
script/build-with-container: add a common packages target
Add a `packages` target to build-with-container.py that requests a build
of packages, whatever package type is native to the distro selected.
For example `./src/script/build-with-container.py -d ubuntu22.04 -e
packages` will automatically select a deb packages build where
`./src/script/build-with-container.py -d centos9 -e packages` will
trigger rpm packages to be built. The underlying package-type specific
targets remain unchanged.
John Mulligan [Fri, 14 Feb 2025 16:44:35 +0000 (11:44 -0500)]
script/build-with-container: support custom tag suffixes
Previously, one could use the `--tag` option to completely override the
container tag generated by the script. However, there are cases where
one may want to add information to the tag rather than override it.
Allow the tag value to start with a plus (+) character that indicates
that the remainder of the string is to be suffixed to the generated tag.
Add a command line option --base-branch that allows the user to supply a
custom base branch name. git doesn't make determining this easy so we
always assume a base branch of 'main' by default - but this option lets
one change that.
John Mulligan [Fri, 14 Feb 2025 16:24:29 +0000 (11:24 -0500)]
src/script: rename CEPH_BRANCH to CEPH_BASE_BRANCH for build container
Previously, we were passing build argument of CEPH_BRANCH, but that was
a bit misleading as we expect the current branch to vary a bit (as users
will be using branches to develop and test the code). What we actually
care about is the base branch ('main', 'squid', etc) as that is fed into
our bootstrap script and we want the option to simple variations based
on the name of said base branch.
Rename CEPH_BRANCH to CEPH_BASE_BRANCH for clarity.
Add a new --current-branch argument that lets the user supply a name for
the current branch. This allows the automatic tag generation to avoid
calling git - something useful if the tree is not using a git checkout
(like a tarball). It also allows you to pull a temporary branch in git
but ignore it and act like the temporary branch is the base branch.
John Mulligan [Tue, 11 Feb 2025 23:36:13 +0000 (18:36 -0500)]
script/build-with-container: add more distro aliases
Add a system to define distro name aliases and use that to define some
additional aliases, primarily to match ubuntu codenames rather than
version numbers. Requested by Zack.
projects / ceph.git / log