Jos Collin [Fri, 7 Nov 2025 12:07:07 +0000 (17:37 +0530)]
Merge PR #65287 into wip-jcollin-testing-20251107.120649-squid
* refs/pull/65287/head:
client: clamp sizes to INT_MAX in sync i/o code paths
client: restrict bufferlist to total write size
src/test: test sync/async i/o code paths with huge (4GiB) buffers
Jos Collin [Fri, 7 Nov 2025 12:07:03 +0000 (17:37 +0530)]
Merge PR #65827 into wip-jcollin-testing-20251107.120649-squid
* refs/pull/65827/head:
pybind/cephfs: fix including of platform_errno.h
pybind: convert ceph errno to host-based errno
src/include: move ceph_to_hostos_errno() to separate header file
qa: set -x for qa/workunits/libcephfs/test.sh
Ville Ojamo [Sat, 25 Oct 2025 08:18:09 +0000 (15:18 +0700)]
doc: Pin pip to <25.3 for RTD as a workaround for pybind
Readthedocs now uses pip 25.3 by default which requires PEP 517.
src/pybind/* does not provide pyproject.toml files for PEP 517.
For an immediate workaround to allow RTD builds to succeed, pin pip
version to earlier than 25.3.
Details for pybind in https://tracker.ceph.com/issues/73645
client: prohibit unprivileged users from setting sgid/suid bits
Prior to fb1b72d, unprivileged users could add mode bits as long as
S_ISUID and S_ISGID were not included in the change.
After fb1b72d, unprivileged users were allowed to modify S_ISUID and
S_ISGID bits only when no other mode bits were changed in the same
operation. This inadvertently permitted unprivileged users to set
S_ISUID and/or S_ISGID bits when they were the sole bits being modified.
This behavior should not be allowed. Unprivileged users should be
prohibited from setting S_ISUID and/or S_ISGID bits under any
circumstances.
This change tightens the permission check to prevent unprivileged
users from setting these privileged bits in all cases.
Aashish Sharma [Mon, 25 Nov 2024 06:37:23 +0000 (12:07 +0530)]
mgr/dashboard: Use a persistent volume to store Loki DB
Current Behavior: The Loki configuration directs all data to /tmp/loki, which is not mounted as a persistent volume. This results in data being stored in a temporary location that gets cleared upon container restart.
Expected Behavior: Loki data should be stored in /loki, which is backed by the persistent volume mapped to /var/lib/ceph/<fsid>/loki.<id>/data, ensuring data persistence across container restarts.
Dan Mick [Sun, 19 Oct 2025 00:45:31 +0000 (17:45 -0700)]
install-deps.sh: install proper compiler version on Debian/Ubuntu
This code used to run in a pbuilder hook (because it needed to run
inside the build environment chroot). When building in a container,
you also want the right compiler installed.
This is necessary at least to build reef on ubuntu focal.
John Mulligan [Mon, 20 Oct 2025 19:04:49 +0000 (15:04 -0400)]
script/build-with-container: optionally source WITH_CRIMSON from env file
Add support for optionally sourcing WITH_CRIMSON from the env file that
can be passed to BWC on the command line. When auto-detecting the
crimson variant we previously only looked at the BWC processes
environment. After speaking with Zack we determined that the Jenkinsfile
only writes the WITH_CRIMSON param into the env file, so we add support
to "peek" in the env file for the WITH_CRIMSON variable.
John Mulligan [Sat, 18 Oct 2025 00:05:09 +0000 (20:05 -0400)]
script/build-with-container: add more detailed variants
Create two new variants 'packages.minimal' or 'packages.crimson'.
The first disables test deps (make check) and crimson deps.
The second only disables test deps and explicitly enables crimson deps.
The existing 'packages' variant now tries to determine if it should
switch to 'packages.minimal' or 'packages.crimson' by checking for
the same env vars install-deps.sh was (WITH_CRIMSON).
John Mulligan [Thu, 2 Oct 2025 17:56:28 +0000 (13:56 -0400)]
Dockerfile.build: improve docker compatibility
Try to fix:
```
Step 6/18 : COPY ceph.spec.in do_cmake.sh install-deps.sh run-make-check.sh src/script/buildcontainer-setup.sh ${CEPH_CTR_SRC}
When using COPY with more than one source file, the destination must be a directory and end with a /
```
Allow the user to control the content of the build image with a
high-level `--image-variant=` switch. Currently the supported values are
`default` (the same maximal image we have been generating) and
`packages` a slimmer image that avoids installing certain test-only
dependencies.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 13 Oct 2025 20:23:10 +0000 (16:23 -0400)]
install-deps.sh: let FOR_MAKE_CHECK variable take precedence
Previously, the FOR_MAKE_CHECK variable could only enable installing
extra (test) dependencies when install-deps.sh was used and it was
ignored if `tty -s` exited true. This change allows FOR_MAKE_CHECK to
take precedence over the tty check and to specify one of true, 1, yes to
enable extra "for make check" deps or false, 0, no to explicitly disable
the extra deps.
Based-on-work-by: Dan Mick <dan.mick@redhat.com> Signed-off-by: John Mulligan <jmulligan@redhat.com>
Abhishek Desai [Thu, 9 Oct 2025 07:49:34 +0000 (13:19 +0530)]
mgr/dashboard : Fixed usage bar for secondary site in rbd mirroing
fixes : https://tracker.ceph.com/issues/73447 Signed-off-by: Abhishek Desai <abhishek.desai1@ibm.com>
(cherry picked from commit 60140b1ccc8006325632320e39fc209724524aef)
In linux, systems calls like write() anyways don't allow writes > 2GiB,
the total write size passed to the Client::_write is clamped to INT_MAX
but bufferlist is not handled. This edge case is patched here.
bug that arises due to buffer list beyond INT_MAX stalls async i/o due to:
unknown file: Failure
C++ exception with description "End of buffer [buffer:2]" thrown in the test body.
2024-05-28T16:17:06.854+0530 7f9a5d24c9c0 2 client.4311 unmount
2024-05-28T16:17:06.854+0530 7f9a5d24c9c0 2 client.4311 unmounting
which results in disconnected inode and cap leaks:
2024-05-28T16:17:11.855+0530 7f9a5d24c9c0 1 client.4311 dump_inode: DISCONNECTED inode 0x10000000001 #0x10000000001 ref 3 0x10000000001.head(faked_ino=0 nref=3 ll_ref=0 cap_refs={4=0,1024=1,4096=1,8192=2} open={3=0} mode=100666 size=0/4294967296 nlink=1 btime=2024-05-28T16:17:03.387546+0530 mtime=2024-05-28T16:17:03.387546+0530 ctime=2024-05-28T16:17:03.387546+0530 change_attr=0 caps=pAsxLsXsxFsxcrwb(0=pAsxLsXsxFsxcrwb) objectset[0x10000000001 ts 0/0 objects 1 dirty_or_tx 0] 0x7f9a2c009530)
2024-05-28T16:17:11.855+0530 7f9a5d24c9c0 2 client.4311 cache still has 0+1 items, waiting (for caps to release?)
This commit changes the way Client::_write accepts data. So, now instead of accepting ptr,
iovec array and iovcnt, the helper now accepts a bufferlist which should be contructed by
the caller itself. The reason behind this change is simple - to declutter the API.
For more context checkout this conversation https://github.com/ceph/ceph/pull/58564#discussion_r2000226752
John Mulligan [Wed, 8 Oct 2025 20:41:36 +0000 (16:41 -0400)]
script/build-with-container: improve error handling for invalid distros
Instead of throwing a long obnoxious traceback at the user if the value
supplied to -d/--distro is invalid do something nicer. For example:
```
$ ./src/script/build-with-container.py -d trixy -e build
usage: build-with-container.py [-h] [--help-build-steps]
build-with-container.py: error: argument --distro/-d: unknown distro: 'trixy' not in centos10, centos10stream, centos8, centos9, centos9stream, rocky9, rockylinux9, rocky10, rockylinux10, fedora41, fc41, fedora42, fc42, fedora43, fc43, ubuntu20.04, ubuntu-focal, focal, ubuntu22.04, ubuntu-jammy, jammy, ubuntu24.04, ubuntu-noble, noble, debian12, debian-bookworm, bookworm, debian13, debian-trixie, trixie
John Mulligan [Wed, 8 Oct 2025 14:23:25 +0000 (10:23 -0400)]
script/build-with-container: be consistent with naming in distro kinds
Update the DistroKind enum and related items so that the naming is
applied consistently. That is: the canonical (no pun indented) form
of the name is "<name><version>" and codenames, such as "jammy" or
"bookworm" are aliases. This matches the previously existing code.
John Mulligan [Thu, 28 Aug 2025 23:39:06 +0000 (19:39 -0400)]
build-with-container: ensure npm dir is set up before configure
When the npm cache path option is passed the npm cache dir is passed
to all container `run` commands, ensure the dir has been created
before the first container command (configure) is used.
John Mulligan [Sat, 15 Mar 2025 16:44:00 +0000 (12:44 -0400)]
install-deps: extract SUDO variable logic into a reusable function
While the function is pretty simple and could be copy-pasted I
prefer to extract things into functions to indicate that the
logic is used/repeated elsewhere to ward off making changes to
one copy vs the other.
Rishabh Dave [Wed, 20 Aug 2025 07:41:04 +0000 (13:11 +0530)]
src/include: move ceph_to_hostos_errno() to separate header file
Including src/include/types.h in src/pybind/cephfs/types.pxd leads to
compilation error: "fatal error: acconfig.h: No such file or directory".
types.h as well as int_types.h include acconfig.h header file.
Move the code to be included in types.pxd to a separate file where
acconfig.h won't be included, thus preventing this error.
Rishabh Dave [Sun, 31 Aug 2025 18:50:19 +0000 (00:20 +0530)]
qa: set -x for qa/workunits/libcephfs/test.sh
LibCephFS unit tests are compiled into different binary files and run
after another but without logging name of the binary being executed,
which can make it bit difficult to find out which binary/test group is
being run. Therefore "set -x" in the script so that binary name/test
group is printed before tests run.
Rishabh Dave [Wed, 2 Apr 2025 15:25:32 +0000 (20:55 +0530)]
mgr/vol: add command to get snapshot path
Fixes: https://tracker.ceph.com/issues/70815 Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 50d28992d99fcd67390815aa42f9da8ffaa82575)
Conflicts:
src/pybind/mgr/volumes/fs/volume.py
- Line where the original patch makes the change is slightly different
in main compared to Squid branch, leading to conflict.
1. Fixes the promql expr used to calculate "In" OSDs in
ceph-cluster-advanced.json.
2. Fixes the color coding for the single state panels used in the OSDs
grafana panel like "In", "Out" etc
according to `dpkg-buildflags`, ubuntu 24 raised this value to
`-D_FORTIFY_SOURCE=3` which causes `error: "_FORTIFY_SOURCE" redefined`
compilation failures because Ceph itself adds `-D_FORTIFY_SOURCE=2`
`_FORTIFY_SOURCE` is a hardening option. both our rpm and debian builds
already specify that via environment variables, so Ceph's cmake should
leave it alone
Anoop C S [Mon, 23 Sep 2024 07:06:55 +0000 (12:36 +0530)]
client: Gracefully handle empty pathname for statxat()
man statx(2)[1] says the following:
. . .
AT_EMPTY_PATH
If pathname is an empty string, operate on the file referred to by
dirfd (which may have been obtained using the open(2) O_PATH flag).
In this case, dirfd can refer to any type of file, not just a
directory.
If dirfd is AT_FDCWD, the call operates on the current working
directory.
. . .
Look out for an empty pathname and use the relative fd's inode in the
presence of AT_EMPTY_PATH flag before calling internal _getattr().
Fixes: https://tracker.ceph.com/issues/68189
Review with: git show -w
Anoop C S [Thu, 17 Oct 2024 16:15:17 +0000 (21:45 +0530)]
libcephfs.h: Fix API documentation for ceph_statxat
flags parameter for ceph_statxat() API is supposed to accept only
AT_STATX_DONT_SYNC and AT_SYMLINK_NOFOLLOW. Modify the corresponding
documentation to reflect the acceptance of above two flags.
Anoop C S [Fri, 20 Sep 2024 08:49:01 +0000 (14:19 +0530)]
client: Gracefully handle empty pathname for chownat()
man fchownat(2)[1] says the following:
. . .
AT_EMPTY_PATH (since Linux 2.6.39)
If pathname is an empty string, operate on the file referred to by
dirfd (which may have been obtained using the open(2) O_PATH flag).
In this case, dirfd can refer to any type of file, not just a
directory. If dirfd is AT_FDCWD, the call operates on the current
working directory.
. . .
Look out for an empty pathname and use the relative fd's inode in the
presence of AT_EMPTY_PATH flag before calling internal _setattr().
Fixes: https://tracker.ceph.com/issues/68189
Review with: git show -w
projects / ceph-ci.git / log