Adam King [Fri, 15 Sep 2023 13:44:58 +0000 (09:44 -0400)]
cephadm: start decorators.py in cephadmlib
Originally, wanted to move all the decorators into
their own files. Unfortunately, that isn't possible
at this time as most of them depend on things that
are still within cephadm.py This includes
and I'm sure I'm missing some others. We'll have to
revisit this again later when more of these things
have moved, or they can be slowly moved as their
dependencies are.
Rewrite the explanation of how a client authenticates against a monitor.
This is a rewrite of a single paragraph, and has been set apart in its
own PR so that it can receive the maximum amount of scrutiny that the
upstream Ceph community can muster.
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
Casey Bodley [Tue, 21 Mar 2023 18:21:26 +0000 (14:21 -0400)]
rgw/sal: StoreBucket no longer wraps RGWBucketEnt
`sal::Bucket` no longer needs to wrap `RGWBucketEnt` to support user
bucket listings, so can be represented by `RGWBucketInfo` alone. the
bucket stats interfaces that relied on RGWBucketEnt internally now
return their result as either `RGWBucketEnt` or `RGWStorageStats`
Casey Bodley [Tue, 21 Mar 2023 17:36:56 +0000 (13:36 -0400)]
rgw/sal: list_buckets() returns RGWBucketEnts
`sal::User::list_buckets()` no longer returns a map of `sal::Bucket`
handles. it now uses `std::span<RGWBucketEnt>` for input and output.
`RGWBucketEnt` contains all of the information we need to satisfy
ListBuckets requests, and also stores the `rgw_bucket` key for use with
`Driver::get_bucket()` where a `sal::Bucket` handle is necessary
`sal::BucketList` contains the span of results and the `next_marker`.
the `is_truncated` flag was removed in favor of `!next_marker.empty()`
the checks for `user->get_max_buckets()` on bucket creation now use a
paginated `check_user_max_buckets()` helper function that limits the
number of allocated entries to `rgw_list_buckets_max_chunk`
Matthew Vernon [Wed, 20 Sep 2023 15:37:17 +0000 (16:37 +0100)]
debian: call dh_python3 for ceph-{base,common,fuse,volume}
In the cases of ceph-base, ceph-common, and ceph-fuse, this picks up
that these packages contain python scripts and adds a necessary
python3 dependency. In the case of ceph-volume it additionally parses
the requirements.txt file.
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
Matthew Vernon [Wed, 20 Sep 2023 15:30:24 +0000 (16:30 +0100)]
debian: specify a dependency on python3 for cephadm
cephadm is a compressed zipapp, and dh3_python3 doesn't understand
this sort of binary file, so fails to produce the required python3
dependency. So specify this explicitly in debian/control
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
debian/ceph-base.docs only referred to a README that doesn't exist, so
remove it. Because dpkg-source doesn't reflect deletions from debian/
cf the orig.tar.gz, also remove the file in dh_auto_clean.
Then do away with the removal of the empty override of dh_installdocs;
the main benefit of which here is that debian/copyright gets installed
in all of the built packages, which otherwise lack a copyright
file.
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
Matthew Vernon [Wed, 20 Sep 2023 15:13:43 +0000 (16:13 +0100)]
debian: dh compat to 12, necessary init/systemd adjustments
Bring the dh compat level to 12, the most recent supported by the
oldest supported Ubuntu LTS release, 20.04. This necessitates changes
to how initscripts & systemd packaging are done.
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
Matthew Vernon [Mon, 11 Sep 2023 15:49:19 +0000 (16:49 +0100)]
debian: remove {Build-,}Depends on essential packages
Unless there's a version requirement (which there isn't here),
packages should not declare a Build-Depends: or Depends: relationship
on essential packages. Policy link:
Matthew Vernon [Mon, 11 Sep 2023 15:48:17 +0000 (16:48 +0100)]
debian: specify interpreters for ceph-mon and ceph-osd postinsts
These were previously missing. The requirement for interpreters is in
Debian policy section 10.4:
https://www.debian.org/doc/debian-policy/ch-files.html#s-scripts
Debian's packaging already adds the #! to these two postinsts. In
practice, a text executible without a #! line will likely be executed
by the calling shell, so a lot of the time we'd get away with it
unless the administrator is using an incompatible shell like tcsh.
This behaviour of shells is documented in POSIX section 1(e)(i)(b)
here:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
Matthew Vernon [Wed, 20 Sep 2023 16:43:30 +0000 (17:43 +0100)]
debian: Build-Depend on g++ 11 or greater
Rely on the packaging system to provide a suitable g++ of version 11
or greater, and removing the corresponding hard-coding from
debian/rules, since cmake will then find a suitable version. This
seems better than trying to hard-code a particular version in
debian/rules, and Debian package building tools like e.g. sbuild will
then do the right thing.
This enables Reef (v18.2.0) to build on Debian bookworm in a clean
chroot.
Fixes: https://tracker.ceph.com/issues/61845 Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
rgw: radosgw-admin bucket check should only print index entries with --check-objects flag
Printing all index entries can be very time consuming for large
buckets and the inability to switch this behavior off makes it
cumbersome to use the command for fixing bucket stats. This was
also preventing the command from outputting recalculated bucket
stats when the --fix flag wasn't specified.
other than the rgw/lifecycle subsuite, all of the other subsuites were just
adding storage classes to the rgw task, but not the s3tests task that
enables the storage class test cases
Casey Bodley [Thu, 29 Jun 2023 14:54:03 +0000 (10:54 -0400)]
qa/rgw: disable 'rgw lc debug interval' in most suites
this has a performance cost, and the teuthology environment can
already be sluggish. if lifecycle expiration/transition testing
was limited to the rgw/lifecycle and rgw/cloud-transition suites,
the others would probably run significantly faster
Casey Bodley [Thu, 29 Jun 2023 14:40:16 +0000 (10:40 -0400)]
qa/s3tests: use cloudtier_tests and lifecycle_tests to control filters
when `extra_attrs` was made additive, these tests were adding both
'not lifecycle_expiration' and 'lifecycle_expiration' to the command
line. use a yaml flag instead to control whether or not the s3tests task
adds the 'not' filter
qa/suites/rbd: disable POOL_APP_NOT_ENABLED health check
Commit 990806e635a1 ("mon, qa: issue pool application warning even
if pool is empty") made it impossible to create a pool without raising
a (bogus) health alert. See [1] for details.
rgw: prevent another leftover bucket index olh entry scenario
If a call to bucket_index_link_olh or bucket_index_unlink_instance
fails, its associated pending xattr may have prevented the olh object
from being removed by another thread. We should do a best effort
cleanup attempt for this case by calling update_olh before returning
an error to the caller.
Patrick Donnelly [Thu, 21 Sep 2023 15:51:31 +0000 (11:51 -0400)]
Merge PR #50503 into main
* refs/pull/50503/head:
mon: do not change pending if strategy is unchanged
mon/MonmapMonitor: do not propose on error in prepare_update
mon/MonmapMonitor: wait for commit before reply
mon: use wait_for_commit to reply
mon: add context list for commit wait
mon: remove unused method
test/mon: add commit benchmark script
mon/MonClient: provide config to target specific rank
Reviewed-by: Laura Flores <lflores@redhat.com> Reviewed-by: Ramana Raja <rraja@redhat.com>
John Mulligan [Tue, 12 Sep 2023 14:25:29 +0000 (10:25 -0400)]
cephadm: move CephContainer/similar to new container_types.py
Part of general cephadm split-up refactoring. I am not happy with the
name 'container_types' but none of the alternatives I could think
of were much better.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 20 Sep 2023 19:38:50 +0000 (15:38 -0400)]
cephadm: fix unit tests executing FileLock type
The FileLock type doesn't play much of a role when running tests so
to prevent issues, always mock it out when using with_cephadm_ctx.
In particular, a future patch revealed a problem with the FileLock code
that I can not understand how it was not hit before, or why this simple
refactoring - not directly related to file locking - triggered it. But
in short, the FakeFilesystem mocking utility only covers some syscalls.
In fact, the fake filesystem was returning an fd that was then passed to
real calls (fcntl and os.close). The latter then triggered issues when
pytest was trying to clean up after it applied it's magic to stdio
objects in sys. The fix is easy - understanding why it happens and how
was hard. I still don't understand why it popped up when it did only
that this is necessary to implement the following patches.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
osd/scrub: modify schedule_result_t to report error class
(which directly translates to the required followup action)
instead of reporting the exact failure. The specific of the failure
were never used by the scrub scheduler.
osd/scrub: correct placement for some scheduler-related methods
Moving some member functions to their corresponding files.
Including ScrubQueue::dump_scrubs()
as it was moved in a previous commit,
and some ScrubJob code.
projects / ceph.git / log