so we need to use different function signatures for talking to different
fmtlib versions.
please see
https://fmt.dev/8.1.0/api.html#_CPPv4I0Dp0EN3fmt5printEvRNSt13basic_ostreamI4CharEERK1SDpRR4Args,
and
https://fmt.dev/9.0.0/api#_CPPv4IDpEN3fmt5printEvRNSt7ostreamE13format_stringIDp1TEDpRR1T
Casey Bodley [Wed, 14 Dec 2022 19:59:58 +0000 (14:59 -0500)]
Merge pull request #49141 from cbodley/wip-58111
rgw: rgw_parse_url_bucket() rejects empty bucket names after 'tenant:'
Reviewed-by: Daniel Gryniewicz <dang@redhat.com> Reviewed-by: Adam C. Emerson <aemerson@redhat.com> Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@cern.ch>
Kefu Chai [Wed, 14 Dec 2022 11:10:54 +0000 (19:10 +0800)]
crimson/osd: include fmt/ranges.h
so we are able to format ss.clone_snaps, which is a
std::map<snapid_t, std::vector<snapid_t>>. before fmtlib v9,
fmtlib is able to fall back to the operator<<() to format the the types
with the operator<<() defined. but after fmtlib v9, we need to
explicitly define them for accessing the specialized formatter.
in this change, fmt/ranges.h is included so we can access the formatter
for std::map and std::vector.
Kefu Chai [Wed, 14 Dec 2022 11:09:49 +0000 (19:09 +0800)]
crimson/os: specialize fmt::formater<> for lba_pin_list_t
since fmt v9, fmt::formatter<> is not specialized for the types with
operator<<(ostream&, ...) anymore. so we need to specialize it manually.
in this change, fmt::formatter<lba_pin_list_t> is defined so
the tree can compile with fmt v9.
Kefu Chai [Wed, 14 Dec 2022 07:17:33 +0000 (15:17 +0800)]
crimson/os: specialize fmt::formater<> for types
since fmt v9, fmt::formatter<> is not specialized for the types with
operator<<(ostream&, ...) anymore. so we need to specialize it manually.
in this change, fmt::formatter<> is specialized for tree_stats_t and
test_item_t so the tree can compile with fmt v9.
Kefu Chai [Wed, 14 Dec 2022 00:32:44 +0000 (08:32 +0800)]
seastar: bump up seastar submodule to recent master
to address the fmtlib v9 compatibility issues.
please note, the submodule is bump up to a commit which includes
* a revert commit on top of upstream master HEAD. the revert is to
work around the a recent change which adds a constraint on the parameter
type of handle_exception(). we are using errorator along with
handle_exception(), but errorator cannot fulfill the requirement of
seastar::Future(), so we need to drop the change before we have a
better solution.
* a commit which defines __NR_pidfd_open if it is not defined.
this should address the FTBFS on CentOS8 where glibc does not have
this macro defined.
Kefu Chai [Wed, 14 Dec 2022 00:27:13 +0000 (08:27 +0800)]
test/cls_fio: specialize fmt::formater<> for bpo::options_description
since fmt v9, fmt::formatter<> is not specialized for the types with
operator<<(ostream&, ...) anymore. so we need to specialize it manually.
in this change, fmt::formatter<bpo::options_description> is defined so
the tree can compile with fmt v9.
Kefu Chai [Wed, 14 Dec 2022 00:20:42 +0000 (08:20 +0800)]
cls/fifo: specialize fmt::formater<> for rados::cls::fifo::info
since fmt v9, fmt::formatter<> is not specialized for the types with
operator<<(ostream&, ...) anymore. so we need to specialize it manually.
in this change, fmt::formatter<rados::cls::fifo::info> is defined so
the tree can compile with fmt v9.
Marcus Watts [Wed, 30 Nov 2022 23:22:30 +0000 (18:22 -0500)]
rgw: added rgw-policy-check
rgw-policy-check - a program to do syntax checking on bucket policy.
This program just reads the policy into memory, so it is not
checking anything except syntax.
Signed-off-by: Marcus Watts <mwatts@redhat.com>
rgw: Fix return value of `rgw-policy-check`
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
rgw: Use ceph initialization in `rgw-policy-check`
Specifically so we can pull in the options from `ceph.conf` and similar.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Kefu Chai [Tue, 13 Dec 2022 10:23:43 +0000 (18:23 +0800)]
crimson/os: define device_spec_t as packed
per C++ standard 20, alignas cannot be applied to scoped enum. despite
that there is a resolution to address this, see
https://cplusplus.github.io/CWG/issues/2354.html, it's not included in
C++20. so we have to use a different way to address
> runtime error: reference binding to misaligned address 0x610000008395
> for type 'device_type_t', which requires 4 byte alignment
otherwise the code would fail to compile with Clang-15, like:
> <source>:3:13: error: 'alignas' attribute cannot be applied to an enumeration
qa: Allow tests to override recovery configs with mClock scheduler enabled
Set osd_mclock_override_recovery_settings option to true for tests that
modify recovery/backfill configuration options. This prevents logging of
the cluster warning when modifying recovery/backfill limits.
Cory Snyder [Wed, 2 Nov 2022 20:06:23 +0000 (20:06 +0000)]
rgw: add 'inline_data' zone placement info option
Adds a new RGW zone placement info option to control whether
an object's first data chunk is placed in the head object.
This allows admins to make a tradeoff between optimizing for
PUT/GET performance vs. DELETE performance for some cluster
configurations.
qa/standalone: Add/Modify tests to verify mclock recovery/backfill limits
- Consolidate all mclock standalone tests under
qa/standalone/misc/mclock-config.sh.
- Revert existing tests in ceph-helpers.sh that verified the earlier hard
override of recovery/backfill limits.
- Add new tests to verify the procedure to change the recovery/backfill
limits with mclock scheduler.
doc: Update mClock config reference doc to reflect new max recovery limits
Document the following:
- New max backfill/recovery defaults for mClock.
- Steps to modify the backfill/recovery defaults.
- Modify defaults using new osd_mclock_override_recovery_settings option
- Steps to mitigate unrealistic OSD bench results to set OSD capacity.
- New capacity threshold options for ssd/hdd
With the earlier cost per io and cost per byte parameters, the
client ops performance was good but the recovery ops were taking
much longer to complete. The new cost parameters optimizes this
further so that the recovery ops progress faster while
maintaining similar client throughput.
This optimization is tested on both a small cluster and a larger
cluster in a scaled environment.
Ronen Friedman [Mon, 24 Jan 2022 13:19:01 +0000 (13:19 +0000)]
osd/scrub: report replicas slow to repond to scrub requests
Implemented timeouts:
1: Slow-Secondary Warning:
Once at least half of the replicas have accepted the reservation, we
start reporting any secondary that takes too long (more than <conf>
milliseconds after the previous response received) to respond to the reservation
request.
(Why? because we have encountered real-life situations where a specific
OSD was systematically very slow to respond (e.g. 5 seconds in one case) to
the reservation requests, slowing the scrub process to a crawl).
2: Reservation Process Timeout:
We now limit the total time the primary waits for the replicas to
respond to the reservation request. If we do not get all the responses
(either Grant or Reject) within <conf> milliseconds, we give up and release all the
reservations we have acquired so far.
(Why? because we have encountered instances where a reservation request
was lost - either due to a bug or due to a network issue.)
Kefu Chai [Sat, 10 Dec 2022 02:31:21 +0000 (10:31 +0800)]
rgw: drop rgw_obj_key::to_str()
rgw_obj_key::to_str() is mainly used by operator<<(ostream&, ..), so
we can just implement it with the specialization of
fmt::formatter<rgw_obj_key>. and let operator<<(ostream&, ..) to
call into fmt::format(..):
1. for better readability and
2. for probably better performance -- we don't need to do deep copy
for constructing a `std::string` from a `char[]`.
3. for better standard compliance -- we don't need to use variable-length
array in C++ code. it is a part of C99 standard. but not a C++ standard.
Kefu Chai [Thu, 8 Dec 2022 04:38:07 +0000 (12:38 +0800)]
rgw: define FMT_HEADER_ONLY using CMakeLists.txt
less repeatings this way. and the macro definition can be populated
to the targets linking against `rgw_common`. this is more maintainable,
as rgw executables and libraries always link against `rgw_common`.
Kefu Chai [Fri, 2 Dec 2022 06:10:02 +0000 (14:10 +0800)]
fmt: pickup v9.1.0
fmt 9.0.0 dropped automatic `std::ostream` insertion operator discovery
when `fmt/ostream.h` to prevent ODR violations. instead of defining
`FMT_DEPRECATED_OSTREAM`, we took efforts to specialize
`fmt::formatter<..>` to be compatible with the new fmtlib. to avoid
breaking the build with fmt v9 and up, let's bump up the fmt submodule.
Kefu Chai [Fri, 2 Dec 2022 09:25:24 +0000 (17:25 +0800)]
rgw: pass fmt::format_string to pretty_print()
otherwise we'd have following compiling failure:
In file included from /var/ssd/ceph/src/rgw/store/rados/rgw_data_sync.cc:13:
/var/ssd/ceph/src/rgw/store/rados/rgw_data_sync.h:345:28: error: call to consteval function 'fmt::basic_format_string<char, std::basic_string<char> &, rgw_obj_key &, unsigned long &, std::basic_string<char> &>::basic_format_string<char[59], 0>' is not a constant expressio
n
fmt::print(*env->ostr, std::forward<T>(t)...);
^
/var/ssd/ceph/src/rgw/store/rados/rgw_data_sync.cc:4136:8: note: in instantiation of function template specialization 'pretty_print<const char (&)[59], std::basic_string<char> &, rgw_obj_key &, unsigned long &, std::basic_string<char> &>' requested here
pretty_print(sc->env, "Syncing object s3://{}/{} version {} in sync from zone {}\n",
^
/var/ssd/ceph/src/rgw/store/rados/rgw_data_sync.cc:4070:3: note: in instantiation of member function 'RGWBucketSyncSingleEntryCR<rgw_obj_key, rgw_obj_key>::operate' requested here
RGWBucketSyncSingleEntryCR(RGWDataSyncCtx *_sc,
^
/var/ssd/ceph/src/rgw/store/rados/rgw_data_sync.cc:4352:27: note: in instantiation of member function 'RGWBucketSyncSingleEntryCR<rgw_obj_key, rgw_obj_key>::RGWBucketSyncSingleEntryCR' requested here
yield spawn(new SyncCR(sc, sync_pipe, entry->key,
^
/var/ssd/ceph/src/rgw/store/rados/rgw_data_sync.h:345:44: note: function parameter 't' with unknown value cannot be used in a constant expression
fmt::print(*env->ostr, std::forward<T>(t)...);
^
/var/ssd/ceph/src/rgw/store/rados/rgw_data_sync.h:343:53: note: declared here
void pretty_print(const RGWDataSyncEnv* env, T&& ...t) {
^
Kefu Chai [Fri, 2 Dec 2022 09:23:23 +0000 (17:23 +0800)]
rgw: specialize fmt::formatter<> for rgw_obj_key
since fmt v9, fmt::formatter<> is not specialized for the types with
operator<<(ostream&, ...) anymore. so we need to specialize it manually.
in this change, fmt::formatter<rgw_obj_key> is defined so the tree can
compile with fmt v9.
Casey Bodley [Mon, 24 Oct 2022 18:09:10 +0000 (14:09 -0400)]
rgw/main: add RGWProcessEnv in AppMain
instead of constructing a separate RGWProcessEnv for each frontend,
initialize a single instance in AppMain and share a reference with each
frontend
RGWRealmReloader now takes a reference to RGWProcessEnv instead of a
reference to the Store pointer itself, and updates RGWProcessEnv::store
on realm reload
because RGWRealmReloader may mutate RGWProcessEnv::store, we no longer have
a separate AppMain::store pointer, so don't worry about keeping that
consistent over reloads
projects / ceph.git / log