Laura Flores [Mon, 4 Mar 2024 15:28:47 +0000 (09:28 -0600)]
Merge pull request #55766 from ceph/pacific-release
Ceph Release Team [Mon, 26 Feb 2024 19:21:09 +0000 (19:21 +0000)]
16.2.15
Signed-off-by: Ceph Release Team <ceph-maintainers@ceph.io>
Laura Flores [Mon, 26 Feb 2024 18:39:32 +0000 (12:39 -0600)]
Merge pull request #55717 from kamoltat/wip-ksirivad-pacific-release-whitelist
pacific: qa/suites: added more whitelisting + fix typo
Kamoltat [Thu, 22 Feb 2024 16:55:26 +0000 (16:55 +0000)]
qa/suites: added more whitelisting + fix typo
Problem:
1. Not enough whitelisting for certain Cephadm failures
2. previous PR that landed has a typo that
causes https://tracker.ceph.com/issues/64452
Solution:
1. Add more whitelisting
2. Fix typo in https://tracker.ceph.com/issues/64452
Fixes: https://tracker.ceph.com/issues/64452
Signed-off-by: Kamoltat <ksirivad@redhat.com>
Yuri Weinstein [Tue, 20 Feb 2024 15:52:42 +0000 (07:52 -0800)]
Merge pull request #55658 from guits/cv-fix-zap-partitions
ceph-volume: fix zap_partitions() in devices.lvm.zap
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Guillaume Abrioux [Tue, 30 Jan 2024 14:17:35 +0000 (14:17 +0000)]
ceph-volume: fix zap_partitions() in devices.lvm.zap
The current logic is unable to find the holders for the
partition being zapped.
This commit fixes this issue.
Fixes: https://tracker.ceph.com/issues/64248
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
Laura Flores [Fri, 16 Feb 2024 22:23:11 +0000 (16:23 -0600)]
Merge pull request #55523 from ljflores/wip-tracker-64343-pacific
pacific: qa/suites/orch: whitelist warnings that are expected in test environments
Laura Flores [Fri, 9 Feb 2024 21:20:26 +0000 (15:20 -0600)]
qa/suites/orch: whitelist warnings that are expected in test environments
Semi-backport of
00fc7960f682a16aa7a02aeb0300780a1bf56476. Some changes
had to be made though for yaml files and warnings that are specific to pacific.
Fixes: https://tracker.ceph.com/issues/64343
Signed-off-by: Laura Flores <lflores@ibm.com>
Laura Flores [Thu, 15 Feb 2024 20:43:09 +0000 (14:43 -0600)]
Merge pull request #55500 from cbodley/wip-64362-pacific
pacific: cmake/modules/BuildRocksDB.cmake: inherit parent's CMAKE_CXX_FLAGS
Yuri Weinstein [Thu, 15 Feb 2024 15:10:38 +0000 (07:10 -0800)]
Merge pull request #55586 from cbodley/wip-64427-pacific
pacific: rgw/putobj: RadosWriter uses part head object for multipart parts
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Casey Bodley [Wed, 14 Feb 2024 14:43:14 +0000 (09:43 -0500)]
rgw/putobj: RadosWriter uses part head object for multipart parts
the cleanup logic in the RadosWrite destructor was using the wrong
`head_obj` to avoid races between cleanup and part re-uploads. it
pointed at the final location of the multipart upload, rather than the
head object of the current part
Fixes: https://tracker.ceph.com/issues/63642
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit
bbbf5c54bdce3061ae9831c49fe065311a2e156d)
Conflicts:
rgw_putobj_processor.* uses sal::RGWObject instead of rgw_obj
Yuri Weinstein [Tue, 13 Feb 2024 16:33:36 +0000 (08:33 -0800)]
Merge pull request #55550 from cbodley/wip-64404-pacific
pacific: rgw/auth: ignoring signatures for HTTP OPTIONS calls
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Tobias Urdin [Tue, 6 Feb 2024 07:50:55 +0000 (07:50 +0000)]
rgw/auth: ignoring signatures for HTTP OPTIONS calls
Before [1] we always sent all HTTP OPTIONS requests to
the S3AnonymousEngine and ignored any provided AWSv4
credentials sent in the request.
That PR changed so that if we got credentials in the
request we instead sent it through the authentication
code in order to solve HTTP OPTIONS requests on tenanted
users to start working (because we need to resolve the
tenant, also called bucket tenant in the code, and we can't
only rely on the bucket name since it will not be found).
We solved this by modifying the canonical HTTP method used
when calculating the AWSv4 signature by instead using the
access-control-request-method header which worked good.
This change did not take into account that when you generated
a presigned URL for a put_object request you can also pass in
extra parameters like a canned ACL [2] to the Params variable
in for example boto3's generated_presigned_url().
Doing that will cause the client to add the x-amz-acl header
to x-amz-signedheaders and also use that in their signature
calculation.
When doing a HTTP OPTIONS calls for CORS on that presigned URL
the browser will never send a x-amz-acl header with the correct
data since that is something that the actual PUT request should
include later, so that HTTP OPTIONS call should pass even though
the signature can never be calculated correctly server-side like
verified against AWS S3 in tracker [3].
This patch as a result skips the signature calculation when doing
EC2 auth using the LocalEngine but we still need to pass the request
there in order to lookup the user to support buckets in a tenant.
For the Keystone EC2 auth we're pretty out of luck in the sense that
Keystone's API itself requires us to send the AWSv4 signature in the
request with the access_key in order to obtain a token, and we cannot
leave the signature out, we also cannot spoof the signature from
rgw -> keystone since we don't have access to the secret_key if it's
not in our cache.
For that approach we simply pass on to get_access_token() that if it's
an HTTP OPTIONS and we find the access_key in the cache we pull that
and ignore verifying signature and pass it on for validation. This means
that the cache must be warm if using Keystone auth and adding extra
params to a presigned URL.
This partly makes some of the commits in [1] redundant for EC2
LocalEngine auth but we still need it for tenanted bucket support.
[1] https://github.com/ceph/ceph/pull/52673
[2] https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl
[3] https://tracker.ceph.com/issues/64308
Fixes: https://tracker.ceph.com/issues/64308
Signed-off-by: Tobias Urdin <tobias.urdin@binero.se>
(cherry picked from commit
fe15b52edb5d228d2ed56679c62cf48493ae2d54)
Conflicts: missing 'rgw/keystone: use secret key from EC2 for sigv4 streaming mode'
src/rgw/rgw_auth_keystone.cc
src/rgw/rgw_auth_keystone.h
Yuri Weinstein [Mon, 12 Feb 2024 15:48:26 +0000 (07:48 -0800)]
Merge pull request #55472 from Matan-B/wip-matanb-mon-health-trim
pacific: mon: fix health store size growing infinitely
Reviewed-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
Yuri Weinstein [Mon, 12 Feb 2024 15:46:36 +0000 (07:46 -0800)]
Merge pull request #55443 from YiteGu/msgr-worker-create-backport
pacific: msg/async: initialize worker in RDMAStack::create_worker() and drop Stack::num_workers
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>
Guillaume Abrioux [Mon, 12 Feb 2024 08:21:45 +0000 (09:21 +0100)]
Merge pull request #55481 from guits/wip-64338-pacific
pacific: ceph-volume: fix zap_partitions() in devices.lvm.zap
Yuri Weinstein [Fri, 9 Feb 2024 22:25:48 +0000 (14:25 -0800)]
Merge pull request #55519 from cbodley/wip-52590
pacific: qa/upgrade: disable a failing ceph_test_cls_cmpomap test case
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Ilya Dryomov [Fri, 9 Feb 2024 20:00:41 +0000 (21:00 +0100)]
Merge pull request #55418 from ceph/wip-yuriw-disable-rbd-upgarde-p2p-pacific
pacific: qa/suites/upgrade/pacific-p2p: run librbd python API tests from pacific tip
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Casey Bodley [Fri, 9 Feb 2024 15:26:38 +0000 (10:26 -0500)]
qa/upgrade: disable a failing ceph_test_cls_cmpomap test case
Fixes: https://tracker.ceph.com/issues/52590
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Yuri Weinstein [Thu, 1 Feb 2024 23:34:35 +0000 (15:34 -0800)]
qa/suites/upgrade/pacific-p2p: run librbd python API tests from pacific tip
This job installs librbd from v16.2.7, upgrades librbd to latest
pacific and then runs librbd python API tests from v16.2.7 against the
upgraded librbd. This isn't expected to work in the general case
and is currently failing in TestImage.test_diff_iterate test because
it was adjusted to do the right thing in commit
f8ced6d1fe66
("test/pybind/rbd: don't ignore from_snapshot in check_diff()").
Co-authored-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Kefu Chai [Fri, 15 Dec 2023 11:03:57 +0000 (19:03 +0800)]
cmake/modules/BuildRocksDB.cmake: inherit parent's CMAKE_CXX_FLAGS
CMake allows us to customize `CMAKE_CXX_FLAGS` by setting CXXFLAGS
environmental variable. and Debian's debhelper also sets CXXFLAGS
when it builds cmake projects for customizing the building flags.
but we fail to populate this setting down when building external
projects. this is important when it comes to the projects which
is critical to the performance. RocksDB is one of them.
in this change, we pass the `CMAKE_CXX_FLAGS` down in
`BuildRocksDB.cmake` so that its `CMAKE_CXX_FLAGS` contains
the same set of `CMAKE_CXX_FLAGS` used by its parent project.
this should help with the performance in the bluestore, where
RocksDB is used.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit
beb1a624921d7589db63dea066935b3aa9ce2698)
Kefu Chai [Fri, 15 Dec 2023 11:01:46 +0000 (19:01 +0800)]
cmake/modules/BuildRocksDB.cmake: use string(APPEND ..) when appropriate
less repeatings this way.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit
620b68a348f07145c49c12668576a89dee8198cb)
Adam King [Thu, 8 Feb 2024 12:07:09 +0000 (07:07 -0500)]
Merge pull request #55429 from adk3798/pacific-mcltf-true
pacific: qa/tasks/cephadm: enable mon_cluster_log_to_file
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Guillaume Abrioux [Fri, 26 Jan 2024 20:35:18 +0000 (21:35 +0100)]
ceph-volume: fix partitions support in disk.get_devices()
The following:
```
is_part = get_file_contents(os.path.join(_sys_dev_block_path, item, 'partition')) == "1"
```
assumes any `/sys/dev/block/x:y/partition` contains '1' which is wrong.
This file actually contains the corresponding partition number.
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit
f72100bbd17539d9774ae72215afefee16f20775)
Guillaume Abrioux [Tue, 3 Oct 2023 23:11:32 +0000 (01:11 +0200)]
ceph-volume: fix util.get_partitions
The current logic makes it report only the first
partitions of devices.
Fixes: https://tracker.ceph.com/issues/63086
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit
b14ff07e6344d9f097259265d468f6300818b053)
Wei Wang [Mon, 29 Jan 2024 08:26:24 +0000 (08:26 +0000)]
mon: fix health store size growing infinitely
The `check_mutes` wrongly marks `changed` to true, trigger `propose_pending` and block following `maybe_trim` logic (`have_pending` will be always be false); as a result, the health store will never be trimmed.
Signed-off-by: Wei Wang <lightmelodies@outlook.com>
(cherry picked from commit
05a85848a01c443d67564a9904a778acfe19cee5)
Kefu Chai [Fri, 26 Feb 2021 03:26:47 +0000 (11:26 +0800)]
common/options: drop ms_async_max_op_threads
the last user of this option was DPDKStack::DPDKStack(). which used it
to initialize its func vector. but this option is not used anymore.
so let's drop it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
ee7184639dcd7b8fc0ed5a189bc764151e601437)
Laura Flores [Tue, 6 Feb 2024 17:54:17 +0000 (11:54 -0600)]
Merge pull request #55109 from k0ste/wip-63978-pacific
pacific: mgr/BaseMgrModule: Optimize CPython Call in Finish Function
Nizamudeen A [Tue, 6 Feb 2024 06:07:40 +0000 (11:37 +0530)]
Merge pull request #55415 from rhcs-dashboard/revert-dt-select
pacifc: Revert "mgr/dashboard: unselect rows in datatables"
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Kefu Chai [Wed, 24 Feb 2021 05:02:43 +0000 (13:02 +0800)]
msg/async: drop Stack::num_workers
no need to store it as a member variable
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
789c1bd62ae4863038afe9fcc014a47c292f02b7)
Kefu Chai [Wed, 24 Feb 2021 04:56:18 +0000 (12:56 +0800)]
msg/async: s/num_workers/workers.size()/
prepare for dropping num_workers, the latter can be dropped in a later
commit.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
52044dcbc7c00758491f095f98c35e886cf379ae)
Kefu Chai [Wed, 24 Feb 2021 04:50:26 +0000 (12:50 +0800)]
msg/async: use range-based loop in NetworkStack
there is no need to reference num_workers here.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
7319340044ffc2394b4782fb8f519ff334e63732)
Kefu Chai [Wed, 24 Feb 2021 04:44:47 +0000 (12:44 +0800)]
msg/async: do not pass worker id to Stack::spawn_worker()
this parameter is not used anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
2ecaa16673f018b3472d640b16a66500c7d3012f)
Kefu Chai [Wed, 24 Feb 2021 04:43:52 +0000 (12:43 +0800)]
async/Stack: pass Worker* to NetworkStack::add_thread()
so we don't need to pass worker id to it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
2d6988feabd8759fc77088b0e0e49cbf8c360602)
Kefu Chai [Wed, 24 Feb 2021 04:38:00 +0000 (12:38 +0800)]
async/rdma: do not reference worker id in RDMAStack::spawn_worker()
it's not needed when creating a new worker.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
ea7686fe99a13779b50e3084b0d2d55a30ea45b0)
Kefu Chai [Wed, 24 Feb 2021 04:34:26 +0000 (12:34 +0800)]
async/dpdk: do not use worker id when creating worker
so we can drop the `i` parameter in a later change.
also restructure DPDKStack::spawn_worker() to capture variables by value
instead of by reference, we cannot assume that the variables allocated on
stack are still available when the function is scheduled on another
stack and core.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
db8d9f6641ea7280e46cd985f8a3bb17fe10808e)
Kefu Chai [Wed, 24 Feb 2021 04:33:33 +0000 (12:33 +0800)]
async/PosixStack: do not reference worker id in ctor
as it's not needed and confusing.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
892bfa097eaf3e1e83cb3fa2ab4b0161bf987851)
Kefu Chai [Wed, 24 Feb 2021 04:06:45 +0000 (12:06 +0800)]
async/rdma: initialize worker in RDMAStack::create_worker()
in
ff65c800b3e1f3f7e3989223b9bde4cbbaf5c076, we moved create_worker()
call out of the constructor to avoid calling virtual functions in
constructor. but this created a regression where RDMAStack's constructor
tries to reference its workers not yet created.
in this change, the workers are initialized right after they are
created.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
103ec86cfab3e6807602b128346212bc8eb0d88d)
Kefu Chai [Wed, 24 Feb 2021 04:05:34 +0000 (12:05 +0800)]
async/rdma: move RDMAStack::create_worker() to .cc
to hide the implementation in .cc file, also speed up the compilation.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
a5ddf1840ba9cc8b30398f86c004918769626a61)
Anthony D'Atri [Sun, 4 Feb 2024 00:09:20 +0000 (19:09 -0500)]
Merge pull request #55440 from zdover23/wip-doc-2024-02-04-backport-55419-to-pacific
pacific: doc/rados: update config for autoscaler
Zac Dover [Fri, 2 Feb 2024 01:53:45 +0000 (11:53 +1000)]
doc/rados: update config for autoscaler
Update doc/rados/configuration/pool-pg-config-ref.rst to account for the
behavior of autoscaler.
Previously, this file was last meaningfully altered in 2013, prior to
the invention of autoscaler. A recent confusion was brought to my
attention on the Ceph Slack whereby a user attempted to alter the
default values of a Quincy cluster, as suggested in this documentation.
That alteration caused Ceph to throw the error "Error ERANGE: 'pgp_num'
must be greater than 0 and lower or equal than 'pg_num', which in this
case is one" and a related "rgw_init_ioctx ERROR" reading in part
"Numerical result out of range". The user removed the
"osd_pool_default_pgp_num" configuration line from ceph.conf and the
cluster worked as expected. I presume that this is because the removal
of this configuration line allowed autoscaler to work as intended.
Fixes: https://tracker.ceph.com/issues/64259
Co-authored-by: David Orman <ormandj@corenode.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit
4dc12092be584da44baca14e31ca33231164235f)
Dan van der Ster [Thu, 2 Nov 2023 23:47:48 +0000 (16:47 -0700)]
qa/tasks/cephadm: enable mon_cluster_log_to_file
Without cluster_log_to_file we have nothing to grep for errors:
2023-10-27T16:06:59.111 DEBUG:teuthology.orchestra.run.smithi150:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/
38cc7fce-74d9-11ee-8db9-
212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1
2023-10-27T16:06:59.141 INFO:teuthology.orchestra.run.smithi150.stderr:grep: /var/log/ceph/
38cc7fce-74d9-11ee-8db9-
212e2dc638e7/ceph.log: No such file or directory
Set mon_cluster_log_to_file = true.
Fixes: https://tracker.ceph.com/issues/63425
Signed-off-by: Dan van der Ster <dan.vanderster@clyso.com>
(cherry picked from commit
822e6b01f909dff79b336bb2fc029de4663b428a)
NitzanMordhai [Thu, 23 Nov 2023 12:01:03 +0000 (12:01 +0000)]
mgr/BaseMgrModule: Optimize CPython Call in Finish Function
Remove CPython overhead packing tuple during the 'finish' function to
improve memory consumption when we deal with long-string outputs.
When modules like Restful return large amounts of output the use
of PyObject_CallFunction without createing PyObject will reduce the
time the memory held by the mgr.
Fixes: https://tracker.ceph.com/issues/59580
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>
(cherry picked from commit
247ace17086ddddd6b7bda44a067d4b1eaa238fd)
Nizamudeen A [Thu, 1 Feb 2024 16:40:21 +0000 (22:10 +0530)]
pacifc: Revert "mgr/dashboard: unselect rows in datatables"
This is already reverted in reef and quincy. Since its not needed
reverting it in pacific as well
Fixes: https://tracker.ceph.com/issues/57829
This reverts commit
3bdb929bab8f3eda32dcf9e46bb54a221ea54d3b.
Signed-off-by: Nizamudeen A <nia@redhat.com>
Yuri Weinstein [Wed, 31 Jan 2024 01:06:49 +0000 (17:06 -0800)]
Merge pull request #55382 from cbodley/wip-pacific-qa-upgrade-ragweed-branch
pacific: qa/upgrade: use ragweed branch for starting ceph release
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Casey Bodley [Tue, 30 Jan 2024 21:48:32 +0000 (16:48 -0500)]
qa/upgrade: use ragweed branch for starting ceph release
don't run new test cases against older ceph versions
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Yuri Weinstein [Wed, 24 Jan 2024 20:06:03 +0000 (12:06 -0800)]
Merge pull request #52978 from k0ste/wip-51653-pacific
pacific: vstart_runner: use FileNotFoundError when os.stat() fails
Reviewed-by: Laura Flores <lflores@redhat.com>
Yuri Weinstein [Wed, 24 Jan 2024 20:05:20 +0000 (12:05 -0800)]
Merge pull request #52977 from k0ste/wip-51654-pacific
pacific: vstart_runner: maintain log level when --debug is passed
Reviewed-by: Laura Flores <lflores@redhat.com>
Yuri Weinstein [Wed, 24 Jan 2024 15:34:14 +0000 (07:34 -0800)]
Merge pull request #54709 from k0ste/wip-63677-pacific
pacific: ceph-volume: fixes fallback to stat in is_device and is_partition
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Teoman Onay <tonay@redhat.com>
Yuri Weinstein [Wed, 24 Jan 2024 15:33:13 +0000 (07:33 -0800)]
Merge pull request #52459 from kamoltat/wip-ksirivad-backport-pacific-48209
pacific: osd/OSDMap: Check for uneven weights & != 2 buckets post stretch mode
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Yuri Weinstein [Wed, 24 Jan 2024 15:31:52 +0000 (07:31 -0800)]
Merge pull request #51314 from rosinL/wip-backport-41306-to-pacific
pacific: isa-l: incorporate fix for aarch64 text relocation
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Yuri Weinstein [Wed, 24 Jan 2024 15:31:00 +0000 (07:31 -0800)]
Merge pull request #51243 from k0ste/wip-54097-pacific
pacific: pybind/mgr/mgr_util: fix to_pretty_timedelta()
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Yuri Weinstein [Wed, 24 Jan 2024 15:29:09 +0000 (07:29 -0800)]
Merge pull request #53355 from k0ste/wip-57260-pacific
pacific: mgr/snap-schedule: use the right way to check the result returned by…
Reviewed-by: Milind Changire <mchangir@redhat.com>
Yuri Weinstein [Wed, 24 Jan 2024 15:23:41 +0000 (07:23 -0800)]
Merge pull request #52928 from guits/wip-62038-pacific
pacific: ceph-volume: add --osd-id option to raw prepare
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Teoman Onay <tonay@redhat.com>
Yuri Weinstein [Wed, 24 Jan 2024 15:22:01 +0000 (07:22 -0800)]
Merge pull request #48311 from k0ste/wip-56635-pacific
pacific: log: Make log_max_recent have an effect again.
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Guillaume Abrioux [Thu, 10 Aug 2023 09:01:22 +0000 (09:01 +0000)]
ceph-volume: support partitions in inventory
This makes ceph-volume report partitions in inventory.
A partition is a valid device for `ceph-volume lvm prepare`
so we should report them in inventory (when using `--list-all`
parameter).
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit
00ba00fdfab8cb4969d46838e44cff3a03fc16ca)
Conflicts:
- file: src/ceph-volume/ceph_volume/util/device.py
comment: pacific don't have --list-all, remove from init
- src/ceph-volume/ceph_volume/util/disk.py
comment: manually resolve conflict in block_types[]
Teoman ONAY [Tue, 3 Oct 2023 19:47:16 +0000 (21:47 +0200)]
ceph-volume: fixes fallback to stat in is_device and is_partition
os.stat (or lstat) cannot distinguish a block device from
a partition.
Fixes: https://tracker.ceph.com/issues/58812
Signed-off-by: Teoman ONAY <tonay@ibm.com>
(cherry picked from commit
52ca4a61d5d69ccfadeef8408a4711c820b98959)
Yuri Weinstein [Tue, 23 Jan 2024 16:09:26 +0000 (08:09 -0800)]
Merge pull request #44462 from pritha-srivastava/wip-52728-pacific
pacific: rgw/sts: code for returning an error when an IAM policy
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Yuri Weinstein [Tue, 23 Jan 2024 16:08:09 +0000 (08:08 -0800)]
Merge pull request #55256 from idryomov/wip-63341-pacific
pacific: librbd: improve rbd_diff_iterate2() performance in fast-diff mode
Reviewed-by: Mykola Golub <mgolub@suse.com>
Yuri Weinstein [Mon, 22 Jan 2024 21:06:16 +0000 (13:06 -0800)]
Merge pull request #53465 from k0ste/wip-50697-pacific
pacific: log: fix the formatting when dumping thread IDs.
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Yuri Weinstein [Mon, 22 Jan 2024 21:05:30 +0000 (13:05 -0800)]
Merge pull request #51250 from k0ste/wip-52839-pacific
pacific: rados: build minimally when "WITH_MGR" is off
Reviewed-by: Laura Flores <lflores@redhat.com>
Yuri Weinstein [Mon, 22 Jan 2024 21:04:41 +0000 (13:04 -0800)]
Merge pull request #48439 from rhcs-dashboard/wip-57839-pacific
pacific: mgr/prometheus: change pg_repaired_objects name to pool_repaired_objects
Reviewed-by: Nizamudeen A <nia@redhat.com>
Yuri Weinstein [Mon, 22 Jan 2024 16:00:47 +0000 (08:00 -0800)]
Merge pull request #55155 from rzarzynski/wip-64006-pacific
pacific: common/weighted_shuffle: don't feed std::discrete_distribution with all-zero weights
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Yuri Weinstein [Mon, 22 Jan 2024 15:58:59 +0000 (07:58 -0800)]
Merge pull request #54558 from k0ste/wip-57794-pacific
pacific: common: intrusive_lru destructor add
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Yuri Weinstein [Mon, 22 Jan 2024 15:57:44 +0000 (07:57 -0800)]
Merge pull request #52981 from k0ste/wip-62154-pacific
pacific: ceph-volume: fix raw list for lvm devices
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Ilya Dryomov [Sat, 20 Jan 2024 21:17:56 +0000 (22:17 +0100)]
Merge pull request #55258 from idryomov/docs-build-fix-pacific
pacific: admin/doc-requirements: bump Sphinx to 5.0.2
Reviewed-by: Zac Dover <zac.dover@proton.me>
Nizamudeen A [Tue, 16 Jan 2024 05:21:56 +0000 (10:51 +0530)]
admin/doc-requirements: bump Sphinx to 5.0.2
```
Running Sphinx v4.5.0
Sphinx version error:
The sphinxcontrib.applehelp extension used by this project needs at least Sphinx v5.0; it therefore cannot be built with this version.
```
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit
a916feeee757e4e196967944feeb1e9b1f92c398)
Conflicts:
admin/doc-requirements.txt [ commit
d61bd5af6596
("admin/doc-requirements: bump sphinx to 4.5.0") not in
pacific ]
Ilya Dryomov [Sat, 20 Jan 2024 15:00:46 +0000 (16:00 +0100)]
PendingReleaseNotes: add rbd_diff_iterate2 note
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
40e8813e9c705838eee42c98de717b20851aed72)
Conflicts:
PendingReleaseNotes [ moved to >=16.2.15 section ]
Ilya Dryomov [Sat, 6 Jan 2024 16:08:04 +0000 (17:08 +0100)]
librbd: try to preserve object map for diff-iterate in fast-diff mode
As an optimization, try to ensure that the object map for the end
version is preloaded through the acquisition of exclusive lock and
as a consequence remains around until exclusive lock is released.
If it's not around, DiffRequest would (re)load it on each call.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
89b0d9e7b40a5f962094428e613315d3697d261f)
Conflicts:
src/librbd/api/DiffIterate.cc [ ImageArea support not in
pacific ]
Ilya Dryomov [Sat, 6 Jan 2024 16:05:39 +0000 (17:05 +0100)]
librbd/object_map: potentially use in-memory object map in DiffRequest
If the object map for the end version is around (already loaded in
memory, either due to the end version being a snapshot or due to
exclusive lock being held), use it to run diff-iterate against the
beginning of time. Since it's the only object map needed in that
case, such calls would be satisfied locally.
Fixes: https://tracker.ceph.com/issues/63341
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
0c4bb58c900efa2356ea8526d3432b2922787afa)
Conflicts:
src/test/librbd/mock/MockObjectMap.h [ commit
87459c23aa05
("librbd,rbd_mirror: do not include RWLock.h unless it is
used") not in pacific ]
Ilya Dryomov [Fri, 5 Jan 2024 12:15:54 +0000 (13:15 +0100)]
librbd/object_map: decouple object map processing in DiffRequest
In preparation for potentially using in-memory object map, decouple
object map processing from loading object maps and place the logic in
prepare_for_object_map() and process_object_map().
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
dabb677ba5923f347c5b4b81b6a86214699a52bf)
Ilya Dryomov [Fri, 5 Jan 2024 11:23:24 +0000 (12:23 +0100)]
common/bit_vector: fix iterator vs reference constness confusion
T (ConstIterator or Iterator) is confused with const T here:
IteratorImpl dereference operator is wrongly overloaded on const
and returns Reference instead of ConstReference for ConstIterator.
This then fails inside bufferlist bowels because Reference is
incompatible with bufferlist::const_iterator.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
45d534553abaca81e26574fd5a7b17b9219c0dd0)
Ilya Dryomov [Thu, 4 Jan 2024 10:44:46 +0000 (11:44 +0100)]
librbd/object_map: make object map in handle_load_object_map() local
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
232ad1a5fb6248d7b3fbfaec5944a90a71a95806)
Ilya Dryomov [Thu, 4 Jan 2024 10:39:20 +0000 (11:39 +0100)]
librbd/object_map: don't resize object map in handle_load_object_map()
Currently it's done in two cases:
- if the loaded object map is larger than expected based on byte size,
it's truncated to expected number of objects
- in case of deep-copy, if the loaded object map is smaller than diff
state, it's expanded to get "track the largest of all versions in the
set" semantics
Both of these cases can be easily dealt with without modifying the
object map. Being able to process a const object map is needed for
working on in-memory object map which is external to DiffRequest.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
275a299cd48d2ddac36608d6633a6b79c8927351)
Ilya Dryomov [Sat, 6 Jan 2024 11:22:35 +0000 (12:22 +0100)]
common/bit_vector: fix IteratorImpl post-increment operator
It's totally broken: instead of returning the current position and
moving to the next position, it returns the next position and doesn't
move anywhere. Luckily it hasn't been used until now.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
2ab5b52f71c88cb55f8ed82f1dfd0115fdd6e022)
Ilya Dryomov [Thu, 28 Dec 2023 09:52:11 +0000 (10:52 +0100)]
librbd: drop DiffIterate::diff_object_map() declaration
This is a leftover from commit
2b3a46801d39 ("librbd: switch
diff-iterate API to use new object-map diff helper").
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
1503b96bf91d358540fc56a69b3cf18aa7eab68e)
Ilya Dryomov [Thu, 28 Dec 2023 09:14:18 +0000 (10:14 +0100)]
librbd: propagate diff-iterate range to parent in fast-diff mode
When getting parent diff, pass the overlap-reduced image extent instead
of the entire 0..overlap range to avoid a similar quadratic slowdown on
cloned images.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
7677d4b1b52ab68484545d0fcd7348f2f8e9f263)
Conflicts:
src/librbd/api/DiffIterate.cc [ ImageArea support not in
pacific ]
Ilya Dryomov [Wed, 27 Dec 2023 17:07:05 +0000 (18:07 +0100)]
librbd/object_map: add support for ranged diff-iterate
Currently diff-iterate in fast-diff mode is performed on the entire
image no matter what image extent is passed to the API. Then, unused
diff just gets discarded as DiffIterate ends up querying only objects
that the passed image extent maps to. This hasn't been an issue for
internal consumers ("rbd du", "rbd diff", etc) because they work on the
entire image, but turns out to lead to quadratic slowdown in some QEMU
use cases.
0..UINT64_MAX range is carved out for deep-copy which is unranged by
definition. To get effectively unranged diff-iterate, 0..UINT64_MAX-1
range can be used.
Fixes: https://tracker.ceph.com/issues/63341
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
0b5ba5fedf704ada74a65108af129eae6baea5c5)
Conflicts:
src/librbd/object_map/DiffRequest.cc [ commit
ef12761bdb03
("librbd: use uint64 for return value of size()") not in
pacific ]
Ilya Dryomov [Wed, 27 Dec 2023 15:18:50 +0000 (16:18 +0100)]
include/intarith: introduce round_down_to()
Same as with round_up_to(), d isn't required to be a power of two.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
94bf3a5d74cfd61e0bd881ca2863fe2205f27767)
Conflicts:
src/librbd/api/DiffIterate.cc [ commit
6eb14774fec0 ("librbd:
build without "using namespace std"") not in pacific ]
Ilya Dryomov [Sat, 23 Dec 2023 14:19:09 +0000 (15:19 +0100)]
test/librbd: expand TestMockObjectMapDiffRequest edge case coverage
For each covered edge case or error, run through the following
scenarios:
- where the edge case concerns snap_id_start
- where the edge case concerns snap_id_end
- where the edge case concerns intermediate snapshot and
snap_id_start == 0 (diff against the beginning of time)
- where the edge case concerns intermediate snapshot and
snap_id_start != 0 (diff from snapshot)
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
9931282bfd2260d654325970555cab8c617e8f14)
Ilya Dryomov [Sat, 23 Dec 2023 13:47:54 +0000 (14:47 +0100)]
librbd/object_map: allow intermediate snaps to be skipped on diff-iterate
In case of diff-iterate against the beginning of time, the result
depends only on the end version. Loading and processing object maps
or intermediate snapshots is redundant and can be skipped.
This optimization is made possible by commit
be507aaed15f ("librbd:
diff-iterate shouldn't ever report "new hole" against a hole") and, to
a lesser extent, the previous commit.
Getting FastDiffInvalid, LoadObjectMapError and ObjectMapTooSmall to
pass required tweaking not just expectations, but also start/end snap
ids and thus also the meaning of these tests. This is addressed in the
next commit.
Fixes: https://tracker.ceph.com/issues/63341
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
23c675f04a4b4bb00794bbc59493d9591625a0a7)
Ilya Dryomov [Fri, 22 Dec 2023 17:50:20 +0000 (18:50 +0100)]
librbd/object_map: resurrect diff-iterate behavior when image is shrunk
The new "track the largest of all versions in the set, diff state is
only ever grown" semantics introduced in commit
330f2a7bb94f ("librbd:
helper state machine for computing diffs between object-maps") don't
make sense for diff-iterate. It's a waste because DiffIterate won't
query beyond the end version size -- this is baked into the API.
Limit this behavior to deep-copy and resurrect the original behavior
from 2015 for diff-iterate.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
19c7c4a5359fa9c1d06cc11187e300251249ad9e)
Conflicts:
src/librbd/object_map/DiffRequest.cc [ commit
ef12761bdb03
("librbd: use uint64 for return value of size()") not in
pacific ]
Ilya Dryomov [Fri, 22 Dec 2023 15:10:12 +0000 (16:10 +0100)]
librbd/object_map: fix diff from snapshot when image is grown
Commit
399a45e11332 ("librbd/object_map: rbd diff between two
snapshots lists entire image content") fixed most of the damage caused
by commit
b81cd2460de7 ("librbd/object_map: diff state machine should
track object existence"), but the case of a "resize diff" when diffing
from snapshot was missed. An area that was freshly allocated in image
resize is the same in principle as a freshly created image and objects
marked OBJECT_EXISTS_CLEAN are no exception. Diff for such objects in
such an area should be set to DIFF_STATE_DATA_UPDATED, however
currently when diffing from snapshot, it's set to DIFF_STATE_DATA.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
34386d29a8b96de79c38465062b8e93e7fc6e184)
Ilya Dryomov [Wed, 20 Dec 2023 11:22:17 +0000 (12:22 +0100)]
librbd/object_map: drop bogus if in handle_load_object_map()
It became redundant with commit
b81cd2460de7 ("librbd/object_map: diff
state machine should track object existence") -- it != end_it condition
in the loop is sufficient.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
4e036d65b9146f28a2f6c0dfb353120baea8d62d)
Ilya Dryomov [Thu, 21 Dec 2023 15:50:31 +0000 (16:50 +0100)]
test/librbd: refactor TestMockObjectMapDiffRequest tests
In preparation for multiple similarly configured MockTestImageCtx
objects being used in a single test, centralize their creation and add
a couple of helpers for setting expectations from a callback.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
718f6b5546890179f66d5ffadbae9e9cb0e6c97b)
Ilya Dryomov [Thu, 21 Dec 2023 17:08:21 +0000 (18:08 +0100)]
test/librbd: improve TestMockObjectMapDiffRequest.InvalidStartSnap
Use a range where only snap_id_start is invalid.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
64a5afcaad7b61a20f630aca064c2953380c70c1)
Pritha Srivastava [Mon, 19 Jul 2021 10:27:52 +0000 (15:57 +0530)]
rgw/sts: code to check IAM policy and return an
appropriate error incase Resource specified in the
IAM policy is incorrect and is discarded. The IAM
policy can be a resource policy or an identity policy.
This is for policies that have already been set.
Fixes: https://tracker.ceph.com/issues/51219
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit
a79fa82939ad83b1a8365f100e7426eb8e144e2e)
Pritha Srivastava [Fri, 18 Jun 2021 10:06:10 +0000 (15:36 +0530)]
rgw/sts: code for returning an error when an IAM policy
resource belongs to someone else's tenant.
While parsing the policy it discards the resource element,
but then when an operation is evaluated, since the resource element
is empty, it doesnt evaluate the resource at all and the policy
ends up erroneously allowing actions on resources in other tenants.
Fixes: https://tracker.ceph.com/issues/51219
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit
be42b2cf04def671023b98c811e77e1e357eb38e)
Joshua Baergen [Thu, 16 Jun 2022 16:14:12 +0000 (10:14 -0600)]
log: Make log_max_recent have an effect again.
The log improvements in
a747aeac13daf3dba43343120659e802cb569f6b
unfortunately left log_max_recent broken because m_max_recent wasn't
used anymore.
Eliminate m_max_recent and set the capacity of the m_recent ring buffer
when log_max_recent changes. In order to call set_capacity(),
ConcreteEntry needed its move constructor set noexcept.
I haven't followed the boost code all the way down but I suspect that
setting the ring buffer capacity to anything less than 1 entry will
probably cause problems, so restrict log_max_recent to >=1.
Also fix a wrong variable used for printing the max new entries during
"log dump".
Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>
(cherry picked from commit
3d59ba1671e3e76326fb706a76b8d9638d782924)
Conflicts:
- file: src/common/options/global.yaml.in
desc: file not exists in pacific
- file: src/common/options.cc
desc: added 'set_min' for log_max_recent
luo rixin [Thu, 11 Jan 2024 11:07:09 +0000 (19:07 +0800)]
isa-l submodule: incorporate fix for aarch64 text relocation
Fixes: https://tracker.ceph.com/issues/48681
Fixes: https://tracker.ceph.com/issues/50384
Signed-off-by: luo rixin <luorixin@huawei.com>
Yuri Weinstein [Thu, 11 Jan 2024 21:17:03 +0000 (13:17 -0800)]
Merge pull request #55113 from k0ste/wip-63980-pacific
pacific: mon/Monitor: during shutdown don't accept new authentication and crea…
Reviewed-by: Laura Flores <lflores@redhat.com>
Yuri Weinstein [Thu, 11 Jan 2024 21:16:28 +0000 (13:16 -0800)]
Merge pull request #55111 from k0ste/wip-63975-pacific
pacific: Tools/rados: Improve Error Messaging for Object Name Resolution
Reviewed-by: Laura Flores <lflores@redhat.com>
Yuri Weinstein [Thu, 11 Jan 2024 21:15:56 +0000 (13:15 -0800)]
Merge pull request #54553 from k0ste/wip-63366-pacific
pacific: mgr: remove out&down osd from mgr daemons
Reviewed-by: Laura Flores <lflores@redhat.com>
Yuri Weinstein [Thu, 11 Jan 2024 21:15:04 +0000 (13:15 -0800)]
Merge pull request #54013 from k0ste/wip-63035-pacific
pacific: mgr: add throttle policy for DaemonServer
Reviewed-by: Laura Flores <lflores@redhat.com>
Radosław Zarzyński [Wed, 30 Aug 2023 13:23:34 +0000 (15:23 +0200)]
test/test_weighted_shuffle: verify weights containing zeros
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit
d02b17ff84c61123ed27d79dc177c2cfbbe6a72f)
Radosław Zarzyński [Wed, 30 Aug 2023 13:19:07 +0000 (15:19 +0200)]
common/weighted_shuffle: don't feed std::discrete_distribution with all-zero weights
This flaw results is the assertions like the following one:
```
/usr/include/c++/11/bits/random.tcc:2667: void std::discrete_distribution<_IntType>::param_type::_M_initialize() [with _IntType = int]: Assertion '__sum > 0' failed.
Aborted (core dumped)
```
The reason behind is that `std::discrete_distribution` sums
the weights and uses the result as a divisor.
Fixes: https://tracker.ceph.com/issues/62645
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit
1b7a7a8df88ffac007dbafdecc131807de66c046)
Laura Flores [Thu, 11 Jan 2024 19:51:16 +0000 (13:51 -0600)]
Merge pull request #55118 from pdvian/wip-63987-pacific
pacific: mon: add exception handling to ceph health mute
Kamoltat [Tue, 4 Oct 2022 20:00:22 +0000 (20:00 +0000)]
doc/rados/operations/health-checks: Added Stretch Mode health warnings
Added stretch mode health warnings in docs for:
1. INCORRECT_NUM_BUCKETS_STRETCH_MODE
2. UNEVEN_WEIGHTS_STRETCH_MODE
Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit
cfc415c0d95c2531c4f75dd215705f6bf3290b86)
Kamoltat [Wed, 10 Jan 2024 09:24:52 +0000 (09:24 +0000)]
qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh: init
Initialize standalone test for stretched clusters,
testing uneven weight warnings and != 2 buckets
warnings.
Added `wait_for_health_gone()` function in ceph-helpers.sh
this function allows us to wait for health condition to
disappear when doing standalone tests.
Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit
2c25b29347c840cef451135bdc220992a91f7e4b)
Signed-off-by: Kamoltat <ksirivad@redhat.com>
Conflicts:
qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh -> trivial fix
Kamoltat [Thu, 22 Sep 2022 03:41:16 +0000 (03:41 +0000)]
osd/OSDMap: Check for uneven weights & != 2 buckets stretch mode
Added additional health warnings in
``OSDMap::check_health()``.
1. Check for uneven weights of the two dividing
buckets if stretch mode is enabled and spit out
warnings if true.
2. Check for != 2 buckets if stretch mode is enabled
and spit out warnings if true.
Fixes: https://tracker.ceph.com/issues/57570
Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit
78baf23ea214dce3c1a7cf3f3515706d03f950ed)