Xiubo Li [Mon, 23 Nov 2020 12:55:01 +0000 (20:55 +0800)]
common: do not dup the options when reexpanding
The old code will store all the options, which has `$pid` in them,
in may_reexpand_meta map. And when reexpanding later, the reexpand
code will dup them with a higher priority(CONF_OVERRIDE).
This will be a problem, if the default value has `$pid` and be
stored in the may_reexpand_meta map, and then the code set a new
different value, which may have no `$pid`, from CLI or config file.
The reexpand will override it with the default value always.
This will do not duplicate the options with CONF_OVERRIDE priority
when reexpanding, just refresh them and call the observers. And the
finalize_reexpand_meta() will always be called after the fork() is
done in child processes.
Xiubo Li [Fri, 13 Nov 2020 08:08:31 +0000 (16:08 +0800)]
global: reexpand the conf meta in all the child processes
Especially for the tools or the daemons whose config options need
to expand the '$pid', they will be always expanded with the parent
processes. We need to reexpand them in child processes just after
the fork is done.
Conflicts:
- src/pybind/mgr/dashboard/module.py
Limiting TLS version seems to work only with CherryPy >= 8.9.1 (Ubuntu 18.04 or SUSE Leap 15.1) based on tests in Teuthology. It failed on Ubuntu 16.04 (CherryPy 3.5.0) and RHEL 7.9 (CherryPy 3.2.2).
- qa/tasks/mgr/test_dashboard.py
Tests must be removed because they will not work in Teuthology because of outdated cherrypy versions on most platforms. Only Ubuntu 18.04 (and downstream SUSE Leap 15.1) will work, but it's not possible to limit QA runs to that specific platforms.
Kotresh HR [Tue, 1 Dec 2020 10:44:17 +0000 (16:14 +0530)]
tasks/cephfs/test_volume_client: Add tests for authorize/deauthorize
1. Add testcase for authorizing auth_id which is not added by
ceph_volume_client
2. Add testcase to test 'allow_existing_id' option
3. Add testcase for deauthorizing auth_id which has got it's caps
updated out of band
Optionally allow authorizing auth-ids not created by ceph_volume_client
via the option 'allow_existing_id'. This can help existing deployers
of manila to disallow/allow authorization of pre-created auth IDs
via a manila driver config that sets 'allow_existing_id' to False/True.
Kotresh HR [Thu, 26 Nov 2020 09:18:16 +0000 (14:48 +0530)]
pybind/ceph_volume_client: Preserve existing caps while authorize/deauthorize auth-id
Authorize/Deauthorize used to overwrite the caps of auth-id which would
end up deleting existing caps. This patch fixes the same by retaining
the existing caps by appending or deleting the new caps as needed.
This patch disallow the ceph_volume_client to authorize the auth_id
which is not created by ceph_volume_client. Those auth_ids could be
created by other means for other use cases which should not be modified
by ceph_volume_client.
Fixes: https://tracker.ceph.com/issues/48555 Signed-off-by: Ramana Raja <rraja@redhat.com> Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 3a85d2d04028a323952a31d18cdbefb710be2e2b)
Neha Ojha [Thu, 3 Dec 2020 19:18:04 +0000 (19:18 +0000)]
messages/MMonCommand, MMonCommandAck: don't log values for "config set" and "config-key set"
This acts like a big hammer to avoid adding sensitive information, like passwords
into mon/mgr/cluster logs when using "config set" and "config-key set" to set keys
whose values should be secure.
Or Friedmann [Sun, 8 Mar 2020 13:34:48 +0000 (15:34 +0200)]
rgw: Disable prefetch of entire head object when GET request with range header
Disable prefetch of entire head object when GET request with range header.
The current behavior for the RGW is getting the whole object although the client asked only for a small bytes offset.
For example: If the client asked for bytes=0-1, The RGW will anyway fetch 0-4194304
Fixes: https://tracker.ceph.com/issues/44508 Signed-off-by: Or Friedmann <ofriedma@redhat.com>
(cherry picked from commit 2be5af0006169cb54547034aa98b7eacb8751d59)
Kamoltat [Mon, 16 Nov 2020 08:52:12 +0000 (08:52 +0000)]
qa/mgr/test_progress: fix bug 48217
Fixes a failing test case regarding osd coming back
after being marked out. The old test case wasn't accounting
for a specific event, therefore this resulted in the failure.
The fix basically accounts for a specific event of osd being
marked in/out.
rgw: in ordered bucket listing skip namespaced entries when possible
When listing non-namespaced entries in the bucket index, the code
would march through the namespaced entries in blocks, requesting all
of them from the CLS layer. When there were many namespaced entries,
it would significantly affect the performance of ordered listing.
This commit adds code to advance the marker passed to lower layers to
skip past namespaced entries. This is challenging in that
non-namespaced entries can appear in the middle of the namespaced
entries. We'll ignore the issue instance tags in names to simplify the
following discussion. Non-namespaced entries are indexed by
"name". Namespaced entries are indexed by _namespace_name, using
underscores to surround the namespace. The challenge comes with
entries such as "_name", where the name begins with an underscore. In
that case we index them by "__name", quoting the underscore with
another.
Now the extra challenge comes due to the lexic ordering of the
following:
ASP
_BAT_cat
__DOG
_eel_FOX
goat
Note that the namespaced entries are in positions 2 and 4, and the
non-namespaced entries are in positions 1, 3, and 5. So when skipping
past the namespaced entries, we have to be careful not to skip past
the non-namespaced entries that begin with underscore.
Xiubo Li [Thu, 12 Nov 2020 01:13:45 +0000 (09:13 +0800)]
client: do not use g_conf().get_val<>() in libcephfs
The g_conf() will try to dereference the "g_ceph_context" to get
the "_conf", but the "g_ceph_context" won't be set in none fuse
use case in libcephfs, so it will crash in when doing
g_conf().get_val<>.
Ken Dreyer [Thu, 3 Dec 2020 17:48:06 +0000 (10:48 -0700)]
mgr/prometheus: don't store exception as e
Python's logging module's exception() method will log the full exception
and stack trace for us, so we do not need to store the exception in the
"e" variable here.
Jason Dillaman [Fri, 11 Sep 2020 19:20:45 +0000 (15:20 -0400)]
librbd: ensure that thread pool lock is held when processing throttled IOs
There previously was a potential race for throttled IOs to complete prior
to the main worker thread finishing the processing of the blocked IO.
Fixes: https://tracker.ceph.com/issues/47371 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2d86e0935aa6f0c392df428676d9ab0a338fccae)
Conflicts:
src/test/librbd/io/test_mock_ImageRequestWQ.cc
- in Octopus, commit 792d6c53fedc695199cc18916347c1b545fe42c2 did a global
replace of Mutex to ceph::mutex, so to fix this for Nautilus, we just need to
do that in test_mock_ImageRequestWQ.cc since the get_pool_lock() method is
returning a Mutex instead of a ceph::mutex
Kevin Meijer [Sat, 14 Nov 2020 18:44:07 +0000 (19:44 +0100)]
mgr/dashboard: Disable sso without python3-saml
Removed the requirement for the python3-saml package when wanting to disable SSO for the dashboard, this is currently relevant since the official container that runs Ceph mgr does not have this package installed.
So when upgrading from an older, non-containerized version, you would be stuck using a non-functional dashboard.
This pull requests changes that and allows the ceph dashboard sso disable command without the requirement of the library so that we SSO can always be disabled again.
Fixes: https://tracker.ceph.com/issues/48237 Signed-off-by: Kevin Meijer <admin@kevinmeijer.nl>
(cherry picked from commit 0c18437d2c786ef1ade8b89e42dbf4b0e163aafe)
Ceph status still reports slow requests on the OSD which is already out.
When orignal PG monitor handled PGSTATS msg, it wouldn't update osd stat
if this OSD is not in OSD map, but current MGR had no checks on that.
Boris Ranto [Wed, 25 Nov 2020 09:27:25 +0000 (10:27 +0100)]
mgr/prometheus: Use mgr.release_name for always on modules
The host_version is not populated properly in the early stages of ceph
mgr start up process. We can use mgr.release_name instead. It is more
stable and it provides the data even if mgr_map does not contain the
versions, yet.
split mempool allocation for bluestore_cache_other
While doing root cause analysis it bluestore_cache_other gives a bit of
a crude estimate, something more helpful would be to have it split into
the following fields:
Conflicts:
src/os/bluestore/BlueStore.cc
Legacy vs. New lock used
src/os/bluestore/bluestore_types.h
Lacked explicit std:: namespace reference in headers which pacific has
qa/mgr/progress: Update the test suite for progress module
Update the test suite to reflect a feature
change that has been merged to master in
progress module where you create an event
when an osd is marked in.
Fixes: http://tracker.ceph.com/issues/40618
with success QA run in sepia:
http://pulpito.ceph.com/kchai-2019-07-16_10:10:01-rados-master-distro-basic-mira/
projects / ceph.git / log