Mike Christie [Thu, 9 Jan 2020 00:03:40 +0000 (18:03 -0600)]
selinux: Fix ceph-iscsi configfs access
This fixes the the following selinux error when using ceph-iscsi's
rbd-target-api daemon (rbd-target-gw has the same issue). They are
a result of the a python library, rtslib, which the daemons use.
Additional Information:
Source Context system_u:system_r:ceph_t:s0
Target Context system_u:object_r:configfs_t:s0
Target Objects
/sys/kernel/config/target/iscsi/iqn.2003-01.com.re
dhat:ceph-iscsi/tpgt_1/attrib/authentication
[
file ]
Source rbd-target-api
Source Path /usr/libexec/platform-python3.6
Port <Unknown>
Host ans8
Source RPM Packages platform-python-3.6.8-15.1.el8.x86_64
Target RPM Packages
Policy RPM selinux-policy-3.14.3-20.el8.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Enforcing
Host Name ans8
Platform Linux ans8 4.18.0-147.el8.x86_64 #1 SMP
Thu Sep 26
15:52:44 UTC 2019 x86_64 x86_64
Alert Count 1
First Seen 2020-01-08 18:39:47 EST
Last Seen 2020-01-08 18:39:47 EST
Local ID 6f8c3415-7a50-4dc8-b3d2-2621e1d00ca3
mgr/prometheus: log time it takes to collect metrics
When debug is enabled, the time it takes to gather the various types of
metrics is logged by type.
Fixes: https://tracker.ceph.com/issues/46444 Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit a5f86cb85fcbf8a9003bf511b57a6d6b4a0e1f41)
Conflicts:
src/pybind/mgr/mgr_util.py
- Some more imports in master compared to octopus;
had to add "import time" (line 6) manually, otherwise make check
would fail with a mypy error ('time is not defined')
Kiefer Chang [Mon, 3 Aug 2020 11:20:08 +0000 (19:20 +0800)]
mgr/dashboard: Fix host attributes like labels are not returned
The problem only happens when a host is reported both from Ceph and the
Orchestrator. There is an error on merging Ceph hosts with Orchestrator
hosts. The unit test is refined to test this case.
Jason Dillaman [Thu, 7 May 2020 20:25:50 +0000 (16:25 -0400)]
librbd: generic helper for tracking in-flight IO and managing flush requests
Layers that potentially queue IOs but not all IOs will need to track all
in-flight IOs to properly ensure that a flush cannot complete while
older IO is still stuck in a queue.
Aleksei Gutikov [Mon, 30 Mar 2020 12:27:45 +0000 (15:27 +0300)]
mgr: decrease pool stats if pg was removed
After merge of placement groups resulting pg contains
objects from itself and merged one.
PGMap::apply_incremental treat this growth as pool stats delta,
but forget to decrease stats for removed pg.
Fixes: https://tracker.ceph.com/issues/44815 Signed-off-by: Aleksei Gutikov <aleksey.gutikov@synesis.ru>
(cherry picked from commit 6090acdae4495e11f117df2330b579744eeada2a)
8987f94416f453829eae6dda08837ef5a42531c6 introduced the osd_lock for the
bench command. Taking the osd_lock in bench can lead to deadlocks, causing the
command to hang as seen in https://tracker.ceph.com/issues/43888.
Fixes: https://tracker.ceph.com/issues/43888 Signed-off-by: Adam Kupczyk <akupczyk@redhat.com> Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit d6be0d3f9a091b8a2e20ea607434314d395a821c)
This will crash the choose_acting() procedure as it will mistakenly
think that peer 3 should continue to perform asynchronous recovery
(e.g., due to num_objects_missing = 1) in contrast to fully
backfill-recovered.
While I did not dig into the real cause, there are a couple of
possible explanations of how num_objects can be off. I think that
if a roll forward or log replay could delete something twice, maybe
there would be an undercount. Or maybe something as simple as a
corruption.
Since _update_calc_stats() is going to fix num_objects_missing
for that peer anyway, let's make sure it always starts with a
clean state.
Jason Dillaman [Tue, 14 Jul 2020 22:49:30 +0000 (18:49 -0400)]
crush/CrushWrapper: rebuild reverse maps after rebuilding crush map
The Objecter will crash when localized reads are enabled and two threads
attempt to rebuild the (invalidated) reverse maps concurrently. This
should address the issue for the Objecter use-case without the need to
add additional locking.
Fixes: https://tracker.ceph.com/issues/44311 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8b866794f5b3674c5e3ad9adceb5e3230d55a0e0)
Switch the default path for the immutable object cache from "/tmp" to
"/tmp/ceph_immutable_object_cache" to prevent the deletion of all
files in the "/tmp" directory and sub-directories for unconfigured
daemons.
Jason Dillaman [Fri, 17 Jul 2020 14:25:20 +0000 (10:25 -0400)]
immutable-object-cache: fix error handling during start up
Previously the daemon would crash if it couldn't remove all the
files from the specified cache directory. It would also crash if
it could not access its specified domain socket file.
Fixes: https://tracker.ceph.com/issues/45169 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f6dcb6c3dbd45574e1a7ca3f5e8f1f52a0d7ae1e)
Jason Dillaman [Wed, 22 Jul 2020 15:25:56 +0000 (11:25 -0400)]
librbd: flush all queued object IO from simple scheduler
Normally IO is tracked via the AioCompletion's async_op but the
scheduler will "complete" writes while the IO might be still
executing. Therefore, prior to shutting down this dispatch layer
we need to wait for all IO to complete.
Fixes: https://tracker.ceph.com/issues/46668 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 75ff8fd14dccaa7d2f11ba8a561ad3c0f410585c)
Matt Benjamin [Thu, 30 Apr 2020 22:59:11 +0000 (18:59 -0400)]
rgw: introduce safe user-reset-stats
Defines cls_user_reset_stats2, a value-returning cls operation
that sets new stats via progressive calls with an accumulator,
avoiding risk of excessive call runtime.
Fixes: https://tracker.ceph.com/issues/41080 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 25a82ed3795ecf6395e3d16fcbb4c29e478aa065)
root [Fri, 26 Jun 2020 10:44:45 +0000 (12:44 +0200)]
rgw: fix double slash (//) killing the gateway
When a bucket is inialized as a static website, a curl request on the bucket with double slash kills the gateway.
The problem is on the URL handling of the subdirectory, which tries to remove the last slash of any URL, so when only / is given as a sub-directory, this results to an empty string.
Casey Bodley [Fri, 29 May 2020 16:31:16 +0000 (12:31 -0400)]
rgw: fix shutdown crash in RGWAsyncReadMDLogEntries
RGWAsyncReadMDLogEntries must not store pointers into coroutine memory,
because it's not guaranteed to outlive our call. store these by-value
instead, and have RGWReadMDLogEntriesCR::request_complete() copy/move
them back on completion
When rgw_bucket_unlink_instance removes the last instance of a name, it
also clears the value of rgw_bucket_olh_entry.key. However, bucket index
resharding uses this key when choosing its shard placement, so an empty
key causes all of these olh entries to be misplaced in shard 0. After
reshard, all of the olh recovery/cleanup logic would be sent to the
correct shard, and these misplaced olh entries would never be cleaned
up.
Preserving the key's name on last unlink allows the olh entry to be
resharded correctly and cleaned up normally.
* extract get_ragweed_branch() out of download() task, for better
readablity.
* use a loop for retry when the first clone fails
* drop the `raise ValueError()` clause as it never happens. we could use
an assert() here, but i don't think it is necessary anyway.
* use sh() instead of run() for better readablity.
* always set ragweed_repo. before this change this variable is
unbounded if `force-branch` is set.
Soumya Koduri [Tue, 16 Jun 2020 12:40:08 +0000 (18:10 +0530)]
rgw: Empty reqs_change_state queue before unregistered_reqs
In RGWHTTPManager::manage_pending_request(), before unregistering
or unlinking the http requests, empty the reqs_change_state list
to avoid use after free.
ofriedma [Wed, 20 May 2020 16:07:03 +0000 (19:07 +0300)]
rgw: fix nginx-rgw docs
Signed-off-by: Or Friedmann <ofriedma@redhat.com> Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit d73b879ac169c46f2dfeba0f4ca7f3a8af272a53)
Nginx (have not cached):
real 0m24.714s
user 0m8.692s
sys 0m10.360s
Nginx (have been cached):
real 0m21.070s
user 0m9.140s
sys 0m10.316s
RGW:
real 0m21.859s
user 0m8.850s
sys 0m10.386s
The results are showing that for objects larger than 512K the cache will increase the performance by twice or more.
For small objs, the overhead of sending the auth request will make the cache less efficient
The result for cached objects in the 10MB test can be explained by net limit of 25 Gb/s(it could reach more)
In Gdal (image decoder/encoder over s3 using range requests) the results were not that different because of Gdal single cpu encoding/decoding.
Gdal have been chosen because of the ability to check the smart cache of the nginx.
https://www.nginx.com/blog/smart-efficient-byte-range-caching-nginx/
Jason Dillaman [Tue, 28 Jul 2020 01:14:18 +0000 (21:14 -0400)]
librbd: update hidden global config when setting pool config override
The new "dev"-level global config setting will be updated when any
pool-level config override is updated. librbd clients will detect
the new global-level config update and trigger a refresh. This avoids
the need for potentially tens of thousands of librbd clients
registering a watch on the pool metadata object or periodically polling
the pool metadata object for updates.
Fixes: https://tracker.ceph.com/issues/46694 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f45df9fe786e8057c491c082e840483759d67e9e)
Conflicts:
src/common/options.cc
- "rbd_quiesce_notification_attempts", "rbd_default_snapshot_quiesce_mode", and
"rbd_plugins" options have not been backported to Octopus, yet
Jason Dillaman [Mon, 27 Jul 2020 19:31:09 +0000 (15:31 -0400)]
librbd: initial config watcher implementation
The config watcher will initially observe all "rbd_" configuration
updates received from the MON that have not been locally overridden
at the pool and/or image level.
rgw: policy: reuse eval_principal to evaluate the policy principal
Since the other edge case when no Principal or a NotPrincipal is supplied also
must be accounted for, which is already done in eval_principal function. Also
reraising the error as Effect::Pass in line with the previous output, though an
Effect::Deny would also work here.
monclient: use "is_connected" check when scheduling tick
When schedule_tick is called for the first time (in init) we are
still not in hunting mode so mon_client_ping_interval and
mon_client_log_interval were used to schedule the tick.
projects / ceph.git / log