Rishabh Dave [Wed, 3 Mar 2021 11:44:22 +0000 (17:14 +0530)]
qa/cephfs: move is_blocklisted() to filesystem.CephCluster
Using self.fs.mon_manager in mount.py can lead to a crash since self.fs
can be None. Move is_blocklisted() to tasks.filesystem.CephCluster where
it can get access to mon_manager without depending on objects
representing Ceph FSs.
Fixes: https://tracker.ceph.com/issues/49511 Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 4d0f56fcc524cfe328d89d0b3706ca22a68a268f)
Xiubo Li [Wed, 30 Dec 2020 01:41:01 +0000 (09:41 +0800)]
client: fire the finish_cap_snap() after buffer being flushed
When the inode has Fb cap and the used reference is none zero, the
cap snap flushing will be delayed, so we need to make sure it will
be invoked after the dirty buffer is flushed to osd.
Adam C. Emerson [Mon, 8 Mar 2021 20:17:53 +0000 (15:17 -0500)]
rgw: Fix probe failure on OSDs not suporting FIFO.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 4e9ec426b15fe60c5b0154980f808076e166dd02) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Thu, 11 Feb 2021 23:27:33 +0000 (18:27 -0500)]
rgw: Make empty datalog fifo markers empty strings
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 4e3a7d5476fa2dd4b9825f4d546c42819f93c7cc) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 10 Feb 2021 22:09:02 +0000 (17:09 -0500)]
rgw: Try to prune empties even if no empties found
Since we won't actually delete empties until much later.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 9bd9b7659fdb7a1a01d5e1523f0d461dbf5eaafe) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 10 Feb 2021 21:18:09 +0000 (16:18 -0500)]
rgw: Wait until a generation has been empty for an hour to delete
This fixes a problem where, while the backing handle remains allocated
while a call completes, the objects it depends on may be deleted
behind it.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 7018c25d47edf7e12b581f7f28c2549fe73bde15) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Tue, 9 Feb 2021 23:10:50 +0000 (18:10 -0500)]
rgw: Leave the zero'th shard of the zero'th generation for cls_lock
Since data sync locks that object, instead of deleting it, truncate
the object and clear the omap.
(cls_lock uses xattrs.)
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 0d4e0abb8a699417ea75a6cd390786189ab964eb) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Thu, 4 Feb 2021 20:48:56 +0000 (15:48 -0500)]
rgw: Don't swallow errors in datalog async trim
Typo and misleading indentation.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit e97de55f46bbe67b523abfb4c30c50f1547f2601) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Tue, 2 Feb 2021 19:09:52 +0000 (14:09 -0500)]
rgw: Fix cursor handling in DataLogBackends::list
Don't assume that the lowest generation not greater than the requested
generation actually is the requested generation.
(Also don't hold the lock after we get a backend.)
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit d7739178e994ce84886d297a29f2250e4bd78daa) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 27 Jan 2021 01:30:58 +0000 (20:30 -0500)]
rgw: Prune datalog generations in the renew loop
Every 150 times through, which is a bit less than an hour between runs
by default.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 8f4291291b0dea4b4701894da0775149266a1373) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Conflicts:
src/rgw/rgw_datalog.cc
Adam C. Emerson [Sun, 22 Nov 2020 04:06:38 +0000 (23:06 -0500)]
rgw: Use LazyFIFO in data changes log
That way we don't start sending ops to open a FIFO until we need it.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 12939a258f8c627d1b7b23c0b9d7c22e98e69d89) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sun, 22 Nov 2020 00:34:07 +0000 (19:34 -0500)]
rgw: Add LazyFIFO to keep from blasting an op-per-shard on startup
LazyFIFO opens the FIFO on first access.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 1cc4a0a4e274700b4ae044db125a8cb3a64253a2) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Tue, 26 Jan 2021 17:24:41 +0000 (12:24 -0500)]
cls/fifo: Don't error in the log if we're being probed for existence
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 4a2575783a050f27b22b7bfe4364520bf29fc6a5) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sat, 23 Jan 2021 01:48:39 +0000 (20:48 -0500)]
rgw: Add and trim datalog generations
This lets us actually change type in mid-stream.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 32b100d797cdf88648530e0162fd103cf279df31) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 27 Jan 2021 01:07:45 +0000 (20:07 -0500)]
rgw: Actually pull logbacking_generations into datalog
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit eb0f8ffcc785146a1fb249f4531620787be216ba) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Tue, 26 Jan 2021 06:27:24 +0000 (01:27 -0500)]
rgw: Clamp FIFO trim to head
Don't try to trim a bunch of parts that don't exist.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 60b729e32602b7401e15957cef976386281c4ccb) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 6 Jan 2021 08:40:50 +0000 (03:40 -0500)]
rgw: Lay groundwork for multigenerational datalog
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 27ca609755a2c0e8fd501be46bc20026aa33b93c) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Mon, 23 Nov 2020 20:29:35 +0000 (15:29 -0500)]
rgw: Add rgw_complete_aio_completion()
To manually complete an asynchronous librados call.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 97c3f2b4e6d0a8d0c2366d6dca4570e063af7953) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 6 Jan 2021 01:00:07 +0000 (20:00 -0500)]
rgw: Generational support for logback switching
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 6b50f6d6def59e3c4b2db2d5311a887127b4804b) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Mon, 4 Jan 2021 00:08:09 +0000 (19:08 -0500)]
rgw: Logback generation data structures
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit b97b207928c60b48fe405ab38be15ba55f927d5c) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sun, 3 Jan 2021 23:32:50 +0000 (18:32 -0500)]
rgw/datalog: make get_oid take generation
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit f7b850f7aa84d9cf24b4eaebbe51c7ee221bbd44) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sat, 21 Nov 2020 23:20:57 +0000 (18:20 -0500)]
rgw: Move get_oid back to RGWDataChangesLog
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit bdd3528e54e399135f602e1f7e94d070d89b8c99) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sat, 21 Nov 2020 22:05:04 +0000 (17:05 -0500)]
rgw/datalog: Pass IoCtx in, don't have each backend make its own
Also don't use svc_cls.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 7f097cf8db433bb4c82a9bafc44e43b84f79bca4) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sat, 21 Nov 2020 20:45:12 +0000 (15:45 -0500)]
rgw: Use refactored log backing tools
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit da6223d281e33e43fa74c50f4d0eedb5ac25ace4) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sat, 21 Nov 2020 06:44:36 +0000 (01:44 -0500)]
rgw: Factor out tool to deal with different log backing
Read through the shards of a log and find out what kind it is.
Also remove a log.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit ed15d03f068c6f6e959f04d9d8f99eac82ebbd29) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Tue, 3 Nov 2020 21:02:26 +0000 (16:02 -0500)]
rgw: Add AioCompletion* versions for the rest of the FIFO methods
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 665573ab8905bfa2e1ede6fc3be9bc80a625cb49) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sat, 21 Nov 2020 22:04:12 +0000 (17:04 -0500)]
cls/log: Take const references of things you won't modify
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 73ea8cec06addc6af2ba354321f1099f657f13c5) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Nizamudeen A [Tue, 23 Mar 2021 07:10:46 +0000 (12:40 +0530)]
mgr/dashboard: Fix for alert notification message being undefined
Prometheus alert notification message in the dashboard always comes up
as undefined. Its because we were showing the alert.summary instead of
alert.description for displaying the message. I couldn't find the
summary field in the ceph_default_alerts.yml file. So removed all the
Summary fields from the dashboard code.
Sage Weil [Fri, 2 Apr 2021 14:58:51 +0000 (09:58 -0500)]
Merge PR #40544 into pacific
* refs/pull/40544/head:
mgr/orchestrator: move PORTS column in 'orch ps' output
cephadm: fix failure when using --apply-spec and --shh-user
cephadm: specify addr on bootstrap's host add
mgr/cephadm: don't have upgrade fail if "." in patch section of version
Daniel Pivonka [Mon, 29 Mar 2021 17:47:11 +0000 (13:47 -0400)]
cephadm: fix failure when using --apply-spec and --shh-user
ssh-copy-id was being run as the root user because cephadm requires sudo
so it was trying to use the root users ssh keys to copy the cephadm ssh key to the hosts in the spec
this would fail if the root user did not has passwordless ssh to the host being added
solution run ssh-copy-id as the user ssh-user
additionally fix the check to not copy the cephadm ssh key to the bootstrap host
Patrick Donnelly [Wed, 10 Mar 2021 22:46:27 +0000 (14:46 -0800)]
qa: reduce dependence on teuthology role list for mds
It's not yet possible to completely remove the dependency on
mds_ids/mds_daemons in the CephFS tests but this commit reduces it
enough for most code paths to work with cephadm.
The main change here is use of CephManager.do_rados, with some
improvements.
Kefu Chai [Tue, 30 Mar 2021 18:32:38 +0000 (02:32 +0800)]
mgr/PyModule: put mgr_module_path before Py_GetPath()
pip comes with _vendor/progress. so there is chance to import the vendored
version of "progress" module instead of the "progress" mgr module, and
fail to import the latter.
in this change, the order of paths are rearranged so the configured
`mgr_module_path` is put before the return value of `Py_GetPath()`.
Aashish Sharma [Mon, 8 Mar 2021 09:44:00 +0000 (15:14 +0530)]
mgr/dashboard: Remove username, password fileds from -Cluster/Manager Modules/dashboard
Username, password fields are empty in Cluster/Manager Modules/dashboard.Since this functionality is when dashboard supported single user-password, now we need to remove these fields from here.
Ali Maredia [Tue, 2 Mar 2021 05:24:14 +0000 (00:24 -0500)]
rgw: add latency to the request summary of an op
Latency of a request added at the end of request
summary rgw log line. This summary line also contains
information about the request like the op, bucket,
object, http status.
Sage Weil [Mon, 29 Mar 2021 13:42:03 +0000 (08:42 -0500)]
mgr/cephadm/upgrade: ignore deployed_by until mgr is upgraded
Until we upgrade the mgr itself, we will never be able to make our
deployed daemons have a deployed_by == target_digests. Ignore those
daemons until the mgr is the right version.
Sage Weil [Sun, 28 Mar 2021 18:07:27 +0000 (13:07 -0500)]
Merge PR #40437 into pacific
* refs/pull/40437/head:
mgr/cephadm: make upgrade progress bar mention target version, not repo digest
doc/cephadm: fix rgw realm and zone flags
mgr/volumes: do not overwrite existant mds specs
mgr/cephadm: no-overwite flag for apply command
mgr/orchestrator: remove image name field from 'orch ps' and 'orch ls'
cephadm: fix parsing of keepalived version (drop leading 'v')
cephadm: keepalived needs --cap-add=NET_RAW
cephadm: fix --cap-add=NET_ADMIN
cephadm: fix quoting for keepalived env var
mgr/cephadm: ha-rgw: use correct port
cephadm: validate fsid during cephadm shell command
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
projects / ceph.git / log