Adam Kupczyk [Wed, 1 Jul 2020 21:09:17 +0000 (23:09 +0200)]
os/bluestore: Add documentation for large bluefs log recovery
Adds additional paragraph to ceph-bluestore-tool documentation,
describing how to use *special* options --bluefs_replay_recovery
and --bluefs_replay_recovery_disable_compact to recover large
bluefs log.
Fixes: https://tracker.ceph.com/issues/46714 Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Sébastien Han [Tue, 18 Aug 2020 13:41:31 +0000 (15:41 +0200)]
ceph-volume: retry when acquiring lock fails
When preaparing the osd device with --mkfs, the ceph-osd binary tries to
acquire an exclusive lock on the device (soon to become an OSD).
Unfortunately, when running in containers, we have seen cases where
there is a race between ceph-osd and systemd-udevd to acquire a lock on
the device. Sometimes systemd-udevd gets the lock and releases it soon
so that the ceph-osd gets sometimes the lock is still held and because
ceph-osd uses LOCK_NB the command fails.
This commit retries if the lock cannot be acquired, up to 5 times for 5
seconds, this should be more than enough to acquire the lock and
proceed with the OSD mkfs.
Unfortunately, this is so transient that we cannot lock earlier from c-v,
this won't do anything.
Patrick Donnelly [Thu, 27 Aug 2020 20:43:01 +0000 (13:43 -0700)]
Merge PR #36833 into nautilus
* refs/pull/36833/head:
nautilus: mgr/volumes: convert uid and gid to integer type
mgr/volumes: Address python breakage in python 2
mgr/volumes: Update doc/cephfs/fs-volumes.rst for nautilus
mgr/volumes: Prevent subvolume recreate if trash is not-empty
mgr/volumes: Disallow subvolume group level snapshots
mgr/volumes: Add test case to ensure subvolume is marked
mgr/volumes: handle idempotent subvolume marks
mgr/volumes: Tests amended and added to ensure subvolume trash functionality
mgr/volumes: Mark subvolume root with the vxattr ceph.dir.subvolume
mgr/volumes: Move incarnations for v2 subvolumes, to subvolume trash
mgr/volumes: maintain per subvolume trash directory
mgr/volumes: make subvolume_v2::_is_retained() object property
mgr/volumes: Use snapshot root directory attrs when creating clone root
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Venky Shankar [Fri, 21 Aug 2020 14:07:37 +0000 (10:07 -0400)]
mgr/volumes: maintain per subvolume trash directory
PR https://github.com/ceph/ceph/pull/36472 introduces changes
that disallow nested nested snapshots in a subtree (subvolume)
and renames across subvolumes. This effect asynchronous purge
in mgr/volumes as subvolume are moved to a trash directory for
asynchronous deletion by purge threads.
To workaround this, start maintaining a subvolume specific
trash directory. Use the trash directory as an index to the
subvolume specific trash directory entry.
This changes subvolume deletion logic which currently relies
on `--retain-snapshots` flag to decide if the subvolume user
directory should get purged or the subvolume base directory
itself. Deleting a subvolume moves the user facing directory
to its specific trash directory. Purge threads take care of
deleting user facing directories (in trash) and the subvolume
base directory if required (when certain conditions are met).
mgr/volumes: Use snapshot root directory attrs when creating clone root
If a subvolumes mode or uid/gid values are changed post a snapshot,
and a clone of a snapshot prior to the change is initiated, the clone
inherits the current source subvolumes attributes, rather than the
snapshots attributes.
Fixing this by using the snapshots subvolume root attributes to create
the clone subvolumes root.
Following attributes are picked from the source subvolume snapshot:
- uid, gid, mode, data pool, pool namespace, quota
Patrick Donnelly [Wed, 26 Aug 2020 23:35:08 +0000 (16:35 -0700)]
Merge PR #36804 into nautilus
* refs/pull/36804/head:
qa/workunits/fs: add test for subvolume
mds: don't move inode with nlink > 1 to global snaprealm if it's in subvolume
mds: disallow hardlink across subvolume
mds: disallow across subvolume rename
mds: disallow creating snapshot on descendent directory of subvolume
mds: add vxattr that marks/clears subvolume flag
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Tatjana Dehler [Tue, 28 Jul 2020 11:18:56 +0000 (13:18 +0200)]
mgr/dashboard: wait longer for health status to be cleared
Because of reasons the cluster needs more time to recover from
HEALTH_WARN while changes are made by `test_pool_update_metadata`.
Lets wait several times for the cluster status to be HEALTH_OK
again.
Fixes: https://tracker.ceph.com/issues/46573 Signed-off-by: Tatjana Dehler <tdehler@suse.com>
(cherry picked from commit 739b365a3f6be9557ccb784819d4ad9ad524880f)
Kefu Chai [Fri, 7 Aug 2020 16:26:21 +0000 (00:26 +0800)]
rgw: hold reloader using unique_ptr
instead of using optional<> for holding reloader, use unique_ptr<>.
as `RGWRealmReloader` is neither
trivially_copy_{assignable,constructible} nor
is_trivially_move_{assignable, constructible}, because of the `Cond`
member variable. but Clang++ and libc++ still tries to rely on a
delgating copy constructor for constructing the
optional<RGWRealmReloader> instance even the optional object is
created `in_place`.
in this change, to workaround this issue, reloader is instead
constructed using make_unique<>
Yaarit Hatuka [Thu, 20 Aug 2020 18:21:11 +0000 (18:21 +0000)]
mgr/devicehealth: fix daemon filtering before scraping device
Scraping health metrics of mon devices was introduced in Nautilus, then
disabled (only in Nautilus) since the 'tell' mechanism of mons was not
reliable.
This commit fixes a bug when filtering the daemons on the device to be
scrapped (and allows scraping osd devices solely).
When:
$ ceph device scrape-health-metrics seagate_123
Error EAGAIN: device seagate_123 not claimed by any active OSD daemons
But:
$ ceph device ls
DEVICE HOST:DEV DAEMONS LIFE EXPECTANCY
seagate_123 hostname:sdc osd.1
Patrick Donnelly [Wed, 19 Aug 2020 20:49:00 +0000 (13:49 -0700)]
Merge PR #36448 into nautilus
* refs/pull/36448/head:
mgr: Add python-enum34 dependency to package for older distributions
mgr/volumes: Add documentation regarding --retain-snapshots option
mgr/volumes: Avoid trashing retained subvolume on create errors
mgr/volumes: Add subvolume v2 test cases
mgr/volumes: Derive v2 from v1 to leverage common methods
mgr/volumes: Introduce v2 subvolumes
mgr/volumes: Use operation type during subvolume open
Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Jason Dillaman [Tue, 28 Jul 2020 01:14:18 +0000 (21:14 -0400)]
librbd: update hidden global config when setting pool config override
The new "dev"-level global config setting will be updated when any
pool-level config override is updated. librbd clients will detect
the new global-level config update and trigger a refresh. This avoids
the need for potentially tens of thousands of librbd clients
registering a watch on the pool metadata object or periodically polling
the pool metadata object for updates.
Fixes: https://tracker.ceph.com/issues/46694 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f45df9fe786e8057c491c082e840483759d67e9e)
Conflicts:
src/common/options.cc
- "rbd_quiesce_notification_attempts", "rbd_default_snapshot_quiesce_mode", and
"rbd_plugins" options have not been backported to Octopus, yet
Jason Dillaman [Mon, 27 Jul 2020 19:31:09 +0000 (15:31 -0400)]
librbd: initial config watcher implementation
The config watcher will initially observe all "rbd_" configuration
updates received from the MON that have not been locally overridden
at the pool and/or image level.
Conflicts:
qa/tasks/vstart_runner.py
src/pybind/mgr/dashboard/run-backend-api-tests.sh
- Bring opt_verbose option and only dump last 1000 lines of logs
root [Fri, 26 Jun 2020 10:44:45 +0000 (12:44 +0200)]
rgw: fix double slash (//) killing the gateway
When a bucket is inialized as a static website, a curl request on the bucket with double slash kills the gateway.
The problem is on the URL handling of the subdirectory, which tries to remove the last slash of any URL, so when only / is given as a sub-directory, this results to an empty string.
rgw: policy: reuse eval_principal to evaluate the policy principal
Since the other edge case when no Principal or a NotPrincipal is supplied also
must be accounted for, which is already done in eval_principal function. Also
reraising the error as Effect::Pass in line with the previous output, though an
Effect::Deny would also work here.
monclient: use "is_connected" check when scheduling tick
When schedule_tick is called for the first time (in init) we are
still not in hunting mode so mon_client_ping_interval and
mon_client_log_interval were used to schedule the tick.
Rishabh Dave [Sat, 1 Aug 2020 18:36:25 +0000 (00:06 +0530)]
ceph-volume: add tests for new functions that run LVM commands
... and for supporting functions. Specifically, these functions are
get_pvs(), get_vgs(), get_lvs(), get_first_pv(), get_first_vg(),
get_first_lv() and make_filters_lvmcmd_ready().
Rishabh Dave [Fri, 3 Jul 2020 06:17:50 +0000 (11:47 +0530)]
ceph-volume: remove class Volumes from api/lvm.py
And remove methods is_lv(), get_lv() and get_lv_from_argument() since
these methods uses class Volumes and get_first_lv() or get_lv() can be
easily used instead when needed in future.
Rishabh Dave [Fri, 3 Jul 2020 05:28:02 +0000 (10:58 +0530)]
ceph-volume: remove class VolumeGroups from api/lvm.py
And remove get_vg() since its based on VolumesGroups and get_first_vg()
or get_vg() can be easily used instead in future when needed. Also,
update tests accordingly.
Rishabh Dave [Fri, 3 Jul 2020 05:15:23 +0000 (10:45 +0530)]
ceph-volume: remove PVolumes from api/lvm.py
And method get_pv() since it's based on PVolumes and get_first_pv() and
get_pvs() can be easily used instead in future. Also, update tests
accordingly.
Conflicts:
src/ceph-volume/ceph_volume/tests/util/test_device.py
The dummy device path in tests "test_is_lvm_memeber",
"test_is_not_lvm_memeber" and "test_is_partition" are slightly
different from master.
Conflicts:
qa/tasks/mgr/dashboard/helper.py
A variable 'maxDiff' has been introduced in master and not been
backported: https://github.com/ceph/ceph/pull/35768
Mike Christie [Thu, 9 Jan 2020 00:03:40 +0000 (18:03 -0600)]
selinux: Fix ceph-iscsi configfs access
This fixes the the following selinux error when using ceph-iscsi's
rbd-target-api daemon (rbd-target-gw has the same issue). They are
a result of the a python library, rtslib, which the daemons use.
Additional Information:
Source Context system_u:system_r:ceph_t:s0
Target Context system_u:object_r:configfs_t:s0
Target Objects
/sys/kernel/config/target/iscsi/iqn.2003-01.com.re
dhat:ceph-iscsi/tpgt_1/attrib/authentication
[
file ]
Source rbd-target-api
Source Path /usr/libexec/platform-python3.6
Port <Unknown>
Host ans8
Source RPM Packages platform-python-3.6.8-15.1.el8.x86_64
Target RPM Packages
Policy RPM selinux-policy-3.14.3-20.el8.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Enforcing
Host Name ans8
Platform Linux ans8 4.18.0-147.el8.x86_64 #1 SMP
Thu Sep 26
15:52:44 UTC 2019 x86_64 x86_64
Alert Count 1
First Seen 2020-01-08 18:39:47 EST
Last Seen 2020-01-08 18:39:47 EST
Local ID 6f8c3415-7a50-4dc8-b3d2-2621e1d00ca3
The disk predictor assumes that it has at least 6 days worth of data to work
with, and the code ensures that `health_data` contains at least 6 elements.
However, it then gets fitlered down into a `predict_datas` array that can
contain less than 6 elements in some cases.
This commit ensures that `predict_datas` contains 6 elements or more before
passing it on to the disk predictor.
projects / ceph.git / log