Kotresh HR [Fri, 19 Feb 2021 11:27:23 +0000 (16:57 +0530)]
mgr/volumes: Bump up AuthMetadataManager's version
With ceph_volume_client and mgr-volumes co-existing
for sometime, the version of both needs to be same.
The ceph_volume_client version <=5 can't decode
'subvolumes' key in auth-metadata file. Hence to
handle version in-compatibility, the version of
ceph_volume_client is bumped up to 6 and the same
needs to be done in mgr-volume's AuthMetadataManager
Kotresh HR [Fri, 19 Feb 2021 11:12:33 +0000 (16:42 +0530)]
pybind/ceph_volume_client: Bump up the version and compat_version to 6
With 'volumes' key updated to 'subvolumes', the version of
ceph_volume_client <= 5 can't decode auth-metadata file. Hence
bumping up ceph_volume_client version and compat_version to 6.
Kotresh HR [Mon, 15 Feb 2021 16:26:51 +0000 (21:56 +0530)]
pybind/ceph_volume_client: Update the 'volumes' key to 'subvolumes' in auth metadata file
The older auth metadata files before nautilus release stores
the authorized subvolumes using the 'volumes' key. As the
notion of 'subvolumes' brought in by mgr/volumes, it makes
sense to use 'subvolumes' key. This patch would be tranparently
update 'volumes' key to 'subvolumes' and newer auth metadata
files would store them with 'subvolumes' key.
Also fails the deauthorize if the auth-id doesn't exist.
These two vxattrs will only exist in local client side, with which
we can easily know which mountpoint the file belongs to and also
they can help locate the debugfs path quickly.
Conflicts:
src/client/Client.cc
- add .hidden member because we still need it
in nautilus
src/client/Client.h
- drop the mirror.info xattr related code
because nautilus does not introduce it
Kotresh HR [Fri, 5 Feb 2021 18:05:22 +0000 (23:35 +0530)]
qa: Fix a few mgr/volume test cases
Recovering dirty auth metadata file might not retain the order,
fixed the comparison in 'test_recover_auth_metadata_during_authorize'
and 'test_recover_auth_metadata_during_deauthorize'.
move all of the etag verifier initialization into a helper function.
none of the errors there should be fatal and fail the download, they
should just turn etag verification off
rgw: rgw_sync_obj_etag_verify accounts for compressed multipart uploads
the etag verifier for multipart uploads uses the manifest to get the
logical offsets for each part. but when compression is enabled, those
are offsets into the compressed data. use the source object's compression
info to translate those compressed part offsets back to their original
offsets
Prasad Krishnan [Fri, 6 Mar 2020 05:08:05 +0000 (05:08 +0000)]
[RGW][Multisite] Add multisite verifier support for MPU objects
The Etag for MPU objects is calculated using a method different from how
it is done for atomic objects. This patch makes use of the RGWObjManifest
to determine the parts in the source cluster and re-computes the ETag in
a similar fashion at the destination cluster during multisite sync for
verification.
Prasad Krishnan [Sun, 23 Feb 2020 06:09:49 +0000 (11:39 +0530)]
RGW:Multisite: Verify if the synced object is identical to source
Introduce an option 'rgw_copy_verify_object' which allows the object
copied from remote cluster through multisite sync is identical to the
source object. This is done by generating the MD5 checksum of the data
being copied and compared to the ETAG stored as part of the object's
attribute.
Kotresh HR [Sat, 23 Jan 2021 17:03:32 +0000 (22:33 +0530)]
ceph_volume_client: Fix failure of test_idempotency
With the test environment, 'args must be encodeable
as a bytearray' error is seen for 'ceph_mds_command'.
Hence removed tuple and passed the JSON formatted string.
Kotresh HR [Tue, 5 Jan 2021 12:55:54 +0000 (18:25 +0530)]
mgr/volumes: Update the 'volumes' key to 'subvolumes' in auth metadata file
The older auth metadata files created by CephVolumeClient stores the
authorized subvolumes using the 'volumes' key as the notion of
'subvolumes' brought in by mgr/volumes. Hence, this would be tranparently
updated to 'subvolumes' and newer auth metadata files would store them
with 'subvolumes' key.
Also fails the deauthorize if the auth-id doesn't exist.
Optionally allow authorizing auth-ids not created by mgr plugin
via the option 'allow_existing_id'. This can help existing deployers
of manila to disallow/allow authorization of pre-created auth IDs
via a manila driver config that sets 'allow_existing_id' to False/True.
Kotresh HR [Tue, 15 Dec 2020 12:01:54 +0000 (17:31 +0530)]
mgr/volumes: Preserve existing caps while authorize/deauthorize auth-id
Authorize/Deauthorize used to overwrite the caps of auth-id which would
end up deleting existing caps. This patch fixes the same by retaining
the existing caps by appending or deleting the new caps as needed.
Kotresh HR [Mon, 4 Jan 2021 13:04:54 +0000 (18:34 +0530)]
mgr/volumes: Disallow authorize existing auth_id
This patch disallow the mgr plugin to authorize the auth_id
which is not created via mgr plugin. Those auth_ids could be
created by other means for other use cases which should not be modified
via mgr plugin.
Kotresh HR [Wed, 18 Nov 2020 10:13:25 +0000 (15:43 +0530)]
mgr/volumes: Persist auth and subvolume metadata
1. Subvolume create and delete operations create and delete subvolume
metadata file respectively.
2. Subvolume authorize creates the auth meta file and persists the
required metadata on subvolume metadata file and auth metdata file
on disk. Subvolume deauthorize clears the required metadata on
both metadata files.
Ramana Raja [Thu, 11 Jul 2019 11:14:35 +0000 (16:44 +0530)]
qa/tasks/cephfs: test `fs subvolume authorize/deauthorize`
Fixes: https://tracker.ceph.com/issues/40401 Signed-off-by: Ramana Raja <rraja@redhat.com> Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 7c98dc1ad35a2244cbf949be9782a3412886b305)
Conflicts:
qa/tasks/cephfs/test_volumes.py: Few of the tests are re-organized
and CLIENTS_REQUIRED should be 2 for these tests which was 1
for existing tests. Resolved the same.
Jan Fajerski [Tue, 22 Dec 2020 13:29:02 +0000 (14:29 +0100)]
ceph-volume: add some flexibility to bytes_to_extents
For batch we want to report the projected sizes before creating any LVM
structures, we use the device size for that. This means we pass this
projected size to lvm/api.py::create_lv. This could result in situations
where we request more extents for the new LV than are actually
available, as LVM takes up some available space on the block device
(plus alignment).
We fix this by adjusting the extent count downwards if its less than 1% off the
requested value.
Fixes: https://tracker.ceph.com/issues/47758 Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit ece6e7eb9a92cb944b2f201a05a5acfb877c2dba)
Resolved some trivial conflicts and replaced the heading of the newly
added section in the documentation to another level, as the
documentation between master and octopus differs quite much.
Jason Dillaman [Tue, 14 Jul 2020 22:49:30 +0000 (18:49 -0400)]
crush/CrushWrapper: rebuild reverse maps after rebuilding crush map
The Objecter will crash when localized reads are enabled and two threads
attempt to rebuild the (invalidated) reverse maps concurrently. This
should address the issue for the Objecter use-case without the need to
add additional locking.
Fixes: https://tracker.ceph.com/issues/44311 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8b866794f5b3674c5e3ad9adceb5e3230d55a0e0)
Ilya Dryomov [Wed, 20 Jan 2021 15:00:18 +0000 (16:00 +0100)]
qa/suites/krbd: add msgr2 modes to most subsuites
basic, rbd and rbd-nomount subsuites are expanded to run with each
of ms_mode=legacy, ms_mode=crc and ms_mode=secure. This increases
the total number of jobs in the suite from 100 to 220.
fsx, singleton and thrash subsuites choose ms_mode at random (from
the above plus ms_mode=prefer-crc).
Conflicts:
qa/suites/krbd/fsx/conf.yaml [ commit cff2e49ff09e
("qa/suites/krbd: fsx with object-map and fast-diff") and
read_from_replica map option not in nautilus ]
qa/suites/krbd/fsx/ms_mode$/*.yaml [ read_from_replica map option
not in nautilus ]
Ilya Dryomov [Mon, 18 Jan 2021 12:49:49 +0000 (13:49 +0100)]
krbd: add support for msgr2
Recognize ms_mode map option and filter initial monitor addresses
accordingly: if ms_mode is not given or ms_mode=legacy, discard v2
addresses, otherwise discard v1 addresses.
Note that nothing was discarded (i.e. v2 addresses were passed to
the kernel) previously. The intent was to preserve that behaviour
in case ms_mode is not given, allowing to change the kernel default
in the future. However, it turns out that mount.ceph helper has
been misguidedly discarding v2 addresses since commit eae01275134e
("mount.ceph: fork a child to get info from local configuration"),
so that ship has sailed.
Conflicts:
doc/man/8/rbd.rst [ crush_location, read_from_replica and
compression_hint map options not in nautilus ]
src/krbd.cc [ commit a1aecba6f44d ("krbd: make wait_for_udev_add()
clearer and a bit more efficient") not in nautilus ]
src/tools/rbd/action/Kernel.cc [ commit 34f539d8af33 ("rbd:
delay parsing of default kernel map options") not in nautilus ]
Kefu Chai [Fri, 23 Aug 2019 08:42:22 +0000 (16:42 +0800)]
install-deps.sh,deb,rpm: move python-saml deps into debian/control and ceph.spec.in
these dependencies are only used for building python-saml which is in
turn used for the SAML support. this feature is tested using
`test_sso.py` while performing dashboard tests. we do not package or
ship python-saml along with other Ceph packages. so let's move these
dependencies to the "make check" sections in ceph.spec.in and
debian/control for simplifying install-deps.sh.
projects / ceph.git / log