Nizamudeen A [Tue, 2 Feb 2021 12:26:13 +0000 (17:56 +0530)]
mgr/dashboard: Host Maintenance Feature
In Cluster -> Hosts, I've added additional button to put the selected host on maintenance or exit out of the maintenance mode. Also for some hosts the ok-to-stop tests may trigger some warnings which requires a --force command to pass along with the maintenance enter command to enter a host into maintenance. In UI this is achieved using a confirmation Modal. In addition to this if the check error is It is NOT safe to stop the host then the host wont be able to put into maintenance mode.
Fixes: https://tracker.ceph.com/issues/49101 Signed-off-by: Nizamudeen A <nia@redhat.com>
for the stated purpose of hiding the %_smp_mflags macro in a higher-level macro.
But, on SUSE, the higher-level macro (%make_build) expands to:
make -O %{_smp_mflags}
The addition of the -O flag makes the build considerably slower and increases
the memory requirement. The exact reason for this is unknown - possibly it's due
to a bug in make, although the same slowness was observed with ninja as well.
In any event, this is a deal-breaker when building in the OBS, because the build
infrastructure there is optimized for builds that do not require huge amounts of
memory and we would rather have a fast build with mixed up compiler messages
than a very slow one with synced compiler messages.
Kefu Chai [Sat, 6 Mar 2021 16:32:42 +0000 (00:32 +0800)]
.github: correct the regex in mileston workflow
also use pull_request_target event so the action is run in the
context of the base of the pull request. this helps us to overcome
the "Resource not accessible by integration" issue where the action
is run in the context of the pull request.
Kefu Chai [Sat, 6 Mar 2021 07:43:33 +0000 (15:43 +0800)]
cmake: make the linkage to pmem::pmemobj public
tools/ceph-dencoder/rbd_types.cc includes Types.h which in turn includes
libpmemobj.h via librbd/cache/pwl/Types.h. and ceph-dencoder pulls in the
rbd_type.cc's linked libraries by linking against rbd_types. but before
this change, rbd_types links against pmem::pmemobj as a PRIVATE library.
so, if we want to pull in rbd_types linkage we should always link
rbd_types as a PUBLIC library. as rbd_types include libpmemobj.h in its
header file.
Kefu Chai [Sat, 6 Mar 2021 04:22:39 +0000 (12:22 +0800)]
cmake: link libpmemobj against libpmem
libpmemobj should link against libpmem, but, in CMake, imported library
does not allow PRIVATE linkage. so pmem::pmem is added to the list of
INTERFACE_LINK_LIBRARIES.
Kefu Chai [Fri, 5 Mar 2021 06:04:23 +0000 (14:04 +0800)]
cmake: support COMPONENTS param in Findpmem.cmake
add two components: pmem and pmemobj to this package. so we can find
them and link against them in a more intuitive way.
before this change the COMPONENTS parameter passed to
find_package(pmem ...)
is dropped on the floor and ignored.
after this change, it is checked and taken into consideration.
also, in this change, the exposed variables are renamed from
PMEM_* to pmem_*
to be consistent with the package name. it's encouraged to be consistent
with the package name when it comes to the INCLUDE_DIR and LIBRARIES
variable names.
Marcus Watts [Tue, 2 Mar 2021 04:10:35 +0000 (23:10 -0500)]
rgw/kms/vault - PendingReleaseNotes pointer
The "new" transit logic requires configure changes to be
effective. Here is a pointer to the revised
documentation for those who already have data encrypted
using the previous version.
Fixes: http://tracker.ceph.com/issues/48746 Signed-off-by: Marcus Watts <mwatts@redhat.com>
Marcus Watts [Tue, 26 Jan 2021 01:49:16 +0000 (20:49 -0500)]
rgw/kms/vault - s3tests for both old and new test logic.
Test both "old" and "new" transit logic with s3tests. Does not test
migration - that will need to be done separately. I've added
a "flavor" parameter so the test logic can tell the difference
between the "old" engine and the "new" engine. The vault
keys creation logic now has options to determine whether
the keys created are exportable (needed for the old transit
engine), or not (should be the case going forward with the
new transit engine.)
Fixes: http://tracker.ceph.com/issues/48746 Signed-off-by: Marcus Watts <mwatts@redhat.com>
Marcus Watts [Fri, 8 Jan 2021 22:49:20 +0000 (17:49 -0500)]
rgw/kms/vault - rework unit test logic for new transit logic.
The "new" transit logic is organized quite differently
than the old logic, so the existing unit test logic was
very broken. Additionally, it's possible to test the
input arguments and send_request() has more of them now,
so add logic to verify most of those arguments are correct.
Fixes: http://tracker.ceph.com/issues/48746 Signed-off-by: Marcus Watts <mwatts@redhat.com>
Marcus Watts [Tue, 5 Jan 2021 02:22:07 +0000 (21:22 -0500)]
rgw/kms/vault - document configuration for new transit logic
Using the new transit logic requires slightly different configuration.
additionally there is some backwards compatibility support, which
also needed documentation.
The existing description of how to configure hashicorp vault
to work with ceph was also incomplete. I've fleshed that out a bit,
including considerably more information on how to use configure
and use the vault secret agent with ceph.
Fixes: http://tracker.ceph.com/issues/48746 Signed-off-by: Marcus Watts <mwatts@redhat.com>
Marcus Watts [Tue, 5 Jan 2021 02:21:33 +0000 (21:21 -0500)]
rgw/kms/vault - new transit logic - fix compat logic
Teuthology passes in a vault uri that ends in /export/encryption-key/
So: we need to handle (and ignore) trailing slashes when deciding
to enable compatibility support.
Fixes: http://tracker.ceph.com/issues/48746 Signed-off-by: Marcus Watts <mwatts@redhat.com>
Marcus Watts [Tue, 8 Dec 2020 23:09:04 +0000 (18:09 -0500)]
rgw/kms/vault - "compat" option
For transit engine:
"compat" option: 0=new only, 1=old & new, 2=old only.
This is just the option parsing itself: not the actual logic
for make_key | reconstitute_key.
Fixes: http://tracker.ceph.com/issues/48746 Signed-off-by: Marcus Watts <mwatts@redhat.com>
Marcus Watts [Mon, 7 Dec 2020 22:55:22 +0000 (17:55 -0500)]
rgw/kms/vault - encryption context - first part
This includes the logic to process the user provided
encryption context, turn it into "canonical json", and
to add in a default arn if it's not present.
Also present here is the start of logic to distinguish
between "prepare_encrypt" and "prepare_decrypt" at a lower
level; as "make_key" and "reconstitute_key" these will be
the functions that separately create a new datakey using
the vault transit operation, and to retrieve a previously
stored datakey.
Fixes: http://tracker.ceph.com/issues/48746 Signed-off-by: Marcus Watts <mwatts@redhat.com>
Marcus Watts [Mon, 7 Dec 2020 22:53:05 +0000 (17:53 -0500)]
rgw/kms/vault - define attribute to store encryption context
For rgw sse:kms use, the aws s3 standard provides an attribute
to store the base-64 encoded canonical json "encryption context".
This should be used to vary the per-object keys used for the
actual object encryption.
Fixes: http://tracker.ceph.com/issues/48746 Signed-off-by: Marcus Watts <mwatts@redhat.com>
Marcus Watts [Mon, 7 Dec 2020 22:48:31 +0000 (17:48 -0500)]
rgw/kms/vault - share get/set attr between rgw_crypt.cc and rgw_kms.cc
In order to pass down and manage "attrs" from crypt logic to kms
logic, it's necessary to share the functions that can get and
set strings in that structure. Eventually, I plan to have
the various engines store and retrieve a per-object "datakey" that
is encrypted (wrapped) by the named kms key.
Fixes: http://tracker.ceph.com/issues/48746 Signed-off-by: Marcus Watts <mwatts@redhat.com>
* refs/pull/38819/head:
win32*.sh: use ninja instead of make
doc: add ceph-dokan documentation
cephfs: Add ceph-dokan, providing Windows support
cephfs: add ceph_may_delete function
cephfs: disable inode emulation on Windows
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Marcus Watts [Mon, 7 Dec 2020 22:28:59 +0000 (17:28 -0500)]
rgw/kms/vault - relax configuration parsing for rgw_crypt_vault_secret_engine
To better manage forwards and backwards compatibility when using vault
transit for rgw object encryption (sse:kms); it is desirable to provide
parameters to control how this works. It was more attractive to overload
the existing rgw_crypt_vault_secret_engine parameter for this purpose
than to invent one or more all-new parameters.
Additionally, the enum support in the configuration parser looks like
it ought to have helpful syntax checking functionality. This is not so;
failure to provide a supported enum results in silently replacing that
with the default option, resulting in confusing and non-obvious behavior
that is not at all helpful.
This change removes the enum constraint on rgw_crypt_vault_secret_engine,
allowing for more useful messages from the rgw code, and the possibility
to also provide additional information on the same line.
Fixes: http://tracker.ceph.com/issues/48746 Signed-off-by: Marcus Watts <mwatts@redhat.com>
Marcus Watts [Mon, 7 Dec 2020 22:20:49 +0000 (17:20 -0500)]
rgw/kms/vault - need libicu to make canonical json for encryption contexts.
for encryption, aws s3 provides an "encryption" context to vary per-object
keys. The encryption context is a base64 encoded json structure, which
must be converted to a determinstic form -- "canonical json". This
requires converting all strings to a normalized canonical form: "utf-8 nfc",
it also requires thta keys in objects be sorted in a fixed order; so some
form of sorting based on nfc.
It turns out that libicu was the best way to produce utf-8 nfc (boost also
provides a mechanism, but it has many quirks). So, here are the hooks
to pull the system libicu into the build.
Fixes: http://tracker.ceph.com/issues/48746 Signed-off-by: Marcus Watts <mwatts@redhat.com>
Sage Weil [Fri, 5 Mar 2021 20:33:50 +0000 (15:33 -0500)]
Merge PR #39817 into master
* refs/pull/39817/head:
qa/suites/rados/cephadm: drop centos/rhel cephadm tests for the moment
qa/sites/rados/cephadm/thrash: rename 3-tasks.yaml/ -> 3-tasks/
qa/suites/rados/cephadm: adjust distros
qa/suites/upgrade: use kubic; test all distros
qa/suites/rados/cephadm/upgrade: use kubic on centos
qa: new kubic distro files; use kubic podman for centos/rhel
* refs/pull/38859/head:
mds: don't start purging inodes in the middle of recovery
mds: purge orphan objects created by lost async file creation
mds: track free prealloc_inos and delegated_inos separately
mds: cleanup code that purges orphan objects created by lost unsafe file creation
mds: subtract inos_to_purge from prealloc_inos when session close is logged
mds: use vector to define old_pools in PurgeItem and inode_backtrace_t
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
This broke cephadm (by triggering CEPHADM_STRAY_DAEMON) because cephadm
assumes that a daemon named rgw.r.z.foo will register as rgw.r.z.foo.
It is not clear to me that there is a way to work around this naming
mismatch that makes much sense. I think it makes more sense to focus on
the use-case that needs daemons to register under unique names and perhaps
control that naming behavior via an option or invest in providing daemons
with unique ids up front.
Lucian Petrut [Fri, 7 Aug 2020 12:07:20 +0000 (12:07 +0000)]
cephfs: Add ceph-dokan, providing Windows support
In order to expose ceph filesystems to Windows hosts, we propose
including ceph-dokan[1][2] in the Ceph tree, while updating it to
work with the latest CephFS and Dokany APIs.
Dokany is a well maintained project (fork of the original Dokan
project), allowing filesystems to be implemented in userspace,
even providing a Fuse compatibility layer.
One reason for not using the FUSE compatibility layer is that it's
only covering the high level API while Ceph is using the low level
FUSE API, which among other things is inode centric.
Changes made by this patch compared to the upstream ceph-dokan:
* support latest stable Dokany API. The upstream version relies on
the legacy unmaintained Dokan API
* return proper error codes, converting standard errno.h values to
NTSTATUS
* minor changes to support latest cephfs API
* drop duplicated ceph code, no longer needed if we're to include it
in tree. This makes it much easier to maintain.
* drop redundant permission checks, leaving it up to libcephfs
* use ceph argparse helpers
* use ceph logging and daemon initialization
* fixed unicode handling
* switched to ceph coding style
* made ceph.conf param optional, using the default path if available
* enabled setting file timestamps
* append support
* configurable timeouts set once per mount
* ensure that the error code is always logged
* various cleanups (removed unused entry points, checks that have
been moved to dokany, simplified conditional statements,
unnecessary conversions in the hot path, etc).
Lucian Petrut [Fri, 26 Feb 2021 08:32:38 +0000 (08:32 +0000)]
cephfs: add ceph_may_delete function
We're adding a new libcephfs function: ceph_may_delete. It checks
if the mount permissions allow deleting a file or directory, without
actually deleting it.
This will allow us to drop the redundant permission checks at
ceph-dokan level, saving about 1500 LOC.
Note that the "DeleteFile" Dokan callback expects us to say if a
delete operation is allowed. The "Cleanup" callback is supposed
to perform the actual file or directory deletion.
Kefu Chai [Sat, 13 Feb 2021 04:57:19 +0000 (12:57 +0800)]
doc/_theme: customize sphinx_rtd_theme
* move the breadcrumbs to the top
* add border around admonition elements
* use different colors and fonts for section headers
* add decoration lines at the bottom of breadcrumbs
* remove left and right borders in tables
* override the injected versions, the name of theme
is different from "sphinx_rtd_theme", but the
versions element is still displayed at the
bottom-left corner as "versions.html" defines.
without overriding .rst-badge CSS styling,
readthedocs puts the injected versions at
the default bottom-right corner, see
https://github.com/readthedocs/readthedocs.org/blob/2a519f1146142d18f6a63b61c2f08984067280e0/readthedocs/api/v2/templates/restapi/footer.html
projects / ceph.git / log