Igor Fedotov [Fri, 20 Mar 2020 17:01:03 +0000 (20:01 +0300)]
os/bluestore: allow main device expansion when DB needs space to
recover.
This patch provides a fix for an expansion "deadlock" state:
OSD is out of space at main disk and DB needs to recover and
flush some additional data to the disk. To expand the main device one needs
to alter freelist parameters in DB which in turn triggers DB recovery.
The following approach is used to fix that:
1) Major set of Freelist Manager's key parameters
(primarily actual size and block count) is kept at device label from now on.
2)Secondary replica of parameters set is preserved at DB mostly to permit
downgrades if any.
3) At first expansion procedure updates bdev label only. No updates to DB takes
place hence read-only mode for DB is enough to execute it.
4) On regular BlueStore mount (second stage of expantion does this as
well) DB is initially loaded in read-only.
5)Freelist manager is started and initiated with parameters from device label.
Original allocation bits are taken from DB, new ones (appeared after
expansion but not present in DB) are virtually treated by FM as free.
Allocator is initialed with expanded size accordingly.
6) DB is reopened in full access mode. During this process BlueFS is able
to request for additional space from main device. This still doesn't need
DB update as BlueFS tracks owned extents via internal logging, not
persisting this info at DB
Hence DB recovery if any completes successfully. DB is fully operational
at this point.
7) Freelist Manager sync function is called to
commit all the changes: expanded size, location bits adjustment for
old and new boundaries.
Regular allocations that needs DB update are operational at this point.
Store is ready.
Adam Kupczyk [Tue, 17 Mar 2020 13:56:58 +0000 (14:56 +0100)]
os/bluestore: open db in read-only when expanding standalone DB/WAL
Replace mount() with cold_open() to force opening db in read-only mode.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com> Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 976079216cec202d47b7193d19b16f5a8578f269)
Igor Fedotov [Mon, 16 Mar 2020 13:40:54 +0000 (16:40 +0300)]
os/bluestore: show space available from bluestore when dumping bluefs devices.
This effectively allows ceph-bluestore-tool to report space available from bluestore
for bluefs-bdev-sizes command, i.e. estimate how utilized main device is.
Robin H. Johnson [Fri, 27 Mar 2020 19:48:13 +0000 (20:48 +0100)]
rgw: reject control characters in response-header actions
S3 GetObject permits overriding response header values, but those inputs
need to be validated to insure only characters that are valid in an HTTP
header value are present.
Credit: Initial vulnerability discovery by William Bowling (@wcbowling)
Credit: Further vulnerability discovery by Robin H. Johnson <rjohnson@digitalocean.com> Signed-off-by: Robin H. Johnson <rjohnson@digitalocean.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> Reviewed-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d8dd5e513c0c62bbd7d3044d7e2eddcd897bd400)
Ilya Dryomov [Fri, 6 Mar 2020 19:16:45 +0000 (20:16 +0100)]
msg/async/crypto_onwire: fix endianness of nonce_t
As a AES-GCM IV, nonce_t is implicitly shared between server and
client. Currently, if their endianness doesn't match, they are unable
to communicate in secure mode because each gets its own idea of what
the next nonce should be after the counter is incremented.
Several RFCs state that the nonce counter should be BE, but since we
use LE for everything on-disk and on-wire, make it LE.
The secure mode uses AES-128-GCM with 96-bit nonces consisting of a
32-bit counter followed by a 64-bit salt. The counter is incremented
after processing each frame, the salt is fixed for the duration of
the session. Both are initialized from the session key generated
during session negotiation, so the counter starts with essentially
a random value. It is allowed to wrap, and, after 2**32 frames, it
repeats, resulting in nonce reuse (the actual sequence numbers that
the messenger works with are 64-bit, so the session continues on).
Because of how GCM works, this completely breaks both confidentiality
and integrity aspects of the secure mode. A single nonce reuse reveals
the XOR of two plaintexts and almost completely reveals the subkey
used for producing authentication tags. After a few nonces get used
twice, all confidentiality and integrity goes out the window and the
attacker can potentially encrypt-authenticate plaintext of their
choice.
We can't easily change the nonce format to extend the counter to
64 bits (and possibly XOR it with a longer salt). Instead, just
remember the initial nonce and cut the session before it repeats,
forcing renegotiation.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com> Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com> Reviewed-by: Sage Weil <sage@redhat.com>
Conflicts:
src/msg/async/ProtocolV2.h [ context: commit ed3ec4c01d17
("msg: Build target 'common' without using namespace in
headers") not in octopus ]
> mgr_util: add CephfsClient implementation
>
> This pulls parts of the VolumesClient implementation into mgr_util to
> make the CephFS specific pieces available to other mgr modules. To
> reduce code duplication the VolumeClient now extends the CephfsClient
> class to add the volume specific methods.
> Signed-off-by: Jan Fajerski <jfajerski@suse.com>
Sage Weil [Thu, 2 Apr 2020 23:36:39 +0000 (18:36 -0500)]
cephadm: create /var/run/ceph dir via unit.run, not unit file
The systemd unit file is shared with non-ceph daemons, which (1) don't
need the /var/run directory, and (2) are based on a uid/gid from a
different container image, which means we can't figure out the right
ceph uid/gid from them to set the ownership properly.
Instead, put it in the unit.run file... and only for ceph daemons when
we have the uid/gid we need.
Daniel-Pivonka [Fri, 27 Mar 2020 18:38:41 +0000 (14:38 -0400)]
mgr/cephadm: add useful error if python3 is not on remote host
show 'Error ENOENT: New host example (example) failed check: ["Can't communicate with
remote host, possibly because python3 is not installed there"]' instead of traceback
with OSError: cannot send(already closed?) when adding host if python3 is not on host
Sage Weil [Fri, 27 Mar 2020 15:39:09 +0000 (10:39 -0500)]
qa/tasks/cephadm: distribute ceph.conf and admin keyring to all nodes
Revert part of 96220c0c0574eb5b896023e1552f528bef9e1ca5 so that we still
distribute a *final* ceph.conf and admin keyring to all nodes, right after
all of the mons are up.
Sebastian Wagner [Fri, 20 Mar 2020 10:25:35 +0000 (11:25 +0100)]
python-common: add `service_name` to `ServiceSpec.to_json`
To make it compatible to `Orchestrator.describe_service`.
Otherwise we have the awkward situation that users need to
pass `service_name` to `describe_service`, but `service_id` to apply
Alfonso Martínez [Wed, 18 Mar 2020 16:19:13 +0000 (17:19 +0100)]
mgr/dashboard: fix error when enabling SSO with cert. file
Also:
* Disabled security setting 'wantNameIdEncrypted': not all Identity Providers support this and we are already requiring encrypted assertions (which is the default).
Fixes: https://tracker.ceph.com/issues/44666 Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 8b0b9277ca3aa0dc5313ac635a06c0f80000b6ac)
clyso [Tue, 31 Mar 2020 12:50:07 +0000 (14:50 +0200)]
doc/mgr/orchestrator: update cephadm shell proposed alias entry
prior change i get following error when using alias as "ceph -v":
INFO:cephadm:Inferring fsid ecfdb924-7341-11ea-943c-020100010027
INFO:cephadm:Using recent ceph image ceph/ceph:v15
/usr/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: "/usr/bin/ceph -v": stat /usr/bin/ceph -v: no such file or directory": unknown.
after:
INFO:cephadm:Inferring fsid ecfdb924-7341-11ea-943c-020100010027
INFO:cephadm:Using recent ceph image ceph/ceph:v15
ceph version 15.2.0 (dc6a0b5) octopus (rc)
zhangdaolong [Tue, 24 Mar 2020 00:51:44 +0000 (08:51 +0800)]
pybind/rbd: fix no lockers are obtained, ImageNotFound exception will be output
No lockers are obtained, ImageNotFound exception will be output,
but tht image is always exist.when lockers number is zero,
Should not output any exceptions。
Fixes: https://tracker.ceph.com/issues/44613 Signed-off-by: zhangdaolong <zhangdaolong@fiberhome.com>
(cherry picked from commit a183aac978dac69f996250324975073a78cb476b)
Avan Thakkar [Tue, 25 Feb 2020 08:49:10 +0000 (14:19 +0530)]
mgr/dashboard: add popover list of managers in landing page Fixes: https://tracker.ceph.com/issues/42979 Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit cdfeb1d196c7d47340baae2be5910b90c889e778)
Jason Dillaman [Thu, 2 May 2019 20:55:44 +0000 (16:55 -0400)]
librbd: re-add support for nautilus clients talking to jewel clusters
We want to support N - 3 client backward compatibility (special case
to support Jewel since it was a LTS release). The "get_snapshot_timestamp"
cls method does not exist in Jewel clusters so librbd should fallback
to excluding the op if it fails.
Note that this N - 3 also needs to apply for downstream releases as well,
which implies we still need Jewel for the time being.
Fixes: http://tracker.ceph.com/issues/39450 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c644121820b83c97e68f9896393a45cd34787672)
Conflicts:
src/test/librbd/image/test_mock_RefreshRequest.cc: tweaked to support pool configs
(cherry picked from commit d1a67c3bda09430bb21f768fb743b3e41c7d411e)
Jason Dillaman [Wed, 20 Mar 2019 18:40:50 +0000 (14:40 -0400)]
librbd: ignore -EOPNOTSUPP errors when retrieving image group membership
The Luminous release did not support adding images to a group (it only
included the bare-minimum support for creating groups). Commit f76df32666b
incorrectly dropped support for ignoring this possible failure. This
prevents Nautilus-release clients from opening images contained within
a Luminous-release cluster.
Neha [Sun, 22 Mar 2020 20:01:23 +0000 (20:01 +0000)]
qa/*/osd-backfill-recovery-log.sh: flush_pg_stats before checking log length
It is possible for the pg dump to not be the latest when we check for newprimary
in _common_test(). This is because mgr_stats_period is 5 seconds, and we may not
have fetched the latest stats just yet. This causes the test to look at the same
stats before and after wait_for_clean.
Sage Weil [Mon, 30 Mar 2020 13:33:32 +0000 (08:33 -0500)]
Merge PR #34255 into octopus
* refs/pull/34255/head:
qa/tasks/ceph.py: fail test if osd devices not found
qa/tasks/ceph.py: use .format to log dicts
qa/tasks/ceph.py: drop roles_to_journals and remote_to_roles_to_journals
qa/tasks/ceph.py: drop block_journal, tmpfs_journal
qa/tasks/ceph.py: cleanup: stop calling get_wwn_id_map()
Tatjana Dehler [Fri, 27 Mar 2020 14:58:42 +0000 (15:58 +0100)]
mgr/dashboard: do not fail on user creation
Accordingly to other Ceph commands do not fail on user
creation with a non-zero error code if the user already
exists.
Instead succeed and return the message 'User <username>
already exists'.
Fixes: https://tracker.ceph.com/issues/44502 Signed-off-by: Tatjana Dehler <tdehler@suse.com>
(cherry picked from commit 9992faa9479b62f05f36352daf89f6e7541f29c9)
projects / ceph.git / log