Sage Weil [Tue, 17 Mar 2020 20:03:32 +0000 (15:03 -0500)]
mgr/balancer: tolerate pgs outside of target weight map
We build a target weight map based on the primary crush weights, and
ignore weights that are 0. However, it's possible that existing PGs are
on other OSDs that have weight 0 because the weight-set weight is >0.
That leads to a KeyError exception when we
pgs_by_osd[osd] += 1
and the key isn't present. Fix by simply populating those keys as we
encounter OSDs. Drop the old initialization loop. The net of this is
we may have OSDs outside of target_by_root (won't matter, as far as I can
tell) and we won't have keys for osds with weight 0 (also won't matter,
as far as I can tell).
Jan Fajerski [Wed, 4 Mar 2020 10:39:40 +0000 (11:39 +0100)]
ceph-volume: available_lvm: vg space takes precedence
This changes available_lvm to check for generic reasons only if no VGs
were found. A VG can contain a (mounted) lv, which triggers the
ro/locked test, despite the VG having space available.
Volker Theile [Wed, 6 Nov 2019 14:02:49 +0000 (15:02 +0100)]
mgr/dashboard: Refactor Python unittests
* Make use of the KVStoreMockMixin class to get rid off duplicate code.
* Fake the index.html file to be able to run tests/test_home.py locally without building the frontend in production mode.
* Encapsulate helper functions in controllers/home.py, otherwise tests/test_feature_toggles.py need to fake the filesystem because load_controllers() will load the home.py controller and fail due missing files in the filesystem.
Tim Serong [Thu, 2 Apr 2020 05:46:57 +0000 (16:46 +1100)]
mgr/PyModule: fix missing tracebacks in handle_pyerror()
In certain cases, errors raised in mgr modules don't actually result in a
proper traceback in the mgr log; all you see is a message like "'Hello'
object has no a ttribute 'dneasdfasdf'", but you have no idea where that
came from, which is a complete PITA to debug.
Here's what's going on: handle_pyerror() calls PyErr_Fetch() to get
information about the error that occurred, then passes that information
back to python's traceback.format_exception() function to get the traceback.
If we write code in an mgr module that explicitly raises an exception
(e.g.: 'raise RuntimeError("that didn't work")'), the error value returned
by PyErr_Fetch() is of type RuntimeError, and traceback.format_exception()
does the right thing. If however we accidentally write code that's just
broken (e.g.: 'self.dneasdfasdf += 1'), the error value returned is not
an actual exception, it's just a string. So traceback.format_exception()
freaks out with something like "'str' object has no attribute '__cause__'"
(which we don't actually ever see in the logs), which in turn dumps us in a
"catch (error_already_set const &)" block, which just prints out the
single line error string.
https://docs.python.org/3/c-api/exceptions.html#c.PyErr_NormalizeException
tells us that "Under certain circumstances, the values returned by
PyErr_Fetch() below can be “unnormalized”, meaning that *exc is a class
object but *val is not an instance of the same class.". And that's exactly
the problem we're having here. We're getting a 'str', not an Exception.
Adding a call to PyErr_NormalizeException() turns the value back into a
proper Exception type and traceback.format_exception() now always does the
right thing.
I've also added calls to peek_pyerror() in the catch blocks, so if anything
else ever somehow causes traceback.format_exception to fail, we'll at least
have an idea of what it is in the log.
Sage Weil [Wed, 11 Sep 2019 22:26:52 +0000 (17:26 -0500)]
mon: disable min pg per osd warning
Now that the pg_autoscaler is on by default, it is "normal" (and okay) to
have a small number of PGs in the cluster if the overall cluster usage is
also low. This setting just results in a health warning out of the box
when you create a pool and haven't written any data yet.
J. Eric Ivancich [Fri, 31 Jan 2020 20:01:40 +0000 (15:01 -0500)]
rgw: fix bug with (un)ordered bucket listing and marker w/ namespace
When listing without specifying a namespace, the returned entries
could be in one or more namespaces. The marker used to continue the
listing may therefore contain a namespace, and that needs to be
preserved. This fixes a bug in both ordered and unordered listings
where it was not preserved.
Shilpa Jagannath [Tue, 26 Nov 2019 08:03:52 +0000 (13:33 +0530)]
rgw: when a period lookup for oldest_realm_epoch returns an ENOENT,
find the oldest one and update RGWMetadataLogHistory. This is to avoid an
empty cursor being passed in to ceph_assert() in PurgePeriodLogsCR::operate()
in case of incomplete period history.
mgr/dashboard: fix COVERAGE_PATH in run-backend-api-tests.sh
As we cannot backport directly https://github.com/ceph/ceph/pull/33407
to nautilus and COVERAGE_PATH is invalid for CentOS7 + py2,
this fix is applied directly to nautilus.
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
Yan, Zheng [Tue, 9 Oct 2018 03:46:56 +0000 (11:46 +0800)]
mds: handle bad purge queue item encoding
The bad encoding was introduced by commit a88f8d5eb4
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit ba06fcbe3e8345dc2c4c4c3dc3bcc18acf5ab076)
Conflicts:
src/mds/PurgeQueue.cc: advance changed to += Fixes: https://tracker.ceph.com/issues/36635
Note: This commit from v13.2.3 fixes a bad backport in v13.2.2. It is
also required in Octopus/Nautilus to handle upgrades.
(cherry picked from commit b73d1989bcbea227017607f8dd6e79633ec11f8f)
Conflicts:
src/mds/PurgeQueue.cc: += changed to advance
Sage Weil [Tue, 3 Mar 2020 16:09:06 +0000 (10:09 -0600)]
common/ceph_time: tolerate mono time going backwards
Some kernels (and possibly some hardware?) can trigger a monotonic clock
that goes back in time. That, in turn, can lead to a negative monotonic
time span. This would trigger an assert.
This this problem seems to be widespread, tolerate the case and interpret
it as a 0-length interval (vs something negative).
Fixes: https://tracker.ceph.com/issues/44078 Fixes: https://tracker.ceph.com/issues/43365 Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit cd00378720459d92394697e2cb8086f30c220312)
Conflicts:
src/pybind/rbd/rbd.pyx
- no "snap_exists", "snap_get_name", "snap_get_id",
"mirror_image_create_snapshot", "mirror_image_get_mode", "config_set",
"config_get", "config_remove", "snap_get_mirror_namespace", in nautilus
- nautilus "mirror_image_enable" does not take any argument
zhangdaolong [Tue, 24 Mar 2020 00:51:44 +0000 (08:51 +0800)]
pybind/rbd: fix no lockers are obtained, ImageNotFound exception will be output
No lockers are obtained, ImageNotFound exception will be output,
but tht image is always exist.when lockers number is zero,
Should not output any exceptions。
Fixes: https://tracker.ceph.com/issues/44613 Signed-off-by: zhangdaolong <zhangdaolong@fiberhome.com>
(cherry picked from commit a183aac978dac69f996250324975073a78cb476b)
Conflicts:
qa/tasks/ceph.py
- when tmpfs journal was enabled, something slightly different was
happening in nautilus compared to master, but this commit is ripping
out the whole thing
Tatjana Dehler [Fri, 8 Nov 2019 12:51:40 +0000 (13:51 +0100)]
mgr/dashboard: fix tests in order to match pg num conventions
Update the tests test_ganesha and test_rbd_mirroring in order
to match the PG num conventions. It prevents the health warning
'POOL_PG_NUM_NOT_POWER_OF_TWO' from being shown.
This is an alternative (and straightforward) way to specify barred space not allowed for 'use
some extra' BlueFS volume selector policy.
Disabled by default since it should depend on RocksDB settings and actual volume size.
Robin H. Johnson [Fri, 27 Mar 2020 19:48:13 +0000 (20:48 +0100)]
rgw: reject control characters in response-header actions
S3 GetObject permits overriding response header values, but those inputs
need to be validated to insure only characters that are valid in an HTTP
header value are present.
Credit: Initial vulnerability discovery by William Bowling (@wcbowling)
Credit: Further vulnerability discovery by Robin H. Johnson <rjohnson@digitalocean.com> Signed-off-by: Robin H. Johnson <rjohnson@digitalocean.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> Reviewed-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d8dd5e513c0c62bbd7d3044d7e2eddcd897bd400)
Ilya Dryomov [Fri, 6 Mar 2020 19:16:45 +0000 (20:16 +0100)]
msg/async/crypto_onwire: fix endianness of nonce_t
As a AES-GCM IV, nonce_t is implicitly shared between server and
client. Currently, if their endianness doesn't match, they are unable
to communicate in secure mode because each gets its own idea of what
the next nonce should be after the counter is incremented.
Several RFCs state that the nonce counter should be BE, but since we
use LE for everything on-disk and on-wire, make it LE.
The secure mode uses AES-128-GCM with 96-bit nonces consisting of a
32-bit counter followed by a 64-bit salt. The counter is incremented
after processing each frame, the salt is fixed for the duration of
the session. Both are initialized from the session key generated
during session negotiation, so the counter starts with essentially
a random value. It is allowed to wrap, and, after 2**32 frames, it
repeats, resulting in nonce reuse (the actual sequence numbers that
the messenger works with are 64-bit, so the session continues on).
Because of how GCM works, this completely breaks both confidentiality
and integrity aspects of the secure mode. A single nonce reuse reveals
the XOR of two plaintexts and almost completely reveals the subkey
used for producing authentication tags. After a few nonces get used
twice, all confidentiality and integrity goes out the window and the
attacker can potentially encrypt-authenticate plaintext of their
choice.
We can't easily change the nonce format to extend the counter to
64 bits (and possibly XOR it with a longer salt). Instead, just
remember the initial nonce and cut the session before it repeats,
forcing renegotiation.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com> Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com> Reviewed-by: Sage Weil <sage@redhat.com>
Conflicts:
src/msg/async/ProtocolV2.cc [ context: commit 697aafa2aad2
("msg/async/ProtocolV2: remove unused parameter") not in
nautilus ]
src/msg/async/ProtocolV2.h [ context: commit ed3ec4c01d17
("msg: Build target 'common' without using namespace in
headers") not in nautilus ]
projects / ceph.git / log