Adam C. Emerson [Sat, 23 Jan 2021 01:48:39 +0000 (20:48 -0500)]
rgw: Add and trim datalog generations
This lets us actually change type in mid-stream.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 32b100d797cdf88648530e0162fd103cf279df31) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 27 Jan 2021 01:07:45 +0000 (20:07 -0500)]
rgw: Actually pull logbacking_generations into datalog
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit eb0f8ffcc785146a1fb249f4531620787be216ba) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Tue, 26 Jan 2021 06:27:24 +0000 (01:27 -0500)]
rgw: Clamp FIFO trim to head
Don't try to trim a bunch of parts that don't exist.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 60b729e32602b7401e15957cef976386281c4ccb) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 6 Jan 2021 08:40:50 +0000 (03:40 -0500)]
rgw: Lay groundwork for multigenerational datalog
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 27ca609755a2c0e8fd501be46bc20026aa33b93c) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Mon, 23 Nov 2020 20:29:35 +0000 (15:29 -0500)]
rgw: Add rgw_complete_aio_completion()
To manually complete an asynchronous librados call.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 97c3f2b4e6d0a8d0c2366d6dca4570e063af7953) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 6 Jan 2021 01:00:07 +0000 (20:00 -0500)]
rgw: Generational support for logback switching
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 6b50f6d6def59e3c4b2db2d5311a887127b4804b) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Mon, 4 Jan 2021 00:08:09 +0000 (19:08 -0500)]
rgw: Logback generation data structures
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit b97b207928c60b48fe405ab38be15ba55f927d5c) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sun, 3 Jan 2021 23:32:50 +0000 (18:32 -0500)]
rgw/datalog: make get_oid take generation
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit f7b850f7aa84d9cf24b4eaebbe51c7ee221bbd44) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sat, 21 Nov 2020 23:20:57 +0000 (18:20 -0500)]
rgw: Move get_oid back to RGWDataChangesLog
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit bdd3528e54e399135f602e1f7e94d070d89b8c99) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sat, 21 Nov 2020 22:05:04 +0000 (17:05 -0500)]
rgw/datalog: Pass IoCtx in, don't have each backend make its own
Also don't use svc_cls.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 7f097cf8db433bb4c82a9bafc44e43b84f79bca4) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sat, 21 Nov 2020 20:45:12 +0000 (15:45 -0500)]
rgw: Use refactored log backing tools
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit da6223d281e33e43fa74c50f4d0eedb5ac25ace4) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sat, 21 Nov 2020 06:44:36 +0000 (01:44 -0500)]
rgw: Factor out tool to deal with different log backing
Read through the shards of a log and find out what kind it is.
Also remove a log.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit ed15d03f068c6f6e959f04d9d8f99eac82ebbd29) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Tue, 3 Nov 2020 21:02:26 +0000 (16:02 -0500)]
rgw: Add AioCompletion* versions for the rest of the FIFO methods
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 665573ab8905bfa2e1ede6fc3be9bc80a625cb49) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Sat, 21 Nov 2020 22:04:12 +0000 (17:04 -0500)]
cls/log: Take const references of things you won't modify
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 73ea8cec06addc6af2ba354321f1099f657f13c5) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Sage Weil [Fri, 2 Apr 2021 14:58:51 +0000 (09:58 -0500)]
Merge PR #40544 into pacific
* refs/pull/40544/head:
mgr/orchestrator: move PORTS column in 'orch ps' output
cephadm: fix failure when using --apply-spec and --shh-user
cephadm: specify addr on bootstrap's host add
mgr/cephadm: don't have upgrade fail if "." in patch section of version
Daniel Pivonka [Mon, 29 Mar 2021 17:47:11 +0000 (13:47 -0400)]
cephadm: fix failure when using --apply-spec and --shh-user
ssh-copy-id was being run as the root user because cephadm requires sudo
so it was trying to use the root users ssh keys to copy the cephadm ssh key to the hosts in the spec
this would fail if the root user did not has passwordless ssh to the host being added
solution run ssh-copy-id as the user ssh-user
additionally fix the check to not copy the cephadm ssh key to the bootstrap host
Kefu Chai [Tue, 30 Mar 2021 18:32:38 +0000 (02:32 +0800)]
mgr/PyModule: put mgr_module_path before Py_GetPath()
pip comes with _vendor/progress. so there is chance to import the vendored
version of "progress" module instead of the "progress" mgr module, and
fail to import the latter.
in this change, the order of paths are rearranged so the configured
`mgr_module_path` is put before the return value of `Py_GetPath()`.
Aashish Sharma [Mon, 8 Mar 2021 09:44:00 +0000 (15:14 +0530)]
mgr/dashboard: Remove username, password fileds from -Cluster/Manager Modules/dashboard
Username, password fields are empty in Cluster/Manager Modules/dashboard.Since this functionality is when dashboard supported single user-password, now we need to remove these fields from here.
Sage Weil [Mon, 29 Mar 2021 13:42:03 +0000 (08:42 -0500)]
mgr/cephadm/upgrade: ignore deployed_by until mgr is upgraded
Until we upgrade the mgr itself, we will never be able to make our
deployed daemons have a deployed_by == target_digests. Ignore those
daemons until the mgr is the right version.
Sage Weil [Sun, 28 Mar 2021 18:07:27 +0000 (13:07 -0500)]
Merge PR #40437 into pacific
* refs/pull/40437/head:
mgr/cephadm: make upgrade progress bar mention target version, not repo digest
doc/cephadm: fix rgw realm and zone flags
mgr/volumes: do not overwrite existant mds specs
mgr/cephadm: no-overwite flag for apply command
mgr/orchestrator: remove image name field from 'orch ps' and 'orch ls'
cephadm: fix parsing of keepalived version (drop leading 'v')
cephadm: keepalived needs --cap-add=NET_RAW
cephadm: fix --cap-add=NET_ADMIN
cephadm: fix quoting for keepalived env var
mgr/cephadm: ha-rgw: use correct port
cephadm: validate fsid during cephadm shell command
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Sage Weil [Thu, 25 Mar 2021 20:05:02 +0000 (15:05 -0500)]
mgr/cephadm: make upgrade progress bar mention target version, not repo digest
The repo digest is super long and meaningless for a human user. Instead,
use the target version (as soon as we know what it is--until then, use
the target image name).
Nathan Cutler [Fri, 26 Mar 2021 10:03:34 +0000 (11:03 +0100)]
rpm: drop extraneous explicit sqlite-libs runtime dependency
Commit 75980798f19b8c11efd75ba4aae3e491d4c99f98 introduced a new package,
libcephsqlite, with a hard RPM dependency on a package "sqlite-libs" which
does not exist in openSUSE.
Since the runtime library dependencies of libcephsqlite are handled by RPM
transparently, this line is not needed.
Adam King [Thu, 18 Mar 2021 17:20:46 +0000 (13:20 -0400)]
mgr/orchestrator: remove image name field from 'orch ps' and 'orch ls'
Now that we're typically using the image digests the name isn't as helpful. We also
end up in scenarios where some images use tags for their name and others use the
digest so the image name comes out as "mix" in orch ls despite it being the same image.
Ilya Dryomov [Wed, 24 Mar 2021 15:23:44 +0000 (16:23 +0100)]
auth: require CEPHX_V2 by default
It's been almost three years and support is present in all relevant
clients.
From the security perspective, roughly the same could be achieved
with "ceph osd set-require-min-compat-client nautilus", but this is
more user friendly as the client gets ENOTSUP instead of spinning on
"feature set mismatch" faults.
Sage Weil [Fri, 26 Mar 2021 12:17:42 +0000 (07:17 -0500)]
Merge PR #40355 into pacific
* refs/pull/40355/head:
mgr/cephadm: Fix dashboard gateway configuration when using IPV6
qa/workunits/cephadm/test_cephadm: specify image separately
mgr/cephadm: retry after JSONDecodeError in wait_for_mgr_restart()
cephadm: prevent podman from breaking socket.getfqdn()
qa/tasks/cephadm: use 'orch apply mon' to deploy mons
qa/suites/rados/cephadm/upgrade: add centos upgrade on latest octopus
mgr/cephadm/upgrade: do not crash if error races with user cancellation
doc/cephfs/nfs: Add note about cephadm NFS-Ganesha daemon port
cephadm: only bootstrap using image that matches cephadm version
mgr/cephadm: redeploy daemons deployed using old image during upgrade
mgr/cephadm: add container digests of mgr that deployed daemon to unit.meta
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Dan van der Ster [Tue, 23 Mar 2021 10:28:37 +0000 (11:28 +0100)]
test_ipaddr: check that we correctly skip loopback
We should skip devices named 'lo' or of the form 'lo:0' regardless
of their IP address.
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch> Related-to: https://tracker.ceph.com/issues/49938
(cherry picked from commit 780125d1ed93cd7b17172752b3e76186a524103b)
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch> Fixes: https://tracker.ceph.com/issues/49938
(cherry picked from commit 6147c0917157efd2d35610e759685656a4989abb)
Brad Hubbard [Thu, 25 Mar 2021 23:57:14 +0000 (09:57 +1000)]
Revert "mgr/dashboard:test prometheus rules through promtool"
Reverts: https://github.com/ceph/ceph/pull/39983
This is currently blocking testing on ubuntu on the eve of a pacific
release. The problems associated with this PR have been resolved
upstream but have not been backported yet and are non-trivial.
Adam Kupczyk [Mon, 22 Mar 2021 10:20:11 +0000 (11:20 +0100)]
os/bluestore: Make Onode::put/get resiliant to split_cache
In
OnodeCacheShard* ocs = c->get_onode_cache();
std::lock_guard l(ocs->lock);
while waiting for lock, split_cache might have changed OnodeCacheShard.
This will result in adding Onode to improper OnodeCacheShard.
Such action is obviously bad, as we will operate in future (at least once) on
different OnodeCacheShard then we got lock for. Particulary sensitive to this
are _trim and split_cache functions, as they iterate over elements.
Sage Weil [Tue, 23 Mar 2021 16:56:59 +0000 (11:56 -0500)]
os/bluestore: separate omap per-pool vs per-pg alerts
Currently the health alert raised does not match the docs, and the docs
do not describe what the health alert indicates.
Octopus added per-pool omap storage. This improves space accounting
and reporting.
Pacific added per-pg omap storage (object hash in key). This speeds up
PG removal.
Separate everthing out into two distinct alerts raised from bluestore
and surfaced as health alerts, with corresponding config options to
disable, and update the docs accordingly.
Also update the fsck options for warn vs error, and raise separate
errors for the per-pg and per-pool cases.
mgr/cephadm: Fix dashboard gateway configuration when using IPV6
Fixes: https://tracker.ceph.com/issues/49957 Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
(cherry picked from commit 1b18f4f9cb28708b544c62b3d07f9e1b4c701e41)
Kefu Chai [Thu, 25 Mar 2021 09:08:48 +0000 (17:08 +0800)]
run-make-check.sh: let ctest generate XML output
to enable XUnit plugin of jenkins to consume the ctest output and
publish it in the dashboard, we need to
* let ctest generate XML output instead of plain text output
* do not fail the test if any test case fails. this allows the publisher
to do its job by checking the XML output.
* prevent ctest from compressing the output. see
https://issues.jenkins.io/browse/JENKINS-21737
Dan van der Ster [Thu, 12 Nov 2020 16:14:37 +0000 (17:14 +0100)]
common/options: bluefs_buffered_io=true by default
Enable bluefs_buffered_io again because it makes a huge user-visible
improvement in metadata intensive scenarios, such as but not limited to
PG deletion.
In our environment, deleting PGs from 4 hybrid OSDs (sharing one SATA SSD block.db) saturates
the block.db at 350MB/s reads and causes slow reqs and flapping on the OSDs.
Those OSDs have 3GB osd_target_memory.
Enabling bluefs_buffered_io drops the SSD IO down to <1MBps and the OSDs
are performant again. (The underlying PG deletion inefficiency is being
solved separately, but the page cache is so much more effective than
the bluestore cache in this scenario).
Lastly, remove the comment about swap. We should separately advise
operators to disable swap on OSD machines, as it is much better in
our experience to OOM and restart than to chug along swapping.
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch> Related-to: https://tracker.ceph.com/issues/45765 Related-to: https://tracker.ceph.com/issues/47044
(cherry picked from commit 5ec8e8e63d409860c35e24a192090ac2b70af8f6)
projects / ceph.git / log