Sage Weil [Mon, 7 Jun 2021 14:02:52 +0000 (10:02 -0400)]
Merge PR #41509 into master
* refs/pull/41509/head:
common/cmdparse: fix CephBool validation for tell commands
mgr/nfs: fix 'nfs export create' argument order
common/cmdparse: emit proper json
mon/MonCommands: add -- seperator to example
qa/tasks/cephfs/test_nfs: fix export create test
mgr: make mgr commands compat with pre-quincy mon
doc/_ext/ceph_commands: handle non-positional args in docs
mgr: fix reweight-by-utilization cephbool flag
mon/MonCommands: convert some CephChoices to CephBool
mgr/k8sevents: fix help strings
pybind/mgr/mgr_module: fix help desc formatting
mgr/orchestrator: clean up 'orch {daemon add,apply} rgw' args
mgr/orchestrator: add end_positional to a few methods
mgr/orchestrator: reformat a few methods
pybind/ceph_argparse: stop parsing when we run out of positional args
pybind/ceph_argparse: remove dead code
pybind/mgr/mgr_module: infer non-positional args
pybind/mgr/mgr_module: add separator for non-positional args
command/cmdparse: use -- to separate positional from non-positional args
pybind/ceph_argparse: adjust help text for non-positional args
pybind/ceph_argparse: track a 'positional' property on cli args
crimson/osd: fix assertion failure in OpSequencer on replay.
Commit b5efdc6f1c9563357d7dfd33a8f379053592a215 has unified
the interruption handling among `InternalClientRequest` and
`ClientRequest`. Unfortunately, a call to `maybe_reset()` of
`OpSequencer` has been overlooked and dropped leading to the
`assert(prev_op > last_unblocked)` assertion failure in
`start_op()`.
This was the root cause of the following problem at Sepia:
Kefu Chai [Mon, 7 Jun 2021 08:00:23 +0000 (16:00 +0800)]
cmake: let vstart-base depend on cython_rados if NOT WIN32
to disable this custom target if NOT WIN32 without explanations is difficult
to maintain, so add a comment, and limit the scope to just
`cython_rados`.
Kefu Chai [Mon, 7 Jun 2021 07:09:42 +0000 (15:09 +0800)]
cmake: do not print warning in each cmake run
cmake prints the warning of
> run-promtool-unittests is skipped due to missing promtool
everytime if the test is skipped per the test. but we only need to
get this message the first time cmake is executed. more importantly,
the value of `PROMTOOL_EXECUTABLE` is cached. so the new error message
does not provide updated information, so just stop doing so if the
cached variable is found.
Kefu Chai [Sat, 5 Jun 2021 09:39:25 +0000 (17:39 +0800)]
crimson/os/seastore: open_collection() returns nullptr if DNE
we check for the existence of meta collection by trying to open it,
if it exists, we continue check for the superblock stored in it, if
the superblock does not exist or corrupted, we consider it as a failure.
before this change, open_collection() always return a valud Collection
even if the store does not have the collection with specified cid. this
behavior could be misleading in the use case above.
after this change, open_collection() looks up the collections stored in
root collection node for the specfied cid, and return nullptr if it does
not exist.
Kefu Chai [Sat, 5 Jun 2021 03:38:17 +0000 (11:38 +0800)]
pybind/ceph_daemon: do not handle get_command_descriptions in admin_socket()
we never pass a single string to admin_socket() as its 2nd parameter.
instead, we always pass a list of string. so no need to handle it as a
special case.
* refs/pull/41697/head:
script: add a few more volume mounts for sepia
script: drop ceph-fuse from docker debugging
script: enable centos debuginfo repo for debugging
script: update repo url for multi-arch builds
script: fetch autobuild.asc key via HTTPS
Kefu Chai [Fri, 4 Jun 2021 17:11:13 +0000 (01:11 +0800)]
qa/suites/orch/rook/smoke: stop testing on ubuntu 18.04
even rook does not really install ceph packages in the host directly, it
uses the ceph container image. but teuthology insists on checking the
existence of debian packages by querying shaman server when it sees a
teuthology facet file which includes:
os_type: ubuntu
os_version: "18.04"
but since we've stopped building ubuntu/bionic packages, teuthology
just complains when we are scheduling test suites which are composed
from facets in qa/suites/orch/rook/smoke.
in this change, the ubuntu_18.04.yaml is dropped because ubuntu/bionic
does not really increase the test coverage of ceph. it helps to test
the rook and container runtime though.
Kefu Chai [Fri, 4 Jun 2021 03:25:12 +0000 (11:25 +0800)]
debian/control: ceph-mgr-modules-core does not Recommend ceph-mgr-rook anymore
per https://www.debian.org/doc/debian-policy/ch-relationships.html
> Recommends
> This declares a strong, but not absolute, dependency.
>
> The Recommends field should list packages that would be found together
> with this one in all but unusual installations.
ceph-mgr-modules-core provides a set of ceph-mgr modules which are
always enabeld. but the rook module enables ceph-mgr to install and
configure a Ceph cluster using Rook. this module is very useful but
it does not have such a strong connection with ceph-mgr-modules-core.
we can always install it separately for using better intergration with
Rook.
Kefu Chai [Wed, 2 Jun 2021 09:54:18 +0000 (17:54 +0800)]
test/objectstore/unittest_alloc_aging: init cct
* initialize the cct use by test, otherwise g_ceph_context is
not set at all.
* instead of using g_ceph_context, use static member variable cct.
less dependency to the global instance.
* setup and teardown the cct for test suite, because global_init()
initialize g_ceph_context, which cannot be set multiple times.
Sage Weil [Fri, 4 Jun 2021 02:04:32 +0000 (22:04 -0400)]
Merge PR #41636 into master
* refs/pull/41636/head:
mgr/cephadm/inventory: do not try to resolve current mgr host
pybind/mgr/mgr_module: make get_mgr_ip() return mgr's IP from mgrmap
mgr/restful: use get_mgr_ip() instead of hostname
Reviewed-by: Adam King <adking@redhat.com> Reviewed-by: Kefu Chai <kchai@redhat.com>
Paul Cuzner [Wed, 2 Jun 2021 23:34:19 +0000 (11:34 +1200)]
mgr/cephadm:fix alerts sent to wrong URL
The path_prefix in prometheus.yml was specifying an
endpoint prefix, which was invalid. This resulted in 404
errors when trying to send alerts to alertmanager and
blocked alerts being sent on to the ceph-dashboard API
receiver. This fix remves this prefix.
Fixes: https://tracker.ceph.com/issues/51073 Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
* refs/pull/39910/head:
test: Add test for mgr hang when osd is full
mgr: Set client_check_pool_perm to false
mds: Add full caps to avoid osd full check
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Sage Weil [Thu, 3 Jun 2021 14:29:00 +0000 (10:29 -0400)]
mgr/cephadm/inventory: do not try to resolve current mgr host
The CNI configuration may set up a private network for the container, which
is mapped to the hostname in /etc/hosts. For example, my test box sets
up 10.88.0.0/24 because I was using crio + kubeadm on this host earlier
(at least I think that's why):
In any case, we should never trust a lookup of our own hostname from inside
a container!
This isn't quite sufficient, though: if this is a single-host cluster, then
we fall back to using get_mgr_ip(). That value may be distorted by the
public_network option on the mgr, but we don't have any other good
options here, and single-node clusters are unlikely to have complex
network configs.
Sage Weil [Wed, 2 Jun 2021 02:31:11 +0000 (22:31 -0400)]
pybind/mgr/mgr_module: make get_mgr_ip() return mgr's IP from mgrmap
The previous approach was convoluted: we tried to do a DNS lookup on the
hostname, which would fail if /etc/hosts had an entry. Which, with podman,
it does. And the IP it has will vary in all sorts of weird ways. For
example, CNI on my host means that I get a dynamic address in 10.88.0.0/24.
Avoid all of that nonsense and use the IP that is in the mgrmap. There
may be multiple IPs (v2 + v1, or maybe even IPv4 + v6 in the future); in
that case, use the first one.
Kefu Chai [Thu, 3 Jun 2021 12:48:53 +0000 (20:48 +0800)]
cmake: increase the MAX_{LINK,COMPILE}_MEM
based on recent observation, quite a few C++ source file take
around more than 3.0GiB to compile. for instance,
test_mock_HttpClient.cc could take up to 6270MiB memory to compile.
Kefu Chai [Thu, 3 Jun 2021 12:41:36 +0000 (20:41 +0800)]
ceph.spec.in: increase the mem_per_job to 3GiB
to lower the number of jobs, we are experiencing build failures on
a builder with 48c96t, 193 free mem. the failures were caused by
OOM killer which kills the c++ compiler
[498376.128969] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/jenkins.service,task=cc1plus,pid=1387895,uid=1110
[498376.145288] Out of memory: Killed process 1387895 (cc1plus) total-vm:3323312kB, anon-rss:3164568kB, file-rss:0kB, shmem-rss:0kB, UID:1110
[498376.315185] oom_reaper: reaped process 1387895 (cc1plus), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[498377.882072] cc1plus invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
before this change, we use the total memory to calculate the number
of jobs, and assume that each job takes at most 2.5GiB mem. in the
case above, the # of job is 96.
after this change, we use the free memory, and increse the mem per job
to 3.0GiB. in the case above, the # of job would be 85.
qa: use dump_metrics as alternative of get_heap_property
"get_heap_property *" asock commands are exposed to operators
to check the tcmalloc internals for understanding the performance
of the memory subsystem. but crimson uses the builtin seastar allocator
which is not backed by tcmalloc. but we can dump the metrics using
the "dump_metrics" asock command which is only available from
crimson-osd.
projects / ceph.git / log