Sage Weil [Tue, 23 Mar 2021 00:20:48 +0000 (19:20 -0500)]
Merge PR #40184 into pacific
* refs/pull/40184/head:
qa/suites/rados/cephadm/orchestrator_cli: random-distro$ -> 0-random-distro$
qa/suites/rados/cephadm/smoke-roleless: distro -> 0-distro
qa/distros/podman: install kubic once per host, in parallel
qa/suites/fs/multiclient: use clients: not all: for pexec
Sage Weil [Tue, 23 Mar 2021 00:20:39 +0000 (19:20 -0500)]
Merge PR #40202 into pacific
* refs/pull/40202/head:
qa/suites/rados/cephadm/upgrade: wait for rgw servicemap entries to refresh
mgr/cephadm: identify iscsi service by the pool
qa/distros/podman: install containernetworking-plugins along with podman
python-common: Validate characters in service_id for container names
qa/suites/rados/cephadm/smoke-roleless: deploy additional daemon types
cephadm: fix a minor typo in logging message
qa/suites/rados/cephadm/dashboard: test on centos
cephadm: use debug verbosity during container exec
mgr/cephadm/upgrade: do not repeat crash message
mgr/cephadm/upgrade: a little less verbose
mgr/cephadm: don't log not-ok-to-stop at ERR level
mgr/cephadm: is presumed -> appears
mgr/cephadm: don't double-log ok-to-stop results
mgr/cephadm/upgrade: include upgrade progress in ceph -s
mgr/cephadm: clean up misc messages
mgr/cephadm/configcheck: do not spam info every minute
mgr/cephadm: stop conflicting daemon when deploying to a specific port
mgr/cephadm: make DaemonPlacement print nicer
mgr/cephadm: fix --force remove comment
mgr/cephadm/schedule: choose an IP from a subnet list
mgr/cephadm: rgw: clean up config and config-key values on removal
mgr/cephadm: rgw: drop .crt extension when storing cert in config-key
mgr/cephadm/services: allow beast/civetweb to bind to a particular IP
python-common: add 'networks' property to ServiceSpec
mgr/cephadm/schedule: match placement ip only combination with port
mgr/cephadm: less noise about refreshing hosts
mgr/cephadm: fall back to service spec port if none on DaemonDescription
mgr/cephadm: fix redeploy when daemons have ip:port
mgr/cephadm/schedule: add test case
qa/suites/rados/cephadm/smoke-roleless: add rgw test on many ports
doc/cephadm/rgw: update docs to show count-per-host
mgr/cephadm: add support for rgw_frontend_type (beast or civetweb)
mgr/cephadm: remove ssl_frontend_ssl_key from RGWSpec
mgr/cephadm: fix beast private key config option
mgr/cephadm: fix rgw ssl cert/key config-key path
mgr/cephadm/schedule: dynamically assign ports for rgw
mgr/cephadm/schedule: only 1 port in DaemonPlacement
mgr/cephadm: move rgw frontend logic into RgwService
mgr/cephadm/schedule: return DaemonPlacement instead of HostPlacementSpec
mgr/cephadm/schedule: remove unused methods
mgr/cephadm: propagate ip:port from CephadmDaemoNDeploySpec to deployment
cephadm: populate ports if known and not included in unit.meta
mgr/cephadm: gather and report ports in 'orch ps' output
qa/suites/rados/cephadm/orchestrator_cli: random-distro$ -> 0-random-distro$
qa/suites/rados/cephadm/smoke-roleless: distro -> 0-distro
qa/distros/podman: install kubic once per host, in parallel
qa/suites/fs/multiclient: use clients: not all: for pexec
mgr/cephadm: add info to 'ceph orch upgrade status' in cephadm
Reviewed-by: Michael Fritch <mfritch@suse.com> Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Sage Weil [Tue, 23 Mar 2021 00:20:30 +0000 (19:20 -0500)]
Merge PR #40279 into pacific
* refs/pull/40279/head:
mgr/cephadm: identify rgw, cepfs-mirror in servicemap
mgr/ServiceMap: adjust 'ceph -s' summary
rgw: register daemons in servicemap by gid; include id
cephadm: fix rbd-mirror auth name
Kefu Chai [Mon, 22 Mar 2021 06:49:13 +0000 (14:49 +0800)]
qa/distros/podman: install containernetworking-plugins along with podman
/etc/cni/net.d/87-podman-bridge.conflist tries to load "bridge",
"firewall", "tuning" and "portmap" plugins, which are provided by
containernetworking-plugins package.
Sage Weil [Thu, 18 Mar 2021 16:45:48 +0000 (11:45 -0500)]
mon/MgrStatMonitor: ignore MMgrReport from non-active mgr
If it's not the active mgr, we should ignore it.
Since the mgr instance is best identified by the gid, add that to the
message. (We can't use the source_addrs for the message since that is
the MgrStandby monc addr, not the active mgr addrs in the MgrMap.)
This fixes a problem where a just-demoted mgr report gets processed and a
new mgr gets a ServiceMap with an epoch >= its pending map. (At least,
that is my theory!)
Josh Durgin [Wed, 17 Mar 2021 18:51:27 +0000 (14:51 -0400)]
common/options: turn off bluestore_fsck_quick_fix_on_mount by default
This option enables 3 conversions:
1) pool stats, added in nautilus
2) per-pool omap, added in octopus
3) per-pg omap (replacing (2)) in pacific
Upgrading the long running cluster in sepia from octopus to pacific
resulted in conversion (3). This conversion isn't particularly useful
yet since the follow-on optimization of pg removal aren't in pacific
yet.
This took 25 minutes for the SSD-based osds with <10GB of omap. That's
a lot of disruption, and some clusters have 10x that much omap data.
Upgrades going from nautilus to pacific will miss the finer-grained
stats granularity, but isn't such an important feature it's worth
causing potential availability problems.
In the future we can orchestrate these format changes via cephadm/rook
to minimize the impact on the whole cluster, e.g. going an osd at a
time or doing it during an off-peak period, and not necessarily at the
same time as an upgrade.
Sage Weil [Sun, 21 Mar 2021 18:25:06 +0000 (13:25 -0500)]
Merge PR #40247 into pacific
* refs/pull/40247/head:
common: reset last_log_sent when clog_to_monitors is updated
logclient: move LogChannel::set_log_to_monitors(bool v) to LogClient.cc
Sage Weil [Sun, 21 Mar 2021 14:38:49 +0000 (09:38 -0500)]
Merge PR #40129 into pacific
* refs/pull/40129/head:
osd: PeeringState: implement an acting_set_writeable() function
osd: PeeringState: fix a boolean conditional direction
osd: PeeringState: fix stretch peering so PGs can go peered but not active
osd: PeeringState: don't add acting-set OSDs to candidate set in stretch mode
osd: PeeringState: fix calc_replicated_acting_stretch() syntax/logic
osd: PeeringState: respect stretch peering constraints for async recovery
osd: PeeringState: add a comment about using size as a proxy for activateable
osd: check for is_stretch_pool() in stretch_set_can_peer()
scripts: some additions to help with local testing
script: set_up_stretch_mode: include OSDs in root=default so pg creation works
myoungwon oh [Wed, 24 Feb 2021 14:16:39 +0000 (23:16 +0900)]
osd: recover unreadable snapshots when handling manifest object
The manifest object needs adjacent clones to increment/decrement
refcount when modifying the object. So, recovering the clones is needed
if the adjacent clones are unreadable.
Kefu Chai [Sat, 20 Mar 2021 05:00:01 +0000 (13:00 +0800)]
install-deps.sh: remove existing ceph-libboost of different version
we install different versions of precompiled ceph-libboost packages
for different branches when building and testing them on ubuntu test
nodes. for instance,
- nautilus: v1.72
- octopus, pacific: v1.73
they share the same set of test nodes. and these ceph-libboost packages
conflict with each other, because they install files to the same places.
in order to avoid the confliction, we should uninstall existing packages
before installing a different version of ceph-libboost packages.
ceph-libboost${version}-dev is a package providing the shared headers of
boost library, so, in this change we check if it is installed before
returning or removing the existing packages.
Sage Weil [Fri, 19 Mar 2021 12:21:18 +0000 (08:21 -0400)]
mgr/ServiceMap: adjust 'ceph -s' summary
- Do not list individual daemon ids as this won't scale for larger
clusters
- Do not contemplate multile daemons of the same type that register with
different "daemon_type" -- not until we actually have any that do that.
- Present counts by various groupings: distinct hosts and rgw zones to
start.
services:
mon: 1 daemons, quorum a (age 4m)
mgr: x(active, since 3m)
osd: 1 osds: 1 up (since 3m), 1 in (since 3m)
cephfs-mirror: 1 daemon active (1 hosts)
rbd-mirror: 2 daemons active (1 hosts)
rgw: 2 daemons active (1 hosts, 1 zones)
Sage Weil [Fri, 19 Mar 2021 12:25:23 +0000 (08:25 -0400)]
rgw: register daemons in servicemap by gid; include id
Registering by gid allows multiple radosgw instances to share an auth
key/identity. Including the id in the metadata allows them to still be
identified by name (even if not uniquely).
Kefu Chai [Fri, 19 Mar 2021 04:05:45 +0000 (12:05 +0800)]
pybind/mgr/dashboard: bump flake8 to 3.9.0
to address the failure of
ERROR: Cannot install -r requirements-lint.txt (line 2) and -r requirements-lint.txt (line 8) because these package versions have conflicting dependencies.
The conflict is caused by:
flake8 3.8.4 depends on pycodestyle<2.7.0 and >=2.6.0a1
autopep8 1.5.6 depends on pycodestyle>=2.7.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
Gerald Yang [Wed, 3 Mar 2021 04:37:15 +0000 (04:37 +0000)]
common: reset last_log_sent when clog_to_monitors is updated
When clog_to_monitors is disabled, "last_log" still keeps increasing by
get_next_seq() if OSD writes info to clog
But "last_log_sent" doesn't increase, if we disable clog_to_monitors for
a bit longer and then re-enabling it, the num_unsent could be bigger than
log_queue_size(), it will trigger an assertion in _get_mon_log_message
We need to reset last_log_sent to last_log before updating clog_to_monitors
Sage Weil [Wed, 17 Mar 2021 19:39:15 +0000 (15:39 -0400)]
mgr/cephadm: stop conflicting daemon when deploying to a specific port
If we are deploying a daemon to bind to a specific port and there is
an existing daemon we are removing that also binds to that port, stop
it first. Unless we are both binding to different IPs.
This resolves the case where daemons bind to * and we redeploy with a
subnet to bind to. It would eventually converge before, but would
throw a bind error in the process and take longer.
Sage Weil [Thu, 11 Mar 2021 23:47:24 +0000 (18:47 -0500)]
mgr/cephadm/schedule: choose an IP from a subnet list
Choose an IP from the subnet list provided by the ServiceSpec.
A few caveats:
- we ignore hosts that don't have IPs in the given subnet
- the subnet matching is STRICT. That is, the CIDR name has to exactly
match what is configured on the host. That means you can't just say 10/8
to match any 10.whatever addres--you need the exactly network on the host
(e.g, 10.1.2.0/24).
- If you modify a servicespec and change the networks when there are
already deployed daemons, we will try to deploy the new instances on
the same ports but bound to a specific IP instead of *. Which will fail.
You need to remove the service first, or remove the old daemons manually
so that creating new ones will succeed.
Sage Weil [Tue, 16 Mar 2021 16:58:03 +0000 (12:58 -0400)]
mgr/cephadm: rgw: drop .crt extension when storing cert in config-key
This will no affect upgrades since we will run the config() method before
prepare_create() any time we deploy a new daemon on this service, which
means we'll re-store the cert in the new key location before we generate
a new rgw_frontends option that references it.
Sage Weil [Thu, 11 Mar 2021 23:40:22 +0000 (18:40 -0500)]
mgr/cephadm/schedule: match placement ip only combination with port
1- We only have an IP to bind to if we also have a port, and
2- If we do, we want an exact match: if the DaemonPlacement has ip of
None, then the DaemonDescription should have None too.
Jason Dillaman [Wed, 17 Mar 2021 19:29:37 +0000 (15:29 -0400)]
test: ignore failures to force-enable lockdep
PR #40062 tweaked the behavior of lockdep to compile it out
of the code entirely for release builds. This fixes several
gtests where lockdep was force-enabled.
Ilya Dryomov [Wed, 17 Mar 2021 10:00:33 +0000 (11:00 +0100)]
qa: krbd_blkroset.t: update for separate hw and user read-only flags
Since kernel 5.12, hardware read-only state and user read-only
policy (BLKROGET/SET ioctls) are tracked separately in the block
layer. As the purpose of our ->set_read_only() method was exactly
that, it was removed.
As a side effect, BLKROSET no longer returns EROFS on an attempt
to make a read-only mapping read-write with "blockdev --setrw".
The policy gets updated, but the device remains read-only as before
because the hardware (== mapping) state is controlled by the driver.
projects / ceph.git / log