mon/ConfigMap: factor out parse_key helper

Signed-off-by: Sage Weil <sage@newdream.net>

mon/ConfigMap: fix stray option leak

The const Option* needs to remain alive only until the next clear().  Keep
the reference in ConfigMap and clean it up then.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr: use new kv subscription for mgr/, device/, config/

Include the config/ prefix (which we weren't loading before).

Before we are active, we collect these changes, and then pass them to
the ActivePyModules ctor.  No change in functionality here, except that
when we make a change from a mgr module, we'll (1) put it in our local
cache store, (2) send the mon command to set it, and (3) get a notification
that updates the same value.  Since this whole process is synchronous (see
ActivePyModules::set_store()), and the notification will generally arrive
*before* the command ack, there is no change in behavior.

If the mon cluster is not yet pacific, we still need to load
kv values the old way.  If it's a mixed-mon cluster (and, e.g., our
current mon has the feature but not all of them do), we'll get this data
both ways, but no harm is done.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: allow subscription to kv/config-key data

Allow subscription to config-key/kv data.  Initially we'll send a full
dump of the prefix.  As changes occur, we'll send incremental diffs,
unless the subscriber is too far behind, in which case we'll send a full
dump again.

There is a new message, MKVData, to support this.

No compat issues since old clients won't subscribe to this stream unless
they know how to handle it.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: convert ConfigKeyService -> KVMonitor

Convert this into a normal PaxosService.  This gives us (1) code uniformity
and also (2) a changelog in the form of the paxos commits for recently
changed keys that we can use to allow clients to subscribe to changes.

For upgrades this is pretty painless:
 - the actual kv data is in the same place
 - an old mon will still see updates made by a new monitor
 - a new mon will still see updates made by an old monitor, but won't get
   a paxosservice version bump.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39363 from rhcs-dashboard/fix-orch-host-e2e

mgr/dashboard: check .badge instead of text for expected label

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge PR #38610 into master

* refs/pull/38610/head:
ceph-volume: pass --filter-for-batch from drive-group subcommand

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #39467 from sebastian-philipp/doc-cephadm-mon-quorum

doc/cephadm: Restoring the MON quorum

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39496 from liewegas/cephadm-refresh-health-update

mgr/cephadm: fix host refresh

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39372 from rhcs-dashboard/avoid-document-write

mgr/dashboard: avoid using document.write()

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39462 from rhcs-dashboard/fix-alerts-mtuMismatch

mgr/dashboard: fix MTU Mismatch alert

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: check .badge instead of text for expected label

this change fixes a regression introduced by
8c5e31ec1a13bc53394eb2cb6880d74db169fac4 which broke the 01-hosts.e2e-spec.ts test
driven by test_dashboard_e2e.sh

Fixes: https://tracker.ceph.com/issues/49205
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #39464 from sebastian-philipp/doc-cephadm-unmanaged

doc/cephadm: Disable automatic deployment of daemons

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39459 from tchaikov/wip-doc-api-command

doc: wrap example command

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #39490 into master

* refs/pull/39490/head:
cephadm: fix selinux mount mis-indent

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #39497 from zdover23/wip-doc-dev-teuthology-intro-workbench-removal-16-Feb-2021

doc/dev: removing workbench docs (part 2)

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39499 from zdover23/wip-doc-dev-teuthology-teuthology-describe-tests-repair-2021-Feb-16

doc/dev: correct "teuthology-describe-tests"

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38030 from p-se/prom-alert-package-drops-leeway

mgr/dashboard: prometheus alerting: add some leeway for package drops and errors

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #39422 from kamoltat/wip-ksirivad-autoscaler-release-notes

PendingReleaseNotes: mgr/pg_autoscaler

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/dev: correct "teuthology-describe-tests"

This tiny commit does this:

s/teuthology-describe-tests/teuthology-describe/

The name of this feature was changed from "teuthology-
describe-tests" to "teuthology-describe", and this
commit brings up to date the title of the section
detailing it.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/dev: removing workbench docs (part 2)

This removes a paragraph in the section "How
Integration Tests Are Run" that described
ceph-workbench. ceph-workbench is deprecated,
so that paragraph is now gone.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/cephadm: fix host refresh

Fixes: 01f60cf4e0a751c314120c02956d4ff941eb71b4
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39323 from sebastian-philipp/cephadm-simplify-cephadmservice-config

mgr/cephadm: Add config() to CephadmService

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39174 from zdover23/wip-doc-37949-grammar-edit

doc/dev: Teuthology guide PR#37949 grammar edit

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39392 from tschoonj/amqp-ssl

rgw: add support for SSL encrypted AMQP connections

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

doc/cephadm: Disable automatic deployment of daemons

Fixes: https://tracker.ceph.com/issues/45767
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/dev: Teuthology guide PR#37949 grammar edit

This PR improves the wording of the technical
information added to the documentation in PR#37949.
This is the second is a series of two PRs, which series
is dedicated to testing a workflow wherein developers
add technical information to the documentation and then
technical writers improve its presentation.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39430 from liewegas/bug-48142

qa/suites/rados/cephadm/upgrade: change starting version by distro

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39493 from rzarzynski/wip-crimson-cond-mon-config

crimson/osd: fetch configuration from monitors conditionally.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39492 from zdover23/wip-doc-dev-teuthology-workflow-cephfs-shell-typo-15-Feb-2021

doc/dev: cephfs-shell typo repair

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: prometheus alerting: add some leeway for package drops and errors (1%)

Fixes: https://tracker.ceph.com/issues/48201
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

crimson/osd: fetch configuration from monitors conditionally.

Before this change we were always attempting to fetch the config
from monitors, even if `--no-mon-config` had been passed. This
was the reason for failures at teuthology like:
http://pulpito.front.sepia.ceph.com/rzarzynski-2021-02-15_21:09:02-rados-master-distro-basic-smithi/5885250

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/cephadm: Restoring the MON quorum

Fixes: https://tracker.ceph.com/issues/49214
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39473 from sebastian-philipp/doc-cephamd-trouble-events

doc/cephadm: Per-service and per-daemon events

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

mgr/cephadm: _add_daemon: remove config_func

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add config() to CephadmService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39465 from sebastian-philipp/install-deps-rm-libboost1.72.list

install-deps: Cope with nonexistent /etc/apt/sources.list.d/ceph-libboost1.72.list

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39249 from varshar16/wip-fix-vstart-ganesha-error

vstart: fix ganesha cluster id and rados url errors

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

doc/dev: cephfs-shell typo repair

This corrects the formatting for a mention of
cephfs-shell. Previously, it was not properly
formatted. This commit adds two backticks before
the string "cephfs-shell", and adds to backticks
after it.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

cephadm: fix selinux mount mis-indent

We want to map this path even if we had to create the empty directory.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #39150 into master

* refs/pull/39150/head:
qa/suites/rados/cephadm/thrash: add cephadm thrashing test

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #39482 into master

* refs/pull/39482/head:
Revert "Merge pull request #37764 from mgfritch/cephadm-no-container-init"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39424 into master

* refs/pull/39424/head:
cephadm: Mounting <empty> folder for selinux only if it is needed

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge PR #39185 into master

* refs/pull/39185/head:
os/bluestore: Add option to check BlueFS reads

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #39281 into master

* refs/pull/39281/head:
mgr/MgrClient; less copying of mgr report struct

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39282 into master

* refs/pull/39282/head:
osdc/Objecter: remove pool_full_try interface
librados: set FULL_TRY flag on ioctx, not objecter
librados/IoCtxImpl: pass extra_op_flags everywhere

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39283 into master

* refs/pull/39283/head:
qa/suites/rados/dashboard: whitelist TELEMETRY_CHANGED

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge PR #39381 into master

* refs/pull/39381/head:
mgr/cephadm: based sleep interval on configured intervals/timeouts

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: Mounting <empty> folder for selinux only if it is needed

There are OSs without </usr/share/empty> folder. And selinux can be
enabled or not.

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Revert "Merge pull request #37764 from mgfritch/cephadm-no-container-init"

This reverts commit f635555fe76322336a1e0e8cc3c199562642ad0e, reversing
changes made to d4d3d17b23fcf6d8808a96b992f56924f495b174.

This PR seems to be (indirectly?) responsible for
  https://tracker.ceph.com/issues/49237

Also, it was causing the rados.py task's follow-up step to wait
for snap trimming to fail: it would time out a 'ceph osd dump --format=json'
command.  :/

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39461 from yuvalif/wip-yuval-fix-49261

rgw/test: allow for duplicate events in pubsub

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/cephadm: Per-service and per-daemon events

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39386 from zdover23/wip-doc-dev-triggering-tests-rewrite-10-Feb-2021

doc/dev: teuthology: improve triggering tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: teuthology: improve triggering tests

- This PR cleans up the language in the "Triggering
  Tests" section of the "Workflow" chapter of the
  "Tests: Integration Tests" (Teuthology) guide.
- Added bash prompts.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39407 from zdover23/wip-doc-dev-testing-qa-changes-rewrite-11-Feb-2021

doc/dev: rewrite 'testing qa changes'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: rewrite 'testing qa changes'

This PR rewrites the "Testing QA Changes (Without
Re-Building Binaries)" section of the Developer
Guide. I also tried to unbury the lede.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge PR #38562 into master

* refs/pull/38562/head:
Update batch.py

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

mgr/dashboard: avoid using document.write()

Fixes: https://tracker.ceph.com/issues/49220
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

install-deps: Cope with nonexistent /etc/apt/sources.list.d/ceph-libboost1.72.list

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgw: add support for SSL encrypted AMQP connections

Fixes: https://tracker.ceph.com/issues/42902
Signed-off-by: Tom Schoonjans <Tom.Schoonjans@rfi.ac.uk>

Merge pull request #39458 from sebastian-philipp/cephadm-spec-store-fix-race

mgr/cephadm: SpecStore: use d-under accessor methods

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard:fix MTU Mismatch alert

This PR intends to fix the expression used for MTU Mismatch alert in prometheus

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/cephadm: SpecStore: use d-under accessor methods

This fixes a conflict between:

* 1fcdad17685a8884135a2cda0cdfe3525e325474
* 2c7f899124207d0d4866e16d9eb5609777ea22e8

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgw/test: allow for duplicate events in pubsub

when tested via radosgw-admin.
also, add verification for the content of the events

Fixes: https://tracker.ceph.com/issues/49261
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #39379 from multi-arch/master

ceph.spec.in: Enable tcmalloc and lttng on IBM Power and Z

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Luke Browning <lbrownin@redhat.com>

Merge pull request #39389 from tchaikov/wip-cmake-ccache

cmake: s/CCACHE_FOUND/CCACHE_EXECUTABLE/

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: wrap example command

otherwise the field name in the field list would be rendered with a
small width.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39411 from tchaikov/wip-rtd-theme

doc: cleanups and group tell commands

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39454 from rzarzynski/wip-crimson-crc-for-temp-msgr

crimson: configure CRC for the temp_mon_client's messenger.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39453 from rzarzynski/wip-crimson-monc-pick-from-addrs

crimson/monc: consider v1 addresses when connecting to a monitor.

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: define command properties using field list

the rendering is more compacted this way, and the parameters can be
aligned.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: silence warnings from openapi sphinx extension

this is a follow-up of 1debd9856565905439ae56f2cfdc16b0549055ec,
we should use the logging from sphinx.util instead from Python standard
library's logging.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: group commands by prefix

as there are lots commands, the toc in sidebar is clutterred with them,
so would be better to group them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: split the argdesc pair at the first '='

there is chance that the value of of the argdesc's kv pair contains
"=". for instance, the goodchars regexp could contain '='. so we should
stop at the first '='.

this change fixes the parsing of "osd pool application set" command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: drop ceph.js completely

these are leftovers of 7a62303b57150814ff465eb611b69b3b2e9f03f1

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: decrease padding in the "docubetter" div

for better UX

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39385 from adk3798/error-redeploy

cephadm: allow redeploy of daemons in error state if container running

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38549 from adk3798/prepare-redeploy

mgr/cephadm: make generate_config internal, prepare before redeploy

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson: configure CRC for the temp_mon_client's messenger.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/monc: improve debugs around Connection::reply.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/monc: consider v1 addresses when connecting to a monitor.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39450 from cbodley/wip-rgw-notify-double-dpp

rgw: remove duplicated ldout in rgw_notify.cc

Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: remove duplicated ldout in rgw_notify.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39373 from varshar16/wip-fix-rook-timezone-error

mgr/rook: Add timezone info

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #39362 from rhcs-dashboard/49216-delete-eof-read-pwd-from-file

mgr/dashboard: delete EOF when reading passwords from file

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: sebastian-philipp <swagner@suse.com>

Merge pull request #39434 from zdover23/wip-doc-dev-about-suites-and-filters-rewrite-2021-Feb-12

doc/dev: rewrite "about suites and filters"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39327 from kotreshhr/qa-fix-mgr-vol-test-cases

qa: Fix a few mgr/volume test cases

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #39409 from trociny/wip-49245

librbd: don't log error if get mirror status fails due to mirroring disabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38883 from sebastian-philipp/cephadm-keep-deleted-specs

mgr/cephadm: Purge deleted services

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #39058 from sebastian-philipp/cephadm-use_repo_digest-default-true

mgr/cephadm: enable use_repo_digest by default for pacific

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39364 from mgfritch/cephadm-log-errs

cephadm: add error messages to log file

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

doc/dev: rewrite "about suites and filters"

This improves the English in the section
"About Suites and Filters" in the new
Teuthology Guide in the Developer Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge PR #39404 into master

* refs/pull/39404/head:
qa/suites/upgrade/octopus-x/stress-split: cephadm-based stress-split upgrade

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #39375 from idryomov/wip-rbd-rm-exclusive

librbd: refuse to release exclusive lock when removing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39368 from MahatiC/wip-ssd-bug-fix

librbd/cache/pwl: Fix user request completion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Xiaoyan Li <xiaoyan.li@intel.com>

qa/suites/rados/cephadm/upgrade: change starting version by distro

centos/rhel have podman 2, which does not like conflicting --cap-add and
--privileged arguments.  cephadm versions prior to 15.2.5 use both args,
however, which means the rhel/centos upgrade test has to start at 15.2.5
to work at all on those distros (with the updated podman).

Fixes: https://tracker.ceph.com/issues/48142
Signed-off-by: Sage Weil <sage@newdream.net>

PendingReleaseNotes: mgr/pg_autoscaler

Added details describing the changes that
occured after merging #39248 into target
upstream.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

qa: Fix a few mgr/volume test cases

Recovering dirty auth metadata file might not retain the order,
fixed the comparison in 'test_recover_auth_metadata_during_authorize'
and 'test_recover_auth_metadata_during_deauthorize'.

Fixes: https://tracker.ceph.com/issues/49192
Signed-off-by: Kotresh HR <khiremat@redhat.com>

librbd: don't log error if get mirror status fails due to mirroring disabled

Fixes: https://tracker.ceph.com/issues/49245
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39037 from liu-chunmei/seastore_collection

crimson/seastore: add collection implementation

Reviewed-by: Samuel Just <sjust@redhat.com>