mgr/cephadm: set health check warning for apply spec failures and daemon place failures in serve

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: set health check warning in `_apply_service_config` for invalid config options and failures to set options

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: refactor `_check_for_strays` to use `remove_health_warning` helper

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: refactor `_refresh_hosts_and_daemons` to use `set_health_warning` and `remove_health_warning` helpers

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: refactor `_update_paused_health` to use `set_health_warning` and `remove_health_warning` helpers

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: refactor `_set_maintenance_health_check` to use `set_health_warning` helper

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: add `set_health_warning` and `remove_health_warning` helpers in module.py

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

Merge pull request #43341 from pdvian/wip-doc-mon-bootstrap

doc/dev/mon-bootstrap: fix typo

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42353 from ifed01/wip-ifed-fix-invalid-offset-repair

os/bluestore: fix writing to invalid offset when repairing

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #39062 from fyzard1991/master

mon/OSDMonitor:Make the pg_num check more accurate

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43289 from tchaikov/wip-52705

os/bluestore: list obj which equals to pend

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Xinze Chi <xinze@xsky.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #42821 from rhcs-dashboard/force-maintenance-workflow

mgr/dashboard: Move force maintenance test to the workflow test suite

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43246 from Daniel-Pivonka/cephadm-iscsi-trust-mgr-ips

mgr/cephadm: populate trusted_ip_list in iscsi-gateway.cfg with mgr ips

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43309 from guits/iscsi_fixes

cephadm/iscsi: fix a typo

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43319 from guits/workaround_tcmurunner_log

cephadm: modify iscsi containers bindmount

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43321 from adk3798/remove-exporter

mgr/cephadm: remove cephadm exporter code

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43226 from TRYTOBE8TME/wip-rgw-amqp-minor-nit

src/test: Minor nit from PR #43025

doc/dev/mon-bootstrap: fix typo

Signed-off-by: Prashant D <pdhange@redhat.com>

Merge pull request #43327 from cyx1231st/wip-seastore-ool-metrics

crimson/os/seastore: introduce ool related metrics with misc improvements

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43302 from ronen-fr/wip-rf-osd-psik

osd: add a missing comma in a list of strings

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #43303 from ronen-fr/wip-rf-osd-modulu

osd: fix the truncation of an int by int division

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43132 from p-se/monitoring-grafana-piechart-update

monitoring: update grafana-piechart-panel plugin

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: p-se <NOT@FOUND>

Merge PR #43177 into master

* refs/pull/43177/head:
osd/PrimaryLogPG: drop ops when pool has EIO flag
osdc/Objecter: set SUPPORTSPOOLEIO flag on all ops
ceph_test_rados_api_aio: test pool EIO flag
osdc/Objecter: return EIO for new linger ops
osdc/Objecter: return EIO for existing ops and linger ops
osdc/Objecter: return EIO for new ops
osd,mon: add EIO pool flag

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43320 from pdvian/wip-doc-cephx-fixtypo

doc/dev/cephx: fix typo

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43316 from smithfarm/wip-spdk-tw

rpm: enable system_pmdk bcond for SUSE builds

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

os/bluestore: list obj which equals to pend

otherwise we could have failures like

scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes."

where the numbers of scrubbed object, clones, dirty and omap are always
less than the total number of corresponding numbers, if the PG contains
object(s) whose hash happens to be 0xffffffff.

in this change, if the calculated hash of the upper bound is greater
than the maximum possible number represented by uint32_t, in addition to
setting the hash of the upper bound hobj to 0xffffffff, we also set the
nspace of hobj of the upper bound to "\xff", so that the upper bound
is greater than an hobj whose hash happens to be 0xfffffff. please note,
the nspace of "\xff" is not an ascii string, so it's not likely to be
less than a real-world nspace of an hobj.

with this new *greater* upper bound, we are able to include the previous
missing hobj when listing the objects in a PG. so the scrub won't be
annoyed when the number of objects does not match.

Fixes: https://tracker.ceph.com/issues/52705
Signed-off-by: Mykola Golub <mykola.golub@clyso.com>
Signed-off-by: Kefu Chai <tchaikov@gmail.com>

os/bluestore: use scope_guard to log latency

simpler this way, and avoid using `goto`.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43150 from sebastian-philipp/doc-cephadm-serves-sub-tree

doc/cephadm: move services under doc/cephadm/services ?

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: Move force maintenance test to the workflow test suite

Fixes: https://tracker.ceph.com/issues/52276
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43221 from badone/wip-install-deps-new-pip-resolver

install-deps.sh: Use legacy pip resolver

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

crimson/os/seastore: introduce ool related metrics with misc improvements

* The number of ool records written;
* Write overhead from journal/ool records;
* Wasted writes from invalided ool records;
* Wasted writes from erased inline extents;
* Distinguish ool and inline extents from metrics;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction: count fresh blocks separately

Correct get_num_fresh_blocks() to be accurate under the effect of
delayed allocations.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction: cleanup comments

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: misc cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup, consolidate metrics about transactional efforts

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup, reset cache metrics as a whole

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/cephadm: populate trusted_ip_list in iscsi-gateway.cfg with mgr ips

Iscsi gateways do not show "UP" in dashboard without this

fixes: https://tracker.ceph.com/issues/52692
Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>

osd/PrimaryLogPG: drop ops when pool has EIO flag

If the op doesn't have the SUPPORTSPOOLEIO flag (i.e., it is an old client),
return with an explicit EIO.

Signed-off-by: Sage Weil <sage@newdream.net>

osdc/Objecter: set SUPPORTSPOOLEIO flag on all ops

This tells us the client understands the pool EIO flag, without having to
burn a feature bit.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph_test_rados_api_aio: test pool EIO flag

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: remove cephadm exporter code

It's functionality has been effectively replaced by the cephadm agent

Signed-off-by: Adam King <adking@redhat.com>

cephadm: fix iscsi test

the current test is wrong because it generates the tcmu-runner part two
times.
given the function `deploy_daemon_units()` in cephadm already writes a
first time the tcmu-runner command, calling a second time
`get_tcmu_runner_container()` from the test makes `deploy_daemon_units()`
write the same command again.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

mon/OSDMonitor: Make the pg_num check more accurate

In check_pg_num function, finding the corresponding osd according to the
current pool's crush rule, and calculating whether the average value of
pg_num on these osd will exceed the value of 'mon_max_pg_per_osd'. Make
the pg_num check more accurate by counting all the pgs on the osd used
by the new pool.

Fixes: https://tracker.ceph.com/issues/47062
Signed-off-by: Jerry Luo <luojierui@chinatelecom.cn>

doc/dev/cephx: fix typo

Signed-off-by: Prashant D <pdhange@redhat.com>

Merge pull request #43285 from rhcs-dashboard/api-endpoint-version-compatibility

mgr/dashboard: make modified API endpoints backward compatible

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

cephadm: modify iscsi containers bindmount

tcmu-runner logs in `/var/log/tcmu-runner.log`, there's no option to
make it log to stdout/stderr so the log is only available from the
container.
Modifying the bindmount from `-v /var/log/ceph/<fsid>/:/var/log/rbd-target-api:z`
to `-v /var/log/ceph/<fsid>/:/var/log:z` makes it at least available
from the host.

Fixes: https://tracker.ceph.com/issues/47480
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

doc/cephadm: move services into services/

This is going to clean up the toctree a bit.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: move service-management to services/index.rst

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43160 from sebastian-philipp/doc-spec-networks

doc/cephadm: Add networks to spec documentation

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

rpm: enable system_pmdk bcond for SUSE builds

ecb8d2cae2c063acf4e7e1bffed887d52117762f disabled the system_pmdk bcond for all
build targets based on the fact that pmdk 1.10 was not available on any of them.

Now that openSUSE Tumbleweed ships pmdk 1.11, we re-enable the system_pmdk bcond
to fix the master build for openSUSE Tumbleweed.

Since openSUSE Tumbleweed is the *only* SUSE build target master supports, there
is no need for greater granularity in the distro conditional here.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cephadm/iscsi: fix a typo

This commit fixes a typo in the comment of unit.run of iscsi service.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #43276 from lxbsz/fix_typo

mds: fix typo in the comments and log when setting new loner

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43301 from athanatos/sjust/wip-seastore-fixes-2021-09-24

crimson/os/seastore/.../btree_range_pin: fix is_parent_of

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43291 from badone/wip-missing-hostname-dependency

rpm, debian: We depend on 'hostname'

Reviewed-by: Kefu Chai <kchai@redhat.com>

rpm, debian: Add hostname as a "make check" dependency

test_hostname.cc depends on the hostname binary.

Fixes: https://tracker.ceph.com/issues/52695
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #43191 from ronen-fr/wip-rf-scrub-locations

osd/scrub: collecting scrub-related files into a separate directory

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43304 from inspur-wyq/wip-doc-3

doc/rbd/rbd-windows.rst: fix typos

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/scrub: collecting scrub-related files into a separate directory

Cleaning src/osd from scrub implementation files. Triggered by:
- the matching Crimson scrub structure;
- the proliferation of scrub related code files (inc. in coming PRs);

scrubber_common.h, which defines the scrubber's interface, remains
in src/osd.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #40984 from ronen-fr/wip-ronenf-scrub-sched

osd/scrub: extracting scrub scheduling code from OSD.cc

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43220 from idryomov/wip-keyring-resolve-error

auth,mon: don't log "unable to find a keyring" error when key is given

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43207 from idryomov/wip-full-try-regressions

librados,neorados,librbd: fix FULL_TRY handling regressions

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mgr/dashboard: replace string version with class

* APIVersion:
  * Moved to a separate file
  * Added doctests
  * Added sentinel values:
    * DEFAULT = 1.0
    * EXPERIMENTAL = 0.1
    * NONE = 0.0
  * Added to_mime_type() helper method
* Controllers.__init__:
  * Added type hints
  * Replaced string versions with APIVersions
* Feedback controller:
  * Replaced with EXPERIMENTAL (probably it should be NONE)

Fixes: https://tracker.ceph.com/issues/52480
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

osd: fix the truncation of an int by int division

The 'r' coeff calculated in OSD::tick() was always 0.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd: add a missing comma in a list of strings

One of OSD::get_tracked_conf_keys() keys was missing
a terminating ','

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #43228 from idryomov/wip-rbd-mirror-snapshot-rx-only

rbd-mirror: unbreak one-way snapshot-based mirroring

Reviewed-by: Mykola Golub <mgolub@suse.com>

mds: fix the set loner log

Once here the current loner has already been dropped if there has,
and the get_wanted_loner() is the new loner it trying to set to.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

mds: fix typo in the comments

Signed-off-by: Xiubo Li <xiubli@redhat.com>

crimson/os/seastore/.../btree_range_pin: fix is_parent_of

The LBA tree implementation only requires that the start addr of
a logical extent be contained within the leaf range.  It's entirely
possible for the end of a logical extent to extend past the end addr
of the containing leaf node.

Fixes: https://tracker.ceph.com/issues/52709
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: whitespace cleanup submit_transaction_direct

Signed-off-by: Samuel Just <sjust@redhat.com>

osdc/Objecter: return EIO for new linger ops

Signed-off-by: Sage Weil <sage@newdream.net>

osdc/Objecter: return EIO for existing ops and linger ops

Signed-off-by: Sage Weil <sage@newdream.net>

osdc/Objecter: return EIO for new ops

Signed-off-by: Sage Weil <sage@newdream.net>

osd,mon: add EIO pool flag

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43282 from Madhu-1/csi-ceph-conf

doc/rbd/rbd-kubernetes: add ceph.conf configmap

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43294 from badone/wip-run-tox-python-common-timedelta-fix

mgr/dashboard: More robust handling of timedelta() representation

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41931 from pritha-srivastava/wip-rgw-sts-51219

rgw/sts: code for returning an error when an IAM policy

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

doc/rbd/rbd-kubernetes: add ceph.conf configmap

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

Merge PR #42384 into master

* refs/pull/42384/head:
mgr/cephadm: kick serve loop if new metadata makes all hosts metadata up to date
mgr/cephadm: fix upgrade to version with agent
mgr/cephadm: delay actions on agent daemons if root cert not created
mgr/cephadm: handle making certs when we have hostname but no address
mgr/cephadm: make agent certs compatible with OpenSSL hostname and ip checks
mgr/cephadm: offline host handling improvements for agent
mgr/cephadm: handle use_agent being turned on and off
mgr/cephadm: better handling of offline hosts with agent
mgr/cephadm: convert networks from set to list + don't reset con on down agent hosts
mgr/cephadm: add ceph volume to metadata agent reports
mgr/cephadm: implement 2-way ssl in mgr -> MgrListener comm line
cephadm: allow mgr listener to handle variable length json strings
mgr/cephadm: cephadm agent 2.0

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43298 from sebastian-philipp/cephadm-check_time_sync-mock

cephadm: TestCheckHost: also mock check_time_sync

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: make modified API endpoints backward compatible

Fixes: https://tracker.ceph.com/issues/52480
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Introducing APIVersion class to handle versioning for API-endpints and making
them backward compatible.

mgr/cephadm: kick serve loop if new metadata makes all hosts metadata up to date

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: fix upgrade to version with agent

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: delay actions on agent daemons if root cert not created

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: handle making certs when we have hostname but no address

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: make agent certs compatible with OpenSSL hostname and ip checks

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: offline host handling improvements for agent

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: handle use_agent being turned on and off

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: better handling of offline hosts with agent

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: convert networks from set to list + don't reset con on down agent hosts

networks needs to be list so they can be encoded in  a json string
resetting con on the hosts where agent isn't reporting (possibly offline hosts)

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: add ceph volume to metadata agent reports

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: implement 2-way ssl in mgr -> MgrListener comm line

Signed-off-by: Adam King <adking@redhat.com>

cephadm: allow mgr listener to handle variable length json strings

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: cephadm agent 2.0

Creates an http endpoint in mgr/cephadm to receive
http requests and an agent that can be deployed on
each host that will gather metadata on the host and
send it to the mgr/cephadm http endpoint. Should save the
cephadm mgr module a lot of time it would have to spend
repeatedly ssh-ing into each host to gather the metadata
and help performance on larger clusters.

Fixes: https://tracker.ceph.com/issues/51004
Signed-off-by: Adam King <adking@redhat.com>

cephadm: Fix TestMaintenance trying to list /var/lib/ceph

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

cephadm: TestCheckHost: also mock `check_time_sync`

Fixes: https://tracker.ceph.com/issues/52722
```
TestCheckHost.test_container_engine fails at cephadm:5834: Error cephadm.Error: No time synchronization is active
```

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

qa/suites/rbd: test case for one-way snapshot-based mirroring

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43037 from ljflores/perf-histogram-formatting

mgr/dashboard: improve formatting of histograms in Telemetry preview form

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge pull request #43295 from liu-chunmei/fix_store_nbd

crimson/store-nbd: fix store_nbd build error for futurized store mkfs

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/store-nbd: fix store_nbd build error for futurized store mkfs

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #43261 from rzarzynski/wip-crimson-mkfs_ertr

crimson/{common,os,osd}: errorate the FuturizedStore::mkfs() paths

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>