Merge pull request #42821 from rhcs-dashboard/force-maintenance-workflow

mgr/dashboard: Move force maintenance test to the workflow test suite

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43246 from Daniel-Pivonka/cephadm-iscsi-trust-mgr-ips

mgr/cephadm: populate trusted_ip_list in iscsi-gateway.cfg with mgr ips

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43309 from guits/iscsi_fixes

cephadm/iscsi: fix a typo

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43319 from guits/workaround_tcmurunner_log

cephadm: modify iscsi containers bindmount

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43321 from adk3798/remove-exporter

mgr/cephadm: remove cephadm exporter code

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43226 from TRYTOBE8TME/wip-rgw-amqp-minor-nit

src/test: Minor nit from PR #43025

Merge pull request #43327 from cyx1231st/wip-seastore-ool-metrics

crimson/os/seastore: introduce ool related metrics with misc improvements

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43302 from ronen-fr/wip-rf-osd-psik

osd: add a missing comma in a list of strings

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #43303 from ronen-fr/wip-rf-osd-modulu

osd: fix the truncation of an int by int division

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43132 from p-se/monitoring-grafana-piechart-update

monitoring: update grafana-piechart-panel plugin

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: p-se <NOT@FOUND>

Merge PR #43177 into master

* refs/pull/43177/head:
osd/PrimaryLogPG: drop ops when pool has EIO flag
osdc/Objecter: set SUPPORTSPOOLEIO flag on all ops
ceph_test_rados_api_aio: test pool EIO flag
osdc/Objecter: return EIO for new linger ops
osdc/Objecter: return EIO for existing ops and linger ops
osdc/Objecter: return EIO for new ops
osd,mon: add EIO pool flag

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43320 from pdvian/wip-doc-cephx-fixtypo

doc/dev/cephx: fix typo

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43316 from smithfarm/wip-spdk-tw

rpm: enable system_pmdk bcond for SUSE builds

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43150 from sebastian-philipp/doc-cephadm-serves-sub-tree

doc/cephadm: move services under doc/cephadm/services ?

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: Move force maintenance test to the workflow test suite

Fixes: https://tracker.ceph.com/issues/52276
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43221 from badone/wip-install-deps-new-pip-resolver

install-deps.sh: Use legacy pip resolver

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

crimson/os/seastore: introduce ool related metrics with misc improvements

* The number of ool records written;
* Write overhead from journal/ool records;
* Wasted writes from invalided ool records;
* Wasted writes from erased inline extents;
* Distinguish ool and inline extents from metrics;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction: count fresh blocks separately

Correct get_num_fresh_blocks() to be accurate under the effect of
delayed allocations.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction: cleanup comments

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: misc cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup, consolidate metrics about transactional efforts

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup, reset cache metrics as a whole

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/cephadm: populate trusted_ip_list in iscsi-gateway.cfg with mgr ips

Iscsi gateways do not show "UP" in dashboard without this

fixes: https://tracker.ceph.com/issues/52692
Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>

osd/PrimaryLogPG: drop ops when pool has EIO flag

If the op doesn't have the SUPPORTSPOOLEIO flag (i.e., it is an old client),
return with an explicit EIO.

Signed-off-by: Sage Weil <sage@newdream.net>

osdc/Objecter: set SUPPORTSPOOLEIO flag on all ops

This tells us the client understands the pool EIO flag, without having to
burn a feature bit.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph_test_rados_api_aio: test pool EIO flag

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: remove cephadm exporter code

It's functionality has been effectively replaced by the cephadm agent

Signed-off-by: Adam King <adking@redhat.com>

cephadm: fix iscsi test

the current test is wrong because it generates the tcmu-runner part two
times.
given the function `deploy_daemon_units()` in cephadm already writes a
first time the tcmu-runner command, calling a second time
`get_tcmu_runner_container()` from the test makes `deploy_daemon_units()`
write the same command again.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

doc/dev/cephx: fix typo

Signed-off-by: Prashant D <pdhange@redhat.com>

Merge pull request #43285 from rhcs-dashboard/api-endpoint-version-compatibility

mgr/dashboard: make modified API endpoints backward compatible

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

cephadm: modify iscsi containers bindmount

tcmu-runner logs in `/var/log/tcmu-runner.log`, there's no option to
make it log to stdout/stderr so the log is only available from the
container.
Modifying the bindmount from `-v /var/log/ceph/<fsid>/:/var/log/rbd-target-api:z`
to `-v /var/log/ceph/<fsid>/:/var/log:z` makes it at least available
from the host.

Fixes: https://tracker.ceph.com/issues/47480
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

doc/cephadm: move services into services/

This is going to clean up the toctree a bit.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: move service-management to services/index.rst

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43160 from sebastian-philipp/doc-spec-networks

doc/cephadm: Add networks to spec documentation

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

rpm: enable system_pmdk bcond for SUSE builds

ecb8d2cae2c063acf4e7e1bffed887d52117762f disabled the system_pmdk bcond for all
build targets based on the fact that pmdk 1.10 was not available on any of them.

Now that openSUSE Tumbleweed ships pmdk 1.11, we re-enable the system_pmdk bcond
to fix the master build for openSUSE Tumbleweed.

Since openSUSE Tumbleweed is the *only* SUSE build target master supports, there
is no need for greater granularity in the distro conditional here.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cephadm/iscsi: fix a typo

This commit fixes a typo in the comment of unit.run of iscsi service.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #43276 from lxbsz/fix_typo

mds: fix typo in the comments and log when setting new loner

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43301 from athanatos/sjust/wip-seastore-fixes-2021-09-24

crimson/os/seastore/.../btree_range_pin: fix is_parent_of

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43291 from badone/wip-missing-hostname-dependency

rpm, debian: We depend on 'hostname'

Reviewed-by: Kefu Chai <kchai@redhat.com>

rpm, debian: Add hostname as a "make check" dependency

test_hostname.cc depends on the hostname binary.

Fixes: https://tracker.ceph.com/issues/52695
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #43191 from ronen-fr/wip-rf-scrub-locations

osd/scrub: collecting scrub-related files into a separate directory

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43304 from inspur-wyq/wip-doc-3

doc/rbd/rbd-windows.rst: fix typos

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/scrub: collecting scrub-related files into a separate directory

Cleaning src/osd from scrub implementation files. Triggered by:
- the matching Crimson scrub structure;
- the proliferation of scrub related code files (inc. in coming PRs);

scrubber_common.h, which defines the scrubber's interface, remains
in src/osd.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #40984 from ronen-fr/wip-ronenf-scrub-sched

osd/scrub: extracting scrub scheduling code from OSD.cc

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43220 from idryomov/wip-keyring-resolve-error

auth,mon: don't log "unable to find a keyring" error when key is given

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43207 from idryomov/wip-full-try-regressions

librados,neorados,librbd: fix FULL_TRY handling regressions

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mgr/dashboard: replace string version with class

* APIVersion:
  * Moved to a separate file
  * Added doctests
  * Added sentinel values:
    * DEFAULT = 1.0
    * EXPERIMENTAL = 0.1
    * NONE = 0.0
  * Added to_mime_type() helper method
* Controllers.__init__:
  * Added type hints
  * Replaced string versions with APIVersions
* Feedback controller:
  * Replaced with EXPERIMENTAL (probably it should be NONE)

Fixes: https://tracker.ceph.com/issues/52480
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

osd: fix the truncation of an int by int division

The 'r' coeff calculated in OSD::tick() was always 0.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd: add a missing comma in a list of strings

One of OSD::get_tracked_conf_keys() keys was missing
a terminating ','

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #43228 from idryomov/wip-rbd-mirror-snapshot-rx-only

rbd-mirror: unbreak one-way snapshot-based mirroring

Reviewed-by: Mykola Golub <mgolub@suse.com>

mds: fix the set loner log

Once here the current loner has already been dropped if there has,
and the get_wanted_loner() is the new loner it trying to set to.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

mds: fix typo in the comments

Signed-off-by: Xiubo Li <xiubli@redhat.com>

crimson/os/seastore/.../btree_range_pin: fix is_parent_of

The LBA tree implementation only requires that the start addr of
a logical extent be contained within the leaf range.  It's entirely
possible for the end of a logical extent to extend past the end addr
of the containing leaf node.

Fixes: https://tracker.ceph.com/issues/52709
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: whitespace cleanup submit_transaction_direct

Signed-off-by: Samuel Just <sjust@redhat.com>

osdc/Objecter: return EIO for new linger ops

Signed-off-by: Sage Weil <sage@newdream.net>

osdc/Objecter: return EIO for existing ops and linger ops

Signed-off-by: Sage Weil <sage@newdream.net>

osdc/Objecter: return EIO for new ops

Signed-off-by: Sage Weil <sage@newdream.net>

osd,mon: add EIO pool flag

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43282 from Madhu-1/csi-ceph-conf

doc/rbd/rbd-kubernetes: add ceph.conf configmap

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43294 from badone/wip-run-tox-python-common-timedelta-fix

mgr/dashboard: More robust handling of timedelta() representation

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41931 from pritha-srivastava/wip-rgw-sts-51219

rgw/sts: code for returning an error when an IAM policy

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

doc/rbd/rbd-kubernetes: add ceph.conf configmap

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

Merge PR #42384 into master

* refs/pull/42384/head:
mgr/cephadm: kick serve loop if new metadata makes all hosts metadata up to date
mgr/cephadm: fix upgrade to version with agent
mgr/cephadm: delay actions on agent daemons if root cert not created
mgr/cephadm: handle making certs when we have hostname but no address
mgr/cephadm: make agent certs compatible with OpenSSL hostname and ip checks
mgr/cephadm: offline host handling improvements for agent
mgr/cephadm: handle use_agent being turned on and off
mgr/cephadm: better handling of offline hosts with agent
mgr/cephadm: convert networks from set to list + don't reset con on down agent hosts
mgr/cephadm: add ceph volume to metadata agent reports
mgr/cephadm: implement 2-way ssl in mgr -> MgrListener comm line
cephadm: allow mgr listener to handle variable length json strings
mgr/cephadm: cephadm agent 2.0

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43298 from sebastian-philipp/cephadm-check_time_sync-mock

cephadm: TestCheckHost: also mock check_time_sync

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: make modified API endpoints backward compatible

Fixes: https://tracker.ceph.com/issues/52480
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Introducing APIVersion class to handle versioning for API-endpints and making
them backward compatible.

mgr/cephadm: kick serve loop if new metadata makes all hosts metadata up to date

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: fix upgrade to version with agent

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: delay actions on agent daemons if root cert not created

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: handle making certs when we have hostname but no address

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: make agent certs compatible with OpenSSL hostname and ip checks

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: offline host handling improvements for agent

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: handle use_agent being turned on and off

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: better handling of offline hosts with agent

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: convert networks from set to list + don't reset con on down agent hosts

networks needs to be list so they can be encoded in  a json string
resetting con on the hosts where agent isn't reporting (possibly offline hosts)

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: add ceph volume to metadata agent reports

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: implement 2-way ssl in mgr -> MgrListener comm line

Signed-off-by: Adam King <adking@redhat.com>

cephadm: allow mgr listener to handle variable length json strings

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: cephadm agent 2.0

Creates an http endpoint in mgr/cephadm to receive
http requests and an agent that can be deployed on
each host that will gather metadata on the host and
send it to the mgr/cephadm http endpoint. Should save the
cephadm mgr module a lot of time it would have to spend
repeatedly ssh-ing into each host to gather the metadata
and help performance on larger clusters.

Fixes: https://tracker.ceph.com/issues/51004
Signed-off-by: Adam King <adking@redhat.com>

cephadm: Fix TestMaintenance trying to list /var/lib/ceph

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

cephadm: TestCheckHost: also mock `check_time_sync`

Fixes: https://tracker.ceph.com/issues/52722
```
TestCheckHost.test_container_engine fails at cephadm:5834: Error cephadm.Error: No time synchronization is active
```

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

qa/suites/rbd: test case for one-way snapshot-based mirroring

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43037 from ljflores/perf-histogram-formatting

mgr/dashboard: improve formatting of histograms in Telemetry preview form

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge pull request #43295 from liu-chunmei/fix_store_nbd

crimson/store-nbd: fix store_nbd build error for futurized store mkfs

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/store-nbd: fix store_nbd build error for futurized store mkfs

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #43261 from rzarzynski/wip-crimson-mkfs_ertr

crimson/{common,os,osd}: errorate the FuturizedStore::mkfs() paths

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43288 from rzarzynski/wip-crimson-skip-our-frames-in-bt

crimson/common: skip first 4 frames when dumping a backtrace.

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

mgr/dashboard: More robust handling of timedelta() representation

The representation of timedelta() appears to have changed recently so
we should try and be more accommodating in that regard.

Fixes: https://tracker.ceph.com/issues/52696
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

install-deps.sh: Use legacy pip resolver

Due to the issue described in https://github.com/pypa/pip/issues/9818 I
am seeing lots of downloads and prolonged runtime when installing
dependencies.

Fixes: https://tracker.ceph.com/issues/52660
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #43275 from adk3798/maint-target

mgr/cephadm: base maintenance mode enter/exit success/failure on returned message

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43121 from mgfritch/cephadm-pull-error

cephadm: raise error during `pull` failure

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>

Merge pull request #43062 from strenuous-life/wip-cephadm-zap-device

mgr/cephadm: osd should not be zap when it is running

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #37544 from kamoltat/wip-mgr-progress-global-efficiency

mgr/progress: optimize global recovery module

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: improve unittest

Signed-off-by: Laura Flores <lflores@redhat.com>

tests/crimson: make the virtual methods of SeaStoreTestState final.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: workaround the segfaulting GCC 11 issue.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd, crimson/os: errorate the FuturizedStore::mkfs() paths.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common: skip first 4 frames when dumping a backtrace.

It's all about these items:

```
 0# print_backtrace(std::basic_string_view<char, std::char_traits<char> >) at /home/rzarzynski/ceph1/build/../src/crimson/common/fatal_signal.cc:80
 1# FatalSignal::signaled(int, siginfo_t const&) at /opt/rh/gcc-toolset-9/root/usr/include/c++/9/ostream:570
 2# FatalSignal::install_oneshot_signal_handler<11>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /home/rzarzynski/ceph1/build/../src/crimson/common/fatal_signal.cc:
62
 3# 0x00007F16BBA13B30 in /lib64/libpthread.so.0
```

They are part of our backtrace handling and typically developers
are not interested in them. Let's be consistent with the classical
OSD and hide them.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/cephadm: Add `networks` to spec documentation

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: add mgr.rst

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: Add `Specifying Networks` to rgw.rst

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>