ceph-volume tests/functional run lvm list after OSD provisioning

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23075 from alfredodeza/wip-24492

ceph-volume batch command

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23083 from dzafman/wip-disable-skip-dd

osd: Change osd_skip_data_digest default to false and make it LEVEL_DEV

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22732 from b-ranto/wip-metric-reset

prometheus: Fix metric resets

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #23040 into master

* refs/pull/23040/head:
ceph-debug-docker: install python3 cephfs/rados packages

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #23008 from tchaikov/wip-rbd-silence-gcc-warning

tools/rbd: assert(g_ceph_context) not g_conf

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22902 from alfonsomthd/24778-dashboard-info-cards-refactor

mgr/dashboard: Dashboard info cards refactoring

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23108 from tchaikov/wip-install-deps-seastar

run-make-check: set WITH_SEASTAR with a non-empty string

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

install-deps: set DEBIAN_FRONTEND for apt-get

so it won't complain if the tty is not an interactive one.

Signed-off-by: Kefu Chai <kchai@redhat.com>

run-make-check: set WITH_SEASTAR with a non-empty string

otherwise "test -n $WITH_SEASTAR" fails, and the dependencies won't be
installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #22966 into master

* refs/pull/22966/head:
mds: clarify doc string for mds_beacon_grace

Reviewed-by: Zheng Yan <zyan@redhat.com>

prometheus: Set the response header for cached response

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Reset the time the data was captured

This was forgotten during the previous patches.

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Format metrics in the collect function

This allows us to clear metrics upon formatting in the collect function.

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Remove the Metrics class

The class was introduced to account for the 'pending' mechanism but that
is no longer being used by the code so we no longer need it since it
serves a purpose of a simple dictionary, now.

This simplifies the code and brings a bit of performance improvements
since the Metric classes in the collect method are created only if they
did not exist before.

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Optimize metrics formatting

It is faster to use join for join multiple strings than the regular
string concatenation since it does not create a new string on each
iteration.

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Use instance instead of inst variable

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Make the cache timeout configurable

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Fix metric resets

This patch changes the way we reset metrics when collecting data in the
prometheus exporter module. With this patch, the exported metrics at any
point in time should align with a freshly started ceph-mgr module.

The patch also introduces locking mechanism to serialize the requests
and to avoid overwriting the metrics during multiple scrapes happenning
at the same time.

Signed-off-by: Boris Ranto <branto@redhat.com>

ceph-volume tests validate the new utils for prompts

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util create prompt helpers

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.api update lvm sizing tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify str_to_int utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm convert and use extents for sizing calculations

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util move the str_to_int utility out of lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch update help docs from auto to batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.main rename auto to batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm rename auto to batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.strategies add stubs for filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.strategies add bluestore with validators

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23093 from ceph/wip-rm24044

ceph-volume: an OSD ID must be exist and be destroyed before reuse

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23038 from linuxbox2/wip-rgwfile-deepstat

rgw_file: deep stat handling

Merge pull request #23044 from tchaikov/wip-crimson-connection

crimson/connection: misc changes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23057 from liewegas/wip-12.2.7-notes

doc/release/luminous: v12.2.6 and v12.2.7 release notes

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23085 from Songweibin/wip-rbd-unused-var

test/librbd: drop unused variable ‘num_aios’

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume: an osd ID must exist and be destroyed before reuse

Fixes: https://tracker.ceph.com/issues/24044
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #22681 into master

* refs/pull/22681/head:
common: use ceph_pthread_getname

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23052 into master

* refs/pull/23052/head:
osd: fix deep scrub with osd_skip_data_digest=true (default) and bluestore

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/release/luminous: v12.2.7 release notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22899 from smithfarm/wip-doc-argonaut

doc: cleanup: prune Argonaut-specific verbiage

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/start/hardware-recommendations.rst: link to Ceph blog

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: fix up install/upgrading-ceph.rst

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: cleanup: prune Argonaut-specific verbiage

Also drop all release-specific upgrading instructions (they only go up to
Firefly, anyway - none of the current releases are covered).

Note that all of this verbiage I am removing here can still be accessed
on docs.ceph.com via e.g. http://docs.ceph.com/docs/firefly/

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23079 from smithfarm/wip-with-seastar-fix

build/ops: install-deps.sh: set with_seastar

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/librbd: drop unused variable ‘num_aios’

Signed-off-by: songweibin <song.weibin@zte.com.cn>

common: Fix description for osd_skip_data_digest

Signed-off-by: David Zafman <dzafman@redhat.com>

common/options: disable osd_skip_data_digest

This is flawed in that it stops doing digests based on the primary's
backend store, not the replicas.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c7d8add701f156eb8bd8c7dc8ae9903bd782e77f)

common/options: make osd_skip_data_digest a DEV option

And document/warn to never turn it off after it's been on.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c854ed9b33a5a293c8ace15718e26f74237e3332)

build/ops: install-deps.sh: set with_seastar

25d5122b5538062e8282251c3d7bae0f9792d596 introduced code into
install-deps.sh which was intended to set a variable "with_seastar" to the
string "true" or "false" depending on whether or not an environment
variable WITH_SEASTAR is defined.

As originally implemented, the conditional is always true, so with_seastar
is set to "true" unconditionally. Fix this by putting $WITH_SEASTAR in
double quotes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-volume util/prepare.py

Replaces PR#21887
Rename function check_id to osd_id_exists in util/prepare.py
Fix boolean analysis of osd_id_exists when --osd-id is provided on command line

Signed-off-by: Ron Allred <rallred@itrefined.com>

Merge PR #23068 into master

* refs/pull/23068/head:
doc: fix some errors in the cephfs-journal-tool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22737 from trociny/wip-24675

librbd: validate data pool for self-managed snapshot support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume lvm.auto create a strategies module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare do not pass the name when creating a vg

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm create_vg should optionally name vgs on the fly

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.main make auto a subcommand

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.auto add --yes prompt to execute strategies

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: fix some errors in the cephfs-journal-tool

Signed-off-by: Zhu Shangzhong <zhu.shangzhong@zte.com.cn>

Merge PR #23059 into master

* refs/pull/23059/head:
common/ceph_strings: add request_flush_mdlog op names to ceph_session_op_name()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22211 into master

* refs/pull/22211/head:
cephfs-journal-tool: get and check layout info during dump/undump

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22705 into master

* refs/pull/22705/head:
Changed Lua library support from all to partial
mds/Mantle: Fixed Mantle's obsolete Lua C++ API

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22751 into master

* refs/pull/22751/head:
test: add tests for ceph_utime() and friends
libcephfs: extend support for changing file access/modification time

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22801 into master

* refs/pull/22801/head:
client: add inst to asok status output

Reviewed-by: Venky Shankar <vshankar@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22804 into master

* refs/pull/22804/head:
mds: remove useless dout_subsys macro

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: validate data pool for self-managed snapshot support

Fixes: https://tracker.ceph.com/issues/24675
Signed-off-by: Mykola Golub <mgolub@suse.com>

doc: releases: release notes for v12.2.6 luminous

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #22945 from cxytt/master

librbd:optionally support FUA (force unit access) on write requests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22637 from zy751713126/resize_equal_size

rbd: error if new size is equal to original size

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20112 from liyichao/aio_fail

librbd: handle aio failure in ManagedLock and PreReleaseRequest

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: fix deep scrub with osd_skip_data_digest=true (default) and bluestore

The osd_skip_data_digest is about skipping the storing and checking of a
full-object data digest.  Here it is also disabling crc checks during
deep-scrub, which prevents us from detecting crc differences across
replicas.  Fix!

Fixes: http://tracker.ceph.com/issues/24922
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23046 from tchaikov/wip-ftbfs-rgw-kefu

rgw: fix FTBFS introduced by abca9805

rgw: fix FTBFS introduced by abca9805

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22809 from tchaikov/wip-with-seastar

run-make-check: enable --with-seastar option

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22964 from tchaikov/wip-import-pub-key-without-hkp-server

install-deps.sh: import ubuntu-toolchain-r's key without keyserver

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

crimson/net: use boost::thread_unsafe_counter for refcounting Connection

seastar is basically single threaded.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: add Connection::get_peer_type()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #22971 into master

* refs/pull/22971/head:
qa: optionally use old fs name during cephfs remount
qa: remove dead code
qa: run asok command on correct machine

qa: optionally use old fs name during cephfs remount

This fixes errors caused by remount done by some tests (test_recovery_pool.py)
where the fs name is not given.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove dead code

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-debug-docker: install python3 cephfs/rados packages

Fixes: http://tracker.ceph.com/issues/24870
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: run asok command on correct machine

The MDS may not be on the same machine where the cluster command is run.

Fixes: http://tracker.ceph.com/issues/24858
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw_file: deep stat handling

A change to the exact-match behavior in leaf stat operations is
required to support instantiation of a full path hierarchy when
one or more intervening directories has not been enumerated.

Fixes: http://tracker.ceph.com/issues/24915
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #21885 into master

* refs/pull/21885/head:
qa: update cluster log health warning message
qa: add tests for client features
mds: evict clients that lack required features
mds: cleanup MDSRank::evict_client
mds: infer client version by client metadata and connection's features
mds: introduce "ceph fs set <fs_name> min_compat_client <release_name>"
mds: tell client why it's rejected
mds: introduce cephfs' own feature bits
mds: make Server::prepare_force_open_sessions() update client metadata

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23018 into master

* refs/pull/23018/head:
qa: use 4 mds cluster for multifs tests

Reviewed-by: Zheng Yan <zyan@redhat.com>

qa: use 4 mds cluster for multifs tests

Fixes: http://tracker.ceph.com/issues/24899
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: update cluster log health warning message

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23033 from linuxbox2/wip-rgw-409

rgw: do not ignore EEXIST in RGWPutObj::execute

Merge pull request #23020 from tchaikov/wip-config-value-friend

common/config_values: friend md_config_impl<>

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

rgw: do not ignore EEXIST in RGWPutObj::execute

The existing logic appears able to cause propagation of a failed
exclusive create to the client, when it should instead have been
retried, due to disagreement about the logical write offset. (The
value of ofs here could be > 0 due to the operation of a stacked
write filter [e.g., compressor], when the RADOS write offset was
0 and hence an exclusive write that should be retried).

Rationale for fix by Casey.

http://tracker.ceph.com/issues/22790

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rbd-ggate: filter out ceph options from args

we can just filter out ceph options from command line args instead of
parsing config in env variables, conf files and argvs.

the config-parsing steps were modeled after the ones in rbd-nbd.cc, but
the whole purpose of reading configurations from env/conf-file is to get
the "rbd_default_pool" for setting cfg->poolname. apparently, this does
not apply to rbd-ggate, where global_init() will take care of all the
trivial work to populate the settings. moreover, `config` is not
read after being initialized in main.cc. so drop it.

Signed-off-by: Mykola Golub <mgolub@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume lvm.auto help menu and dynamic strategy loading

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.auto move auto as a sub-command of lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify Size() behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk capture real size when inspecting disks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk create a Size class for sizing representations and calculations

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume main remove auto as a top-level command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm extract sizing method into its own utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

common/config_values: friend md_config_impl<>

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #22995 into master

* refs/pull/22995/head:
qa: use root to find client admin socket

Reviewed-by: Nathan Cutler <ncutler@suse.com>