python-common drive-groups: allow strings for wal and db size

This enables user to pass sizes like "10G", which batch now understands.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume inventory: add option to filter unwanted devices

Some device we never want to pass to the batch subcommand. For now this
includes devices that have a partition or are mounted on the machine.
One goal is to filter the root device, so it is not included on a batch
command and thus would contribute to its implicit sizing calculation.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: address review comments

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: batch: fix size retrieval for lvs

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: include encryption in batch report

Fixes: https://tracker.ceph.com/issues/44783
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume lvm batch: use namedtuple instead of tuple

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: address review comments, mostly tidying, clarification

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

idempotency must result in the same outcome

...not should

Co-authored-by: Joshua Schmid <jschmid@suse.de>
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: batch test should pass --journal-devices with filestore

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: make --journal optional, add --journal-slots

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

doc: update ceph-volume lvm batch docs

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

py-common/deployment: drop workarounds for old batch

This commit drops two workarounds in drive groups for the old batch
behaviour. OSDs on LVs no longer need to be created individually by
create (related to functional testing) and ceph-volume batch now filters
unavailable drives itself.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume batch: add deprecation warning for auto behaviour

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume batch: add ceph.conf mocking to pass tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume batch: use disk.Size for size args

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume batch: Fix osd_ids passing and improve plan formatting

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume batch: track rel_size in percent, more tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume batch: improve backwards compatibility

This restores legacy batch behavior and also adds some initial test and
adjusts existing tests to changes.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: batch - enable legacy auto behaviour

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: batch - major refactor

This completely refactors the batch code in order to make use of the
create/prepare code path for creating OSDs instead of having a second
code path doing this. This not only eases the maintenance burden but
also adds various features and fixes bugs. This subcommand can now
handle LVs, replace OSDs, reuse VGs and has a better notion of
idempotency.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: Device - available_lvm if 10 extents are free.

This changes the available_lvm notion to only require 10 free extents
instead of 5GB.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: Device - add vg_free property

This new property returns the free space in any VGs present. If no VGs
are on the device we project how much space a VG will have.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: prepare/create - size args as Size class

This add the disk.Size class as all size related argument types. We
often create this class form args like this anyway and it enables users
to pass not only bytes but also strings like 50G.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: disk.Size - add cast to bool

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: api/lvm - add VolumeGroup.free_percent property

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: util.device -  add vg_free_percent property

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: api/lvm - query LV units in bytes

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: lvm/common - refactor common arg specification

This makes it easier to create valid Namespace objects/arg lists when
programmatically calling create/prepare.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: batch: fix argument help message

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37311 from mgariepy/fix-47502

[ceph-volume]: remove unneeded call to get_devices()

Merge PR #36685 into master

* refs/pull/36685/head:
mds: allow quota for root directory

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #37009 into master

* refs/pull/37009/head:
client: remove the unecessary client_lock for objector->read/write()
client: fix the comment for RWRef

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37266 into master

* refs/pull/37266/head:
qa: ignore logrotate state rename error

Reviewed-by: Rishabh Dave <ridave@redhat.com>

ceph-volume: remove unneeded call to get_devices()

there is no need to probe the device to generate the argparse help
message.

also removing the test on the function as it's not there anymore.

Signed-off-by: Marc Gariepy <gariepy.marc@gmail.com>
Fixes: https://tracker.ceph.com/issues/47502

Merge pull request #37374 from wjwithagen/wjw-fix-crypto-alloca

librbd: Use stdlib.h instead of alloca.h

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: Use stdlib.h instead of alloca.h

src/librbd/crypto/BlockCrypto.cc:5:10: fatal error: 'alloca.h' file not found
         ^~~~~~~~~~
1 error generated.

Including <stdlib.h> is enough

fixes: https://tracker.ceph.com/issues/47614
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #37385 from idryomov/wip-krbd-read-only-override

rbd: make common options override krbd-specific options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37384 from tchaikov/wip-crimson-doc

doc/dev/crimson: add instructions on how to build crimson

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rbd: make common options override krbd-specific options

ceph-csi has added support for passing custom map and unmap options via
mapOptions and unmapOptions storage class parameters.  However, it also
uses --read-only for implementing ROX (ReadOnlyMany) PVs.  If the user
supplies "mapOptions: rw", they will get around the intended read-only
restriction (at least on the block device).

ceph-csi could be patched to use "-o ro", but it actually makes sense
for common options to win over device type-specific equivalents.

Fixes: https://tracker.ceph.com/issues/47625
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #37236 from jschmid1/osd_rm_doc_fix

docs: remove docs should say osd_id instead of svc_id

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson: add instructions on how to build crimson

and add a link to Scylla's document on debugging with GDB

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37275 from bk201/wip-47494

mgr/dashboard: display devices' health information within a tabset

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37305 from zdover23/wip-doc-3-dev-guide-basic-workflow-date-update-sep-2020

doc/dev: updated date from 2016

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/dev: added the other four tests

This PR adds a list of tests that are performed
when a PR hits Github.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37186 from guits/guits-cephadm-filter-dangling-images

cephadm: filter dangling images in get_last_local_ceph_image()

Merge pull request #37133 from jschmid1/ptrace_priv

cephadm: capadd and privileged are mutex with podman >= 2.0

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #37310 into master

* refs/pull/37310/head:
client: refresh rstat for quota check of rename

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37041 from bk201/wip-47356

mgr/dashboard: fix security scopes of some NFS-Ganesha endpoints

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37244 from ceph/47543-bucket-list-perf

mgr/dashboard: fix perf. issue when listing large amounts of buckets

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #37242 from dang/wip-dang-47520

Fix missing req_state for librgw

Merge pull request #37330 from idryomov/wip-backport-create-issue-key

doc: SubmittingPatches-backports.rst: backport scripts read key from file

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #37136 from p-se/cd-add-per-rbd-graphs

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

SubmittingPatches-backports.rst: backport scripts read key from file

The --key option was removed in commit 1b5f37bbcd4a
("backport-create-issue: read redmine key from file").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #37192 from rhcs-dashboard/fix-46898-master

mgr/dashboard/api: move/create OSD histogram in separate endpoint

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #37303 from tchaikov/wip-crimson-osd

crimson: do not set_value() twice and fix Message leak

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/mon: move handle_auth_done() to "v1 and v2" section

crimson::mon::Connection::handle_auth_done() is shared by both v1 and v2
authentication implementation, so move it to "v1 and v2" section

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37326 from tchaikov/wip-47324

test/crimson: better error message when bind fails

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #37323 from tchaikov/wip-47593

crimson/osd: add_recovery(oid) before recover_object(oid)

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

test/crimson: better error message when bind fails

See also: https://tracker.ceph.com/issues/47324

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add_recovery(oid) before recover_object(oid)

in 7736bddc534e5df2daf704c4bbc92a0927b0db4d, we assumed that the object
to be recovered did not exist in `recovering` before
`recover_object(oid)` was called. but this turns out not true. so, in
this change, `add_object(oid)` is called before `recover_object(oid)`
gets called.

Fixes: https://tracker.ceph.com/issues/47593
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use recoverying.at() when appropriate

this helps to avoid the confusion of "where the recovery is added" and
"are we adding a new instance of recovery here".

should call add_recovery() explicitly when we need to add a new recovery
instance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add RecoveryBackend:add_recovering()

before this change, get_recovery() can also be used for adding a
recovery instance to `recovering`. this behavior is error-prone and
confusing.

after this change, add_recovery() is used in the place where we
want to add a new instance of recovery instance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37290 from tchaikov/wip-run-make-check

run-make-check.sh: fix syntax error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37106 from sebastian-philipp/cephadm-overwritable-templates

mgr/cephadm: make j2 teamplates overwritable

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #37121 from Songweibin/wip-rbd-metadata

librbd: async version of metadata_set/remove

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37083 from orozery/librbd-data-cryptor

librbd: add data encryptor/decryptor using openssl

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37110 from CongMinYin/qa_immutable_object_cache_check

qa: check immutable object cache content

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37237 from MahatiC/wip-fix-rwl-path

librbd/cache: Fix pwl_path read issue

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37206 from bk201/wip-47387

mgr/rook: fix listing daemons

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

client: refresh rstat for quota check of rename

Fixes: https://tracker.ceph.com/issues/47224
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #37043 from s0nea/wip-dashboard-47331

mgr/dashboard: log in non-admin users successfully if the telemetry notification is shown

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #37293 from changchengx/osd_comment

osd: fix wrong comment about setting auth_client for messenger

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not add ref when constructing MessageRef

crimson::osd::PG::send_cluster_message() accepts a `Message*`
pointer, and then hand it over to `shard_services.send_to_osd()`,
which expects a `Ref<Message>`. so the raw pointer is used to
construct an `intrusive_ptr<Message>`, which increment the
refcount of that Message instance by one. but that Message
was owned by nobody before that, so we end up with an
`intrusive_ptr<Message>` of 2 refcount, and only a single
owner. hence the memory leak.

in this change, instructs the constructor to not add the refcount.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: do not set_value() again

once the continuation consuming the stored value of the associated
future, we cannot set_value() again. otherwise, ASan complains that we
are accessing the memory on heap after it is freed.

in this change, std::optional<> is used for holding
promise<auth_result_t>, once the promise is fulfilled, `auth_done` is
reset to prevent another call of `set_value()` or `set_exception()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev: updated date from 2016

This updates a date from 2016 to 2020,
so that readers can be confident that the
procedure that they're reading has been recently
tested.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/mon: only finish auth when auth is completed

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: s/AuthResult/auth_result_t/

auth_result_t is merely an enum, so make it like so.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: extract mon::Client::_finish_auth()

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: correct comment about setting auth_client for messenger

client_messenger's auth_client has not been set until call
monc->init();

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #37152 from rhcs-dashboard/bottom-links-component

mgr/dashboard: Creating a new Login Page Legal Links Component

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: display devices' health information within a tabset

Wrap all devices' health information within a tabset
instead of displaying them from top to bottom.

Add more guard in the HTML template to prevent referencing undefined
variables.

Fixes: https://tracker.ceph.com/issues/47494
Fixes: https://tracker.ceph.com/issues/43177
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37291 from badone/wip-docker-test-helper-fixes

test/docker-test: helper fixes and support more distro releases

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Add support for fedora 32, 33 and Ubuntu 20.04

Enable creation and use of these OS images in docker-test.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

tests: Make sure install-deps is run noninteractively

This gets past things like tzconfig stopping for user input.
Remove redundant install of python-virtualenv.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #37200 from tchaikov/wip-crimson-with-lock

crimson/common: add helpers for seastar::with_lock()

Reviewed-by: Samuel Just <sjust@redhat.com>

run-make-check.sh: fix syntax error

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37211 from tchaikov/wip-zbd

rpm,install-deps,run-make-check: enable WITH_ZBD build on demand

Reviewed-by: Neha Ojha <nojha@redhat.com>

test/centos-8: Install git before running install-deps

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

test/docker-test: Fix permissions issue when using podman

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #36776 into master

* refs/pull/36776/head:
systemd: Support Graceful Reboot for AIO Node

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge PR #37227 into master

* refs/pull/37227/head:
qa/cephfs: don't pass args to destroy() in recreate()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37233 into master

* refs/pull/37233/head:
qa/mgr: revert a patch from commit 04ed58f

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37271 from cyx1231st/wip-seastore-fix-non-repeatable-read

crimson/seastore: fix potential non-repeatable-read from RootBlock

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #36927 from idryomov/wip-krbd-noudev

krbd: optionally skip waiting for udev events

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #37213 into master

* refs/pull/37213/head:
mgr/rook: Pass pod namespace to list_namespaced_pod()

Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #37101 from LenzGr/master-documentation

doc: Updated `HACKING.rst` and `README.rst`

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37137 from tspmelo/wip-fix-iscsi-tests

mgr/dashboard: Fix iSCSI backend unit-test

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #37262 from trociny/wip-rbd-nbd-quiesce-hook

rbd-nbd: fix typo in mini help

Reviewed-by: Jason Dillaman <dillaman@rehdat.com>

Merge pull request #35785 from rhcs-dashboard/wip-45957-consolidate_Osd_Endpoints

mgr/dashboard: Consolidate Osd mark endpoints

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>