osd: Move creation of 'master_set' to scrub_compare_maps

Moving it to scrub_compare_maps allows us to pass it to multiple
subsequently called functions.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 63ef97b3db81c0c725119943c8594a4a0d12154e)

Merge pull request #21717 from croit/luminous-23915

luminous: mon: handle bad snapshot removal reqs gracefully

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21568 from cbodley/wip-22937

luminous: rgw: beast frontend can listen on multiple endpoints

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #21083 from pdvian/wip-23227-luminous

luminous: common: fix BoundedKeyCounter const_pointer_iterator

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: 'make check' builds radosgw and its cls dependencies

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a029d20b86c7d64626baf14bf0217c142e5e17cf)

rgw: beast frontend can listen on multiple endpoints

rgw_frontends options port= and endpoint= can be given multiple times,
and the beast frontend will listen on each endpoint

Fixes: http://tracker.ceph.com/issues/22779
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1bff073b0e3e19e341fc0a08918abb510f0711b4)

Conflicts: rgw_asio_frontend.cc (boost::asio::string_view not available,
                                 replaced with BOOST_ASIO_STRING_VIEW_PARAM
                                 which falls back to 'const std::string&')

rgw: civetweb accepts multiple port= entries

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 22df0fbffdfa8b1963092cbf67e1ca8e18f01595)

Merge pull request #21746 from alfredodeza/backport-rm23874

luminous  ceph-volume failed ceph-osd --mkfs command doesn't halt the OSD creation process

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests mkfs bluestore fails on non-zero exit status

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 978552a51f000952e0c820422369522063fd581a)

ceph-volume tests allow setting return values on stubbed process.call

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b57e90105721d2817908b60ec39c2b4cbb1ca8f3)

ceph-volume util.prepare raise on non-zero exit statuso

Just for bluestore objectstore that uses process.call

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mon: handle bad snapshot removal reqs gracefully

Snapshot deletion requests on snap ids larger than the snap_seq of
the pool will leave the pool in a state with snap_seq being less
than max(removed_snaps).

This is bad because further deletion requests to a pool in this state
might crash the mon in some cases: the deletion also inserts the new
snap_seq into the removed_snaps set -- which might already exist
in this case and trigger an assert.

Such bad requests will be generated by rbd clients without a fix for
issue #21567.

The change in OSDMonitor prevents pools from getting into this state
and may prevent old broken clients from incorrectly deleting snaps.
The change in osd_types avoids a crash if a pool is already in this
state.

Fixes #18746

Signed-off-by: Paul Emmerich <paul.emmerich@croit.io>
(cherry picked from commit f42a6ba4c37cb10869718ec296352ae55254b44a)

 Conflicts:
src/osd/osd_types.cc

rgw: frontend config uses multimap

this allows us to configure multiple values for a given key without
resorting to string formatting to cram them into a single key/value pair

ex. port=80 port=443s instead of port=80+443s

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit cf1b84195ebb8b0abf733889a3fa994a341be1b1)

Conflicts:
        rgw_frontend.cc: missing overload of get_str_vec()

Merge pull request #21654 from alfredodeza/backport-rm23451

luminous ceph-volume include physical devices associated with an LV when listing

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc/ceph-volume include information for devices in listing

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 009d49a4f5cf8c97a156b710342d4786b734747b)

ceph-volume tests verify associated pvs reporting on listing

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f57cc594c282cf06a75b2278ca1c2b2d38f87aa3)

ceph-volume tests update to include devices on listing

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 08dea1b7d1d92d893c2f0a5e6e750d838d9cdbe4)

ceph-volume lvm.listing include associated physical devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit be1e6c2ae9a6e6a77fcb5297badb1aa58548180c)

ceph-volume api.lvm include lv_uuid on pvs listings

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 12a7a48c53b16f52a76ad74773aaa67f8bfc7c52)

12.2.5

Merge pull request #21565 from alfredodeza/backport-wip-ceph-volume

luminous: ceph-volume: Nits noticed while studying code

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #21574 from alfredodeza/backport-cv-doc-updates

luminous: ceph-volume update man page for prepare/activate flags

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc/man/ceph-volume update flags for activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a454abc863817cd393e01e8c135174c1e1839be2)

doc/man/ceph-volume update flags for prepare

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ae96b9f0a283a65ddf6887461f3515a5a756572b)

Merge pull request #21492 from b-ranto/wip-mgr-rgw-luminous

[luminous] Backport recent prometheus rgw changes

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Remove unused and wrongly-described _lsblk_type

Signed-off-by: Dan Mick <dan.mick@redhat.com>
(cherry picked from commit a97419f5d2b23131242a3602f2fa5f91ac7bc74d)

Fix mixing of format() and %-interpolation

Signed-off-by: Dan Mick <dan.mick@redhat.com>
(cherry picked from commit 9e9a07c9ae41cc72e587dfd6551fd83389cc35e6)

Spelling/typos

Signed-off-by: Dan Mick <dan.mick@redhat.com>
(cherry picked from commit 15ef400bb35e70cbb27144d595759d31a075aa37)

Merge pull request #21538 from alfredodeza/backport-rm23678

luminous: ceph-volume: allow skipping systemd interactions on activate/create

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: performance counter browsers

Signed-off-by: Rubab-Syed <rubab.syed21@gmail.com>
(cherry picked from commit 25ecfbd9cfb620463de1545ca0be50001c805460)
Conflicts: Removed dashboard changes

mgr: expose have_mon_connection to python modules

So that they can warn their end users if the data
we have is probably stale.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 5633328e28c0f7a5122229737c2ca132174a4349)

ceph-volume lvm.activate add --no-systemd flag to skip starting/enabling services

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 75685e33bfc192f3a3b16ad57b8cd40313bd8bc6)

ceph-volume lvm.common add --no-systemd flag to skip enabling/starting services

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 832da49ca8f1a57807f8a892cd734aa9b37216a6)

ceph-volume tests functional python tests for --no-systemd in activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8848d037086f889a880f5ea1d3bb2254e16faf7b)

Merge pull request #21533 from alfredodeza/backport-rm23067

luminous: ceph-volume Implement an 'activate all' to help with dense servers or migrating OSDs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #21506 from liewegas/wip-rados-test-luminous

qa/workunits/rados/test-upgrade-to-mimic.sh: fix tee output

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind/mgr/prometheus: fix creation of osd_metadata metric

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 05a37d4a61c427ed532138ab0c13a9dd4d0f65fb)

mgr/prometheus: fix and complete PG state names

...and reformat into a flat list in the same
order as found in osd_types.cc so that it's
easy to cross check.

Fixes: https://tracker.ceph.com/issues/23584
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 48acc6c7981ef6cbb962883cd36e2666a16d9c23)

mgr/prometheus: skip OSD output if missing from CRUSH devices

Apparently this occurs as a transient state while adding
OSDs.

Fixes: https://tracker.ceph.com/issues/23166
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit e8e5bffbb0842cc9c41bb64e7e9b5c6e2898fc0c)

ceph-volume lvm.activate error if no bluestore OSDs are found

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b1920d6e731f194c4829810d849dba6b087ce3a4)

ceph-volume tests/functional add 'activate all' for xenial/filestore/dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d6524297ac423ffc7ea82dc9598d27b4cc61a0ae)

ceph-volume tests/functional add 'activate all' for xenial/bluestore/dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 452081e57e5de9bb3a1391f22ffcac1bb712f2e9)

ceph-volume tests/functional add 'activate all' for centos7/filestore/dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5513f4de4ee0b4802bc1c0d1af754dd2d80486ed)

ceph-volume tests/functional add 'activate all' for centos7/bluestore/dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 16421c9058011e115cf043de4cf279cbbd0ac0ce)

ceph-volume tests/functional add 'activate all' in filestore playbook

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ef09f3996b8a165112afc4e6912a119015c798be)

ceph-volume tests/functional add 'activate all' in bluestore playbook

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 797518a54603ad8fc9b57a66eab7cf7e55ea0299)

ceph-volume tests ensure activate all behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2d8f48983d8674376a57ad98db16700bc00f8e29)

ceph-volume lvm.activate warn when no osds are found with --all

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 967938a0cf4353e99eb55815cde4ee90d6d4b79b)

ceph-volume tests add process.call unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6975e221d9f0c1b17531ca6fed87882b514e6d95)

ceph-volume lvm.activate add 'activate all' functionality

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fb0008ea8babe0065c08d3377bdabd36f3c2fa33)

ceph-volume process.call add conditional verbosity on failure

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 50c9d5331b0c42d568baa47a2d52f199b3b4d818)

ceph-volume systemd.systemctl add helpers for daemon status

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2e8249f9e233d408fc45b772e45995c00f5180d7)

ceph-volume lvm.listing create a helper that lists osds

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cee7612ce400ca4bc3e6e5dbcc1a158178edafe5)

ceph-volume lvv.activate allo to configure the CLI with --all

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4f305eb661553bebc1a86d18c11f596fa6206828)

Merge pull request #21509 from ceph/backport-rm23757

luminous: ceph-volume: allow parallel creates

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #21505 from ceph/backport-rm23581

luminous: ceph-volume: automatic VDO detection

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #21519 from ceph/evelu-luminous-readonly

luminous: ceph-volume: Using --readonly for {vg|pv|lv}s commands

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: Using --readonly for {vg|pv|lv}s commands

The actual code is detecting {vg|pv|lv}s by running the usual {vg|pv|lv}s commands.
Those calls expect lvmetad to be aware of the actual state of them.
This works actually pretty well in most of the cases.

When ceph-volume is run from a container,
this code also works from the container itself but don't on the host.

On the host side, running {vg|pv|lv}s commands reports nothing,
making ceph-volume reporting "No valid Ceph devices found"
The root cause is lvmetad not receiving the udev notification and so,
{vg|pv|lv}s commands reports the 'known' state instead of the 'real' state.

This is a major issue as it means that it exists cases or maybe races where
ceph-volume reports "No valid Ceph devices found" while the disk
actually have some Ceph data & metadata on them.
This could be interpreted like disks are free/available while they are not.
This will mislead users or configuration tools trying to understand the
current state of a node.

In July 2015, as per https://www.redhat.com/archives/lvm-devel/2015-July/msg00086.html,
a new option called "--readonly" have been added to lvm2.
One of the most interesting part of it is : "disable using lvmetad so VGs are read from disk"

In our case, that feature is really interesting as it means that
everytime ceph-volume calls a {vg|pv|lv}s command, it will read the
metadata from the disks instead of considering the lvmetad status.

This patch change all the {vg|pv|lv}s call to use --readonly.
It solves the bug exposed here and doesn't affect the traditional use-case.
The main benefit of this patch is to avoid a false report of a disk not having metadata.

(cherry picked from commit 7be00cd19297ec80b4a629e2fc1e79c64205402c)
Fixes: https://tracker.ceph.com/issues/23693
Signed-off-by: Erwan Velu <erwan@redhat.com>

ceph-volume: allow parallel creates

Currently the create code decides the vg_name "ceph-$cluster_fsid" as
the primary vg_name and creates a new name if this already exists.

If this code is run N times in parallel, the script will try to
create N times the vg with the name "ceph-$cluster_fsid" and it
will fail to create the N osds successfully.

Creating vgs with names like "ceph-$uuid4" lets our scripts to run
without any problems.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>
(cherry picked from commit 201817f27ecfe195216284d9e4c344038cb654b6)

qa/workunits/rados/test-upgrade-to-mimic.sh: fix tee output

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume util.prepare preserve order on incoming mount flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 89f8f82402f9eda7cf76ebe860e50dd8e1047d76)

ceph-volume tests update (preserved) order on mount flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1a92ecbede09f084ab34ef556de46b435fa3e9d1)

doc/dev/ceph-volume add VDO tag information

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit eeff3463543c12874ca4dfc3c3852af539d318f2)

ceph-volume tests ensure non-duplicate flags in mount calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 61bed1e615a111f1c978d474fda390ba980bfb85)

ceph-volume util.prepare allow and dedupe extra flags for mounting

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 81c20e6b58b77671ffaa0f8030f18747e1a23569)

ceph-volume lvm.prepare detect vdo from device for mounting filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6b87580d4fe8e86de958ac258971cb2f90a70455)

ceph-volume lvm.activate detect vdo from tags for mounting filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0deef5f53c61b4215817008c04589a4f4b5cc3b3)

ceph-volume util.prepare allow extra ad-hoc mount flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d815c4f45429fd73d2506bd874656de438fe74fd)

ceph-volume lvm.prepare capture vdo-ness on LVM tags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c6a79ef21ccc61a7634c05db5a705fc3ef46fbb9)

ceph-volume tests for is_vdo detection

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 75f532597aa5ad5b84e993dc60c60d4a147af191)

ceph-volume api.lvm helpers to detect vdo from a dm path

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5b97d8e54ce3dcf2a56872cdf0cbc0139ca0a723)

Merge pull request #21426 from ivancich/wip-fix-existing-mclock-cost-luminous

osd: remove cost from mclock op queues; cost not handled well in dmcl…

prometheus: Implement rgw_metadata metric

This maps rgw id to its hostname and ceph version running on the node.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit c7cbc24803da9474c9d1d15a52da8d0eed2d49af)

Merge pull request #21397 from dzafman/wip-23654

luminous: Special scrub handling of hinfo_key errors

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: Change shard digests to hex like object info digests

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 22ddc6da5fd03e65a7baf96748a8893cc258d977)

test: Luminous backport specific changes

osd-scrub-repair.sh:
    Remove legacy_snaps and redirect_target to keep test more in sync with master
    We don't have omap_digest bluestore handling
        Additional cases of omap_digest_mismatch_oi changes
        omap_digest still set
        ROBJ10 output still applies
osd-scrub-snaps.sh:
    Add head_exists in snapset
    Add snapset for object with head_mismatch

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #21216 from smithfarm/wip-23545-luminous

luminous: journal: Message too long error when appending journal

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21471 from liewegas/wip-librbd-upgrade-luminous

test/pybind/test_rbd: allow v1 images for testing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test/pybind/test_rbd: allow v1 images for testing

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21452 from yuriw/wip-fix-p2p-luminous

qa/tests - added tag: v12.2.2 to be used by client.1 ...

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests - added tag: v12.2.2 to be used by client.1 under workload_luminous to avoid failures.

added workunits/rados/test-upgrade-to-mimic.sh, and add --gtest_filter=-CReadOpsTest.Exec to api_c_read_operations

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21328 from tchaikov/wip-11332-luminous

luminous: mon: update PaxosService::cached_first_committed in PaxosService::may…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21365 from k0ste/luminous_backports

luminous: mgr: prometheus: fix PG state names

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #21394 from ifed01/wip-ifed-backport2

luminous:  BlueStore.cc: _balance_bluefs_freespace: assert(0 == "allocate failed, wtf");

Reviewed-by: Sage Weil <sage@redhat.com>

test: Use jq in a compatible way and for easier diff analysis

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 458babe7ee2e5fc0c4519a94249cbc1bd7cec9b6)

osd: remove cost from mclock op queues; cost not handled well in dmclock library

The current version of the dmclock library does not handle operation
cost well. Therefore cost should not be passed into the library when
enqueuing operations; instead 0 should be passed in.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #21388 from theanalyst/wip-luminous-usage-log

luminous: usage log fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21389 from theanalyst/wip-luminous-23687

Wip luminous 23687

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #21393 from theanalyst/wip-luminous-recalc

luminous: rgw recalculate stats option added

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21390 from theanalyst/wip-luminous-23690

rgw: fix mutlisite read-write issues

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21233 from smithfarm/wip-jinja-mgr-luminous

luminous: build/ops: deb: move python-jinja2 dependency to mgr

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: James Page <james.page@ubuntu.com>

Merge pull request #21283 from liewegas/wip-pgpool-enc-luminous

luminous: osd/osd_types: fix pg_pool_t encoding for hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21307 from smithfarm/wip-23606-luminous

luminous: tests: ENGINE Error in 'start' listener <bound  in rados

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #21330 from tchaikov/wip-23610-luminous

luminous: osd: resend osd_pgtemp if it's not acked

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21352 from pdvian/wip-23634-luminous

luminous: doc: outline upgrade procedure for mds cluster

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore: do not assert on insufficient allocation during bluefs
rebalance

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 64abc7ba39fab57cc9e19463219e8a70d284eda0)

test/store_test: add test case to for excessive fragmentation

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit eb0443a1034b42d006d2a1d2bd5cb91881f07df7)

os/bluestore: fix lack of extent demotion in StupidAllocator::init_rm_free

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 89aeea84f9ad24b2f9ca5ea3ee1dd9146e7eb664)

unittest_interval_set: test with flat_map<> too

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 70ddb0c18b2969c26dbe855e1573f3f028d9f113)

include/interval_set: cbegin/cend -> begin/end

c{begin,end} don't exist for btree_map<>.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7bbb7958df45b36556a0e469fe55f9236d036972)