cephadm: fix prometheus namespace spelling

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #42994 from ceph/wip-crimson-monmap-more-srcs

monmap, crimson: add support for mon_host_override

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #39849 from pritha-srivastava/wip-rgw-sts-principal-tags

rgw/sts: adding code for aws:RequestTags as part

Merge pull request #42982 from adk3798/async-mock-py3-8

mgr/cephadm: don't fail unit tests if can't import AsyncMock

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42830 from ifed01/wip-ifed-fix-bluefs-replay-crc

os/bluestore: accept undecodable multi-block bluefs transactions on log

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #43016 from athanatos/sjust/wip-fix-build

crimson/os/seastore/lba_manager/btree/lba_btree: fix FTBFS on gcc 9

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore/lba_manager/btree/lba_btree: fix FTBFS on gcc 9

gcc-9 doesn't seem to consider interator nothrow move constructible with
the default move constructor implementation yielding the following build
failure:

m/el8/BUILD/ceph-17.0.0-7373-gfc349212/src/seastar/include/seastar/core/future.hh:584:58: error: static assertion failed: Types must be no-throw move constructible
  584 |     static_assert(std::is_nothrow_move_constructible<T>::value,

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #42942 from zdover23/wip-doc-mgr-progress

doc/mgr: add progress module documentation

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42948 from llamerada-jp/fix-radosgw-admin-document

rgw: fix document

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mgr/cephadm: run coroutines in a new event loop

`asyncio.run` does not exist until python37 or later

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/tasks: Addition of new code for session tags in STS

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

qa/tasks: Addition of two new parameters for sts-tests

Addition of SUB and AZP parameter for some new sts-tests

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

rgw/sts: configuration steps to add tags to a user in
Keycloak.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: documentation related to tag, list tags
and untag REST APIs for a role.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: adding documentation for session tags,
all kinds of tag keys and related examples.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: documentation related to supporting all claims
in the JWT in the condition of the trust policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: code changes to store multi-valued tags
for objects and buckets (to be used as s3:ResourceTags
in Identity and Resource policies).

Test code changes as suggested by Yuval Lifshitz.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: code to add role tags as aws:PrincipalTags and
iam:ResourceTags.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: added code to tag, untag and list role tags.

The role tags can be used as iam:ResourceTags in AssumeRoleWithWebIdentity
or as aws:PrincipalTags in a session using temporary credentials.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: code to add s3:ResourceTag to IAM environment
for objects and buckets, that will be used in evaluation of
Condition element in the IAM policies.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: code for runtime evaluation of Condition
element of an IAM policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: code for aws:TagKeys to be used as Condition
element of a role's trust and permission policy.

This also includes implementation of ForAnyValue and
ForAllValues to be used in conjunction with
StringLike, StringEquals and StringEqualsIgnoreCase
which are used alongwith aws:TagKeys.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: code to use session tags in AssumeRoleWithWebIdentity call,
as aws:PrincipalTags in an IAM policy's Condition element.

The incoming session tags in AssumeRoleWithWebIdentity call
are passed along with the session credentials (temporary creds)
as aws:PrincipalTags, and those can be used as Condition element
of an IAM Policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: adding code for aws:RequestTags as part
of session tags implementation.

Session Tags can be passed in the web token in
AssumeRoleWithWebIdentity call by configuring
them in the IDP. These tags can be used as Conditions
in the trust policy of a role as aws:RequestTag, based
on which a federated user is allowed to assume a role.
The trust policy should have a statement for 'sts:TagSession'
alongwith 'sts:AssumeRoleWithWebIdentity' in case principal
tags are passed in the web token.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: adding code to enable usage of all
token claims as 'Condition' of a role's
trust policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #42995 from rhcs-dashboard/fix-kcli-dashboard-e2e-doc

doc/dashboard: Fix kcli pool and network requirements documentation

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43004 from gregsfortytwo/wip-nodaemon

vstart: make "--crimson" imply "--nodaemon"

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43003 from gregsfortytwo/wip-error-code-value

crimson: osd: do not capture std::error_code by reference

Reviewed-by: Xuehan Xu <xuxuehan@360.cn>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

vstart: make "--crimson" imply "--nodaemon"

If you want to run multiple crimson-osd instances, you need nodaemon. Default
to nodaemon when using crimson.

Add a new "--crimson-foreground" which preserves the old behavior, in case
it's needed.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

crimson: osd: do not capture std::error_code by reference

I can't find any reason to assume it has a longer lifetime than
the future we're creating has.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #42831 from ifed01/wip-ifed-fix-52311

os/bluestore: fix using incomplete bluefs log when dumping it.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #41912 from trociny/wip-crushdiff

tools/crushdiff: new tool to test crushmap change

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: add asyncmock to python venv

for running unittests on python <= 3.8

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: don't fail unit tests if can't import AsyncMock

AsyncMock was added in python 3.8. We don't want unit tests to
be failing if somebody tries running them with pytohn 3.7. Our
Jenkins job should be unaffected and still run the test.

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #42985 from cyx1231st/wip-seastore-lba-metrics

crimson/os/seastore: misc improvements to metrics

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/dashboard: Fix kcli pool and network requirements documentation

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

monmap, crimson: unify handling of mon_host and most_host_override.

This commit unifies the code for `mon_host` and `mon_host_override`
handling but only in the `WITH_SEASTAR` domain. The next step is to
extend to to the classical world but there is a subtle difference
in error handling.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42874 from cbodley/wip-51491

rgw: use existing s->bucket in s3 website retarget()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

crimson/os/seastore: drop Transaction::src_t::INIT category from metrics

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: measure insert/erase operations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: fix logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42973 from liewegas/doc-foundation-osnexus

doc/foundation: add osnexus

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42966 from cyx1231st/wip-seastore-lba-cleanup

crimson/os/seastore/lba_manager: misc cleanup

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #42970 from sebastian-philipp/monitoring-custom-webhook

doc/cephadm: monitoring: Add "Adding Alertmanager webhooks"

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

doc/foundation: add osnexus

Signed-off-by: Sage Weil <sage@newdream.net>

monmap, crimson: add support for the mon_host_override configurable.

NOTE: crimson still misses the `CephContext::get_mon_addrs()` support.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/cephadm: monitoring: Further Reading

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

.github/labeler: Add monitoring

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: Add "Adding Alertmanager webhooks"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: Add "Setting up Grafana"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: move "deploying w/o" up

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: default placements

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42848 from ronen-fr/wip-ronenf-argv2vec-users

common: modify 'main()s' to use new argv_to_vec() signature

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42959 from idryomov/wip-krbd-escape-match-sysattr

krbd: escape udev_enumerate_add_match_sysattr values

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #42913 from yaarith/update-sudoers-smartctl

rpm, debian: move smartmontools and nvme-cli to ceph-base

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42770 from ktdreyer/no-lsb-release

Remove dependency on lsb_release

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

crimson/os/seastore/lba_manager: drop the unecessary lookup in leaf

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: drop the impossible case that the insert key exists after handle_split

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: end iterator should be pointing to the leaf-node size already

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: misc improvements on debugging

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: check depth when load lba node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: mark iterator(depth_t) private

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: cleanup LBABtree::iterate_repeat()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: drop unused code

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42961 from tchaikov/wip-docker-test

test/docker-test.sh: add support for fedora 34 and debian stretch

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

test/docker-test.sh: add support for fedora 34 and debian strech

and drop fedora-31, fedora32, debian-jessie and ubuntu-16.04 since
they are not maintained anymore. see

- https://fedoraproject.org/wiki/End_of_life
- https://wiki.debian.org/LTS
- https://wiki.ubuntu.com/Releases

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

krbd: escape udev_enumerate_add_match_sysattr values

libudev uses fnmatch(3) for matching attributes, meaning that shell
glob pattern matching is employed instead of literal string matching.
Escape glob metacharacters to suppress pattern matching.

Fixes: https://tracker.ceph.com/issues/52425
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42957 from athanatos/sjust/wip-lba-rewrite-working

crimson: add object_info, log support to crimson-store-nbd, fix lba btree bugs

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore/lba_manager/btree: don't assert deltas are empty

We use the contents to update counters when transactions conflict,
deltas may validly be empty there.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: retire_extent_addr: add addr asserts

retire_extent_addr can only be called on absolute or record-relative
addrs.  Record-relative addrs are only valid on extents allocated as
part of the current transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: fix get_val() paddr value from iterator

This was causing a stray RETIRED_PLACEHOLDER to be created resulting in
a segment cleaner segfault in release on an invalid segment and a crash
upon adding a duplicate lba btree pin since the returned addr didn't
match the addr of the INITIAL_PENDING extent on the transaction.

Fixes: https://tracker.ceph.com/issues/52434
Fixes: https://tracker.ceph.com/issues/52435
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd: add log emulation to fs_driver

Signed-off-by: Samuel Just <sjust@redhat.com>

doc/mgr: add progress module documentation

This PR ingests the work added to the documentation
in PR#29335.

The technical information in this PR concerns the
installation and use of the "Progress Module".

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Signed-off-by: kamoltat <ksirivad@redhat.com>

Merge PR #42899 into master

* refs/pull/42899/head:
src/client: fix dump mds twice

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #42751 into master

* refs/pull/42751/head:
cephfs-mirror: shutdown ClusterWatcher on termination

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #42687 into master

* refs/pull/42687/head:
qa: test the "ms_mode" options in kclient workflows

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #42620 into master

* refs/pull/42620/head:
mds: switch mds_lock to fair mutex
common/Timer: make SafeTimer a template
common/fair_mutex: add is_locked_by_me support
common: do not compile condition_variable_debug in none debug mode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38481 into master

* refs/pull/38481/head:
qa/vstart_runner: inherit methods instead of duplicating them
qa/ceph_manager: make it possible to reuse few methods
qa/vstart_runner: don't use "shell=False" in run_ceph_w()
qa/ceph_manager: minor refactor

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42955 from tchaikov/wip-fio-tag

cmake: use ceph repo with tag

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/tools/store_nbd: add object_info emulation to fs_driver

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: move int future refcounting to trace

We probably want to add a dedicated subsys for this, but for now it's just
too much for debug level.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd/fs_driver: collid should be the msb

This way, objects in a collection will sort together.  Probably
a typo.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39377 from ifed01/wip-ifed-fix-huge-omap-rename

os/bluestore: cap omap naming scheme upgrade transaction

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #42918 from ifed01/wip-ifed-fix-52398

test/store_test: fix broken test case

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42926 from zdover23/wip-doc-2021-08-26-nomad-cleanup-cherry-picker

doc: clean rbd-nomad.rst

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cmake: use ceph repo with tag

to address following warning:

fatal: reference is not a tree: 7f02f21f53ccd5e2448086f8e9015489693dd2dc
CMake Error at /home/jenkins-build/build/workspace/ceph-pull-requests/build/fio_ext-prefix/tmp/fio_ext-gitclone.cmake:40 (message):
  Failed to checkout tag: '7f02f21f53ccd5e2448086f8e9015489693dd2dc'

it seems that the shallow option does not work with a sha1 tag option,
let's continue using the ceph repo with a tag.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mailmap, organizationmap: update Mykola Golub affiliation

Signed-off-by: Mykola Golub <mykola.golub@clyso.com>

qa/suites/rados: add crushdiff test

Signed-off-by: Mykola Golub <mykola.golub@clyso.com>

Merge pull request #42944 from dengchl01/dengchl01

mgr/restful/api/crush.py: delete unused reference

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #39933 from linuxbox2/wip-bl-opslog

rgw: permit logging of list-bucket (and any other no-bucket op)

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42934 from tchaikov/wip-fio-cxx

cmake: use upstream repo for fio

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/rbd/rbd-nomad: fix indentation in config snippets

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: clean rbd-nomad.rst

This PR improves the rbd-nomad.rst file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

rgw: fix document

add the description of --show-config option in radosgw-admin

Signed-off-by: Ushitora Anqou <ushitora@anqou.net>

Merge pull request #42901 from athanatos/sjust/wip-lba-rewrite

crimson/os/seastore: replace lba btree implementation with simpler, more correct iterator based implementation

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/restful/api/crush.py: delete unused reference

This commit delete unused reference of the crush page.

Signed-off-by: Charlie Developer <dengchl01@inspur.com>

Merge PR #42530 into master

* refs/pull/42530/head:
qa: ignore expected metadata cluster log error

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

crimson/os/seastore/lba_manager: replace btree implementation

Replace previous implementation with one based around an internal
iterator interface.  Besides simplifying the implementation and
removing duplicate lookups in the allocation pathway, this implementation
should correct a design problem in the prior implementation wherein
LBALeafNode::find_hole couldn't see the first element of the subsequent
node and therefore assumed that there was one at get_meta().end.

This patch removes the btree logic from lba_btree_node_impl.* leaving
the LBAInternalNode and LBALeafNode layout in lba_btree_node.*.
lba_btree.h/cc now have the main btree update/query logic.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #42941 from kamoltat/ksirivad-add-release-notes-quincy

PendingReleaseNotes: add autoscaler profile

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/os/seastore/seastore_types: add lba_root_t::set_depth

Signed-off-by: Samuel Just <sjust@redhat.com>