crimson: switch try_bind() to bind().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: bind() respects now ms_bind_retry_{delay,count}.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: make try_bind() aware about address_not_available.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: decouple listen_ertr from bind_ertr.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: decouple listen_ertr from FixedCPUServerSocket.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: s/do_bind/do_listen/ in SocketMessenger.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: simplify error handling in FixedCPUServerSocket::listen().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43047 from idryomov/wip-rbd-mirror-peer-profile-doc

doc: adjust client.rbd-mirror-peer mon cap and add details about site names

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42840 from TRYTOBE8TME/wip-rgw-kafka-timeout

Wip rgw kafka timeout

Merge pull request #42875 from pleiadesian/feature-lua

examples: add elasticsearch and prometheus examples

Merge pull request #42870 from ktdreyer/cmake-thread-libs-init

cmake: link Threads::Threads instead of CMAKE_THREAD_LIBS_INIT

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #42960 from wjwithagen/wjw-fix-cephfs-lock

tools/cephfs_mirror: fix lock declaratie/locking

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #42962 from tchaikov/wip-cmake-ppc

arch,cmake: compile ppc.c on all powerpc machines

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #43043 from tchaikov/crimson/os/cleanup

crimson/os: use structured binding in loop

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

doc: adjust client.rbd-mirror-peer mon cap

On top of "profile rbd" permissions, "profile rbd-mirror-peer" also
allows getting rbd/mirror and setting rbd/mirror/peer/* config keys.
This is what "rbd mirror pool peer bootstrap create" does.

Fixes: https://tracker.ceph.com/issues/50970
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: add some details about rbd-mirror site names

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43035 from zdover23/wip-doc-2021-09-03-rados-bootstrap-options

doc: rados/ceph-conf.rst bootstrap options

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #43012 into master

* refs/pull/43012/head:
script: ceph-debug-docker.sh deals with the centos-repos -> centos-linux-repos transition.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43040 from sebastian-philipp/qa-deadsymlinks

qa/tox.ini: check for dead symlinks

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #42836 from ivancich/wip-fix-bi-test-error

rgw: fix bucket index list test error

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/os: use structured binding in loop

also avoid using `map[key] = val` for setting an item in map, as, if
he key does not exist in map, `map[key]` would have to create a value
using its default ctor, and then call the `operator=(bufferlist&&)` to
set it.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge PR #43032 into master

* refs/pull/43032/head:
backport-create-issue: set backport priority

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #43039 from sebastian-philipp/rm-kubic

qa/distros: Remove stale kubic distros

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #43021 from sebastian-philipp/rhel_8.3_container_tools_rhel8.yaml

qa/distros/podman: Add rhel_8.3_container_tools_rhel8.yaml

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tox.ini: check for dead symlinks

IMO the amount of symlinks we have to manually maintain
is tedious and error prone. Any ideas on improving thing?

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

qa/distros: Remove stale kubic distros

Cause they're broken

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

qa/distros/podman: Add rhel_8.rhel_8.4_container_tools_3.0.yaml

mainly for the cephfs suite

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43030 from dsavineau/cephadm_fix_prometheus_spelling

cephadm: fix prometheus namespace spelling

Reviewed-by: Sage Weil <sage@newdream.net>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #42873 into master

* refs/pull/42873/head:
qa/tasks/rook: add OSD creation to Rook QA

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #43018 from cyx1231st/wip-seastore-onode-tree-alloc-hint

crimson/onode-staged-tree: improve laddr hints during allocation

Reviewed-by: Samuel Just <sjust@redhat.com>

doc: rados/ceph-conf.rst bootstrap options

This is the editorial syntax and elegance PR for the "Bootstrap Options"
section in the "Configuring Ceph" chapter of the RADOS Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #43031 from rzarzynski/wip-crimson-watch-reconnect

crimson/osd: fix Watch::connect() behaviour on reconnect.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43033 from markhpc/wip-cyanstore-moreops

src/crimson: Add various missing ops to cyanstore

Reviewed-by: Samuel Just <sjust@redhat.com>

backport-create-issue: set backport priority

Set backport issue priority based upon the priority of the original issue.

Fixes: https://tracker.ceph.com/issues/52387
Signed-off-by: Cory Snyder <csnyder@iland.com>

crimson/osd: fix Watch::connect() behaviour on reconnect.

It's perfectly legal for a client to reconnect to particular `Watch`
using different socket / `Connection` than original one. This shall
include proper handling of the watch timer which is currently broken
as, when reconnecting, we don't cancel the timer. This leaded to the
following crash at Sepia:

```
rzarzynski@teuthology:/home/teuthworker/archive/rzarzynski-2021-09-02_07:44:51-rados-master-distro-basic-smithi/6372357$ less ./remote/smithi183/log/ceph-osd.4.log.gz
...
DEBUG 2021-09-02 08:10:45,462 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.5087.0:93 7.1e 7:7c7084bd:::repobj:head {watch reconnect cookie 94478891024832 gen 1} snapc 0={} ondisk+write+know
n_if_redirected e40) v8]): got obc lock
...
DEBUG 2021-09-02 08:10:45,462 [shard 0] osd - do_op_watch
INFO  2021-09-02 08:10:45,462 [shard 0] osd - found existing watch by client.5087
DEBUG 2021-09-02 08:10:45,462 [shard 0] osd - do_op_watch_subop_watch
INFO  2021-09-02 08:10:45,462 [shard 0] osd - found existing watch watch(cookie 94478891024832 30s 172.21.15.150:0/3544196211) by client.5087
...
INFO  2021-09-02 08:10:45,462 [shard 0] osd - op_effect: found existing watcher: 94478891024832,client.5087
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-7406-g9d30203c/rpm/el8/BUILD/ceph-
17.0.0-7406-g9d30203c/src/seastar/include/seastar/core/timer.hh:95: void seastar::timer<Clock>::arm_state(seastar::timer<Clock>::time_point, std::optional<typename Clock::duration>) [with Clock = seastar::l
owres_clock; seastar::timer<Clock>::time_point = std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long int, std::ratio<1, 1000> > >; typename Clock::duration = std::chrono::duration<long
 int, std::ratio<1, 1000> >]: Assertion `!_armed' failed.
Aborting on shard 0.
Backtrace:
 0# 0x000055CC052CF0B6 in ceph-osd
 1# FatalSignal::signaled(int, siginfo_t const&) in ceph-osd
 2# FatalSignal::install_oneshot_signal_handler<6>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) in ceph-osd
 3# 0x00007FA58349FB20 in /lib64/libpthread.so.0
 4# gsignal in /lib64/libc.so.6
 5# abort in /lib64/libc.so.6
 6# 0x00007FA581A98C89 in /lib64/libc.so.6
 7# 0x00007FA581AA6A76 in /lib64/libc.so.6
 8# 0x000055CC0BEEE9DD in ceph-osd
 9# crimson::osd::Watch::connect(seastar::shared_ptr<crimson::net::Connection>, bool) in ceph-osd
10# 0x000055CC00B1D246 in ceph-osd
11# 0x000055CBFFEF01AE in ceph-osd
...
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

src/crimson: Add various missing ops to cyanstore

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #43015 from ktdreyer/build-doc-venv

admin/build-doc: use venv module again

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: fix prometheus namespace spelling

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #42994 from ceph/wip-crimson-monmap-more-srcs

monmap, crimson: add support for mon_host_override

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #39849 from pritha-srivastava/wip-rgw-sts-principal-tags

rgw/sts: adding code for aws:RequestTags as part

Merge pull request #42982 from adk3798/async-mock-py3-8

mgr/cephadm: don't fail unit tests if can't import AsyncMock

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42830 from ifed01/wip-ifed-fix-bluefs-replay-crc

os/bluestore: accept undecodable multi-block bluefs transactions on log

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

crimson/onode-staged-tree: add logs for hint

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement hint based on the splitting node

Instead of the hint from the inserting key.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement hint in NodeImpl::rebuild_extent()

Get the hint from its first key when rebuilding a node extent.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43016 from athanatos/sjust/wip-fix-build

crimson/os/seastore/lba_manager/btree/lba_btree: fix FTBFS on gcc 9

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore/lba_manager/btree/lba_btree: fix FTBFS on gcc 9

gcc-9 doesn't seem to consider interator nothrow move constructible with
the default move constructor implementation yielding the following build
failure:

m/el8/BUILD/ceph-17.0.0-7373-gfc349212/src/seastar/include/seastar/core/future.hh:584:58: error: static assertion failed: Types must be no-throw move constructible
  584 |     static_assert(std::is_nothrow_move_constructible<T>::value,

Signed-off-by: Samuel Just <sjust@redhat.com>

admin/build-doc: use venv module again

Commit 690ca1c5628af629a25a80611c20e56095f13352 accidentally
re-introduced virtualenv. Switch back to the venv module in stdlib.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #42942 from zdover23/wip-doc-mgr-progress

doc/mgr: add progress module documentation

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/rook: add OSD creation to Rook QA

This commit adds OSD creation to the Rook QA tasks. The Rook task will
explicitly wait for the mgr to start and the CLI to work (instead of
implicitly doing so while waiting for 'ceph osd dump' to work).
Then it will do `ceph orch apply osd --all-available-devices` to create
OSDs on the rest of the PVs.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

script: ceph-debug-docker.sh deals with the centos-repos -> centos-linux-repos transition.

See: https://gist.github.com/rzarzynski/19f11e2180faeb82c42d1787b7e0e8dd.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42948 from llamerada-jp/fix-radosgw-admin-document

rgw: fix document

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mgr/cephadm: run coroutines in a new event loop

`asyncio.run` does not exist until python37 or later

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/tasks: Addition of new code for session tags in STS

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

qa/tasks: Addition of two new parameters for sts-tests

Addition of SUB and AZP parameter for some new sts-tests

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

rgw/sts: configuration steps to add tags to a user in
Keycloak.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: documentation related to tag, list tags
and untag REST APIs for a role.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: adding documentation for session tags,
all kinds of tag keys and related examples.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: documentation related to supporting all claims
in the JWT in the condition of the trust policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: code changes to store multi-valued tags
for objects and buckets (to be used as s3:ResourceTags
in Identity and Resource policies).

Test code changes as suggested by Yuval Lifshitz.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: code to add role tags as aws:PrincipalTags and
iam:ResourceTags.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: added code to tag, untag and list role tags.

The role tags can be used as iam:ResourceTags in AssumeRoleWithWebIdentity
or as aws:PrincipalTags in a session using temporary credentials.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: code to add s3:ResourceTag to IAM environment
for objects and buckets, that will be used in evaluation of
Condition element in the IAM policies.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

src/rgw: Connection timeout implementation

Checking the idleness of a connection and deleting the connection if idle for 30 secs.
Added a manual test for checking the behavior.
This PR fixes: https://tracker.ceph.com/issues/50074

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

rgw/sts: code for runtime evaluation of Condition
element of an IAM policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: code for aws:TagKeys to be used as Condition
element of a role's trust and permission policy.

This also includes implementation of ForAnyValue and
ForAllValues to be used in conjunction with
StringLike, StringEquals and StringEqualsIgnoreCase
which are used alongwith aws:TagKeys.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: code to use session tags in AssumeRoleWithWebIdentity call,
as aws:PrincipalTags in an IAM policy's Condition element.

The incoming session tags in AssumeRoleWithWebIdentity call
are passed along with the session credentials (temporary creds)
as aws:PrincipalTags, and those can be used as Condition element
of an IAM Policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: adding code for aws:RequestTags as part
of session tags implementation.

Session Tags can be passed in the web token in
AssumeRoleWithWebIdentity call by configuring
them in the IDP. These tags can be used as Conditions
in the trust policy of a role as aws:RequestTag, based
on which a federated user is allowed to assume a role.
The trust policy should have a statement for 'sts:TagSession'
alongwith 'sts:AssumeRoleWithWebIdentity' in case principal
tags are passed in the web token.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: adding code to enable usage of all
token claims as 'Condition' of a role's
trust policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #42995 from rhcs-dashboard/fix-kcli-dashboard-e2e-doc

doc/dashboard: Fix kcli pool and network requirements documentation

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

examples: add elasticsearch and prometheus examples

Add examples on how to use lua scripting to push request fields from the RGW to Elasticsearch and Prometheus.

Signed-off-by: Zulai Wang <zl31wang@gmail.com>

Merge pull request #43004 from gregsfortytwo/wip-nodaemon

vstart: make "--crimson" imply "--nodaemon"

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43003 from gregsfortytwo/wip-error-code-value

crimson: osd: do not capture std::error_code by reference

Reviewed-by: Xuehan Xu <xuxuehan@360.cn>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

vstart: make "--crimson" imply "--nodaemon"

If you want to run multiple crimson-osd instances, you need nodaemon. Default
to nodaemon when using crimson.

Add a new "--crimson-foreground" which preserves the old behavior, in case
it's needed.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

crimson: osd: do not capture std::error_code by reference

I can't find any reason to assume it has a longer lifetime than
the future we're creating has.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #42831 from ifed01/wip-ifed-fix-52311

os/bluestore: fix using incomplete bluefs log when dumping it.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #41912 from trociny/wip-crushdiff

tools/crushdiff: new tool to test crushmap change

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: add asyncmock to python venv

for running unittests on python <= 3.8

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: don't fail unit tests if can't import AsyncMock

AsyncMock was added in python 3.8. We don't want unit tests to
be failing if somebody tries running them with pytohn 3.7. Our
Jenkins job should be unaffected and still run the test.

Signed-off-by: Adam King <adking@redhat.com>

test/rgw: fix and add to rgw/cls bi_list tests

Make sure marker is cleared. Put end-of-list check inside the
conditional with the rest of the test. Add some additional testing.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: improve bucket index listing cls logging

Adding more level 20 logging for bucket index listing.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: fix bucket index listing count bug

Fix bugs surrounding calculation of number of entries returned and
whether the end of a listing range has been reached.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #42985 from cyx1231st/wip-seastore-lba-metrics

crimson/os/seastore: misc improvements to metrics

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/dashboard: Fix kcli pool and network requirements documentation

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

monmap, crimson: unify handling of mon_host and most_host_override.

This commit unifies the code for `mon_host` and `mon_host_override`
handling but only in the `WITH_SEASTAR` domain. The next step is to
extend to to the classical world but there is a subtle difference
in error handling.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42874 from cbodley/wip-51491

rgw: use existing s->bucket in s3 website retarget()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

crimson/os/seastore: drop Transaction::src_t::INIT category from metrics

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: measure insert/erase operations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: fix logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42973 from liewegas/doc-foundation-osnexus

doc/foundation: add osnexus

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42966 from cyx1231st/wip-seastore-lba-cleanup

crimson/os/seastore/lba_manager: misc cleanup

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #42970 from sebastian-philipp/monitoring-custom-webhook

doc/cephadm: monitoring: Add "Adding Alertmanager webhooks"

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

doc/foundation: add osnexus

Signed-off-by: Sage Weil <sage@newdream.net>

monmap, crimson: add support for the mon_host_override configurable.

NOTE: crimson still misses the `CephContext::get_mon_addrs()` support.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/cephadm: monitoring: Further Reading

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

.github/labeler: Add monitoring

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: Add "Adding Alertmanager webhooks"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: Add "Setting up Grafana"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: move "deploying w/o" up

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: default placements

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>