qa: spawn MDS daemons before creating fs

This avoids unnecessary MDS_ALL_DOWN messages because the MDS daemons
have not yet been spawned.

Fixes: https://tracker.ceph.com/issues/47518
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37197 into master

* refs/pull/37197/head:
doc: add "fs authorize" subcommand to ceph man page

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37120 from tchaikov/wip-rados-type-hintings

pybind/rados: add more type hintings

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37143 from dvanders/dvanders_flush

ceph.in: ignore failures to flush stdout

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37100 from rhcs-dashboard/fix-47400-master

ceph: ignore BrokenPipeError when printing help

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37045 from tchaikov/wip-crimson-bt

common/BackTrace: extract demangle() out

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/BackTrace: let abi::__cxa_demangle() do the malloc

also use the returned length for constructing the string_view to be
appended.

we could reuse the buffer across multiple demangle() call for saving the
calls to malloc()/free(). but the upside of this change is that it's
simpler.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/BackTrace: extract demangle() out

so it can be reused by crimson

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add "fs authorize" subcommand to ceph man page

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37187 from kamoltat/patch-1

doc/dev/macos: remove nonexistence flag `--with-toolchain`

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #37199 into master

* refs/pull/37199/head:
qa/cephfs: add a note to test_nfs.py

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37159 into master

* refs/pull/37159/head:
mon/MDSMonitor: return zero when mds is absent for "mds fail"

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/cephfs: add a note to test_nfs.py

A note that tells that test_nfs.py is not yet compatible with
vstart_runner.py.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37167 from neha-ojha/wip-47239

qa/tasks/ceph_manager.py: remove redundant check in raw_cluster_cmd_result

Reviewed-by: Rishabh Dave <ridave@redhat.com>

doc/dev/macos: remove nonexistence flag `--with-toolchain`

` --with-toolchain` is an old option from llvm 6

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #37109 from theanalyst/doc/releases/v15.2.5

doc: releases: notes for v15.2.5 octopus

Reviewed-By: Neha Ojha <nojha@redhat.com>
Reviewed-By: Nathan Cutler <ncutler@suse.com>

Merge pull request #37166 from adamemerson/wip-undetailed-time

common/ceph_time: Don't define public things in time_detail

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/MDSMonitor: return zero when mds is absent for "mds fail"

... instead of EINVAL.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37093 from guits/guits-bz1877672

ceph-volume: fix simple activate when legacy osd

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #35956 from ishanrai05/hover

mgr/dashboard: Display description on mouse hover

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35924 from pritha-srivastava/wip-rgw-sts-error-messages

rgw/sts: adding error log messages to STS code.

Merge pull request #36255 from pritha-srivastava/wip-rgw-sts-user-as-owner

rgw/sts: adding code for federated user as owner in case of STS.

ceph-volume: fix simple activate when legacy osd

`ceph-volume simple activate --all` relies on the presence of json files
in `/etc/ceph/osd` that was created with `ceph-volume simple scan`
command.

In a cluster lifecycle, it is very likely an OSD which was deployed with
ceph-disk at some point gets removed or replaced. It means the corresponding
json file in `/etc/ceph/osd` becomes unrelevant. It makes `ceph-volume
simple activate --all` fails because it tries to mount non existing
partitions.
The idea here is to simply warn the user that the osd described in the
json file doesn't exist anymore and exit properly instead of throwing an
error.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1877672
Closes: https://tracker.ceph.com/issues/47493
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #36898 from sebastian-philipp/doc-fix-orch-apply-mgr

doc/mgr: fix outdated `orch apply` docs.

Merge pull request #35986 from TRYTOBE8TME/wip-rgw-sts-new

STS Testing Files

Merge pull request #37172 from tchaikov/wip-crimson-object-context-lock

crimson/common/tri_mutex: update the class comment

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd/object_context: s/rwlock/lock/

so it's less confusing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common/tri_mutex: update the class comment

to explain the reason why we have tri_mutex, and how it is related to
pipelined read / write. and the mutual exclusion between read, write
and rmw operations.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37131 from tchaikov/wip-47311

crimson/object_context: make obc interruptible

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

ceph: ignore BrokenPipeError when printing help

ignore BrokenPipeError which is thrown when piping the output of ceph
CLI to a tool which might close its stdin before ceph CLI sends the
whole help message.

Follow approach suggested by Kefu: https://github.com/python/cpython/commit/7b0ed43af55c1e2844aa0ccd5e088b2ddd38dbdb
This doesn't manage the clean-up/exit logic, as that's deferred to the
last part of the __main__ code.

Fixes: https://tracker.ceph.com/issues/47400
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

common/ceph_time: Don't define public things in time_detail

Defining things in a detail section and then using them outside turned
out to not be the best idea.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

qa/tasks/ceph_manager.py: remove redundant check in raw_cluster_cmd_result

Fixes 530982129ec131ef78e2f9989abfaeddb0959c65. The check for cephadm is no
longer needed since it was moved to run_cluster_cmd.

Fixes: https://tracker.ceph.com/issues/47239
Signed-off-by: Neha Ojha <nojha@redhat.com>

rgw/sts: adding documentation for a shadow user created for
a user that authenticates with oidc provider.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: adding code for federated user as owner in case of STS.

A new user under the namespace 'oidc' is created for every federated
user in case of AssumeRoleWithWebIdentity.

In case of AssumeRole, the user that needs cross account access becomes
the owner.

In both cases buckets are created in the tenant that the role belongs to.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: adding error log messages to STS code.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

common/ceph_time: Don't indent under namespaces

This is a violation of our guidelines and since I'm making a larger
change anyway.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

crimson/osd: replace "ceph_abort_msg()" with assert()

these are programming errors, and are easy to detect. also assert() does
not return, so compiler won't complain at seeing a branch does not
return value in a function that returns value.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/object_context: make obc interruptible

* add tri_mutex::abort() to pass given exception to all waiters
* add ObjectContext::interrupt() to abort all pending consumers
  of current object context

Fixes: https://tracker.ceph.com/issues/47311
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: use tri_mutex for guarding ObjectContext

before this change, a seastar::shared_mutex, a RWState and a
shared_promise are used for tracking the consumers of ObjectContext.
and all of the consumers are put into writers if the predicate function
evaluates to "false", and is awaken if the predicate function evaluates
to "true" afterwards in a polling loop waiting on the shared_promise,
which is in turn fulfilled once the last consumer of the given category
relinquishes the lock.

this approach has couple issues:

* it is heavy weighted. seastar::shared_mutex already tracks each of
  the waiters' continuation using separate promise<>, and it does try
  to reschedule them once a given consumer releases the last lock.
  so it's like a design of a customized shared_mutex over a
  shared_mutex.
* it is complicated. 3 variables for tracking the different
  consumers of ObjectContext.

in this change,

* `tri_mutex` is introduced as a variant of the original
  `seastar::shared_mutex` to track two different shared users in
  addition to an exclusive user.
* replace `shared_mutex` with `tri_mutex` in `ObjectContext`, to
  simplify the design.
* move recovery_read_marker into `ObjectContext`. assuming all
  pending actions will be added as a waiter for the related
  object context before they acquire the lock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add ObjectContext::wait_recovery_read()

instead of reusing ObjectContext::get_recovery_read() for both
sync call and async call. just add a new method for the async call
for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37113 from dillaman/wip-47390

librbd: ensure local site is included in mirror image global status

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa : STS Testing Files (AssumeRole and GetSessionToken)

Added required files for testing of AssumeRole and GetSessionToken API's and modified s3tests.py to handle the same.

(cherry picked from commit c2c90eaf524087925b8a0993098a87132d4940bd)

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

Merge pull request #37126 from tchaikov/wip-crimson-test

test/crimson: cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #35167 from BryceCao/wip-inspect-zonegroup-when-create-existing-bucket

rgw : add check for zonegroup when create existing bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #36978 from cbodley/wip-47028

rgw: add check for ACL when create existing bucket

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge PR #36125 into master

* refs/pull/36125/head:
doc: cephfs/fs-nfs-exports: more cleanups
doc: cephfs/fs-nfs-exports: command requires enabled orch module
doc: cephfs/fs-nfs-exports.rst: write "RADOS" in ALL-CAPS
doc: cephfs/fs-nfs-exports: use "NFS Ganesha" everywhere
doc: cephfs: clarify that pseudo root path must be absolute
doc: cephfs: fix misleading "ceph fs volume create" command
doc: cephfs: fix misleading "ceph nfs create" command

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37086 from trociny/wip-qa-rbd-nbd-unmap

qa/workunits/rbd: yet another attempt to improve rbd-nbd unmap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37038 from trociny/wip-rbd-nbd-quiesce-error

rbd-nbd: return quiesce hook exit code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37107 from dillaman/wip-47405

qa/workunits/mon: fixed excessively large pool PG count

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37115 from sebastian-philipp/cephadm-refactor-cephamservice-create

mgr/cephadm: CephadmService.create -> CephadmService.prepare_create

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35945 from jschmid1/osdspec_limit_fixes

mgr/cephadm: allow to map existing daemons to osdspecs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph.in: ignore failures to flush stdout

Catch an IOError exception when flushing ceph stdout.

Fixes: https://tracker.ceph.com/issues/47442
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #36997 from linuxbox2/wip-rgwlc-mtime

rgwlc:  fix (post-parallel) non-current expiration

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #37125 from tchaikov/wip-rgw-const-char

rgw: convert string constant to string_view

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/workunits/mon: fixed excessively large pool PG count

Fixes: https://tracker.ceph.com/issues/47405
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: CephadmService.create -> CephadmService.prepare_create

Refactored this to simplify the control flow. It became obvious that

* `CephadmService.make_daemon_spec`
* `CephadmService.prepare_create`
* `CephadmService.generate_config`

are basically the same thing. They're all called directly
before the deployment of a daemon. All of them should be
unified. This PR makes this refactorization simpler

By renaming `create` to `prepare_create`, we make `create`
no longer being the entrypoint to call
`create_daemon`. Thus all the functions above
return some data structures.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37044 from votdev/update_on_demand

mgr/dashboard: Update datatable only when necessary

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36845 from bk201/wip-44803

mgr/dashboard: allow getting fresh inventory data from the orchestrator

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36365 from bk201/wip-44637

mgr/dashboard: add e2e tests for Orchestrator components

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37127 from tchaikov/wip-crimson-errorator

crimson/common: add specialization for futurize::invoke(Func, monostate)

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/common: add comment to explain the partial specialization

it might be confusing why we don't use explicit specialization for
defining errorator::futurize::stored_to_future.

quote from item 16, § 17.7.3, n4659:

In an explicit specialization declaration for a member of a class
template or a member template that appears in namespace scope, the
member template and some of its enclosing class templates may remain
unspecialized, except that the declaration shall not explicitly
specialize a class member template if its enclosing class templates are
not explicitly specialized as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37098 from tchaikov/wip-47179

common/admin_socket: always validate the parameters

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/common: add specialization for futurize::invoke(Func, monostate)

this is a leftover of 260a702ba983f1bca29d4c8d1e28f3eef46c6699. where we
bumped up the Seastar API level to 5, in which seastar::internal::monostate
is used to represent the stored state of a future instead of a tuple<>.

to address FTBFS with GCC-9.2.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #36987 into master

* refs/pull/36987/head:
mgr: PyModuleRegistry::unregister_client() can run endlessly

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37069 from tchaikov/wip-seastar-lv5

crimson/common: use seastar API level 5

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/crimson: use ASSERT_EQ() for verify critial results

the following EXPECT_EQ() statements actually depends on this test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: use better names than ret

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: use range-based loop when appropriate

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: remove unused variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: convert string constant to string_view

instead of converting string constant to char*, construct string_views
from string constants

to silence GCC warnings like:

src/rgw/services/svc_sys_obj_cache.cc:512:7: warning: ISO C++ forbids converting a string constant to 'char*' [-Wwrite-strings]
  512 |     { "cache list name=filter,type=CephString,req=false",
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup alien fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-nbd: initialize quiesce state variable

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37097 from tchaikov/wip-buffer-offsetof

common/buffer: use homebrew BUF_OFFSETOF to replace offsetof()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37117 from bstillwell/zstd-upgrade-v1_4_5

zstd: upgrade to v1.4.5

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37096 from majianpeng/osd-pg-remove-unused-ocde

osd/PG: remove unused function.

Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/rados: add more type hintings

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36918 from tchaikov/wip-pybind-type-annotations

pybind/rados: use PEP484 style type annotations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37105 from tchaikov/wip-47404

mgr/zabbix: indent the output of "zabbix config-show"

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37012 from trociny/wip-46803

librbd: track complete async operation requests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

zstd: upgrade to v1.4.5

Since the v1.4.0 release there have been a few improvements to Zstandard
including improved compression ratios, faster compression, and faster
decompression.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

Merge pull request #37114 from yuriw/wip-yuriw-crontab-master

qa/tests: running rados with -p 80 to produce better test results on …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: releases: notes for v15.2.5 octopus

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: return success on existing bucket

RGWRadosStore::create_bucket() only returns EEXIST errors when a
conflict is detected and the recreation should fail. in other cases,
return success and use the 'bool *existed' flag to notify the caller of
its prior existence

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35543 from sebastian-philipp/qa-cephadm-iscsi

qa/cephadm: Add iSCSI

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Georgios Kyratsas <gkyratsas@suse.com>
Reviewed-by: Matthew Oliver <moliver@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36936 from mgfritch/cephadm-keyring-rm

mgr/cephadm: remove keyring during daemon post-remove

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32581 from rishabh-d-dave/wip-djf-15070

mon, cephfs: Add auth caps for CephFS fsids

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

mgr/dashboard: display description on mouse hover

Fixes: https://tracker.ceph.com/issues/36607
Signed-off-by: Ishan Rai <ishanrai05@gmail.com>

qa/tests: running rados with -p 80 to produce better test results on master

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #37021 from tchaikov/wip-osd-pass-by-ref

osd: pass pg_log_t by lvalue reference

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37011 from tchaikov/wip-rgw-cache-list

rgw: set command desc for "cache list"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37002 from tchaikov/wip-alpine-cleanup

alpine: remove .editorconfig

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: ensure local site is included in mirror image global status

Ensure we always return local status for a mirrored image. The Python
bindings expect it to be included for backwards compatibility. Previously
the local status was not included if at least one remote site was reporting
status.

Fixes: https://tracker.ceph.com/issues/47390
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rados: do not encode bytes

the annotations ensures the type of input parameter, so no need to
encode a bytes object anymore

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in: read bytes from stdin

the data read from stdin is used as the input parameter for calls like
`cluster.osd_command(...)` and `cluster.mon_command`. all of them
expect a bytes `inbuf`. in Python2, this sys.stdin.read() returns a str,
and we don't differentiate str from byte back then. but we need enforce
the type now for better readablity and type correctness.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: pass a str to "ping_monitor()"

monitors' names are not numbers, they are referenced by names in
general. their ranks are numbers though, but that's different story.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: define unicode alias

for silencing warnings like:

WARNING: Cannot resolve forward reference in type annotations of "rados.Rados.conf_get": name 'unicode' is not defined

because cython < 3.0 with language_level = "3", translates "str" to "unicode"
to be python2 compatible, but we've migrated to python3. and the specified
"language_level" is "3'. see also
https://github.com/cython/cython/issues/1370

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: drop redundant type in docstring

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: remove duplicated statement

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: s/unicode/str/

we've migrated to python3, so no "unicode" anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: use PEP484 style type annotations

* use PEP484 style typo annotations
* drop homebrew `requires()` decorator
* drop tests for `requires()` decorator
* add test for type annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>