librbd: reorder exclusive-lock pre-release state steps

The exclusive-lock dispatch layer should be locked and flushed to
ensure no IO is waiting for a refresh. Once that is complete, interlock
with the refresh state machine and re-flush one last time w/ the
refresh dispatch layer skipped.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid blocking writes when initializing exclusive-lock

The exclusive-lock dispatch layer will already block IOs as required
so this second layer of blocking just increases the complexity and
the potential for deadlocks when attempting to flush.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: skip flush from exclusive-lock dispatch layer on init/shutdown

If the exclusive-lock layer is being initialized/shut down at image
open/close, there is no IO flowing so there is no need to flush.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: assign a unique flush source to each internal component

This will allow improved tracking and bypassing of a flush request
that might cause IO deadlocks in dispatch layers.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove unncessary templating from io::ImageDispatchSpec

This was a remnant of the original implimentation for the image
dispatch spec. Now it more closely aligns with the object dispatch
spec.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: queued IOs should retry acquiring exclusive lock

If the IO that attempts to acquire the exclusive lock fails,
any queued IO will not be retried leading to a deadlock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36397 from dzafman/wip-39012

distinguish unfound + impossible to find, vs start some down OSDs to get

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37132 from lixiaoy1/dirty_cache_feature

librbd: add DIRTY_CACHE in IMPLICIT_ENABLE

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36586 from MahatiC/wip-ssd-integration

librbd/cache: SSD cache integration framework

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37141 from sebastian-philipp/cephadm-fix-rm-util.load_from_store

mgr/cephadm: fix RemoveUtil.load_from_store()

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge PR #37197 into master

* refs/pull/37197/head:
doc: add "fs authorize" subcommand to ceph man page

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37120 from tchaikov/wip-rados-type-hintings

pybind/rados: add more type hintings

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37143 from dvanders/dvanders_flush

ceph.in: ignore failures to flush stdout

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37100 from rhcs-dashboard/fix-47400-master

ceph: ignore BrokenPipeError when printing help

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37045 from tchaikov/wip-crimson-bt

common/BackTrace: extract demangle() out

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/BackTrace: let abi::__cxa_demangle() do the malloc

also use the returned length for constructing the string_view to be
appended.

we could reuse the buffer across multiple demangle() call for saving the
calls to malloc()/free(). but the upside of this change is that it's
simpler.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/BackTrace: extract demangle() out

so it can be reused by crimson

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add "fs authorize" subcommand to ceph man page

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37187 from kamoltat/patch-1

doc/dev/macos: remove nonexistence flag `--with-toolchain`

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #37199 into master

* refs/pull/37199/head:
qa/cephfs: add a note to test_nfs.py

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: Add doc and PendingRealeaseNotes

Add help about new list_unfound fields in troubleshooting-pg.rst
Add info in PendingReleaseNotes

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #37159 into master

* refs/pull/37159/head:
mon/MDSMonitor: return zero when mds is absent for "mds fail"

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/cephfs: add a note to test_nfs.py

A note that tells that test_nfs.py is not yet compatible with
vstart_runner.py.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37167 from neha-ojha/wip-47239

qa/tasks/ceph_manager.py: remove redundant check in raw_cluster_cmd_result

Reviewed-by: Rishabh Dave <ridave@redhat.com>

doc/dev/macos: remove nonexistence flag `--with-toolchain`

` --with-toolchain` is an old option from llvm 6

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #37109 from theanalyst/doc/releases/v15.2.5

doc: releases: notes for v15.2.5 octopus

Reviewed-By: Neha Ojha <nojha@redhat.com>
Reviewed-By: Nathan Cutler <ncutler@suse.com>

Merge pull request #37166 from adamemerson/wip-undetailed-time

common/ceph_time: Don't define public things in time_detail

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/MDSMonitor: return zero when mds is absent for "mds fail"

... instead of EINVAL.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37093 from guits/guits-bz1877672

ceph-volume: fix simple activate when legacy osd

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #35956 from ishanrai05/hover

mgr/dashboard: Display description on mouse hover

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35924 from pritha-srivastava/wip-rgw-sts-error-messages

rgw/sts: adding error log messages to STS code.

Merge pull request #36255 from pritha-srivastava/wip-rgw-sts-user-as-owner

rgw/sts: adding code for federated user as owner in case of STS.

ceph-volume: fix simple activate when legacy osd

`ceph-volume simple activate --all` relies on the presence of json files
in `/etc/ceph/osd` that was created with `ceph-volume simple scan`
command.

In a cluster lifecycle, it is very likely an OSD which was deployed with
ceph-disk at some point gets removed or replaced. It means the corresponding
json file in `/etc/ceph/osd` becomes unrelevant. It makes `ceph-volume
simple activate --all` fails because it tries to mount non existing
partitions.
The idea here is to simply warn the user that the osd described in the
json file doesn't exist anymore and exit properly instead of throwing an
error.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1877672
Closes: https://tracker.ceph.com/issues/47493
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #36898 from sebastian-philipp/doc-fix-orch-apply-mgr

doc/mgr: fix outdated `orch apply` docs.

Merge pull request #35986 from TRYTOBE8TME/wip-rgw-sts-new

STS Testing Files

Merge pull request #37172 from tchaikov/wip-crimson-object-context-lock

crimson/common/tri_mutex: update the class comment

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd/object_context: s/rwlock/lock/

so it's less confusing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common/tri_mutex: update the class comment

to explain the reason why we have tri_mutex, and how it is related to
pipelined read / write. and the mutual exclusion between read, write
and rmw operations.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37131 from tchaikov/wip-47311

crimson/object_context: make obc interruptible

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

ceph: ignore BrokenPipeError when printing help

ignore BrokenPipeError which is thrown when piping the output of ceph
CLI to a tool which might close its stdin before ceph CLI sends the
whole help message.

Follow approach suggested by Kefu: https://github.com/python/cpython/commit/7b0ed43af55c1e2844aa0ccd5e088b2ddd38dbdb
This doesn't manage the clean-up/exit logic, as that's deferred to the
last part of the __main__ code.

Fixes: https://tracker.ceph.com/issues/47400
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

common/ceph_time: Don't define public things in time_detail

Defining things in a detail section and then using them outside turned
out to not be the best idea.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

qa/tasks/ceph_manager.py: remove redundant check in raw_cluster_cmd_result

Fixes 530982129ec131ef78e2f9989abfaeddb0959c65. The check for cephadm is no
longer needed since it was moved to run_cluster_cmd.

Fixes: https://tracker.ceph.com/issues/47239
Signed-off-by: Neha Ojha <nojha@redhat.com>

rgw/sts: adding documentation for a shadow user created for
a user that authenticates with oidc provider.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: adding code for federated user as owner in case of STS.

A new user under the namespace 'oidc' is created for every federated
user in case of AssumeRoleWithWebIdentity.

In case of AssumeRole, the user that needs cross account access becomes
the owner.

In both cases buckets are created in the tenant that the role belongs to.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: adding error log messages to STS code.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

common/ceph_time: Don't indent under namespaces

This is a violation of our guidelines and since I'm making a larger
change anyway.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

librbd/cache: Rename namespaces and move files

Signed-off-by: Lisa Li <xiaoyan.li@intel.com>
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

librbd/cache: Establish the framework to integrate RWL and SSD

- Create WriteLogCache class
- Rename ReplicatedWriteLog files to AbstractWriteLog and
  modify the I/O method names
- fix the test
- Modify CMakeLists.txt to add newly created classes

Signed-off-by: Lisa Li <xiaoyan.li@intel.com>
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

crimson/osd: replace "ceph_abort_msg()" with assert()

these are programming errors, and are easy to detect. also assert() does
not return, so compiler won't complain at seeing a branch does not
return value in a function that returns value.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/object_context: make obc interruptible

* add tri_mutex::abort() to pass given exception to all waiters
* add ObjectContext::interrupt() to abort all pending consumers
  of current object context

Fixes: https://tracker.ceph.com/issues/47311
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: use tri_mutex for guarding ObjectContext

before this change, a seastar::shared_mutex, a RWState and a
shared_promise are used for tracking the consumers of ObjectContext.
and all of the consumers are put into writers if the predicate function
evaluates to "false", and is awaken if the predicate function evaluates
to "true" afterwards in a polling loop waiting on the shared_promise,
which is in turn fulfilled once the last consumer of the given category
relinquishes the lock.

this approach has couple issues:

* it is heavy weighted. seastar::shared_mutex already tracks each of
  the waiters' continuation using separate promise<>, and it does try
  to reschedule them once a given consumer releases the last lock.
  so it's like a design of a customized shared_mutex over a
  shared_mutex.
* it is complicated. 3 variables for tracking the different
  consumers of ObjectContext.

in this change,

* `tri_mutex` is introduced as a variant of the original
  `seastar::shared_mutex` to track two different shared users in
  addition to an exclusive user.
* replace `shared_mutex` with `tri_mutex` in `ObjectContext`, to
  simplify the design.
* move recovery_read_marker into `ObjectContext`. assuming all
  pending actions will be added as a waiter for the related
  object context before they acquire the lock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add ObjectContext::wait_recovery_read()

instead of reusing ObjectContext::get_recovery_read() for both
sync call and async call. just add a new method for the async call
for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37113 from dillaman/wip-47390

librbd: ensure local site is included in mirror image global status

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa : STS Testing Files (AssumeRole and GetSessionToken)

Added required files for testing of AssumeRole and GetSessionToken API's and modified s3tests.py to handle the same.

(cherry picked from commit c2c90eaf524087925b8a0993098a87132d4940bd)

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

Merge pull request #37126 from tchaikov/wip-crimson-test

test/crimson: cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #35167 from BryceCao/wip-inspect-zonegroup-when-create-existing-bucket

rgw : add check for zonegroup when create existing bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #36978 from cbodley/wip-47028

rgw: add check for ACL when create existing bucket

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge PR #36125 into master

* refs/pull/36125/head:
doc: cephfs/fs-nfs-exports: more cleanups
doc: cephfs/fs-nfs-exports: command requires enabled orch module
doc: cephfs/fs-nfs-exports.rst: write "RADOS" in ALL-CAPS
doc: cephfs/fs-nfs-exports: use "NFS Ganesha" everywhere
doc: cephfs: clarify that pseudo root path must be absolute
doc: cephfs: fix misleading "ceph fs volume create" command
doc: cephfs: fix misleading "ceph nfs create" command

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

test: Modify test to check new feature might_have_unfound added to list_unfound

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test list_unfound with asserts to check for new fields

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix typos in descriptions

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Feature to add might_have_unfound info to list_unfound requests

fixes: https://tracker.ceph.com/issues/39012

Signed-off-by: David Zafman <dzafman@redhat.com>

Fix to raise BOOST_MPL_LIMIT_LIST_SIZE from 20 to 30

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #37086 from trociny/wip-qa-rbd-nbd-unmap

qa/workunits/rbd: yet another attempt to improve rbd-nbd unmap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37038 from trociny/wip-rbd-nbd-quiesce-error

rbd-nbd: return quiesce hook exit code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: Move scrubber info out of recovery_state and add fields to scrubber::dump()

Add scrubber::dump() function which is a more obvious place when adding fields
Indicate that the scrubber info in recovery_state is deprecated

Signed-off-by: David Zafman <dzafman@redhat.com>

rbd: add DIRTY_CACHE in IMPLICIT_ENABLE

Add DIRTY_CACHE IN RBD_FEATURES_IMPLICIT_ENABLE, so that when creating
images, it won't create an image with the DIRTY_CACHE feature.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37107 from dillaman/wip-47405

qa/workunits/mon: fixed excessively large pool PG count

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37115 from sebastian-philipp/cephadm-refactor-cephamservice-create

mgr/cephadm: CephadmService.create -> CephadmService.prepare_create

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35945 from jschmid1/osdspec_limit_fixes

mgr/cephadm: allow to map existing daemons to osdspecs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph.in: ignore failures to flush stdout

Catch an IOError exception when flushing ceph stdout.

Fixes: https://tracker.ceph.com/issues/47442
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #36997 from linuxbox2/wip-rgwlc-mtime

rgwlc:  fix (post-parallel) non-current expiration

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #37125 from tchaikov/wip-rgw-const-char

rgw: convert string constant to string_view

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/cephadm: fix RemoveUtil.load_from_store()

* Run `json.loads` only once
* `nodename` -> `hostname`
* Remove network calls from `repr(OSD(...)`

Fixes: https://tracker.ceph.com/issues/47438
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/workunits/mon: fixed excessively large pool PG count

Fixes: https://tracker.ceph.com/issues/47405
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: CephadmService.create -> CephadmService.prepare_create

Refactored this to simplify the control flow. It became obvious that

* `CephadmService.make_daemon_spec`
* `CephadmService.prepare_create`
* `CephadmService.generate_config`

are basically the same thing. They're all called directly
before the deployment of a daemon. All of them should be
unified. This PR makes this refactorization simpler

By renaming `create` to `prepare_create`, we make `create`
no longer being the entrypoint to call
`create_daemon`. Thus all the functions above
return some data structures.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37044 from votdev/update_on_demand

mgr/dashboard: Update datatable only when necessary

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36845 from bk201/wip-44803

mgr/dashboard: allow getting fresh inventory data from the orchestrator

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36365 from bk201/wip-44637

mgr/dashboard: add e2e tests for Orchestrator components

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37127 from tchaikov/wip-crimson-errorator

crimson/common: add specialization for futurize::invoke(Func, monostate)

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/common: add comment to explain the partial specialization

it might be confusing why we don't use explicit specialization for
defining errorator::futurize::stored_to_future.

quote from item 16, § 17.7.3, n4659:

In an explicit specialization declaration for a member of a class
template or a member template that appears in namespace scope, the
member template and some of its enclosing class templates may remain
unspecialized, except that the declaration shall not explicitly
specialize a class member template if its enclosing class templates are
not explicitly specialized as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37098 from tchaikov/wip-47179

common/admin_socket: always validate the parameters

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/common: add specialization for futurize::invoke(Func, monostate)

this is a leftover of 260a702ba983f1bca29d4c8d1e28f3eef46c6699. where we
bumped up the Seastar API level to 5, in which seastar::internal::monostate
is used to represent the stored state of a future instead of a tuple<>.

to address FTBFS with GCC-9.2.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #36987 into master

* refs/pull/36987/head:
mgr: PyModuleRegistry::unregister_client() can run endlessly

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37069 from tchaikov/wip-seastar-lv5

crimson/common: use seastar API level 5

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/crimson: use ASSERT_EQ() for verify critial results

the following EXPECT_EQ() statements actually depends on this test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: use better names than ret

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: use range-based loop when appropriate

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: remove unused variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: convert string constant to string_view

instead of converting string constant to char*, construct string_views
from string constants

to silence GCC warnings like:

src/rgw/services/svc_sys_obj_cache.cc:512:7: warning: ISO C++ forbids converting a string constant to 'char*' [-Wwrite-strings]
  512 |     { "cache list name=filter,type=CephString,req=false",
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup alien fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-nbd: initialize quiesce state variable

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37097 from tchaikov/wip-buffer-offsetof

common/buffer: use homebrew BUF_OFFSETOF to replace offsetof()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37117 from bstillwell/zstd-upgrade-v1_4_5

zstd: upgrade to v1.4.5

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37096 from majianpeng/osd-pg-remove-unused-ocde

osd/PG: remove unused function.

Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/rados: add more type hintings

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36918 from tchaikov/wip-pybind-type-annotations

pybind/rados: use PEP484 style type annotations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37105 from tchaikov/wip-47404

mgr/zabbix: indent the output of "zabbix config-show"

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37012 from trociny/wip-46803

librbd: track complete async operation requests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

zstd: upgrade to v1.4.5

Since the v1.4.0 release there have been a few improvements to Zstandard
including improved compression ratios, faster compression, and faster
decompression.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>