rgw/sts: adding error log messages to STS code.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #37113 from dillaman/wip-47390

librbd: ensure local site is included in mirror image global status

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37126 from tchaikov/wip-crimson-test

test/crimson: cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #35167 from BryceCao/wip-inspect-zonegroup-when-create-existing-bucket

rgw : add check for zonegroup when create existing bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #36978 from cbodley/wip-47028

rgw: add check for ACL when create existing bucket

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge PR #36125 into master

* refs/pull/36125/head:
doc: cephfs/fs-nfs-exports: more cleanups
doc: cephfs/fs-nfs-exports: command requires enabled orch module
doc: cephfs/fs-nfs-exports.rst: write "RADOS" in ALL-CAPS
doc: cephfs/fs-nfs-exports: use "NFS Ganesha" everywhere
doc: cephfs: clarify that pseudo root path must be absolute
doc: cephfs: fix misleading "ceph fs volume create" command
doc: cephfs: fix misleading "ceph nfs create" command

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37086 from trociny/wip-qa-rbd-nbd-unmap

qa/workunits/rbd: yet another attempt to improve rbd-nbd unmap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37038 from trociny/wip-rbd-nbd-quiesce-error

rbd-nbd: return quiesce hook exit code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37107 from dillaman/wip-47405

qa/workunits/mon: fixed excessively large pool PG count

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37115 from sebastian-philipp/cephadm-refactor-cephamservice-create

mgr/cephadm: CephadmService.create -> CephadmService.prepare_create

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35945 from jschmid1/osdspec_limit_fixes

mgr/cephadm: allow to map existing daemons to osdspecs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36997 from linuxbox2/wip-rgwlc-mtime

rgwlc:  fix (post-parallel) non-current expiration

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #37125 from tchaikov/wip-rgw-const-char

rgw: convert string constant to string_view

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/workunits/mon: fixed excessively large pool PG count

Fixes: https://tracker.ceph.com/issues/47405
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: CephadmService.create -> CephadmService.prepare_create

Refactored this to simplify the control flow. It became obvious that

* `CephadmService.make_daemon_spec`
* `CephadmService.prepare_create`
* `CephadmService.generate_config`

are basically the same thing. They're all called directly
before the deployment of a daemon. All of them should be
unified. This PR makes this refactorization simpler

By renaming `create` to `prepare_create`, we make `create`
no longer being the entrypoint to call
`create_daemon`. Thus all the functions above
return some data structures.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37044 from votdev/update_on_demand

mgr/dashboard: Update datatable only when necessary

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36845 from bk201/wip-44803

mgr/dashboard: allow getting fresh inventory data from the orchestrator

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36365 from bk201/wip-44637

mgr/dashboard: add e2e tests for Orchestrator components

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37127 from tchaikov/wip-crimson-errorator

crimson/common: add specialization for futurize::invoke(Func, monostate)

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/common: add comment to explain the partial specialization

it might be confusing why we don't use explicit specialization for
defining errorator::futurize::stored_to_future.

quote from item 16, § 17.7.3, n4659:

In an explicit specialization declaration for a member of a class
template or a member template that appears in namespace scope, the
member template and some of its enclosing class templates may remain
unspecialized, except that the declaration shall not explicitly
specialize a class member template if its enclosing class templates are
not explicitly specialized as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37098 from tchaikov/wip-47179

common/admin_socket: always validate the parameters

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/common: add specialization for futurize::invoke(Func, monostate)

this is a leftover of 260a702ba983f1bca29d4c8d1e28f3eef46c6699. where we
bumped up the Seastar API level to 5, in which seastar::internal::monostate
is used to represent the stored state of a future instead of a tuple<>.

to address FTBFS with GCC-9.2.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #36987 into master

* refs/pull/36987/head:
mgr: PyModuleRegistry::unregister_client() can run endlessly

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37069 from tchaikov/wip-seastar-lv5

crimson/common: use seastar API level 5

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/crimson: use ASSERT_EQ() for verify critial results

the following EXPECT_EQ() statements actually depends on this test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: use better names than ret

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: use range-based loop when appropriate

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: remove unused variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: convert string constant to string_view

instead of converting string constant to char*, construct string_views
from string constants

to silence GCC warnings like:

src/rgw/services/svc_sys_obj_cache.cc:512:7: warning: ISO C++ forbids converting a string constant to 'char*' [-Wwrite-strings]
  512 |     { "cache list name=filter,type=CephString,req=false",
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup alien fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-nbd: initialize quiesce state variable

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37097 from tchaikov/wip-buffer-offsetof

common/buffer: use homebrew BUF_OFFSETOF to replace offsetof()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37117 from bstillwell/zstd-upgrade-v1_4_5

zstd: upgrade to v1.4.5

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37096 from majianpeng/osd-pg-remove-unused-ocde

osd/PG: remove unused function.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36918 from tchaikov/wip-pybind-type-annotations

pybind/rados: use PEP484 style type annotations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37105 from tchaikov/wip-47404

mgr/zabbix: indent the output of "zabbix config-show"

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37012 from trociny/wip-46803

librbd: track complete async operation requests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

zstd: upgrade to v1.4.5

Since the v1.4.0 release there have been a few improvements to Zstandard
including improved compression ratios, faster compression, and faster
decompression.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

Merge pull request #37114 from yuriw/wip-yuriw-crontab-master

qa/tests: running rados with -p 80 to produce better test results on …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: return success on existing bucket

RGWRadosStore::create_bucket() only returns EEXIST errors when a
conflict is detected and the recreation should fail. in other cases,
return success and use the 'bool *existed' flag to notify the caller of
its prior existence

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35543 from sebastian-philipp/qa-cephadm-iscsi

qa/cephadm: Add iSCSI

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Georgios Kyratsas <gkyratsas@suse.com>
Reviewed-by: Matthew Oliver <moliver@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36936 from mgfritch/cephadm-keyring-rm

mgr/cephadm: remove keyring during daemon post-remove

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32581 from rishabh-d-dave/wip-djf-15070

mon, cephfs: Add auth caps for CephFS fsids

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/tests: running rados with -p 80 to produce better test results on master

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #37021 from tchaikov/wip-osd-pass-by-ref

osd: pass pg_log_t by lvalue reference

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37011 from tchaikov/wip-rgw-cache-list

rgw: set command desc for "cache list"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37002 from tchaikov/wip-alpine-cleanup

alpine: remove .editorconfig

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: ensure local site is included in mirror image global status

Ensure we always return local status for a mirrored image. The Python
bindings expect it to be included for backwards compatibility. Previously
the local status was not included if at least one remote site was reporting
status.

Fixes: https://tracker.ceph.com/issues/47390
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rados: do not encode bytes

the annotations ensures the type of input parameter, so no need to
encode a bytes object anymore

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in: read bytes from stdin

the data read from stdin is used as the input parameter for calls like
`cluster.osd_command(...)` and `cluster.mon_command`. all of them
expect a bytes `inbuf`. in Python2, this sys.stdin.read() returns a str,
and we don't differentiate str from byte back then. but we need enforce
the type now for better readablity and type correctness.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: pass a str to "ping_monitor()"

monitors' names are not numbers, they are referenced by names in
general. their ranks are numbers though, but that's different story.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: define unicode alias

for silencing warnings like:

WARNING: Cannot resolve forward reference in type annotations of "rados.Rados.conf_get": name 'unicode' is not defined

because cython < 3.0 with language_level = "3", translates "str" to "unicode"
to be python2 compatible, but we've migrated to python3. and the specified
"language_level" is "3'. see also
https://github.com/cython/cython/issues/1370

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: drop redundant type in docstring

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: remove duplicated statement

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: s/unicode/str/

we've migrated to python3, so no "unicode" anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: use PEP484 style type annotations

* use PEP484 style typo annotations
* drop homebrew `requires()` decorator
* drop tests for `requires()` decorator
* add test for type annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse.py: pass a str not a list to mon_command()

no need to construct a list as the cmd. and more importantly,

the document of Rados.mon_command() reads:

  Rados.mon_command(self, cmd, inbuf, timeout=0, target=None):
  ...
  :param cmd: JSON formatted string.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: construct cmd as an array of a single entry

otherwise what monitor receives would be something like

mon_command({ " p r e f i x " :   " o s d   m a p " ...

difficult to parse/grep by machine/human.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: add type annotations to Rados.mon_command()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37052 from cbodley/wip-rgw-rest-op-err

rgw: RGWRESTOp no longer tracks separate error code

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mgr/cephadm: merge update_keyring_caps -> create_keyring

keyring is removed during daemon remove, which obsoletes the need to
update the keyring caps after get-or-create

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: disallow removing the mon keyring

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: remove keyring during daemon post-remove

Fixes: https://tracker.ceph.com/issues/47298
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: pass `DaemonDescription` during pre/post remove

add more context than simply passing the `daemon_id`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add `post_remove` daemon action

invoked after a daemon has been removed

Signed-off-by: Michael Fritch <mfritch@suse.com>

rbd-nbd: return quiesce hook exit code

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd-nbd: fix potential race when waiting unqiesce

There was a window between issuing quiesce_complete and checking
the quiesce variable unset by the unqiesce callback, when the lock
was not holding. If during that window the unquiesce following the
next quiesce callbacks were called we would miss unquiesce event.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: track complete async operation requests

to prevent duplicate maintenance operations due to RPC hiccups.

Fixes: https://tracker.ceph.com/issues/46803
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36825 from trociny/wip-47068

librbd: task finisher should distinguish quiesce tasks with different ID

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/cephfs: add tests for "fs authorize" subcommand

Make caps FS-specific affects "fs authorize" subcommand. Let's add few
tests to verify its behaviour.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: backup mount object before running tests

Right now, only client IDs are stashed and restored but with the recent
changes (addition of more attributes to mount objects, specifically),
this is not enough. Saving and restoring these details before and after
tests respectively ensures that mount commands rus smoothly. Not doing
this typically leads to mount command failure for the second test in the
testsuite under execution since the client IDs are saved and restored in
CephFSTestCase.setUp and CephFSTestCase.tearDown respectively but the
rest of the details are not.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: add tests for multi-FS auth tests

Add testsuite for testing authorization on Ceph cluster with multiple
file systems and enable it to be executable with Teuthology framework.

Also add helper methods required to setup the test environment for
multi-FS tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

mgr/zabbix: indent the output of "zabbix config-show"

Fixes: https://tracker.ceph.com/issues/47404
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: allow iscsi to be named by teuthology

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/cephadm: Also fetch iscsi logs

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/cephadm: Add iSCSI

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37094 from adk3798/iscsi-m

cephadm: remove z option from iSCSI dev and configfs mount

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37084 from tchaikov/wip-crimson-fmt

crimson/osd: fix fmt::format() errors and cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37099 from tchaikov/wip-crimson-cause-sv

crimson/osd: pass the the cause by string_view

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37031 from sseshasa/wip-pgpush-recovery-class

osd: Introduce new PGOpQueueable class for recovery push/reply messages.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #36745 from ifed01/wip-ifed-bluefs-remove-obsolete

os/bluestore: get rid of obsolete stuff in bluefs.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #36822 from sebastian-philipp/cephadm-container-init

cephadm: Add --container-init

Merge pull request #36211 from kshtsk/wip-remote-write-file

qa/tasks: use remote write file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37029 from tchaikov/wip-pybind-finer-grainer-test

test/pybind: split aio_test into smaller pieces

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32169 from zjcmszh/cmpext_op

pybind: add aio_cmpext()

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37036 from changchengx/unused_var

common: remove data_dir_option in func global_init

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: pass the the cause by string_view

in future, string_view can be used almost every where string can be
used. in this change, `const char*` is instead passed to the constructor
of system_error, as we can ensure that the string_view instances are
always constructed from a `const char*` ended with `\0`.

we need this change for two reasons:

* better performance
* prefer for the world where string_view rules.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/admin_socket: always validate the parameters

this change addresses the regression introduced by
65267d55cd3714c436b188aaa8b2049ad2b21225.

before this change, we reply the client with a failure only if
both the prefix and the arguments do not match.

after this change, both the prefix and argument are checked.

Fixes: https://tracker.ceph.com/issues/47179
Signed-off-by: Kefu Chai <kchai@redhat.com>

ciommon/admin_socket: extract find_matched_hook() out

this function is complicated and isolated enough to be a separated
method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/buffer: use homebrew BUF_OFFSETOF to replace offsetof()

because buffer::raw is not a standard layout type. so `offsetof()`
does not always work. let's use a homebrew macro to calculate the
address of `bptr_storage`.

this change silences the warning like:

src/common/buffer.cc:2156:15: warning: ‘offsetof’ within non-standard-layout type ‘ceph::buffer::v15_2_0::raw’ is conditionally-supported [-Winvalid-offsetof]
 2156 |      offsetof(buffer::raw, bptr_storage));

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: remove unused function.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cephadm: remove z option from iSCSI dev and configfs mount

selinux complains if you try to deploy iSCSI with the z option
on the /dev and /sys/kernel/config mounts

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #36935 from mgfritch/cephadm-ceph-service

mgr/cephadm: introduce `CephService` class

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37070 from zdover23/wip-doc-3-hacking-into-docs

doc/dev: add HACKING.rst to Dev Guide

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: RGWCreateBucket uses s->bucket_exists instead of EEXIST

RGWStore::create_bucket() only returns EEXIST in case of conflicts that
should be returned to the client as errors. the code that applies
changes to metadata should instead use the s->bucket_exists flag

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37089 from cbodley/wip-cmake-osd-fifo

cmake: add cls_fifo as a dependency of osd

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

qa/cephfs: add methods to read/write on CephFS mounts

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: modify delete_all_filesystems() in filesystem.py

Modify filesystem.Filesystem.delete_all_filesystems() method to make it
more succinct, move it to class MDSCluster instead and update every call
to it accordingly.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

rgw : add check for ACL when create existing bucket

Fixes: https://tracker.ceph.com/issues/47028
Signed-off-by: caolei <halei15848934852@163.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #36980 from adk3798/fix-iscsi

cephadm: fix iSCSI unit.run file

Reviewed-by: Matthew Oliver <moliver@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>