Merge PR #38631 into master

* refs/pull/38631/head:
MDSLogContext: fix typo in MDSLogContextBase and add more comments

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38672 into master

* refs/pull/38672/head:
mds: increase the l_mds_openino_backtrace_fetch when fetching the backtrace

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38692 into master

* refs/pull/38692/head:
pybind/cephfs: pass command as size-1 array of strings

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38668 into master

* refs/pull/38668/head:
osdc: Fix return type of map waiting functions
monc: Use post rather than defer
common/async: Hold lock in constructor of blocked_result
osdc: Don't keep wrapping completion in wait_for_latest_osdmap
neorados: Replace unsafe uses of dispatch with post

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38707 from ceph/wip-yuriw-octopus-x-4-master

qa/tests: initial check in for octopus-x/parallel tests (try 2)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/cephadm: include first mgr host when applying mgrs

This prevents the first mgr from being shut down due to lack of
appropriate placements.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/tests: add light octopus-x/copy (cephadm based)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #37721 into master

* refs/pull/37721/head:
test: add tests for new snapshot APIs
pybind/cephfs: python bindings for new snapshot APIs
client: Snapshot cephfs APIs
mds: include snapshot metadata in trace reply to client
mds: include snap metadata on-disk on snapshot creation
mds: include metadata in SnapInfo structure

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38468 from IlsooByun/fix_lc_mp_iter

rgw: lc: fix multipart iteration problem

Merge PR #38611 into master

* refs/pull/38611/head:
qa/cephfs/nfs: Add test for readonly export

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/cephfs: pass command as size-1 array of strings

Fixes: https://tracker.ceph.com/issues/48701
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38187 from zdover23/wip-doc-dev-unit-tests-5-of-7

doc/dev: update unit-tests tox section

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38414 from pcuzner/fix-drivegroup-example

doc: Update drivegroup spec examples

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

doc/dev: update unit-tests tox section

This commit
- alters the structure of the
  unit-test section of the Developer Guide
  that discusses the tox tool, which is
  used to run certain unit tests
- corrects some grammar
- updates the list of locations of tox.ini
  files
- This also incorporates N. Cutler's suggestions.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38621 from anthonyeleven/anthonyeleven/bluestore-doc-cleanup

doc/rados/configuration: clean up BlueStore config and enhance DB sizing

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

qa/cephfs/nfs: Add test for readonly export

Fixes: https://tracker.ceph.com/issues/48622
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #38499 from aclamk/wip-bs-onode-prune-deleted

os/bluestore: Prune deleted onodes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38650 from dmick/wip-dmick-docker-mirror

qa/distros: configure podman to pull from docker-mirror

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #38666 into master

* refs/pull/38666/head:
doc: Fix mds capabilities for manual deployment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37156 from CloudFerro/wip-allow-dynamic-levels

os/bluestore: allow dynamic levels

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #38669 into master

* refs/pull/38669/head:
mds: fix typo in debug logs

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #38674 from mkogan1/rgw-fix-t48688

rgw: fix 'this->ondisk_version' will always evaluate to 'true'

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38653 from dillaman/wip-48647

librbd: clear implicitly enabled feature bits when creating images

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38670 from Rethan/doc-fix-max_part-description

doc/rbd-nbd: fix description for command rbd-nbd's parameter --max_part

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix 'this->ondisk_version' will always evaluate to 'true'

fix:
```
src/rgw/CMakeFiles/radosgw.dir/rgw_rest_metadata.cc.o
../src/rgw/rgw_rest_metadata.cc:267:13: warning: address of
'this->ondisk_version' will always evaluate to 'true'
[-Wpointer-bool-conversion]
                                       &ondisk_version);
```
Fixes: https://tracker.ceph.com/issues/48688
Signed-off-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #38259 from rhcs-dashboard/fix-jwt-token

mgr/dashboard: Use secure cookies to store JWT Token

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mds: increase the l_mds_openino_backtrace_fetch when fetching the backtrace

Signed-off-by: Xiubo Li <xiubli@redhat.com>

doc/rbd-nbd: fix description for command rbd-nbd's parameter --max_part

The parameter --max_part should actually override module param max_part,
not as nbds_max as it said.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

mds: fix typo in debug logs

Signed-off-by: Xiubo Li <xiubli@redhat.com>

osdc: Fix return type of map waiting functions

Fixes: https://tracker.ceph.com/issues/47294
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #38573 from batrick/i48600

osd: remove invalid put on message

Reviewed-by: Kefu Chai <kchai@redhat.com>

monc: Use post rather than defer

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/async: Hold lock in constructor of blocked_result

More correct, though not the cause of error.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osdc: Don't keep wrapping completion in wait_for_latest_osdmap

Also remove unsafe use of dispatch.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

neorados: Replace unsafe uses of dispatch with post

Dispatch may block the calling thread, so don't use it directly in a
called function. (In handlers is okay, though.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

doc: Fix mds capabilities for manual deployment

Now matches auth-config-ref.rst, and actually works.

Signed-off-by: Ivan Jager <aij+git@mrph.org>

Merge pull request #38646 from xxhdx1985126/wip-crimson-bug-fix

crimson/osd: don't get recovery read lock in PGRecvery::on_local_recover()

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make sure the read lock is acquired when trying to get recovery read

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: don't get recovery read lock in PGRecvery::on_local_recover()

Now we've changed drop_recovery_read to only care about the recovery_read flag, so
we shouldn't get read lock when acquiring recovery read, otherwise there would be
a chance in which the read lock can't get released

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #38291 from aserdean/windows_common_proccess_name

common: Add get_process_name implementation for Windows

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38630 from tchaikov/wip-crimson-cleanups

crimson/osd: cleanups to drop some do_with() calls

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38522 from tchaikov/wip-mon-health-cleanup

mon/HealthMonitor: split large method into smaller ones and cleanups

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #38579 into master

* refs/pull/38579/head:
mds: ensure dirfrags are fetched once
mds: refactor variable names

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38540 into master

* refs/pull/38540/head:
client: support client oc's options can update at runtime

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38495 into master

* refs/pull/38495/head:
doc: add documentation for new multifs feature
qa: update tests concerning setting of multifs
mon,mds: mark multifs stable

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

doc/rados/configuration: clean up BlueStore config and enhance DB sizing guidance

librbd: drop explicit masking of implicit feature bits

Now that the create image state machine is handling the masking
of implicit features, all callers to the state machine can skip
the need to perform the masking themselves.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/image: mask out all implicit features when creating an image

This will ensure that all paths to the create image state machine
properly handle this condition. Previously, it was up to the callers
of the state machine to clear the implicit feature bits.

Fixes: https://tracker.ceph.com/issues/48647
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/image: use internal feature flag mask when creating images

When validating the provided features for an image, utilize the
RBD_FEATURES_INTERNAL flag instead of just RBD_FEATURE_OPERATION.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/api: do not explicitly add the MIGRATION feature bit

This feature bit is automatically added to the source and destination
images via the cls method on the OSD.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38613 from dillaman/wip-48522

librbd/api: avoid retrieving more than max mirror image info records

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Use secure cookies to store JWT Token

This PR intends to store the jwt token in secure cookies instead of local storage

Fixes: https://tracker.ceph.com/issues/44591
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 36703c63381e6723fff57266235f8230e6af1d92)

Merge pull request #38643 from tchaikov/wip-doc-dev

doc/dev/generatedocs.rst: fix highlight syntax

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/distros: configure podman to pull from docker-mirror

This also requires a change to cephadm.py to search for
registry entries for docker.io by 'prefix' or 'location',
and to only add the mirror if a mirror isn't already present

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #38638 from idryomov/wip-krbd-stable-writes-attr

qa: krbd_stable_pages_required.sh: move to stable_writes attribute

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38635 from ronen-fr/wip-ronenf-crimsclang-tree

crimson/seastore: fixing Clang errors and warnings

Merge pull request #38602 from dillaman/wip-48486

librbd: explicitly disable readahead for writearound cache

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38645 from cyx1231st/wip-seastar-onode-tree-fix-clang-link

crimson/onode-staged-tree: fix explicit template instantiation for clang

MDSLogContext: fix typo in MDSLogContextBase and add more comments

Fix typos in MDSLogContextBase's definition and function '::complete',
also add more comments about setting safe_pos.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

common: Add get_process_name implementation for Windows

This patch adds get_process_name for Windows based on
GetModuleFileName.

Signed-off-by: Alin Gabriel Serdean <aserdean@cloudbasesolutions.com>

crimson/onode-staged-tree: fix explicit template instantiation for clang

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/dev/generatedocs.rst: fix highlight syntax

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38618 from neha-ojha/wip-14-2-16

doc/releases/nautilus.rst: add release notes for 14.2.16

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38622 from neha-ojha/wip-15-2-8

doc/releases/octopus.rst: add release notes for 15.2.8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: add tests for new snapshot APIs

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/cephfs: python bindings for new snapshot APIs

Signed-off-by: Venky Shankar <vshankar@redhat.com>

client: Snapshot cephfs APIs

Introduce calls to create and delete a snapshot. The difference
b/w this and the traditional mkdir() call to create a snapshot
is the additional metadata field that can be persistent along
side a snapshot (on creation). However, rmsnap() does nothing
special and is included for completeness.

Additionally, get_snap_info() API allows to fetch snapshot info
which includes things like snaphot ID and metadata (if any, which
was set via mksnap() API).

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: include snapshot metadata in trace reply to client

MDS sends (custom user) metadata associated with a snapshot in
trace reply to the client. This is required to implement API for
fetching snapshot info (snap-id, metadata, etc...).

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: include snap metadata on-disk on snapshot creation

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: include metadata in SnapInfo structure

This is a free-flowing key/value metadata attached to a snapshot
and persisted in `SnapInfo` structure. The idea is to allow
custom metadata to be persisted when creating a snapshot.

Initial users for this feature would be cephfs-mirror daemon. The
idea is to persist the primary filesystems snapshot ID on snapshot
creation in secondary filesystem. This allows identification of
snapshot renames and recreates.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #38351 from yuvalif/wip-yuval-fix-46128

rgw/notifications: automatically delete topics associated with a bucket

Merge pull request #38246 from yuvalif/wip-yuval-fix-48321

rgw/notifications: fix tag based filtering to work on all ops

Merge pull request #38190 from yuvalif/wip-yuvali-fix-48293

rgw/amqp/kafka: prevent concurrent shutdowns from happening

Merge pull request #37924 from yuvalif/allow_packages_in_lua

rgw/lua: support packages via luarocks

Merge pull request #38284 from rhcs-dashboard/48355-read-pwd-from-file

mgr/dashboard: CLI commands: read passwords from file

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: knortema <knortema@redhat.com>

Merge pull request #38473 from votdev/issue_48474_health_check

mgr/dashboard: Display a warning message in Dashboard when debug mode is enabled

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38599 from tchaikov/wip-47977

ceph.in: try harder if asok path does not exist

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

crimson/osd: do not repeat type name when constructing an instance

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not use __func__ in lambda

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #38447 into master

* refs/pull/38447/head:
ceph-volume: disable cache for blkid calls

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #38636 from ceph/fix-labeler-master

.github/labeler: fix yaml syntax

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38634 from tchaikov/wip-install-deps

install-deps.sh: do not install libboost-all-dev and nlohmann-json-dev

Reviewed-by: Josh Durgin <jdurgin@redat.com>

Merge pull request #38601 from linuxbox2/more-librgw-zipper9

More librgw zipper9

Merge pull request #23396 from linuxbox2/wip-rgw-xattrs-2

rgw_file:  expose RGW user-defined attributes

Merge pull request #37138 from lxbsz/iscsi_qa

qa: add ceph iscsi test suite support

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd/api: avoid retrieving more than max mirror image info records

This could otherwise result in an assertion failure in the API if
it failed to retrieve the status on an image and therefore required
a second iteration through the loop.

Fixes: https://tracker.ceph.com/issues/48522
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa: add ceph iscsi test suite support

Currently this only supports the gwcli tests on CentOS/RHEL 8 with
2 ceph iscsi gateways.

Fixes: https://tracker.ceph.com/issues/47439
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: krbd_stable_pages_required.sh: move to stable_writes attribute

bdi/stable_pages_required attribute was deprecated in 5.10 and now
always returns 0.  The replacement is queue/stable_writes.  (It is
also writeable, so we can simplify these test cases somewhat in the
future.)

Fixes: https://tracker.ceph.com/issues/48232
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks: get the correct clone_dir for each client

The clone_dir will be overrided in the loop.

Fixes: https://tracker.ceph.com/issues/47439
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks: allow none client prefixed node to run the cram

Fixes: https://tracker.ceph.com/issues/47439
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks: fix not enough values to unpack value error

Fixes: https://tracker.ceph.com/issues/47439
Signed-off-by: Xiubo Li <xiubli@redhat.com>

crimson/seastore: fixing Clang errors and warnings

Errors: capturing structured-binding "special ref"

Warnings: unused captures

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

.github/labeler: fix yaml syntax

Introduced by https://github.com/ceph/ceph/pull/38617

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

crimson/osd: return plain value instead of future

seastar allows us to use plain value instead of a future<> when the
future is available right away.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use move() to avoid memcpy()

try to move away from the returned map of xattrs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not use a now() future if no shard to push

in that case, seastar::parallel_for_each() returns a
`make_ready_future<>()`, so no need to optimize for this case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: extract push_func as a separate method

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not use do_with() for capturing target pg_shards

it's perfectly fine to check for empty, and pass it to
seastar::parallel_for_each().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not use the parallel_for_each() in prep_push()

the caller of prep_push() also loops in target pg shards, so there is no
need to do the loop in prep_push(). in this change, prep_push() just
takes care of a single pg_shard for a single oid instead of collecting
all push ops for all pg_shards to be recovered.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: create the collected push ops use shared_ptr

do_with() uses a unique_ptr for keeping the captured variables alive. so
it is practically the same thing in concept. but `do_with()` is more
distracting and difficult to reason about, it increases the indent level
also.

so use a lw_shared_ptr() for collecting the push ops map.

Signed-off-by: Kefu Chai <kchai@redhat.com>