Merge PR #37776 into master

* refs/pull/37776/head:
cephfs: Fix variable declartion in capture list in lambda

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37735 into master

* refs/pull/37735/head:
client: fix inode ll_ref reference count leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37629 into master

* refs/pull/37629/head:
qa/cephfs: add session_timeout option support
qa/cephfs: move the cephfs's opertions setting to create()
qa/cephfs: add 'cephfs:' section support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37652 into master

* refs/pull/37652/head:
qa/tasks: tear down the background process before unmounting
qa/tasks: switch to _kill_background() helper to terminate the daemons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37718 from smithfarm/wip-rn-nautilus

doc/releases: add Nautilus 14.2.12 release notes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37755 from rhcs-dashboard/notification-style

mgr/dashboard: Adding more style to the notification sidebar

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37738 from ifed01/wip-ifed-fix-death-tests

test/store_test: use 'threadsafe' style for death tests.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37765 from tchaikov/wip-mgr-common-cleanup

mgr,common: use make_unique<> and fmt::format()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37431 from sseshasa/wip-handle-future-sched-ops

osd: Handle possible future work items returned from mclock scheduler.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephfs: Fix variable declartion in capture list in lambda

/home/jenkins/workspace/ceph-master-compile/src/tools/cephfs_mirror/Mirror.cc:529:33: error: 'mirror_action' in capture list does not name a variable
    m_cond.wait(locker, [this, &mirror_action] {return !mirror_action.action_in_progress;});
                                ^
/home/jenkins/workspace/ceph-master-compile/src/tools/cephfs_mirror/Mirror.cc:529:57: error: reference to local binding 'mirror_action' declared in enclosing function 'cephfs::mirror::Mirror::run'
    m_cond.wait(locker, [this, &mirror_action] {return !mirror_action.action_in_progress;});
                                                        ^
/home/jenkins/workspace/ceph-master-compile/src/tools/cephfs_mirror/Mirror.cc:526:27: note: 'mirror_action' declared here
  for (auto &[filesystem, mirror_action] : m_mirror_actions) {
                          ^
/home/jenkins/workspace/ceph-master-compile/src/tools/cephfs_mirror/Mirror.cc:529:26: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
    m_cond.wait(locker, [this, &mirror_action] {return !mirror_action.action_in_progress;});
                         ^~~~
1 warning and 2 errors generated.

Tracker: https://tracker.ceph.com/issues/47973
Fixes: https://github.com/ceph/ceph/pull/37313
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #37678 from ideepika/wip-drop-hammer-from-qa

qa: drop hammer branch qa tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37193 from rhcs-dashboard/log_download

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36449 from s0nea/wip-dashboard-osd-flags-indiv

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37440 from Devp00l/wip-44433

mgr/dashboard: Add clay plugin support

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37512 from tspmelo/wip-ng-validation

mgr/dashboard: Extract ng-bootstrap-form-validation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37748 from Rethan/msg_remove_dispatch_strategy

msg: move DispatchStrategy into test/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37404 from badone/wip-docker-test-allow-persistent-containers

test/docker-test: Allow persistent containers

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/DaemonHealthMetricCollector: replace boost::format with fmt::format

the latter is easier to use, and future proof in the sense that {fmt}
was included as a part of C++20 standard

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Formatter: replace boost::format with fmt::format

the latter is easier to use, and future proof in the sense that {fmt}
was included as a part of C++20 standard

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: use make_unique<> when appropriate

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/cephfs: add session_timeout option support

When the mds revoking the Fwbl caps, the clients need to flush
the dirty data back to the OSDs, but the flush may make the OSDs
to be overloaded and slow, which may take more than 60 seconds to
finish. Then the MDS daemons will report the WRN messages.

For the teuthology test cases, let's just increase the timeout
value to make it work.

Fixes: https://tracker.ceph.com/issues/47565
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/cephfs: move the cephfs's opertions setting to create()

Fixes: https://tracker.ceph.com/issues/47565
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/cephfs: add 'cephfs:' section support

Fixes: https://tracker.ceph.com/issues/47565
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #36754 from rzarzynski/wip-bs-filewriter-appender

common/bl, os/bluestore: eradicate bufferlist rebuilds caused by FileWriter of BlueFS

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/docker-test: Allow persistent containers

It can be useful to allow the container to persist for debugging
purposes. Also show the command used to create the container.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

msg: move DispatchStrategy into test/

Since commit cc9a914, xio related code were all removed, which made
direct_messenger become the only user of DispatchStraegy. So move
these code there.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

client: fix inode ll_ref reference count leak

Fixes: https://tracker.ceph.com/issues/47918
Signed-off-by: sepia-liu <liuwei_coder@163.com>

Merge pull request #37760 from athanatos/sjust/wip-fix-lba-leaf

crimson/seastore: remove invalid mutate_mapping assert

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #36204 into master

* refs/pull/36204/head:
client: add command_lock support
client: defer swapping the newmap to mdsmap
client: try to grab the client_lock when needed
client: clean up the client code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: remove invalid mutate_mapping assert

A leaf node can have fewer than min_capacity items when it is the root.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: test mutate_mapping on more split/merge states

incref_mapping has a slightly different relationship with merge than
decref_mapping, update split/merge test to ensure that we call into
mutate_mapping at each step as the nodes fill, split, empty, and merge.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #37715 from anthonyeleven/rados-doc-improvements

doc/rados: operations.rst: clarity, detail, modernization

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #37702 into master

* refs/pull/37702/head:
mon/MDSMonitor: do not ignore mds's down:dne request

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37663 into master

* refs/pull/37663/head:
msg/MFSMap: remove unused variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37313 into master

* refs/pull/37313/head:
cephfs-mirror: restart mirroring on blocklist
cephfs-mirror: public method to fetch peer list
cephfs-mirror: dump rados address in "fs mirror status" asok command
cephfs-mirror: handle blocklist errors during rewatch
cephfs-mirror: periodically drive mirror enable, disable and peer updates
cephfs-mirror: identify a ceph filesystem using (id, name)
cephfs-mirror: introduce Filesystem type
cephfs-mirror: record failed FSMirror instances
cephfs-mirror: protect FSMirror::is_stopping() under lock
cephfs-mirror: unregister asok commands outside lock
cephfs-mirror: spawn peer replayers on peer changes
cephfs-mirror: s/blacklist/blocklist/i
cephfs-mirror: s/dir_name/dir_path/

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36004 into master

* refs/pull/36004/head:
mds: detect mds-mds messages not of type MMDSOp
messages: inherit MMDSPing and MMDSMetrics from MMDSOp

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36544 into master

* refs/pull/36544/head:
vstart_runner: omit result line only when testcases are run one by one
vstart_runner: don't quit running testsuite on a test failure

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Extract ng-bootstrap-form-validation

Remove ng-bootstrap-form-validation and
extract part of its code into the dashboard.

Fixes: https://tracker.ceph.com/issues/47721
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35355 from ofriedma/wip-ofriedma-beast-timeout

rgw: Add request timeout to beast

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37666 from pritha-srivastava/wip-rgw-gc-list-loop

rgw/gc: fixing the condition when marker for a queue is

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37409 from pritha-srivastava/wip-rgw-gc-perf-counter

rgw/gc: fix for incrementing the perf counter 'gc_retire_object'

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #37752 into master

* refs/pull/37752/head:
vstart.sh: fix fs set max_mds bug

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Adding more style to the notification sidebar

This commit improves the notification sidebar styling by adding a border on the left side and the color of the border aligns with the type of the warning (succes, info or danger).

Fixes: https://tracker.ceph.com/issues/47950
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #36199 from ofriedma/wip-ofriedma-lc-fixes-tags-and

rgw: fix expiration header returned even if there is only one tag in …

mgr/dashboard: Created a Download and Copy-to-Clipboard option for the logs

Created Download Button Component.
Added a Download and Copy-to-Clipboard button for the Cluster and Audit Logs in the Log section. Also added a flag in the current Copy2ClipboardButton directive to indicate the incoming string is actually a log.

Fixes: https://tracker.ceph.com/issues/47498
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37745 from tchaikov/wip-crimson-recovery-cleanup

crimson/osd: add more privacy

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

vstart.sh: fix fs set max_mds bug

Fix a bug where the name used when creating a volume and the name used when setting max_mds were different.
Fixes: https://tracker.ceph.com/issues/47946
Signed-off-by: Jinmyeong Lee <jinmyeong.lee@linecorp.com>

Merge pull request #37557 from wjwithagen/wjw-fix-warnings-clang-iterators

mds,osd: fix clang warnings to prevent unneeded copies

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: detect mds-mds messages not of type MMDSOp

* Detect mds-mds messages not of type MMDSOp.
* Return true from MetricAggregator::ms_dispatch2 only when the peer is an MDS and the metrics is handled.

Fixes: https://tracker.ceph.com/issues/46426
Signed-off-by: Jos Collin <jcollin@redhat.com>

messages: inherit MMDSPing and MMDSMetrics from MMDSOp

The inter-MDS messages MMDSPing and MMDSMetrics should inherit from
MMDSOp before the MDS dispatch the message.

Fixes: https://tracker.ceph.com/issues/46426
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #37709 from tchaikov/wip-common-pick-address

common/pick_address: fail if cannot bind with specified network family

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crimson/osd: add more privacy

* reorder the member variable and member function definitions so the
  private members are listed at the end of class. as requested by
  the CodingStyle
* mark some member variables and member functions private, as long as
  they are not used by other classes or non-friend classes.
* document UrgentRecovery
* refactor BackgroundRecovery::get_scheduler_params() to dedup
  the comments

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37739 from ivancich/wip-rados-ls-flush

tools/rados: flush formatter periodically during json output of `rados ls`

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge PR #37717 into master

* refs/pull/37717/head:
doc/cephfs: add sample cephfs-shell.conf

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37622 from ivancich/wip-rgw-orphan-list-comma

rgw: rgw-orphan-list should use "plain" formatted `rados ls` output

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

tools/rados: flush formatter periodically during json output of `rados ls`

While `rados ls` is emitting object info through a json formatter,
flush the formatter after there are at least 4096 bytes are buffered
for output.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

doc/rados/operations: clarity, detail, modernization

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #37615 from rhcs-dashboard/remove-last-change-column

mgr/dashboard: Remove the "Last change" column in Pools table

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

test/store_test: use 'threadsafe' style for death tests.

Hopefully Fixes: https://tracker.ceph.com/issues/47328
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: assign flags to single OSDs

Add the possibility to assign the flags ['noup',
'nodown', 'noin', 'noout'] to single OSDs.

Fixes: https://tracker.ceph.com/issues/40739
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37088 from Devp00l/wip-37408

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37416 from tspmelo/wip-rbd-clone

mgr/dashboard: Disable RBD clone action when conditions are not met

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37396 from SMIL-IT/fix-keycount

rgw: fix: S3 API KeyCount incorrect return.

mgr/dashboard: remove custom class `oa-hr-small`

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: remove unnecessary blank

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

doc/releases: add Nautilus 14.2.12 release notes

Also fix missing Octopus 15.2.5 in "doc/releases/general.rst" and
"doc/releases/releases.yml"

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #37727 from dillaman/wip-librbd-readresult-clip

librbd: remove io::ReadResult::set_clip_length

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37726 from dillaman/wip-librbd-remove-image-cache

librbd: remove remainder of ImageCache API hooks

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/tasks: tear down the background process before unmounting

If the background process keeps running by opening the mountpoint
directory, the unmount will fail with BUSY.

Fixes: https://tracker.ceph.com/issues/46883
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks: switch to _kill_background() helper to terminate the daemons

Fixes: https://tracker.ceph.com/issues/46883
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #37614 from smithfarm/wip-revert-suse-podman-requires

Revert "rpm: on SUSE, podman is required for cephadm to work" and "spec: Podman (temporarily) requires apparmor-abstractions on suse"

Reviewed-by: Michael Fritch <mfritch@suse.com>

doc/cephfs: add sample cephfs-shell.conf

And elaborate the description about it.

Fixes: https://tracker.ceph.com/issues/43028
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37403 from athanatos/sjust/wip-seastore-gc

crimson/seastore: add online gc to seastore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37731 from adk3798/cephadm-logs

doc: document additional journal args in cephadm logs command

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: document additional journal args in cephadm logs command

Signed-off-by: Adam King <adking@redhat.com>

test/crimson/seastore/test_btree_lba_manager: fix get_overlap

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: add single transaction split/merge test

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: always return absolute or record relative addrs

insert and lookup_range should return absolute or record-relative addrs,
never block relative.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: use node_[un]resolve_vals

Implement node_[un]resolve_vals to transform values to/from block relative
representations when copying out of/in to newly created blocks.  This can
happen when splitting a node which has had entries added during the
same transaction, or itself was created during the transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: add support for node dependent values

node_[un]resolve_vals can be used by implementations to transform vals
during split and merge.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: add test for large single-transaction split/merge

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: fix lookup_range, scan_* references with weak transactions

Weak transactions won't keep references in memory as we traverse
the tree, update each recursive call to ensure lifecycle of
this.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: extract common test logic for transaction_maanager initialization

This should make it somewhat easier to avoid breaking tests that
depend on transaction_manager.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: add online gc

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: close and reopen tm after mkfs

We'll just require this generally.  Most of the time, mkfs will be a
standalone operation anyway.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: add get_physical_extent_if_live

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add public, cache aware get_extent_by_type variants

Signed-off-by: Samuel Just <sjust@redhat.com>

os/seastore/journal: replay_segment: skip replay on deltas to released segments

See comment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix get_journal_seq

current_journal_segment_seq was misleading -- it's actually the value
for the next one.  Fix naming and update get_journal_seq.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add journal_replay check and debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add segment_header_t operator<<

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add duplicate_for_write debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/journal: check offset against segment length in replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: no need for p2roundup, to_write must already be aligned

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: simplify new extent paddr mapping during complete_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: improve encode_record debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use scan_segment to add extent scan

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: generalize replay scan

We'll need to do at least two forms of scan:
- deltas for replay
- extents for gc

Most of the mechanics are common, however, so this patch
hoists the common machinery into scan_segemnt.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: record extents types and laddr in record metadata

We'll need this to scan segments for live blocks.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use metadata size for block base rather than assuming one block

Uncaught because it's rare for the metadata prefix to exceed 4k.

Signed-off-by: Samuel Just <sjust@redhat.com>