Merge PR #24794 into master

* refs/pull/24794/head:
mds: move BatchOp to separate translation unit
mds: use auto ref during iteration
mds: answering all pending getattr/lookups targeting the same inode in one go

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28702 into master

* refs/pull/28702/head:
qa: update json format from session listing
mds: recall caps from quiescent sessions
mds: use Session::dump method uniformly
mds: use auto to deduce iterator type
mds: simplify method definition
mds: remove useless debug message
mds: use const get_request_count
mds: use session_info_t socket inst for addr
mds: refactor session lookup
mds: add explicit trim flag
mds: alphabetize tracked config keys
common: provide method to get half-life
common: correct variable name

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #29005 into master

* refs/pull/29005/head:
client: nfs-ganesha with cephfs client, removing dir reports not empty

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #29594 into master

* refs/pull/29594/head:
mds: reject sessionless messages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29752 into master

* refs/pull/29752/head:
mds: make MDSIOContextBase delete itself when shutting down

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30361 into master

* refs/pull/30361/head:
qa/tasks/mon_thrash: sync force requires some force flags

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #30357 from zhangsw/rgw-cleanup-bilog-status

rgw: add missing bilog status to help info.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29081 from wintamute/patch-1

Documentation: Add missing ceph-volume lvm batch argument to ceph-volume.rst

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge PR #30352 into master

* refs/pull/30352/head:
mon: disable min pg per osd warning

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30284 from Damenly/missing_cmath

tests: add missing header cmath to test/mon/test_mon_memory_target.cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Add missing ceph-volume lvm batch argument to ceph-volume.rst

The optional argument '--osds-per-device' is missing. Since this is quite helpful when creating NVMe based OSDs, it should be documented.

Signed-off-by: Andreas Krebs <wintamute@gmail.com>

mgr/dashboard: Increase the global value of jasmine timeout (#30359)

mgr/dashboard: Increase the global value of jasmine timeout

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #29761 from wjwithagen/wjw-fix-venv-python-version

test/venv: do not hardwire to py2.7 for tox tests

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix calculation of PG Status percentage (#30343)

mgr/dashboard: Fix calculation of PG Status percentage

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #30285 from IlsooByun/slow_op_report

mgr: do not reset reported if a new metric is not collected

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: update json format from session listing

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30112 from liewegas/wip-autoscaler-on

common: default pg_autoscale_mode=on for new pools

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: recall caps from quiescent sessions

This introduces two new config options [1,2] that dictate when a session
is considered quiescent by the MDS. (Options are documented fully in
options.cc.) When a session is quiescent, the MDS will preemptively
recall caps to reduce the outstanding capabilities which optimizes for
reducing work during failover.

[1] mds_session_cache_liveness_magnitude
[2] mds_session_cache_liveness_decay_rate

Fixes: https://tracker.ceph.com/issues/22446
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use Session::dump method uniformly

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use auto to deduce iterator type

This prevents accidental type mismatches.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: simplify method definition

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove useless debug message

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use const get_request_count

So it can be used from a const dump method. Make the elist mutable
instead.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use session_info_t socket inst for addr

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: refactor session lookup

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add explicit trim flag

So that the intent of the caller is clear to trim due to the cache being
too full.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: alphabetize tracked config keys

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: provide method to get half-life

Print the half-life in Formatter output as it is human understandable
and the value used when configuring the counters.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: correct variable name

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29574 into master

* refs/pull/29574/head:
mds: Reorganize class members in FSMapUser header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29883 into master

* refs/pull/29883/head:
mds: Reorganize class members in InoTable header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30090 into master

* refs/pull/30090/head:
mds: Reorganize class members in Anchor header

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #30143 into master

* refs/pull/30143/head:
mds: Reorganize class members in LocalLock header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30205 into master

* refs/pull/30205/head:
mds: Reorganize class members in LogEvent header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #30191 into master

* refs/pull/30191/head:
cls/hello: disable write returning data clobbering demo
messages/MOSDOpReply: assert no write result payloads
osd/PrimaryLogPG: assert write result == 0
osd/PrimaryLogPG: pass TMAPGET errors up
osd/PrimaryLogPG: set result=0 for successful writes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29792 from sebastian-philipp/orchestrator-api-replace

mgr/orchestrator: Document OSD replacement

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

common: default pg_autoscale_mode=on for new pools

Signed-off-by: Sage Weil <sage@redhat.com>

cls/hello: disable write returning data clobbering demo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29820 into master

* refs/pull/29820/head:
osd/PeeringState: log_weirdness during peering
osd/PeeringState: send new message types
osd/PeeringState: give require_osd_release to BufferedRecoveryMessages
osd: add new notify, query, and info messages
osd/PeeringState: use send_info() for replica activation ack
osd/PeeringState: remove old info_map member
osd/PeeringState: send infos via message_map (not info_map)
osd/PeeringState: remove old query_map member
osd/PeeringState: send queries via message_map (not query_map)
osd/PeeringState: use return value from discover_all_missing
osd/PeeringState: give PeeringCtxWrapper a BufferedRecoveryMessages ref
osd/PeeringState: send notifies via message_map (not notify_list)
osd/PeeringState: add message_map to PeeringCtx/BufferedRecoveryMessages
osd: dispatch_context inside PG lock

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #30146 from ivancich/wip-fix-min-unordered-bkt-list

rgw: fix minimum of unordered bucket listing

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>

mgr/dashboard: Display the duration of each e2e test

Now we are able to see which tests are taking
too much time and may need improvements.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Increase the global value of jasmine timeout

Some tests are taking longer than the default jasmine timeout to finish
and are causing a failure.

This commit will increase the timeout to 5 minutes,
removing the need to increase it for each failing test.

Signed-off-by: Tiago Melo <tmelo@suse.com>

qa/tasks/mon_thrash: sync force requires some force flags

AFAICS this has been the case for basically forever.  Not sure why/how
the mon_thrash task hasn't had a problem with that?

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30106 from smithfarm/wip-more-cbs

script/ceph-backport.sh: make the script idempotent

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: key-value-table doesn't render booleans (#30203)

 mgr/dashboard: key-value-table doesn't render booleans

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Fix error message not displayed in pool form (#30137)

mgr/dashboard: Fix error message not displayed in pool form

Reviewed-by: Volker Theile <vtheile@suse.com>

rgw: add missing bilog status to help info.

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Merge pull request #28936 from tchaikov/wip-unaligned-access

cmake: check for unaligned access

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mds: make MDSIOContextBase delete itself when shutting down

Fixes: https://tracker.ceph.com/issues/41346
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

Merge pull request #28939 from trociny/wip-37529

rbd-mirror: add namespace support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon: disable min pg per osd warning

Now that the pg_autoscaler is on by default, it is "normal" (and okay) to
have a small number of PGs in the cluster if the overall cluster usage is
also low.  This setting just results in a health warning out of the box
when you create a pool and haven't written any data yet.

Fixes: https://tracker.ceph.com/issues/41735
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30192 from dzafman/wip-41669

Make dumping of reservation info congruent between scrub and recovery

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30347 from neha-ojha/wip-add-backend

qa/suites/rados/perf: test min recommended osd_memory_target

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #30319 from zhangsw/rgw-fix-zone-rm

rgw: fix incorrect "radosgw-admin zonegroup rm" info.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

qa/suites/rados/perf: test min recommended osd_memory_target

Add bluestore-basic-min-osd-mem-target.yaml to objectstore.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #30346 into master

* refs/pull/30346/head:
script: raw_input was renamed to input in py3

Reviewed-by: Sage Weil <sage@redhat.com>

script: raw_input was renamed to input in py3

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30014 into master

* refs/pull/30014/head:
doc: update CephFS overview in introductory page
doc: filesystem to file system

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30221 from cbodley/wip-rgw-dns-name-iends-with

rgw: fix dns name comparison for virtual hosting

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30310 from rzarzynski/wip-crimson-op_effects

crimson/osd: extend OpsExecuter to carry about op effects.

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix calculation of PG Status percentage

We were reading a wrong value for the total of PGs.

Fixes: https://tracker.ceph.com/issues/41536
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: E2E enhancements (#30017)

mgr/dashboard: E2E enhancements

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29606 from sseshasa/bluestoreCfgObserver

os/bluestore: Add config observer for osd memory specific options.

Merge pull request #29793 from sebastian-philipp/ansible-default-realm-zonegroup

mgr/ansible: Change default realm and zonegroup

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #30321 from tchaikov/wip-run-cbt.sh

script/run-cbt.sh: always use python3

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Update protractor dependencies

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Enable waitforAngular in E2E

Convert waitForTableData into a method,
using it as a decorator was causing problems.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add protractor-fast-fail plugin

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add click call inside waitClickable

All waitClickable calls were being followed by a click.
This will remove the need for the click call.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix error in pool list component

Sometimes the pools list result comes without the tiers field.
This was causing an error in the code.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix random failure in images e2e test

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove duplicated method

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove helper.po.ts

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove E2E warning

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Extract delete method into page-helper

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Use suites in protractor

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Split user-mgmt e2e files

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Move some e2e into new "ui" folder

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Improve clickCheckbox with some waits

Signed-off-by: Tiago Melo <tmelo@suse.com>

script/ceph-backport.sh: help the user set up the script

This commit adds several options:

    --setup
    --setup-advice
    --usage-advice
    --troubleshooting-advice

It replaces the "comment block at the beginning of the script" which was lamely
masquerading as documentation.

It also implements heuristic deduction of GitHub remotes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

script/ceph-backport.sh: check Backport ticket assignee

Don't work on tickets assigned to someone else.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Improve images e2e test (#30212)

mgr/dashboard: Improve images e2e test

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

mgr/dashboard: Wait for breadcrumb text is present in e2e tests (#30267)

mgr/dashboard: Wait for breadcrumb text is present in e2e tests

mgr/dashboard: Rendering datatable causes Angular runtime error (#30209)

mgr/dashboard: Rendering datatable causes Angular runtime error

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

script/run-cbt.sh: always use python3

run-cbt.sh will be used by the "ceph-perf-pull-requests" jenkins job,
which will prepare the runtime dependencies by installing python3
modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30318 from tchaikov/wip-run-make.sh

script/run-make.sh: do not pass cmake options twice

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: fix incorrect "radosgw-admin zonegroup rm" info.

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

script/run-make.sh: do not pass cmake options twice

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30292 from Reperator/wip-doc-dashboard-fix-typo

doc: Fixes typo for ceph dashboard command.

Merge pull request #30281 from sebastian-philipp/doc-orchestrator-feature-update

doc/orchestrator: Sync status with reality

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #30231 from xiexingguo/wip-blue-balance

os/bluestore: add bluestore_bluefs_max_free; smooth space balancing a bit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30315 from tchaikov/wip-crimson-perf-test

test/crimson: add acceptable section to tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: check for unaligned access

C and C++ allow us to cast a pointer with an alignment requirement to
one with a different alignment requirement. but if the destination
type's alignment is greater/stricter than the source type, the behavior
is undefined. on amd64, it's allowed. and on aarch64, GCC offers an
option named `-munaligned-access` to enable this behavior. so we cannot
take this as guaranteed.

in denc.h and messenger implementations, we just cast the pointers to
the type we want to access without memcpy(), which is, strictly
speaking, not quite standard compliant.

so, to be on the safe side, we detect the support of unaligned access at
compile time.

please note, this would fail the cross-compile. as the target platform
might not support unaligned access, and the produced executable does not
run on the build platform. so this test is skipped if the building
host's architecture is different than that of target host.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: extract checks into CephChecks.cmake

also use case chosen in the macro definition as per suggested by cmake
document. see
https://cmake.org/cmake/help/latest/command/macro.html#invocation

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #29809 into master

* refs/pull/29809/head:
common/config_proxy: hold lock while accessing mutable container

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson: add acceptable section to tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30124 from rzarzynski/wip-crimson-conn-in-msg

crimson/osd: Message has non-null ref to SocketConnection now.

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: add missing header cmath to test/mon/test_mon_memory_target.cc

While building ceph, the complier complains:
=================================================================
[ 86%] Building CXX object src/test/mon/CMakeFiles/ceph_test_mon_memory_target.dir/test_mon_memory_target.cc.o
/root/ceph/src/test/mon/test_mon_memory_target.cc: In int main(int, :
/root/ceph/src/test/mon/test_mon_memory_target.cc:66:16:  was not declared in this scope
   66 |   auto stdev = sqrt(sump / results.size());
      |                ^~~~
make[2]: *** [src/test/mon/CMakeFiles/ceph_test_mon_memory_target.dir/build.make:63: src/test/mon/CMakeFiles/ceph_test_mon_memory_target.dir/test_mon_memory_target.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:15575: src/test/mon/CMakeFiles/ceph_test_mon_memory_target.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
=================================================================

This's caused by lack of header cmath which is required by
std::sqrt().
Fix it by add the header then build continues.

Fixes: https://tracker.ceph.com/issues/41739
Signed-off-by: Su Yue <Damenly_Su@gmx.com>

Merge pull request #30309 from neha-ojha/wip-41661

qa/suites/rados/perf/objectstore: do not symlink to qa/objectstore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30255 from xiexingguo/wip-bluestore-gift-ratio

os/bluestore: fix space balancing overflow

Reviewed-by: Sage Weil <sage@redhat.com>