Merge pull request #29778 from cbodley/wip-41212

vstart: move [client.rgw] config into [client]

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mgr/dashboard: User Management E2E tests (#29641)

mgr/dashboard: User Management E2E tests

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: run-backend-api-tests.sh CI improvements (#29504)

mgr/dashboard: run-backend-api-tests.sh CI improvements

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #29590 from Aran85/fix_proc_replica_log

osd: merge replica log on primary need according to replica log's crt

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29747 from liewegas/wip-39546

osd/PeeringState: do not complain about past_intervals constrained by oldest epoch

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29624 from NancySu05/osdmonitor_markmedown

mon:C_AckMarkedDown has not handled the Callback Arguments

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29738 from ifed01/wip-ifed-alloc-cleanup

os/bluestore: minor improvements/cleanup around allocator

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #29614 from votdev/issue_41205

mgr/dashboard: Access control database does not restore disabled users correctly

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #29146 from badone/wip-tracker-40835-OSDCap.PoolClassRNS-abort

osd/OSDCap: Check for empty namespace

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25697 from Aran85/fix-onode-trim

os/bluestore: more aggressive deferred submit when onode trim skipping

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #28488 from liuchang0812/show-pool-id-in-pool-ls-cmd

mon: show pool id in pool ls command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24636 from rzarzynski/wip-denc-container_base

denc: slightly optimize container_base::bound_encode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29756 from Aran85/fix-repair-object

osd: clear PG_STATE_CLEAN when repair object

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #29806 into master

* refs/pull/29806/head:
mgr/BaseMgrModule: tolerate Int or Long for health 'count'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29298 from zhangsw/rgw-fix-bug-listobjv2-startafter

rgw: continuationToken or startAfter shouldn't be returned if not specified

Merge PR #29780 into master

* refs/pull/29780/head:
osd/PeeringState: semi-colon after DECLARE_LOCALS
osd/PeeringState: on_new_interval on child PG after split

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #29774 into master

* refs/pull/29774/head:
qa/standalone/scrub/osd-scrub-snaps: snapmapper omap is now 'm'

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #29807 into master

* refs/pull/29807/head:
mgr/pg_autoscaler: fix race with pool deletion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17719 from mikulely/fix-usage-stats

rgw: distinguish different get_usage for usage log

Reviewed-by: Robin H. Johnson <rjohnson@digitalocean.com>

mgr/dashboard: run-backend-api-tests.sh CI improvements

As there is now a jenkins job to run this script
(see https://github.com/ceph/ceph-build/pull/1351),
this refactoring adapt the script to be run in a jenkins job as well as locally.

Signed-off-by: alfonsomthd <almartin@redhat.com>

Merge pull request #29544 from tchaikov/wip-doc-search-CSP

doc: always load resources via HTTPS

Reviewed-by: Tiago Melo <tmelo@suse.com>

doc: always load resources via HTTPS

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29755 from xiexingguo/wip-inc-recovery-4

osd: do not invalidate clear_regions of missing item at boot

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/BaseMgrModule: tolerate Int or Long for health 'count'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29804 from alfredodeza/wip-rm41378

ceph-volume tests set the noninteractive flag for Debian

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #29744 into master

* refs/pull/29744/head:
qa/run-standalone.sh: fix python path
qa/standalone/mon/health-mute.sh: fix up rachet test
qa/standalone/mon/health-mute.sh: s/kill daemons/kill_daemons/

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #29749 into master

* refs/pull/29749/head:
mon/HealthMonitor: remove unused label

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29757 into master

* refs/pull/29757/head:
osd: always initialize local variable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29763 into master

* refs/pull/29763/head:
qa/suites/rados: whitelist POOL_APP_NOT_ENABLED warning

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/pg_autoscaler: fix race with pool deletion

The pool_stats map comes from a get('df') that may not include a pool
because it was just deleted.

Fixes: https://tracker.ceph.com/issues/41386
Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume tests set the noninteractive flag for Debian, to avoid prompts in apt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #28378 into master

* refs/pull/28378/head:
qa/tasks: introduce Thrasher base class
qa/tasks: Fix typo
qa/tasks: manage thrashers
qa/tasks: start DaemonWatchdog when ceph starts
qa/tasks: make watch and bark handle more daemons
qa/tasks: move DaemonWatchdog to new file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29773 from dillaman/wip-41352

pybind/mgr/rbd_support: fix missing variable in error path

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #29659 from jan--f/c-v-simple-functional-no-lvm-zap

ceph-volume: don't try to test lvm zap on simple tests

qa/tasks: introduce Thrasher base class

* Introduced a Thrasher base class.
* Updated thrashers to inherit from Thrasher.
* Replaced the magic variable e with Thrasher.exception as per the discussion.
  Now the exception variable sets by default as the thrashers are inheriting
  from the Thrasher class.

Fixes: https://github.com/ceph/ceph/pull/28378#discussion_r309337928
Fixes: https://tracker.ceph.com/issues/41133
Signed-off-by: Jos Collin <jcollin@redhat.com>

rgw: distinguish different get_usage for usage log

get_usage op via s3 endpoint are not the same as get_usage
via admin endpoint in the rgw usage log categories.

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

mon/HealthMonitor: remove unused label

move the whole sanity checks into `HealthMonitor::preprocess_command()`.

this change silences warning of:

warning: label 'reply' defined but not used [-Wunused-label]

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PeeringState: semi-colon after DECLARE_LOCALS

This screws with emacs' auto-indent.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: on_new_interval on child PG after split

This was broken as part of the PeeringState refactor, see
ace6655f955837b104f307183fc0b988b7880c04.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29717 into master

* refs/pull/29717/head:
os/bluestore: prefix omap of temp objects by real pool

Reviewed-by: Sage Weil <sage@redhat.com>

qa/standalone/scrub/osd-scrub-snaps: snapmapper omap is now 'm'

...due to per-pool omap.

Fixes 91f533be71e5fd9b2c0135a5b54d663425a1d9c4

Fixes: https://tracker.ceph.com/issues/41353
Signed-off-by: Sage Weil <sage@redhat.com>

vstart: move [client.rgw] config into [client]

common rgw config was moved into [client.rgw] with the intent to
cover radosgw in multisite tests too, but this broke the default case
where radosgws are named after [client.rgw.port]. move the common stuff
to [client] so it does actually apply to both

Fixes: https://tracker.ceph.com/issues/41212
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29759 into master

* refs/pull/29759/head:
qa/tasks: fixed typo in the comment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29625 from kamoltat/wip-test-progress-fix-bug-#29385

mgr/progress/module.py: s/events/_events/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29704 from tchaikov/wip-ceph-objectstore-tool

ceph-objectstore-tool: return 0 if incmap is sane

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #29655 from trociny/wip-40923

qa/workunits/rbd: stress test `rbd mirror pool status --verbose`

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

pybind/mgr/rbd_support: fix missing variable in error path

Fixes: https://tracker.ceph.com/issues/41352
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29376 from Songweibin/wip-rbd-display-id

rbd/action: display image id in rbd du/list output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29743 from smithfarm/wip-ceph-backport-https

script/ceph-backport.sh: carry https through to logical conclusion

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #29653 from lixiaoy1/AsyncOpTracker_dup

librbd: Remove duplicated AsyncOpTracker in librbd/Utils.h

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29094 from cbodley/wip-40806

radosgw-admin: bucket sync status not 'caught up' during full sync

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

osd/PeeringState: do not complain about past_intervals constrained by oldest epoch

The start of the required interval has a floor set by the oldest osdmap
epoch we have.  That can lead to an invalid/empty required interval
(because the start is >= the end), but the PG may still have past
intervals.  That can be cause by a slow PG deletion.

No need to complain about this harmless condition.

Fixes: https://tracker.ceph.com/issues/39546
Signed-off-by: Sage Weil <sage@redhat.com>

scripts/ceph-backport.sh: always use https://tracker.ceph.com

Completing the wave of fixes to this script in the wake of
https://tracker.ceph.com/issues/38764, this commit replaces
"http" with "https" in the comments and puts the Redmine endpoint
into a variable, along with some other cleanups.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #29762 from alfredodeza/bz-1738379

ceph-volume: use the OSD identifier when reporting success

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

qa/suites/rados: whitelist POOL_APP_NOT_ENABLED warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: User Management E2E tests

Test breadcrumbs for user management pages
Test creating, editing and deleting a user
Test creating, editing and deleting a role

Fixes: https://tracker.ceph.com/issues/41231
Fixes: https://tracker.ceph.com/issues/41232
Fixes: https://tracker.ceph.com/issues/41233
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

mgr/dashboard: login screen language dropdown enhancement (#29636)

mgr/dashboard: login screen language dropdown enhancement

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29737 from tspmelo/wip-label-alignment

mgr/dashboard: Vertically align the "Refresh" label

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Exclude some folders in the frontend project (#29708)

mgr/dashboard: Exclude some folders in the frontend project

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume devices.lvm zap use the identifier to report success

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests check success message when zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Hosts Page Service Links Test (#29516)

mgr/dashboard: Hosts Page Service Links Test

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29728 from tchaikov/wip-41330

cmake,run-make-check.sh: disable SPDK by default

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

qa/tasks: fixed typo in the comment

Signed-off-by: Jos Collin <jcollin@redhat.com>

mon: show pool id in pool ls command

Fixes: http://tracker.ceph.com/issues/40287
Signed-off-by: Chang Liu <liuchang0812@gmail.com>

osd: always initialize local variable

to silence a GCC warning like:

../src/osd/OSD.cc:4608:24: warning: ‘sender_delta_ub’ may be used
uninitialized in this function [-Wmaybe-uninitialized]
 4608 |       ceph::signedspan sender_delta_ub;
      |                        ^~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: add a process that handle Callback Arguments
in C_AckMarkedDown,we add a process that handle _finish(int) Callback Arguments

Fixes: https://tracker.ceph.com/issues/41217
Signed-off-by: NancySu05 <su_nan@inspur.com>

cmake,run-make-check.sh,deb,rpm: disable SPDK by default

but we still enable it in `run-make-check.sh`

* cmake: disable SPDK by default
* run-make-check.sh: enable WITH_SPDK so at least we can ensure it
  builds
* deb,rpm: add uuid-dev / libuuid-devel as a "make check" dependency

Fixes: https://tracker.ceph.com/issues/41330
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: make WITH_SPDK a dependent option

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: clear PG_STATE_CLEAN when repair object

there is a race be found, when we repair object on clean state,
we queue a DoRecovery peering event, but before the peering event
dequeue,a snaptrim event on the missing object's snap dequeue,
then we will get pass the context< SnapTrimmer >().can_trim()
and go to get the context of the missing object(snapdir)

we can avoid this by clear clean state when we found missing..

Fixes: https://tracker.ceph.com/issues/41348
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

osd: do not invalidate clear_regions of missing item at boot

Seems we'll always mark clear_regions as all dirty
when reading pg logs and missing items off the disk,
which as a result turns incremental recovery off by default.
Also using std::move seems to be a bit more efficient
and robust here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29753 from tchaikov/wip-sudo-sysctl

run-make-check.sh: run sudo with absolute path

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #29726 from xiexingguo/wip-inc-recovery-1

osd/PrimaryLogPG: fix dirty range of write_full

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29729 from xiexingguo/wip-inc-recovery-2

osd/PrimaryLogPG: finish_copyfrom - dirty omap if necessary

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

run-make-check.sh: run sudo with absolute path

there is chance that `/sbin` is not listed in `secure_path`, see
sudoers(5) for more details

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: merge replica log on primary need according to replica log's crt

Fixes: https://tracker.ceph.com/issues/41194
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

osd/OSDCap: Check for empty namespace

Fixes: http://tracker.ceph.com/issues/40835
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

qa/run-standalone.sh: fix python path

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/mon/health-mute.sh: fix up rachet test

Make sure we provide time for the mute to get cleared out by tick().

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29518 into master

* refs/pull/29518/head:
mount.ceph: properly handle -o strictatime

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29411 into master

* refs/pull/29411/head:
mds: fix InoTable::force_consume_to()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29543 into master

* refs/pull/29543/head:
mds: move some MDCache member init to header

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #29429 from dillaman/wip-40834

doc/rbd: initial kubernetes / ceph-csi integration documentation

Reviewed-by: ShyamsundarR <srangana@redhat.com>
Reviewed-by: Humble Chirammal <hchiramm@redhat.com>
Reviewed-by: Madhu Rajanna <madhupr007@gmail.com>

qa/standalone/mon/health-mute.sh: s/kill daemons/kill_daemons/

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29719 into master

* refs/pull/29719/head:
osd/PrimaryLogPG: kill obsolete ondisk_{read,write}_lock comments

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: E2E Test for RBD Mirroring and Images (#29381)

mgr/dashboard: E2E Test for RBD Mirroring and Images

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

os/bluestore: avoid constant name replica in bitmap allocator.
This was introduced by Adam's https://github.com/ceph/ceph/pull/29425

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: implement original allocator fragmentation inspection.

Using admin socket interface: bluestore allocator fragmentation
[block|bluefs-db|bluefs-wal|bluefs-slow]

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: simplify Allocator::get_fragmentation() signature
Remove alloc_unit parameter.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/allocator_test: additional coverage for get_fragmentation_score

Tests the function results agains the same cases as we do for
get_fragmentation.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Vertically align the "Refresh" label

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Exclude some folders in the frontend project

This will remove an warning in VS Code [1] and, more important, will reduce the
number of files that TS will watch when compiling or linting.

[1] To enable project-wide JavaScript/TypeScript language features,
exclude large folders with source files that you do not work on.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29685 from smithfarm/wip-41296

rpm: always build ceph-test package

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

Merge pull request #29556 from Devp00l/wip-41165

mgr/dashboard: CephFS client tab switch

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Verify fields on ISCSI page (#29608)

mgr/dashboard: Verify fields on ISCSI page

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: clean-up python unit tests (#28696)

mgr/dashboard: clean-up python unit tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

osd/PrimaryLogPG: finish_copyfrom - dirty omap if necessary

otherwise we'll at risk of losing track of omap entries.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PrimaryLogPG: fix dirty range of write_full

A write_full operation may implicitly truncate the object down,
hence we need to mark the truncated part as dirty as well since
follow-up randomized writes may still be able to (re)extend the
object size and leave some holes against the truncated part,
which as a result might cause problems during incremental-mode
recovery.

Note that write_update_size_and_usage would reset oi.size
and that's why we move the mark_data_region_dirty call before that.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29584 from tchaikov/wip-qa-test_envlibrados_for_rocksdb-cmake3

qa/workunits/rados/test_envlibrados_for_rocksdb: install newer cmake

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/workunits/rados/test_envlibrados_for_rocksdb: install newer cmake

rocksdb needs CMake 3.5.1 since
https://github.com/facebook/rocksdb/pull/5662

Fixes: https://tracker.ceph.com/issues/41253
Signed-off-by: Kefu Chai <kchai@redhat.com>