mgr/dashboard: run-backend-api-tests.sh CI improvements (#29504)

mgr/dashboard: run-backend-api-tests.sh CI improvements

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #29590 from Aran85/fix_proc_replica_log

osd: merge replica log on primary need according to replica log's crt

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29747 from liewegas/wip-39546

osd/PeeringState: do not complain about past_intervals constrained by oldest epoch

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29624 from NancySu05/osdmonitor_markmedown

mon:C_AckMarkedDown has not handled the Callback Arguments

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29738 from ifed01/wip-ifed-alloc-cleanup

os/bluestore: minor improvements/cleanup around allocator

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #29614 from votdev/issue_41205

mgr/dashboard: Access control database does not restore disabled users correctly

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #29146 from badone/wip-tracker-40835-OSDCap.PoolClassRNS-abort

osd/OSDCap: Check for empty namespace

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25697 from Aran85/fix-onode-trim

os/bluestore: more aggressive deferred submit when onode trim skipping

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #28488 from liuchang0812/show-pool-id-in-pool-ls-cmd

mon: show pool id in pool ls command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24636 from rzarzynski/wip-denc-container_base

denc: slightly optimize container_base::bound_encode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29756 from Aran85/fix-repair-object

osd: clear PG_STATE_CLEAN when repair object

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #29806 into master

* refs/pull/29806/head:
mgr/BaseMgrModule: tolerate Int or Long for health 'count'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29298 from zhangsw/rgw-fix-bug-listobjv2-startafter

rgw: continuationToken or startAfter shouldn't be returned if not specified

Merge PR #29780 into master

* refs/pull/29780/head:
osd/PeeringState: semi-colon after DECLARE_LOCALS
osd/PeeringState: on_new_interval on child PG after split

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #29774 into master

* refs/pull/29774/head:
qa/standalone/scrub/osd-scrub-snaps: snapmapper omap is now 'm'

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #29807 into master

* refs/pull/29807/head:
mgr/pg_autoscaler: fix race with pool deletion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17719 from mikulely/fix-usage-stats

rgw: distinguish different get_usage for usage log

Reviewed-by: Robin H. Johnson <rjohnson@digitalocean.com>

mgr/dashboard: run-backend-api-tests.sh CI improvements

As there is now a jenkins job to run this script
(see https://github.com/ceph/ceph-build/pull/1351),
this refactoring adapt the script to be run in a jenkins job as well as locally.

Signed-off-by: alfonsomthd <almartin@redhat.com>

Merge pull request #29544 from tchaikov/wip-doc-search-CSP

doc: always load resources via HTTPS

Reviewed-by: Tiago Melo <tmelo@suse.com>

doc: always load resources via HTTPS

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29755 from xiexingguo/wip-inc-recovery-4

osd: do not invalidate clear_regions of missing item at boot

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/BaseMgrModule: tolerate Int or Long for health 'count'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29804 from alfredodeza/wip-rm41378

ceph-volume tests set the noninteractive flag for Debian

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #29744 into master

* refs/pull/29744/head:
qa/run-standalone.sh: fix python path
qa/standalone/mon/health-mute.sh: fix up rachet test
qa/standalone/mon/health-mute.sh: s/kill daemons/kill_daemons/

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #29749 into master

* refs/pull/29749/head:
mon/HealthMonitor: remove unused label

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29757 into master

* refs/pull/29757/head:
osd: always initialize local variable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29763 into master

* refs/pull/29763/head:
qa/suites/rados: whitelist POOL_APP_NOT_ENABLED warning

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/pg_autoscaler: fix race with pool deletion

The pool_stats map comes from a get('df') that may not include a pool
because it was just deleted.

Fixes: https://tracker.ceph.com/issues/41386
Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume tests set the noninteractive flag for Debian, to avoid prompts in apt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #28378 into master

* refs/pull/28378/head:
qa/tasks: introduce Thrasher base class
qa/tasks: Fix typo
qa/tasks: manage thrashers
qa/tasks: start DaemonWatchdog when ceph starts
qa/tasks: make watch and bark handle more daemons
qa/tasks: move DaemonWatchdog to new file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29773 from dillaman/wip-41352

pybind/mgr/rbd_support: fix missing variable in error path

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #29659 from jan--f/c-v-simple-functional-no-lvm-zap

ceph-volume: don't try to test lvm zap on simple tests

qa/tasks: introduce Thrasher base class

* Introduced a Thrasher base class.
* Updated thrashers to inherit from Thrasher.
* Replaced the magic variable e with Thrasher.exception as per the discussion.
  Now the exception variable sets by default as the thrashers are inheriting
  from the Thrasher class.

Fixes: https://github.com/ceph/ceph/pull/28378#discussion_r309337928
Fixes: https://tracker.ceph.com/issues/41133
Signed-off-by: Jos Collin <jcollin@redhat.com>

rgw: distinguish different get_usage for usage log

get_usage op via s3 endpoint are not the same as get_usage
via admin endpoint in the rgw usage log categories.

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

mon/HealthMonitor: remove unused label

move the whole sanity checks into `HealthMonitor::preprocess_command()`.

this change silences warning of:

warning: label 'reply' defined but not used [-Wunused-label]

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PeeringState: semi-colon after DECLARE_LOCALS

This screws with emacs' auto-indent.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: on_new_interval on child PG after split

This was broken as part of the PeeringState refactor, see
ace6655f955837b104f307183fc0b988b7880c04.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29717 into master

* refs/pull/29717/head:
os/bluestore: prefix omap of temp objects by real pool

Reviewed-by: Sage Weil <sage@redhat.com>

qa/standalone/scrub/osd-scrub-snaps: snapmapper omap is now 'm'

...due to per-pool omap.

Fixes 91f533be71e5fd9b2c0135a5b54d663425a1d9c4

Fixes: https://tracker.ceph.com/issues/41353
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29759 into master

* refs/pull/29759/head:
qa/tasks: fixed typo in the comment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29625 from kamoltat/wip-test-progress-fix-bug-#29385

mgr/progress/module.py: s/events/_events/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29704 from tchaikov/wip-ceph-objectstore-tool

ceph-objectstore-tool: return 0 if incmap is sane

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #29655 from trociny/wip-40923

qa/workunits/rbd: stress test `rbd mirror pool status --verbose`

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

pybind/mgr/rbd_support: fix missing variable in error path

Fixes: https://tracker.ceph.com/issues/41352
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29376 from Songweibin/wip-rbd-display-id

rbd/action: display image id in rbd du/list output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29743 from smithfarm/wip-ceph-backport-https

script/ceph-backport.sh: carry https through to logical conclusion

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #29653 from lixiaoy1/AsyncOpTracker_dup

librbd: Remove duplicated AsyncOpTracker in librbd/Utils.h

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29094 from cbodley/wip-40806

radosgw-admin: bucket sync status not 'caught up' during full sync

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

osd/PeeringState: do not complain about past_intervals constrained by oldest epoch

The start of the required interval has a floor set by the oldest osdmap
epoch we have.  That can lead to an invalid/empty required interval
(because the start is >= the end), but the PG may still have past
intervals.  That can be cause by a slow PG deletion.

No need to complain about this harmless condition.

Fixes: https://tracker.ceph.com/issues/39546
Signed-off-by: Sage Weil <sage@redhat.com>

scripts/ceph-backport.sh: always use https://tracker.ceph.com

Completing the wave of fixes to this script in the wake of
https://tracker.ceph.com/issues/38764, this commit replaces
"http" with "https" in the comments and puts the Redmine endpoint
into a variable, along with some other cleanups.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #29762 from alfredodeza/bz-1738379

ceph-volume: use the OSD identifier when reporting success

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

qa/suites/rados: whitelist POOL_APP_NOT_ENABLED warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: login screen language dropdown enhancement (#29636)

mgr/dashboard: login screen language dropdown enhancement

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29737 from tspmelo/wip-label-alignment

mgr/dashboard: Vertically align the "Refresh" label

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Exclude some folders in the frontend project (#29708)

mgr/dashboard: Exclude some folders in the frontend project

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume devices.lvm zap use the identifier to report success

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests check success message when zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Hosts Page Service Links Test (#29516)

mgr/dashboard: Hosts Page Service Links Test

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29728 from tchaikov/wip-41330

cmake,run-make-check.sh: disable SPDK by default

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

qa/tasks: fixed typo in the comment

Signed-off-by: Jos Collin <jcollin@redhat.com>

mon: show pool id in pool ls command

Fixes: http://tracker.ceph.com/issues/40287
Signed-off-by: Chang Liu <liuchang0812@gmail.com>

osd: always initialize local variable

to silence a GCC warning like:

../src/osd/OSD.cc:4608:24: warning: ‘sender_delta_ub’ may be used
uninitialized in this function [-Wmaybe-uninitialized]
 4608 |       ceph::signedspan sender_delta_ub;
      |                        ^~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: add a process that handle Callback Arguments
in C_AckMarkedDown,we add a process that handle _finish(int) Callback Arguments

Fixes: https://tracker.ceph.com/issues/41217
Signed-off-by: NancySu05 <su_nan@inspur.com>

cmake,run-make-check.sh,deb,rpm: disable SPDK by default

but we still enable it in `run-make-check.sh`

* cmake: disable SPDK by default
* run-make-check.sh: enable WITH_SPDK so at least we can ensure it
  builds
* deb,rpm: add uuid-dev / libuuid-devel as a "make check" dependency

Fixes: https://tracker.ceph.com/issues/41330
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: make WITH_SPDK a dependent option

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: clear PG_STATE_CLEAN when repair object

there is a race be found, when we repair object on clean state,
we queue a DoRecovery peering event, but before the peering event
dequeue,a snaptrim event on the missing object's snap dequeue,
then we will get pass the context< SnapTrimmer >().can_trim()
and go to get the context of the missing object(snapdir)

we can avoid this by clear clean state when we found missing..

Fixes: https://tracker.ceph.com/issues/41348
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

osd: do not invalidate clear_regions of missing item at boot

Seems we'll always mark clear_regions as all dirty
when reading pg logs and missing items off the disk,
which as a result turns incremental recovery off by default.
Also using std::move seems to be a bit more efficient
and robust here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29753 from tchaikov/wip-sudo-sysctl

run-make-check.sh: run sudo with absolute path

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #29726 from xiexingguo/wip-inc-recovery-1

osd/PrimaryLogPG: fix dirty range of write_full

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29729 from xiexingguo/wip-inc-recovery-2

osd/PrimaryLogPG: finish_copyfrom - dirty omap if necessary

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

run-make-check.sh: run sudo with absolute path

there is chance that `/sbin` is not listed in `secure_path`, see
sudoers(5) for more details

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: merge replica log on primary need according to replica log's crt

Fixes: https://tracker.ceph.com/issues/41194
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

osd/OSDCap: Check for empty namespace

Fixes: http://tracker.ceph.com/issues/40835
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

qa/run-standalone.sh: fix python path

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/mon/health-mute.sh: fix up rachet test

Make sure we provide time for the mute to get cleared out by tick().

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29518 into master

* refs/pull/29518/head:
mount.ceph: properly handle -o strictatime

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29411 into master

* refs/pull/29411/head:
mds: fix InoTable::force_consume_to()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29543 into master

* refs/pull/29543/head:
mds: move some MDCache member init to header

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #29429 from dillaman/wip-40834

doc/rbd: initial kubernetes / ceph-csi integration documentation

Reviewed-by: ShyamsundarR <srangana@redhat.com>
Reviewed-by: Humble Chirammal <hchiramm@redhat.com>
Reviewed-by: Madhu Rajanna <madhupr007@gmail.com>

qa/standalone/mon/health-mute.sh: s/kill daemons/kill_daemons/

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29719 into master

* refs/pull/29719/head:
osd/PrimaryLogPG: kill obsolete ondisk_{read,write}_lock comments

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: E2E Test for RBD Mirroring and Images (#29381)

mgr/dashboard: E2E Test for RBD Mirroring and Images

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

os/bluestore: avoid constant name replica in bitmap allocator.
This was introduced by Adam's https://github.com/ceph/ceph/pull/29425

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: implement original allocator fragmentation inspection.

Using admin socket interface: bluestore allocator fragmentation
[block|bluefs-db|bluefs-wal|bluefs-slow]

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: simplify Allocator::get_fragmentation() signature
Remove alloc_unit parameter.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/allocator_test: additional coverage for get_fragmentation_score

Tests the function results agains the same cases as we do for
get_fragmentation.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Vertically align the "Refresh" label

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Exclude some folders in the frontend project

This will remove an warning in VS Code [1] and, more important, will reduce the
number of files that TS will watch when compiling or linting.

[1] To enable project-wide JavaScript/TypeScript language features,
exclude large folders with source files that you do not work on.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29685 from smithfarm/wip-41296

rpm: always build ceph-test package

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

Merge pull request #29556 from Devp00l/wip-41165

mgr/dashboard: CephFS client tab switch

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Verify fields on ISCSI page (#29608)

mgr/dashboard: Verify fields on ISCSI page

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: clean-up python unit tests (#28696)

mgr/dashboard: clean-up python unit tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

osd/PrimaryLogPG: finish_copyfrom - dirty omap if necessary

otherwise we'll at risk of losing track of omap entries.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PrimaryLogPG: fix dirty range of write_full

A write_full operation may implicitly truncate the object down,
hence we need to mark the truncated part as dirty as well since
follow-up randomized writes may still be able to (re)extend the
object size and leave some holes against the truncated part,
which as a result might cause problems during incremental-mode
recovery.

Note that write_update_size_and_usage would reset oi.size
and that's why we move the mark_data_region_dirty call before that.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29584 from tchaikov/wip-qa-test_envlibrados_for_rocksdb-cmake3

qa/workunits/rados/test_envlibrados_for_rocksdb: install newer cmake

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/workunits/rados/test_envlibrados_for_rocksdb: install newer cmake

rocksdb needs CMake 3.5.1 since
https://github.com/facebook/rocksdb/pull/5662

Fixes: https://tracker.ceph.com/issues/41253
Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: kill obsolete ondisk_{read,write}_lock comments

The ondisk_{read,write}_lock infrastructure was long gone with
https://github.com/ceph/ceph/pull/20177 merged - c244300ef33a044ad71fea7d92d77f33b5d41851,
to be specific. Hence the related comments must die since they
could be super-misleading.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29433 from dillaman/wip-41029

pybind/mgr/rbd_support: ignore missing support for RBD namespaces

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

os/bluestore: prefix omap of temp objects by real pool

For recovery or backfill, we use temp object at destination
if the whole object context can not sent in one shot.
And since https://github.com/ceph/ceph/pull/29292, we now segregate
omap keys by object's binding pool.
However, https://github.com/ceph/ceph/pull/29292 does not
work well for recovery or backfill process that need
to manipulate temp objects because we always (deliberately)
assign a negative pool id for each temp object which is
(obviously) different from the corresponding target object,
and we do not fix it when trying to rename the temp object
back to the target object at the end of recovery/backfill,
as a result of which we totally lose track of the omap
portion of the recovered object.

Fix by prefixing all omap-related stuff of temp objects
by using its real(pg's) pool.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds: move some MDCache member init to header

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>