doc: Document network performance monitoring

Signed-off-by: David Zafman <dzafman@redhat.com>

osd doc mon mgr: To milliseconds for config value, user input and threshold out

Signed-off-by: David Zafman <dzafman@redhat.com>

osd mon mgr: Convert all network ping time output to milliseconds

To output milliseconds (usec / 1000), treat as fixed point integers

Signed-off-by: David Zafman <dzafman@redhat.com>

common: Add support routines to generate strings for fixed point

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add basic test for network ping tracking

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add debug_heartbeat_testing_span to allow quicker testing

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add debug_disable_randomized_ping config for use in testing

Signed-off-by: David Zafman <dzafman@redhat.com>

osd mgr: Add osd_mon_heartbeat_stat_stale option to time out ping info
after 1 hour

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Indicate when an osd with slow ping time is down

Signed-off-by: David Zafman <dzafman@redhat.com>

osd mon: Add last_update to osd_stat_t heartbeat info

Ignore old heartbeat info which hasn't updated

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: After first interval populate vectors so 5min/15min values aren't 0

Signed-off-by: David Zafman <dzafman@redhat.com>

osd mgr: Store last pingtime for possible graphing

Signed-off-by: David Zafman <dzafman@redhat.com>

osd mgr: Add minimum and maximum tracking to network ping time

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Add documentation and release notes

Signed-off-by: David Zafman <dzafman@redhat.com>

osd mgr mon: Add mon_warn_on_slow_ping_ratio config as 5% of osd_heartbeat_grace

Compute network ping threshold based on ratio (5% of 20 seconds is 1 second)
Make the threshold value used part of dump_osd_network for osd and mgr
Keep mon_warn_on_slow_ping_time (default 0) to optionally override the ratio

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: Add "dump_osd_network" mgr admin request to get a sorted report

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add "dump_osd_network" osd admin request to get a sorted report

Signed-off-by: David Zafman <dzafman@redhat.com>

osd mon: Track heartbeat ping times and report health warning

Fixes: http://tracker.ceph.com/issues/40640
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #29715 into master

* refs/pull/29715/head:
qa: fix broken ceph.restart marking of OSDs down
qa: add debugging failed osd-release setting

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29821 into master

* refs/pull/29821/head:
qa: stop DaemonWatchdog for each cluster in daemon roles

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #29575 into master

* refs/pull/29575/head:
objclass, osd: improve const-correctness of PGLSFilter.
common: add bl::contents_equal() override for void* + size_t.
osd: refactor manufacturing of PGLSFilter.
osd: don't carry PGLSFilter between multiple ops in MOSDOp.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #28727 into master

* refs/pull/28727/head:
test/crimson: resolve name collision
test: switch to ldout; let users specify mon debug level
test: add new ElectionLogic unit test framework
elector: const-ify a bunch of functions
elector: swap order of parameters in ElectionLogic::receive_propose
elector: Update Elector and ElectionLogic function documentation
elector: persist the epoch in bump_epoch()
elector: make some more ElectionLogic members private
elector: fix privacy and restore dout in Elector
elector: don't clear peer_info in bump_epoch()
elector: split ElectionLogic into its own compilation unit
elector: move all the elector callouts into the Elector
elector: make ElectionLogic private to Elector; undo most public shenanigans
elector: create declare_standlone_victory in Elector/Logic for Monitor
elector: make ElectionLogic::declare_victory private
elector: route _bump_epoch through the interface-to-be
elector: rename handle_propose_logic -> receive_propose
elector: hoist handle_victory into ElectionLogic
elector: hoist handle_ack into ElectionLogic
elector: hoist victory into ElectionLogic
elector: hoist expire into ElectionLogic
elector: hoist start into ElectionLogic
elector: hoist participating into ElectionLogic
elector: hoist init into ElectionLogic
elector: hoist defer into ElectionLogic
elector: split handle_propose in two and hoist into ElectionLogic
elector: hoist bump_epoch into ElectionLogic
elector: store accessors for ElectionLogic
elector: hoist Elector data bits out into a new ElectionLogic class
mon: Rearrange Paxos::dispatch to be a little cleaner

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #15183 into master

* refs/pull/15183/head:
kv/rocksdb: support rmrange unconditionally
cls/rgw: rgw_bi_log_trim() uses cls_cxx_map_remove_range()
cls/log: cls_log_trim() uses cls_cxx_map_remove_range()
test/cls: add cls_log.trim_by_marker test
test/cls: test_cls_log doesn't allocate ObjectOperations
test/cls: test_cls_log uses fixture for temporary pool
test/cls: add cls_rgw.bi_log_trim test
cls/rgw: expose cls_rgw_bilog_list/trim() for single shard
test/cls: test_cls_rgw uses cls_rgw_obj_key
test/cls: test_cls_rgw doesn't allocate ObjectOperations
test/cls: test_cls_rgw uses fixture for temporary pool
objclass: add cls_cxx_map_remove_range()
librados: add rados_write_op_omap_rm_range2()
osdc: add Objecter omap_rm_range()
osd: add CEPH_OSD_OP_OMAPRMKEYRANGE to do_osd_ops()
osd: add omap_rmkeyrange() to PGTransaction
os: add bufferlist overload for omap_rmkeyrange()
tracing: add do_osd_op_pre_omaprmkeyrange
rados: add CEPH_OSD_OP_OMAPRMKEYRANGE

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29778 from cbodley/wip-41212

vstart: move [client.rgw] config into [client]

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mgr/dashboard: User Management E2E tests (#29641)

mgr/dashboard: User Management E2E tests

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: run-backend-api-tests.sh CI improvements (#29504)

mgr/dashboard: run-backend-api-tests.sh CI improvements

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #29590 from Aran85/fix_proc_replica_log

osd: merge replica log on primary need according to replica log's crt

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29747 from liewegas/wip-39546

osd/PeeringState: do not complain about past_intervals constrained by oldest epoch

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29624 from NancySu05/osdmonitor_markmedown

mon:C_AckMarkedDown has not handled the Callback Arguments

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29738 from ifed01/wip-ifed-alloc-cleanup

os/bluestore: minor improvements/cleanup around allocator

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #29614 from votdev/issue_41205

mgr/dashboard: Access control database does not restore disabled users correctly

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #29146 from badone/wip-tracker-40835-OSDCap.PoolClassRNS-abort

osd/OSDCap: Check for empty namespace

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25697 from Aran85/fix-onode-trim

os/bluestore: more aggressive deferred submit when onode trim skipping

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #28488 from liuchang0812/show-pool-id-in-pool-ls-cmd

mon: show pool id in pool ls command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24636 from rzarzynski/wip-denc-container_base

denc: slightly optimize container_base::bound_encode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29756 from Aran85/fix-repair-object

osd: clear PG_STATE_CLEAN when repair object

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #29806 into master

* refs/pull/29806/head:
mgr/BaseMgrModule: tolerate Int or Long for health 'count'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29298 from zhangsw/rgw-fix-bug-listobjv2-startafter

rgw: continuationToken or startAfter shouldn't be returned if not specified

Merge PR #29780 into master

* refs/pull/29780/head:
osd/PeeringState: semi-colon after DECLARE_LOCALS
osd/PeeringState: on_new_interval on child PG after split

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #29774 into master

* refs/pull/29774/head:
qa/standalone/scrub/osd-scrub-snaps: snapmapper omap is now 'm'

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #29807 into master

* refs/pull/29807/head:
mgr/pg_autoscaler: fix race with pool deletion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: stop DaemonWatchdog for each cluster in daemon roles

Fixes: https://tracker.ceph.com/issues/41398
Introduced-by: 08b99eef277b00a3ea423cbf085bd114a805813f
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix broken ceph.restart marking of OSDs down

Sage noticed `osd down` was not being performed. Bug was that the role
format had changed so splitting no longer worked correctly.

Fixes: https://tracker.ceph.com/issues/40773
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17719 from mikulely/fix-usage-stats

rgw: distinguish different get_usage for usage log

Reviewed-by: Robin H. Johnson <rjohnson@digitalocean.com>

qa: add debugging failed osd-release setting

See-also: https://tracker.ceph.com/issues/40773
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: run-backend-api-tests.sh CI improvements

As there is now a jenkins job to run this script
(see https://github.com/ceph/ceph-build/pull/1351),
this refactoring adapt the script to be run in a jenkins job as well as locally.

Signed-off-by: alfonsomthd <almartin@redhat.com>

Merge pull request #29544 from tchaikov/wip-doc-search-CSP

doc: always load resources via HTTPS

Reviewed-by: Tiago Melo <tmelo@suse.com>

doc: always load resources via HTTPS

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29755 from xiexingguo/wip-inc-recovery-4

osd: do not invalidate clear_regions of missing item at boot

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/BaseMgrModule: tolerate Int or Long for health 'count'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29804 from alfredodeza/wip-rm41378

ceph-volume tests set the noninteractive flag for Debian

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #29744 into master

* refs/pull/29744/head:
qa/run-standalone.sh: fix python path
qa/standalone/mon/health-mute.sh: fix up rachet test
qa/standalone/mon/health-mute.sh: s/kill daemons/kill_daemons/

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #29749 into master

* refs/pull/29749/head:
mon/HealthMonitor: remove unused label

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29757 into master

* refs/pull/29757/head:
osd: always initialize local variable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29763 into master

* refs/pull/29763/head:
qa/suites/rados: whitelist POOL_APP_NOT_ENABLED warning

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/pg_autoscaler: fix race with pool deletion

The pool_stats map comes from a get('df') that may not include a pool
because it was just deleted.

Fixes: https://tracker.ceph.com/issues/41386
Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume tests set the noninteractive flag for Debian, to avoid prompts in apt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #28378 into master

* refs/pull/28378/head:
qa/tasks: introduce Thrasher base class
qa/tasks: Fix typo
qa/tasks: manage thrashers
qa/tasks: start DaemonWatchdog when ceph starts
qa/tasks: make watch and bark handle more daemons
qa/tasks: move DaemonWatchdog to new file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29773 from dillaman/wip-41352

pybind/mgr/rbd_support: fix missing variable in error path

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #29659 from jan--f/c-v-simple-functional-no-lvm-zap

ceph-volume: don't try to test lvm zap on simple tests

qa/tasks: introduce Thrasher base class

* Introduced a Thrasher base class.
* Updated thrashers to inherit from Thrasher.
* Replaced the magic variable e with Thrasher.exception as per the discussion.
  Now the exception variable sets by default as the thrashers are inheriting
  from the Thrasher class.

Fixes: https://github.com/ceph/ceph/pull/28378#discussion_r309337928
Fixes: https://tracker.ceph.com/issues/41133
Signed-off-by: Jos Collin <jcollin@redhat.com>

rgw: distinguish different get_usage for usage log

get_usage op via s3 endpoint are not the same as get_usage
via admin endpoint in the rgw usage log categories.

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

mon/HealthMonitor: remove unused label

move the whole sanity checks into `HealthMonitor::preprocess_command()`.

this change silences warning of:

warning: label 'reply' defined but not used [-Wunused-label]

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: resolve name collision

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd/PeeringState: semi-colon after DECLARE_LOCALS

This screws with emacs' auto-indent.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: on_new_interval on child PG after split

This was broken as part of the PeeringState refactor, see
ace6655f955837b104f307183fc0b988b7880c04.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29717 into master

* refs/pull/29717/head:
os/bluestore: prefix omap of temp objects by real pool

Reviewed-by: Sage Weil <sage@redhat.com>

qa/standalone/scrub/osd-scrub-snaps: snapmapper omap is now 'm'

...due to per-pool omap.

Fixes 91f533be71e5fd9b2c0135a5b54d663425a1d9c4

Fixes: https://tracker.ceph.com/issues/41353
Signed-off-by: Sage Weil <sage@redhat.com>

vstart: move [client.rgw] config into [client]

common rgw config was moved into [client.rgw] with the intent to
cover radosgw in multisite tests too, but this broke the default case
where radosgws are named after [client.rgw.port]. move the common stuff
to [client] so it does actually apply to both

Fixes: https://tracker.ceph.com/issues/41212
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29759 into master

* refs/pull/29759/head:
qa/tasks: fixed typo in the comment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29625 from kamoltat/wip-test-progress-fix-bug-#29385

mgr/progress/module.py: s/events/_events/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29704 from tchaikov/wip-ceph-objectstore-tool

ceph-objectstore-tool: return 0 if incmap is sane

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #29655 from trociny/wip-40923

qa/workunits/rbd: stress test `rbd mirror pool status --verbose`

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

pybind/mgr/rbd_support: fix missing variable in error path

Fixes: https://tracker.ceph.com/issues/41352
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29376 from Songweibin/wip-rbd-display-id

rbd/action: display image id in rbd du/list output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29743 from smithfarm/wip-ceph-backport-https

script/ceph-backport.sh: carry https through to logical conclusion

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #29653 from lixiaoy1/AsyncOpTracker_dup

librbd: Remove duplicated AsyncOpTracker in librbd/Utils.h

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29094 from cbodley/wip-40806

radosgw-admin: bucket sync status not 'caught up' during full sync

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

osd/PeeringState: do not complain about past_intervals constrained by oldest epoch

The start of the required interval has a floor set by the oldest osdmap
epoch we have.  That can lead to an invalid/empty required interval
(because the start is >= the end), but the PG may still have past
intervals.  That can be cause by a slow PG deletion.

No need to complain about this harmless condition.

Fixes: https://tracker.ceph.com/issues/39546
Signed-off-by: Sage Weil <sage@redhat.com>

scripts/ceph-backport.sh: always use https://tracker.ceph.com

Completing the wave of fixes to this script in the wake of
https://tracker.ceph.com/issues/38764, this commit replaces
"http" with "https" in the comments and puts the Redmine endpoint
into a variable, along with some other cleanups.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #29762 from alfredodeza/bz-1738379

ceph-volume: use the OSD identifier when reporting success

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

qa/suites/rados: whitelist POOL_APP_NOT_ENABLED warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: User Management E2E tests

Test breadcrumbs for user management pages
Test creating, editing and deleting a user
Test creating, editing and deleting a role

Fixes: https://tracker.ceph.com/issues/41231
Fixes: https://tracker.ceph.com/issues/41232
Fixes: https://tracker.ceph.com/issues/41233
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

mgr/dashboard: login screen language dropdown enhancement (#29636)

mgr/dashboard: login screen language dropdown enhancement

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29737 from tspmelo/wip-label-alignment

mgr/dashboard: Vertically align the "Refresh" label

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Exclude some folders in the frontend project (#29708)

mgr/dashboard: Exclude some folders in the frontend project

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume devices.lvm zap use the identifier to report success

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests check success message when zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Hosts Page Service Links Test (#29516)

mgr/dashboard: Hosts Page Service Links Test

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29728 from tchaikov/wip-41330

cmake,run-make-check.sh: disable SPDK by default

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

qa/tasks: fixed typo in the comment

Signed-off-by: Jos Collin <jcollin@redhat.com>

mon: show pool id in pool ls command

Fixes: http://tracker.ceph.com/issues/40287
Signed-off-by: Chang Liu <liuchang0812@gmail.com>

osd: always initialize local variable

to silence a GCC warning like:

../src/osd/OSD.cc:4608:24: warning: ‘sender_delta_ub’ may be used
uninitialized in this function [-Wmaybe-uninitialized]
 4608 |       ceph::signedspan sender_delta_ub;
      |                        ^~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: add a process that handle Callback Arguments
in C_AckMarkedDown,we add a process that handle _finish(int) Callback Arguments

Fixes: https://tracker.ceph.com/issues/41217
Signed-off-by: NancySu05 <su_nan@inspur.com>

cmake,run-make-check.sh,deb,rpm: disable SPDK by default

but we still enable it in `run-make-check.sh`

* cmake: disable SPDK by default
* run-make-check.sh: enable WITH_SPDK so at least we can ensure it
  builds
* deb,rpm: add uuid-dev / libuuid-devel as a "make check" dependency

Fixes: https://tracker.ceph.com/issues/41330
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: make WITH_SPDK a dependent option

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: clear PG_STATE_CLEAN when repair object

there is a race be found, when we repair object on clean state,
we queue a DoRecovery peering event, but before the peering event
dequeue,a snaptrim event on the missing object's snap dequeue,
then we will get pass the context< SnapTrimmer >().can_trim()
and go to get the context of the missing object(snapdir)

we can avoid this by clear clean state when we found missing..

Fixes: https://tracker.ceph.com/issues/41348
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

osd: do not invalidate clear_regions of missing item at boot

Seems we'll always mark clear_regions as all dirty
when reading pg logs and missing items off the disk,
which as a result turns incremental recovery off by default.
Also using std::move seems to be a bit more efficient
and robust here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29753 from tchaikov/wip-sudo-sysctl

run-make-check.sh: run sudo with absolute path

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #29726 from xiexingguo/wip-inc-recovery-1

osd/PrimaryLogPG: fix dirty range of write_full

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>