Merge PR #29514 into master

* refs/pull/29514/head:
mgr/hello: typos
mgr/hello: a better command example
mgr/hello: boilerplate for handling config options
mgr/hello: fully document our options

Reviewed-by: Tim Serong <tserong@suse.com>

Merge PR #29546 into master

* refs/pull/29546/head:
mgr/telemetry: clear the event after being awaken by it

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Prevent clone when layering not enabled on pare… (#29317)

mgr/dashboard: Prevent clone when layering not enabled on parent image

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: fix RBD snapshot protected label (#29534)

mgr/dashboard: fix RBD snapshot protected label

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Verify fields on Crush map page (#29452)

mgr/dashboard: Verify fields on Crush map page

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Verify fields on Monitors page (#29450)

mgr/dashboard: Verify fields on Monitors page

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: E2E Dashboard Page Test Suite (#29432)

mgr/dashboard: E2E Dashboard Page Test Suite

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Verify manager modules have editing functionali… (#29407)

mgr/dashboard: Verify manager modules have editing functionality

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/telemetry: clear the event after being awaken by it

otherwise telemetry will have a busy-loop once it's signaled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29494 from dzafman/wip-scrub-test

test: Bump sleep time for slower machines

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28813 from smanjara/wip-user-rename-working

rgw : Bucket mv, bucket chown and user rename utilities

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29425 from aclamk/wip-bluestore-monitor-allocations

[bluestore][tools] Inspect allocations in bluestore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/rados/operations/health-checks: document BlueStore fragmentation and BlueFS space available features

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

mgr/dashboard: fix RBD snapshot protected label

Fixes: https://tracker.ceph.com/issues/41153
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #29438 from wjwithagen/wjw-fix-ceph-backport.sh-https-tracker

script/ceph-backport.sh: Use secure access for tracker.ceph.com

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/hello: typos

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/hello: a better command example

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/hello: boilerplate for handling config options

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: E2E Dashboard Page Test Suite

Fixes: https://tracker.ceph.com/issues/40549
Fixes: https://tracker.ceph.com/issues/40550
Fixes: https://tracker.ceph.com/issues/40561
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

Merge pull request #29525 from tchaikov/wip-mgr/dashboard/run-backend-api-tests.sh

mgr/dashboard: add python-common to $PYTHONPATH

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #29451 from rhcs-dashboard/wip-41047-master

github: Add CODEOWNERs for designated code-owner reviews

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: fix HACKING.rst is not rendered on github (#29521)

mgr/dashboard: fix HACKING.rst is not rendered on github

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #29461 from tchaikov/wip-mgr/dashboard/test_mgr_module

qa/tasks/mgr/dashboard/test_mgr_module: sync w/ telemetry

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/dashboard/test_mgr_module: sync w/ telemetry

* use primitive types instead of `JLeaf(the_type)` as they are
  equivalent in this context
* remove fields which are added only if certain channels are
  activated.
* allow unknown fields, as we are including various stuff
  in the report, for instance, osdmap, usage, crash info, etc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: add python-common to $PYTHONPATH

fix the regression introduced by 8c50be5df6, so ceph-mgr's python
modules are able to import python-common.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29382 from MI-OSiRIS/prometheus_scrape_interval

mgr/prometheus: Cast collect_timeout (scrape_interval) to float

mgr/dashboard: fix HACKING.rst is not rendered on github

The file is not rendered because of inconsistent section markers.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #29116 into master

* refs/pull/29116/head:
osd: move heartbeat connection cleanup to helper
osd: break con<->session cycle when removing heartbeat peers
osd: mark down heartbeat connections on shutdown
crimson/: move get_mnow() to ShardServices, pass to heartbeat
crimson/osd: stubs for get_mnow, get_hbstamps
crimson/osd/heartbeat: adapt to new MOSDPing fields
crimson/osdmap_service: add get_mnow(), get_up_epoch()
osd/PeeringState: take HeartbeatStamps refs for current interval
osd: track clock delta between peer OSDs
osd: add get_mnow() interface to OSDService, PG, PeeringState
osd: record startup_time
osd: some minor refactoring/cleanup in handle_osd_ping

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #29498 into master

* refs/pull/29498/head:
qa/tasks/ceph_manager: remove race from all_active_or_peered()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

script/ceph-backport.sh: Use secure access for tracker.ceph.com

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #29509 into master

* refs/pull/29509/head:
doc: indicate imperative mood for commit titles

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: indicate imperative mood for commit titles

This is convention in the kernel and Ceph but is not clearly documented.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Write E2E test to verify that Manager modules have editing functionality

Fixes: https://tracker.ceph.com/issues/40823
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

Merge pull request #29501 from tchaikov/wip-journal/test_JournalRecorder.cc

journal: always shutdown JournalRecoreder before destructing it

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27859 from jan--f/update-bluestore-cache-doc

doc: update bluestore cache settings and clarify data fraction

test/objectstore: Allocator_test. Add test for dumping free regions and fragmentation_score.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

BlueStore/allocator: Add command to inspect how much BlueStore's block can go to BlueFS.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

journal: always shutdown JournalRecoreder before destructing it

otherwise when we destruct `journal::JournalRecorder::m_object_locks`,
it/they would be still being waited by some condition variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29475 from tchaikov/wip-cmake3-for-xenial

install-deps.sh: use chacra for cmake repo

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/hello: fully document our options

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: remove race from all_active_or_peered()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move heartbeat connection cleanup to helper

Signed-off-by: Sage Weil <sage@redhat.com>

osd: break con<->session cycle when removing heartbeat peers

Signed-off-by: Sage Weil <sage@redhat.com>

osd: mark down heartbeat connections on shutdown

This ensures that heartbeat_reset() gets call and we clean up the
ref loop between the Connections and Sessions.

Signed-off-by: Sage Weil <sage@redhat.com>

crimson/: move get_mnow() to ShardServices, pass to heartbeat

get_mnow isn't clearly at home in OSDMapService, and the other methods
are needed from PeeringState, so let's consolidate on ShardServices
for now.  We probably ought OSDMapService state out of OSD into its
own module at some point.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: stubs for get_mnow, get_hbstamps

Signed-off-by: Sage Weil <sage@redhat.com>

crimson/osd/heartbeat: adapt to new MOSDPing fields

Signed-off-by: Sage Weil <sage@redhat.com>

crimson/osdmap_service: add get_mnow(), get_up_epoch()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: take HeartbeatStamps refs for current interval

Signed-off-by: Sage Weil <sage@redhat.com>

osd: track clock delta between peer OSDs

We need to keep track of the monotonic clock deltas between peer OSDs in
order to be able to exchange timestamps across messages.  We need an
upper and lower bound on this delta, depending on the context where it is
used.

We can use the existing ping messages to get this by assuming that a ping
message is sent instantly to get a bound, and to share our delta in a
follow-up reply to share the other bound.  The ping sender will get both
bounds with a single ping + ping_reply exchange.  The ping receiver will
get the delta value from the next round's ping.

Include up_from in the ping messages to ensure we don't mix up different
instances of the same OSD.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: add get_mnow() interface to OSDService, PG, PeeringState

Single helper to provide a monotonic time offset from startup_time.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: record startup_time

Signed-off-by: Sage Weil <sage@redhat.com>

osd: some minor refactoring/cleanup in handle_osd_ping

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29337 into master

* refs/pull/29337/head:
mon: enable telemetry module by default
mgr/telemetry: force re-opt-in if the report contents change
mgr/telemetry: less noise in the log
mgr/telemetry: wake up serve on config change
mgr/telemetry: track telemetry report revisions

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29457 from tchaikov/wip-consolidate-run-tox

cmake: one run_tox.sh to rule them all

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

script/run_tox.sh: add --help option

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #29492 into master

* refs/pull/29492/head:
doc: Fix for new ceph-devel mailing list

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

cmake: pass tox envs to run_tox.sh explicitly

* add two optional keyword parameters to `add_tox_test()`
* use the default `TOX_PATH` parameter instead passing it explicitly,
  as it is always `${CMAKE_CURRENT_SOURCE_DIR}`
* drop the code to guess the tox envs in `run_tox.sh`, and always pass
  them explicitly in CMake script using the `--tox-envs` argument.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29471 from tchaikov/wip-googletest

googletest: pick up change to suppress CMP0048 warning

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test: Bump sleep time for slower machines

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Fix for new ceph-devel mailing list

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #29458 from mikulely/rgw-drop-flush

rgw: drop dead flush_read_list declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27675 from jan--f/c-v-console-logging-to-stderr

ceph-volume: print most logging messages to stderr

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

github: Add CODEOWNERs for component/team reviews

Adds [CODEOWNERS](https://help.github.com/en/articles/about-code-owners)
file and assigns ownership of /src/pybind/mgr/dashboard to
@ceph/dashboard alias. That will also automatically add @ceph/dashboard
team to all reviews modifying any file in the above location.

Besides, if "Require review from Code Owners" is marked in the Branch
Protection Rules of any branch, where this CODEOWNERS file is propagated
to, at least one review from a designated CODEOWNER will be mandatory.

If Ceph teams' visibility is changed to public, teams can also be
@mentioned (both from Ceph and other projects), so it's easier to reach
to component-team members.

Fixes: https://tracker.ceph.com/issues/41047
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

cmake: move tests to where they are implemented

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not add gtest-parallel_ext if already added

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use script/run_tox.sh for running tox for dashboard

* adapt script/run_tox.sh to use different envs for dashboard's tox
  test.
* use script/run_tox.sh for running dashboard's tox test
* remove pybind/mgr/dashboard/run-tox.sh

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules/AddCephTest.cmake: make ${name}-venv a target

so we can prepare the venv for tox with `make tests`. and it's optional,
as run_tox.sh will check for it and setup a venv if it's not around.

Signed-off-by: Kefu Chai <kchai@redhat.com>

src: remove unused run-tox.sh

they are not used anymore after b9c9f74f

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29393 from ifed01/wip-ifed-bluestore-header-cleanup

os/bluestore: remove unused _tune_cache_size() method declaration

Reviewed-by: Mark Nelson <mnelson@redhat.com>

mgr/dashboard: update npm packages (#28343)

mgr/dashboard: update npm packages

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29403 from rhcs-dashboard/wip-41013-master

install-deps.sh: remove failing error catching

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29476 from tchaikov/wip-test/journal-close-object

test/journal: always close object

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Added test to verify user rename failure.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

test/journal: always close object

otherwise when we destruct `TestObjectRecorder::Handler::lock` or
`journal::JournalRecorder::m_object_locks`, it/they would be still
being waited by some condition variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29474 from wjwithagen/wjw-fix-rbd_ggate_Server.cc

rbd-ggate: fix compile errors from ceph::mutex update

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd-ggate: fix compile errors from ceph::mutex update

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

install-deps.sh: use chacra for cmake repo

two reasons:

- do not rely on kitware repo so we have better control of the
  cmake3: the `debian` directory is tracked by
  https://github.com/tchaikov/ceph-cmake
- chacra repo also offers aarch64 build of cmake3

Signed-off-by: Kefu Chai <kchai@redhat.com>

googletest: pick up change to suppress CMP0048 warning

newer cmake complains about googletest should have set CMP0048

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29472 from wjwithagen/wjw-fix-src/osdc/ObjectCacher.cc

osdc: Fix a missing ':' for the correct namespace

Reviewed-by: Kefu Chai <kchai@redhat.com>

osdc: Fix a missing ':' for the correct namespace

Clang complains:
/home/jenkins/workspace/ceph-master/src/osdc/ObjectCacher.cc:1848:22: error: no viable constructor or deduction guide for deduction of template arguments of 'unique_lock'
    std::unique_lock l{lock, std:adopt_lock};

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #29322 from rzarzynski/wip-objclass-cleanup

objclass, osd: clean up the cls-host interface. Turn ClassHandler into singleton

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29113 from tchaikov/wip-ceph-mutex

src/: s/Mutex/ceph::mutex/

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

common/Mutex: remove Mutex

as it's replaced by `ceph::mutex`

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Cond.h: remove Cond

as it's replaced by ceph::condition_variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: support is_locked() in non-debug mode

`PG` print details info in the prefix of logging messages if the PG is
being locked by current thread. but `ceph::mutex` is an alias of
`std::mutex` in non-debug mode, so neither `mutex::is_locked_by_me()` nor
`mutex::is_locked()` is supported in non-debug mode. to continue supporting
this feature, `PG::locked_by` is added to memorize the thread id of the owner
of the lock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ceph_context.cc: include necessary header

for using functions like `lockdep_unregister_ceph_context()`

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/testmsgr: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librados: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/compressor: do not include unused header

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/common/Throttle: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/obj_bencher: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth: remove unused forwarde declaration

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librados_test_stub: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

libcephfs: remove unused #include

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls_kvs: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush: remove unused #include

Signed-off-by: Kefu Chai <kchai@redhat.com>

fusetrace: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rbd_nbd: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rbd_ggate: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>