mgr/dashboard: adapt bucket tenant tests to new behaviour

Fixes: https://tracker.ceph.com/issues/41175
Signed-off-by: alfonsomthd <almartin@redhat.com>

Merge pull request #29400 from wjwithagen/wjw-fix-do_freeBSD.sh

do_freebsd.sh: update build scripts to resemble Jenkins scripts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29495 from ifed01/wip-ifed-finisher-improve

common/Finisher: remove some lock acquisitions.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29477 from tchaikov/wip-osd-pg-as-mutex

osd: pg as a mutex

Reviewed-by: Greg Farnum gfarnum@redhat.com
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #29486 from runsisi/wip-fix-verbose

ceph.in: fix verbose print

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29465 from penglaiyxy/wip_bluestore_caculated_revert

os/bluestore: no need to add tail length (revert PR#29185)

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29488 from majianpeng/bluestore-remove-lock

os/bluestore: no need protected by OpSequencer::qlock.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29385 from kamoltat/wip-qa-tasks-mgr-test-progress-bug-fix

qa/tasks/mgr/test_progress.py: fix bug in 9b4dbf0

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29365 from liewegas/wip-telemetry-post

telemetry/server: misc fixes

Merge pull request #29541 from yuriw/wip-yuriw-upgrade-rbd-master

qa/tests: added rbd task on ec

Merge PR #27542 into master

* refs/pull/27542/head:
qa/tasks: Fix ambiguous store_thrash, thrash_store

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #29493 into master

* refs/pull/29493/head:
qa/tasks/mgr/mgr_test_case: get mgrmap from 'mgr dump', not status
qa/tasks/ceph_manager: no newlines in 'ceph -s' output
mon: make mon summary more concise in 'ceph -s'
mon/MgrStatMonitor: set initial service_map 'modified' to cluster mkfs
mon: remove double-nesting of "osdmap" for ceph status
mon/MgrMap: make print_summary (used by 'ceph -s') more concise

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29511 into master

* refs/pull/29511/head:
common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env vars
common/config: let diff show non-build defaults
common/config: do no include multiple 'default' values

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29517 into master

* refs/pull/29517/head:
osd: include PastInterals in pg_notify_t

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #29559 from adamemerson/wip-putobj-crash

rgw: Move upload_info declaration out of conditional

Reviewed-By: Casey Bodley <cbodley@redhat.com>

rgw: Move upload_info declaration out of conditional

That way we don't deconstruct it right after assigning a reference to
part of it.

Fixes: https://tracker.ceph.com/issues/41172
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #29449 from cbodley/wip-vstart-client-rgw

vstart: move common rgw config to [client.rgw]

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

qa/tests: added rbd task on ec

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

osd: include PastInterals in pg_notify_t

We use a pair<pg_notify_t,PastIntervals> everywhere a pg_notify_t is used.
This is silly; just make it a member instead.

Include some minor compat cruft so we can speak to pre-octopus OSDs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29538 from cbodley/wip-rgw-data-changes-shared-unlock

rgw: fix unlock of shared lock in RGWDataChangesLog

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29468 from dillaman/wip-41032

pybind/mgr/rbd_support: use image ids to detect duplicate tasks

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #29514 into master

* refs/pull/29514/head:
mgr/hello: typos
mgr/hello: a better command example
mgr/hello: boilerplate for handling config options
mgr/hello: fully document our options

Reviewed-by: Tim Serong <tserong@suse.com>

qa/tasks/mgr/mgr_test_case: get mgrmap from 'mgr dump', not status

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29546 into master

* refs/pull/29546/head:
mgr/telemetry: clear the event after being awaken by it

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Prevent clone when layering not enabled on pare… (#29317)

mgr/dashboard: Prevent clone when layering not enabled on parent image

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: fix RBD snapshot protected label (#29534)

mgr/dashboard: fix RBD snapshot protected label

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Verify fields on Crush map page (#29452)

mgr/dashboard: Verify fields on Crush map page

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Verify fields on Monitors page (#29450)

mgr/dashboard: Verify fields on Monitors page

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: E2E Dashboard Page Test Suite (#29432)

mgr/dashboard: E2E Dashboard Page Test Suite

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Verify manager modules have editing functionali… (#29407)

mgr/dashboard: Verify manager modules have editing functionality

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/telemetry: clear the event after being awaken by it

otherwise telemetry will have a busy-loop once it's signaled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29494 from dzafman/wip-scrub-test

test: Bump sleep time for slower machines

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28813 from smanjara/wip-user-rename-working

rgw : Bucket mv, bucket chown and user rename utilities

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29425 from aclamk/wip-bluestore-monitor-allocations

[bluestore][tools] Inspect allocations in bluestore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: fix unlock of shared lock in RGWDataChangesLog

std::shared_mutex expects a call to unlock_shared() after lock_shared().
use the std::shared_lock guard to make it more obviously correct

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rados/operations/health-checks: document BlueStore fragmentation and BlueFS space available features

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

mgr/dashboard: fix RBD snapshot protected label

Fixes: https://tracker.ceph.com/issues/41153
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #29438 from wjwithagen/wjw-fix-ceph-backport.sh-https-tracker

script/ceph-backport.sh: Use secure access for tracker.ceph.com

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/hello: typos

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/hello: a better command example

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/hello: boilerplate for handling config options

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: E2E Dashboard Page Test Suite

Fixes: https://tracker.ceph.com/issues/40549
Fixes: https://tracker.ceph.com/issues/40550
Fixes: https://tracker.ceph.com/issues/40561
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

Merge pull request #29525 from tchaikov/wip-mgr/dashboard/run-backend-api-tests.sh

mgr/dashboard: add python-common to $PYTHONPATH

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #29451 from rhcs-dashboard/wip-41047-master

github: Add CODEOWNERs for designated code-owner reviews

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: fix HACKING.rst is not rendered on github (#29521)

mgr/dashboard: fix HACKING.rst is not rendered on github

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #29461 from tchaikov/wip-mgr/dashboard/test_mgr_module

qa/tasks/mgr/dashboard/test_mgr_module: sync w/ telemetry

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/dashboard/test_mgr_module: sync w/ telemetry

* use primitive types instead of `JLeaf(the_type)` as they are
  equivalent in this context
* remove fields which are added only if certain channels are
  activated.
* allow unknown fields, as we are including various stuff
  in the report, for instance, osdmap, usage, crash info, etc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: add python-common to $PYTHONPATH

fix the regression introduced by 8c50be5df6, so ceph-mgr's python
modules are able to import python-common.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29382 from MI-OSiRIS/prometheus_scrape_interval

mgr/prometheus: Cast collect_timeout (scrape_interval) to float

mgr/dashboard: fix HACKING.rst is not rendered on github

The file is not rendered because of inconsistent section markers.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #29116 into master

* refs/pull/29116/head:
osd: move heartbeat connection cleanup to helper
osd: break con<->session cycle when removing heartbeat peers
osd: mark down heartbeat connections on shutdown
crimson/: move get_mnow() to ShardServices, pass to heartbeat
crimson/osd: stubs for get_mnow, get_hbstamps
crimson/osd/heartbeat: adapt to new MOSDPing fields
crimson/osdmap_service: add get_mnow(), get_up_epoch()
osd/PeeringState: take HeartbeatStamps refs for current interval
osd: track clock delta between peer OSDs
osd: add get_mnow() interface to OSDService, PG, PeeringState
osd: record startup_time
osd: some minor refactoring/cleanup in handle_osd_ping

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #29498 into master

* refs/pull/29498/head:
qa/tasks/ceph_manager: remove race from all_active_or_peered()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

script/ceph-backport.sh: Use secure access for tracker.ceph.com

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #29509 into master

* refs/pull/29509/head:
doc: indicate imperative mood for commit titles

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: indicate imperative mood for commit titles

This is convention in the kernel and Ceph but is not clearly documented.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Write E2E test to verify that Manager modules have editing functionality

Fixes: https://tracker.ceph.com/issues/40823
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env vars

If a kubernetes pod spec specifies a limit of X, then the pod gets both
the limits.memory and requests.memory resource fields set, and rook passes
those as POD_MEMORY_LIMIT and POD_MEMORY_REQUEST environment variables.

This is a problem if only the limit is set, because we will end up
setting our osd_memory_target (and, in the future, other *_memory_targets)
to the hard limit, and the daemon will inevitably reach that threshold
and get killed.

Fix this by also looking at the POD_MEMORY_LIMIT value, and applying the
ratio (default: .8) to it, and setting our actual target to the min of
that and the POD_MEMORY_REQUEST.

Also, set the "default" target to ratio*limit, so that it will apply in
general when no request is specified.

When both request and limit are 10M, we then see

        "osd_memory_target": {
            "default": "800000000000",
            "env": "800000000000",
            "final": "800000000000"
        },

In a more "normal" situation where limit is 10M and request is 5M, we get

        "osd_memory_target": {
            "default": "800000000000",
            "env": "500000000000",
            "final": "500000000000"
        },

If only limit is specified (to 10M), we get

        "osd_memory_target": {
            "default": "800000000000",
            "final": "800000000000"
        },

Fixes: https://tracker.ceph.com/issues/41037
Signed-off-by: Sage Weil <sage@redhat.com>

common/config: let diff show non-build defaults

Notably, this includes things from set_val_default(), which may be set
at runtime.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/rbd_support: use image ids to detect duplicate tasks

This helps to to avoid the case where new tasks were not being scheduled
when an image name was re-used after having a task created under the
same name.

Fixes: https://tracker.ceph.com/issues/41032
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29501 from tchaikov/wip-journal/test_JournalRecorder.cc

journal: always shutdown JournalRecoreder before destructing it

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/config: do no include multiple 'default' values

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27859 from jan--f/update-bluestore-cache-doc

doc: update bluestore cache settings and clarify data fraction

test/objectstore: Allocator_test. Add test for dumping free regions and fragmentation_score.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

BlueStore/allocator: Add command to inspect how much BlueStore's block can go to BlueFS.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

journal: always shutdown JournalRecoreder before destructing it

otherwise when we destruct `journal::JournalRecorder::m_object_locks`,
it/they would be still being waited by some condition variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: no newlines in 'ceph -s' output

This gets dumped to the log, making it hard to read.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: make mon summary more concise in 'ceph -s'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29475 from tchaikov/wip-cmake3-for-xenial

install-deps.sh: use chacra for cmake repo

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/hello: fully document our options

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: remove race from all_active_or_peered()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move heartbeat connection cleanup to helper

Signed-off-by: Sage Weil <sage@redhat.com>

osd: break con<->session cycle when removing heartbeat peers

Signed-off-by: Sage Weil <sage@redhat.com>

osd: mark down heartbeat connections on shutdown

This ensures that heartbeat_reset() gets call and we clean up the
ref loop between the Connections and Sessions.

Signed-off-by: Sage Weil <sage@redhat.com>

crimson/: move get_mnow() to ShardServices, pass to heartbeat

get_mnow isn't clearly at home in OSDMapService, and the other methods
are needed from PeeringState, so let's consolidate on ShardServices
for now.  We probably ought OSDMapService state out of OSD into its
own module at some point.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: stubs for get_mnow, get_hbstamps

Signed-off-by: Sage Weil <sage@redhat.com>

crimson/osd/heartbeat: adapt to new MOSDPing fields

Signed-off-by: Sage Weil <sage@redhat.com>

crimson/osdmap_service: add get_mnow(), get_up_epoch()

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: take HeartbeatStamps refs for current interval

Signed-off-by: Sage Weil <sage@redhat.com>

osd: track clock delta between peer OSDs

We need to keep track of the monotonic clock deltas between peer OSDs in
order to be able to exchange timestamps across messages.  We need an
upper and lower bound on this delta, depending on the context where it is
used.

We can use the existing ping messages to get this by assuming that a ping
message is sent instantly to get a bound, and to share our delta in a
follow-up reply to share the other bound.  The ping sender will get both
bounds with a single ping + ping_reply exchange.  The ping receiver will
get the delta value from the next round's ping.

Include up_from in the ping messages to ensure we don't mix up different
instances of the same OSD.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: add get_mnow() interface to OSDService, PG, PeeringState

Single helper to provide a monotonic time offset from startup_time.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: record startup_time

Signed-off-by: Sage Weil <sage@redhat.com>

osd: some minor refactoring/cleanup in handle_osd_ping

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29337 into master

* refs/pull/29337/head:
mon: enable telemetry module by default
mgr/telemetry: force re-opt-in if the report contents change
mgr/telemetry: less noise in the log
mgr/telemetry: wake up serve on config change
mgr/telemetry: track telemetry report revisions

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29457 from tchaikov/wip-consolidate-run-tox

cmake: one run_tox.sh to rule them all

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

script/run_tox.sh: add --help option

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Finisher: kill the need for lock for ContextQueue::empty()

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29492 into master

* refs/pull/29492/head:
doc: Fix for new ceph-devel mailing list

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

cmake: pass tox envs to run_tox.sh explicitly

* add two optional keyword parameters to `add_tox_test()`
* use the default `TOX_PATH` parameter instead passing it explicitly,
  as it is always `${CMAKE_CURRENT_SOURCE_DIR}`
* drop the code to guess the tox envs in `run_tox.sh`, and always pass
  them explicitly in CMake script using the `--tox-envs` argument.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29471 from tchaikov/wip-googletest

googletest: pick up change to suppress CMP0048 warning

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tasks/mgr/test_progress.py: resolve dictionary runtime error

Think the problem is how we are using dictionary.item() in
for loop causing runtime error therefore the event didn't
get created so we convert it to list(dictionary).

Fixes http://pulpito.ceph.com/kchai-2019-07-28_14:30:09-rados-wip-kefu2-testing-2019-07-28-1941-distro-basic-mira/4160881/

Also the same fix as: #28840

Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

qa/tasks/mgr/test_progress.py: fix bug in 9b4dbf0

follow-up-fix for 9b4dbf0

basically we wanna look at the list that has inprogress events to inprogress+complete

Fixes: http://pulpito.ceph.com/kchai-2019-07-28_14:30:09-rados-wip-kefu2-testing-2019-07-28-1941-distro-basic-mira/4160881/
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

test: Bump sleep time for slower machines

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Fix for new ceph-devel mailing list

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #29458 from mikulely/rgw-drop-flush

rgw: drop dead flush_read_list declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: do not check ContextQueue for empty before taking the content.

This saves one lock acquisition. And empty() call is useless as move_to
checks for emptyness anyway()

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mon/MgrStatMonitor: set initial service_map 'modified' to cluster mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

common/Finisher: rename ContextQueue::swap method.
Actually it doesn't conform to swap semantic.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mon: remove double-nesting of "osdmap" for ceph status

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMap: make print_summary (used by 'ceph -s') more concise

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27675 from jan--f/c-v-console-logging-to-stderr

ceph-volume: print most logging messages to stderr

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>