Merge pull request #28070 from badone/wip-lazy-omap-test

Wip lazy omap test

Reviewed-by: Greg Farnum gfarnum@redhat.com

Merge pull request #28949 from adamemerson/wip-40416

Don't crash on copy when metadata directive not supplied

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #28940 into master

* refs/pull/28940/head:
common/config: Remove unused code.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28863 from votdev/fix_various_issues

mgr/dashboard: Fix issues in user form

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

rgw: Don't crash on copy when metadata directive not supplied

Fixes: http://tracker.ceph.com/issues/40416
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #28897 from ceph/fix-dashboard-hacking_link

mgr/dashboard: Fix link format to HACKING.rst

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28372 from pcuzner/add-mgr-metadata

mgr/prometheus: Add mgr metdata to prometheus exporter module

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge PR #28941 into master

* refs/pull/28941/head:
osd: scrub_purged_snaps: fix log message tense
osd/SnapMapper: reset iterators at finish of run()
osd/SnapMapper: drop useless _init()
osd: do not hold osd_lock while queueing stray purged_snaps
osd: more debug output during scrub hang

Reviewed-by: Neha Ojha <nojha@redhat.com>

osd: Modify lazy omap stats to only show one copy

Only show stats from the primary to get more in line with other pg stats
representations.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Add mgr metdata to prometheus exporter module

Add's metadata and status information for the mgr
daemon(s), together with the status of the mgr
modules.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

tests: Add test for lazy omap stat collection

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #28910 from yuvalif/wip-yuval-pubsub-test-fixes

rgw/pubsub: tests enhancements and fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28931 from yuriw/wip-yuriw-40452-master

qa/tests: added ragweed coverage to stress-split* upgrade suites

Merge pull request #28932 from yuriw/wip-yuriw-40452-master_2

qa/tests: added ragweed coverage to stress-split* upgrade suites

common/config: Remove unused code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: scrub_purged_snaps: fix log message tense

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: reset iterators at finish of run()

These iterators implicitly hold locks in FileStore, which can lead to a
deadlock if they are still alive and the caller tries to take new locks.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: drop useless _init()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27965 into master

* refs/pull/27965/head:
doc: remove recommendation for kernel.pid_max

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #17614 into master

* refs/pull/17614/head:
doc: cover more cache modes in rados/operations/cache-tiering.rst

Reviewed-by: Sage Weil <sage@redhat.com>

doc: cover more cache modes in rados/operations/cache-tiering.rst

Fixes: http://tracker.ceph.com/issues/14153
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28850 from liewegas/wip-40634

mon/AuthMonitor: fix initial creation of rotating keys

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #28758 into master

* refs/pull/28758/head:
mgr/progress: Created first unit test for progress module
mgr/progress: makes changes to unit test PR
mgr/progress: Created first unit test for progress module

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #28842 into master

* refs/pull/28842/head:
test/msgr: make ceph_perf_msgr_client/server work.

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: do not hold osd_lock while queueing stray purged_snaps

Signed-off-by: Sage Weil <sage@redhat.com>

osd: more debug output during scrub hang

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28873 from albIN7/liv3

rgw: fix bugs in listobjectsv1

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28913 from cyx1231st/wip-seastar-msgr-fix-perfasync

test/crimson: fix interpretability with perf_async_msgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: set python binary for teuthology venv (#28914)

mgr/dashboard: set python binary for teuthology venv

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

rgw: fix bugs in listobjectsv1

Fixes: https://tracker.ceph.com/issues/40663
Signed-off-by: Albin Antony <aantony@redhat.com>

Merge pull request #28920 from tchaikov/wip-cmake-pybind

cmake: avoid rebuilding extensions, and using python-config

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28884 from DimStar77/depchain

ceph.spec.in: Drop systemd BuildRequires in case of building for SUSE

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28843 from tchaikov/wip-armhf-build

cmake, pybind: fix build on armhf

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: Fix npm vulnerabilities (#28903)

mgr/dashboard: Fix npm vulnerabilities

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: set python binary for teuthology venv

* Added env. var. to set python binary when creating teuthology venv.
* venv/bin/activate: run after setting PATH (as it does the last PATH export).

Signed-off-by: alfonsomthd <almartin@redhat.com>

Merge pull request #28888 from joscollin/wip-fix-typo

doc: fix typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #28848 into master

* refs/pull/28848/head:
common/blkdev: only pay attention to first 2 bits of smartctl return code
common/blkdev: include stdout and stderr on join error
common/blkdev: fix syntax of error message
common/blkdev: fix smartctl output in error path
common/blkdev: include bad json from smartctl when unparseable
mgr/devicehealth: sort keys when dumping metrics

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge PR #28052 into master

* refs/pull/28052/head:
doc/rgw: document use of 'realm pull' instead of 'period pull'

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

qa/tests: added ragweed coverage to stress-split* upgrade suites

added ceph config set global bluestore_warn_on_legacy_statfs fals
changed user "foo" to "ragweed" in ragweed.py

Fixes: http://tracker.ceph.com/issues/40452
Fices: http://tracker.ceph.com/issues/40467
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/tests: added ragweed coverage to stress-split* upgrade suites

added ceph config set global bluestore_warn_on_legacy_statfs fals
chnaged user "foo" to "ragweed" in ragweed.py

Fixes: http://tracker.ceph.com/issues/40452
Fices: http://tracker.ceph.com/issues/40467
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 5e7e1f382ffbe1515ce6629dda480041260af448)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28791 from dcasier/bug-40100-2

Return an error, for Bluestore OSD, if WAL or DB are defined in the tags of the OSD but not present on the system

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #28152 from alimaredia/wip-rgw-thread-yield-context-prepare-bucket-index-transaction

rgw: thread option_yield through bucket index transaction prepare

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: change optional_yield to null_yield

In copy_obj_data() in the AtomicObjectProcessor
constructor

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: thread yield context through writing of head obj

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: thread option_yield through bucket index transaction prepare

Signed-off-by: Ali Maredia <amaredia@redhat.com>

pybind: use distutils.sysconfig for compiling flags

this allows maintainer to override the compiling flags when
cross-compiling Ceph.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28823 from cbodley/wip-qa-rgw-swift-server

qa/rgw: more fixes for swift task

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: Silence Alertmanager alerts (#27277)

mgr/dashboard: Silence Alertmanager alerts

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

test/crimson: default bind to 127.0.0.1 for perf

Async-messenger's accepting in v2 will fail if it binds to a blank ip.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: add v1-crc-enabled for msgr perf

* fix a bug that perf_crimson_msgr and perf_async_msgr cannot connect
  to each other in v1 due to CRC checks;
* add an option to enable v1 CRC checks, because they are enabled in
  real world, and we might want to see the performance impact;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: track msg with tid in perf_async_msgr

fix perf_async_msgr by setting tid for perf_crimson_msgr to track
messages.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #28901 into master

* refs/pull/28901/head:
qa/standalone/scrub/osd-scrub-repair: fix 'scrub ok' grep
osd/osd_types: remove 'snap_context' from SnapSet::dump()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake/modules/Distutils.cmake: add setup as a dependency

so the extension will be rebuilt if setup.py is updated

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules/Distutils.cmake: avoid rebuilding cython ext

use the combination of add_custom_command() and add_custom_target() to
build cython extensions to avoid rebuilding the cython extension
everytime we run "make"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules/Distutils.cmake: add byproduct

so add_custom_target() has better idea what is generated. if any target
will use it, it can be used as a dependency.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add an arg for name of cython extension

so we can deduce the path of cython extension

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix closing tag in app.component.html (#28899)

mgr/dashboard: fix closing tag in app.component.html

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Upgrade ceph-iscsi config to version 10 (#28720)

mgr/dashboard: Upgrade ceph-iscsi config to version 10

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: check embedded Grafana dashboard references (#28234)

mgr/dashboard: check embedded Grafana dashboard references

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

rgw/pubsub: fix amqp topic bug. add disabled end2end push tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

rgw/pubsub: fix duplicates due to multiple zone synching

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

rgw/pubsub: run pubsub tests even if multisite fails

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #28859 from tchaikov/wip-40646

ceph.spec.in: pin to gcc-c++-8.2.1

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28838 from tchaikov/wip-rgw-printf-size_t

rgw/rgw_client_io_filters.h: print size_t the portable way

Merge pull request #28617 from zhangsw/rgw-fix-lifecycle-tag

rgw: Object tags shouldn't work with deletemarker or multipart expiration

Merge pull request #28256 from theanalyst/rgw-lc-placement-target-fixes

rgw: lc: check for valid placement target before processing transitions

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #28905 from adamemerson/wip-fix-crimson-test-build

rgw: Fix narrowing conversion error

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28192 from tianshan/fix_39989

rgw: fix list bucket with start maker and delimiter '/' will miss next object…

Merge pull request #28902 from tchaikov/wip-fix-crimson-test-build

test/crimson: fix FTBFS

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

test/crimson: fix FTBFS

Connection::send_message(Message*)

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: Fix narrowing conversion error

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph.spec.in,install-dep.sh: pin to gcc-c++-8.2.1

as an intermediate solution before
https://bugzilla.redhat.com/show_bug.cgi?id=1726630 gets fixed.

Fixes: https://tracker.ceph.com/issues/40646
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix npm vulnerabilities

Currently npm audit reports we have 12 vulnerabilities (4 moderate, 8 high).
This will fix most of them, leaving only 1 moderate that still doesn't have a
fix.

Fixes: http://tracker.ceph.com/issues/40677
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28890 from cyx1231st/wip-seastar-msgr-msgreuse

crimson/net: prevent reusing the sent messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: change to use make_message()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: check front_msg correctly during sweep

In order to check whether the front_msg is unchanged, we need to make sure:
* The sent message is not reused;
* The message to be checked is not freed;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: prevent reusing the sent messages

We are tracking sent messages by seq, so reusing messages will cause
undefined behavior because seq should not be shared between different
messages.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/dashboard: fix closing tag

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #28858 from callithea/wip-issue-40395

mgr/dashboard: Extend E2E test section

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Fix link format

Fix link (compodoc section) with wrong format.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: use mds_mem.dn for fs dentries (#28505)

mgr/dashboard: use mds_mem.dn for fs dentries

Reviewed-by: Volker Theile <vtheile@suse.com>

Packaging: Drop systemd BuildRequires in case of building for SUSE

There is already pkgconfig(systemd) present, which, in SUSE's packaging,
translates to the same package (systemd.pc is shipped as part of the main
systemd package). Not explicitly mentioning 'systemd' as package name allows
the openSUSE Build Service though to find shortcuts by using the
bootstrap packages, i.e. systemd-mini (ABI/API complete).

Signed-off-by: Dominique Leuenberger <dimstar@opensuse.org>

Merge pull request #28876 from tchaikov/wip-cmake-vstart-with-cython_cephfs

cmake: add cython_cephfs to vstart target

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28845 from ifed01/wip-ifed-fix-alloc-dump

os/bluestore: more smart allocator dump when lacking space for bluefs.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

doc: fix typo

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #28651 from ifed01/wip-ifed-remove-assert-bs-tool

os/bluestore/bluestore-tool: minor fixes around migrate

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #28732 from tchaikov/wip-32bit-size_t

osdc/Striper: specialize std::min<>

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28829 from tchaikov/wip-dmclock-uint

dmclock: pick up fix to replace uint

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #27834 from rzarzynski/wip-nss-drop-cms

rgw, common, build: drop NSS support

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/standalone/scrub/osd-scrub-repair: fix 'scrub ok' grep

The log now also has a 'purged_snaps scrub ok' message that (generally)
precedes the first scrubbed PG.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove 'snap_context' from SnapSet::dump()

We no longer have a snaps field with real values, so dumping this as a
"snap_context" is silly.  Instead, just dump the seq.

Adjust qa/standalone/scrub/osd-scrub-repair.sh accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore-tool: omit device from both source and taget
lists when migrating.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore/bluestore-tool: fix error output

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore/bluestore-tool: do not assert when migrate command fails.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #28835 from croit/fix-24420

debian/control: add python-routes dependency for dashboard

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Time diff service

The time diff service can easily calculate the difference between two
dates and print it out in a human readable duration format. It also can
calculate a end or start date based on a date and a duration string.

Fixes: https://tracker.ceph.com/issues/36722
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Allow test selection overwrite

Currently it's not easily possible to mock a selection during table
action tests, that's why I added a method that is used to create the
selection that is used through out the table action scenario tests.
You can simply overwrite it to get a different base object as selection.

Fixes: https://tracker.ceph.com/issues/36722
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Handle unset settings

It's now easy to deal with unset settings.

Fixes: https://tracker.ceph.com/issues/36722
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Silence Alertmanager alerts

Now you can silence alerts through the dashboard. You can now create,
recreate, edit and expire a silence. You can create a silence based on a
selected alert.

The silence form will help you create a silence that silences an alert.
It is provided with functionality to check if the silences, that
you are about to create, will or will not match an active alert or even
a rule.

It also provides help choosing the right values for the right chosen
matcher attribute name, through the use of type ahead values.

The dashboard will now use the Prometheus and the Alertmanager API

Fixes: https://tracker.ceph.com/issues/36722
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge PR #28877 into master

* refs/pull/28877/head:
doc/dev/corpus.rst: minor tweaks

Reviewed-by: Sage Weil <sage@redhat.com>