qa: vstart_runner fails because of string index out of range

Fix problem in LocalRemote::_perform_checks_and_return_list_of_args() when LocalCephManager::raw_cluster_cmd() is called with a empty command parameter.

Fixes: https://tracker.ceph.com/issues/40729
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #28758 into master

* refs/pull/28758/head:
mgr/progress: Created first unit test for progress module
mgr/progress: makes changes to unit test PR
mgr/progress: Created first unit test for progress module

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #28842 into master

* refs/pull/28842/head:
test/msgr: make ceph_perf_msgr_client/server work.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28873 from albIN7/liv3

rgw: fix bugs in listobjectsv1

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28913 from cyx1231st/wip-seastar-msgr-fix-perfasync

test/crimson: fix interpretability with perf_async_msgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: set python binary for teuthology venv (#28914)

mgr/dashboard: set python binary for teuthology venv

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

rgw: fix bugs in listobjectsv1

Fixes: https://tracker.ceph.com/issues/40663
Signed-off-by: Albin Antony <aantony@redhat.com>

Merge pull request #28920 from tchaikov/wip-cmake-pybind

cmake: avoid rebuilding extensions, and using python-config

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28884 from DimStar77/depchain

ceph.spec.in: Drop systemd BuildRequires in case of building for SUSE

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28843 from tchaikov/wip-armhf-build

cmake, pybind: fix build on armhf

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: Fix npm vulnerabilities (#28903)

mgr/dashboard: Fix npm vulnerabilities

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: set python binary for teuthology venv

* Added env. var. to set python binary when creating teuthology venv.
* venv/bin/activate: run after setting PATH (as it does the last PATH export).

Signed-off-by: alfonsomthd <almartin@redhat.com>

Merge pull request #28888 from joscollin/wip-fix-typo

doc: fix typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #28848 into master

* refs/pull/28848/head:
common/blkdev: only pay attention to first 2 bits of smartctl return code
common/blkdev: include stdout and stderr on join error
common/blkdev: fix syntax of error message
common/blkdev: fix smartctl output in error path
common/blkdev: include bad json from smartctl when unparseable
mgr/devicehealth: sort keys when dumping metrics

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge PR #28052 into master

* refs/pull/28052/head:
doc/rgw: document use of 'realm pull' instead of 'period pull'

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #28791 from dcasier/bug-40100-2

Return an error, for Bluestore OSD, if WAL or DB are defined in the tags of the OSD but not present on the system

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #28152 from alimaredia/wip-rgw-thread-yield-context-prepare-bucket-index-transaction

rgw: thread option_yield through bucket index transaction prepare

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: change optional_yield to null_yield

In copy_obj_data() in the AtomicObjectProcessor
constructor

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: thread yield context through writing of head obj

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: thread option_yield through bucket index transaction prepare

Signed-off-by: Ali Maredia <amaredia@redhat.com>

pybind: use distutils.sysconfig for compiling flags

this allows maintainer to override the compiling flags when
cross-compiling Ceph.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28823 from cbodley/wip-qa-rgw-swift-server

qa/rgw: more fixes for swift task

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: Silence Alertmanager alerts (#27277)

mgr/dashboard: Silence Alertmanager alerts

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

test/crimson: default bind to 127.0.0.1 for perf

Async-messenger's accepting in v2 will fail if it binds to a blank ip.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: add v1-crc-enabled for msgr perf

* fix a bug that perf_crimson_msgr and perf_async_msgr cannot connect
  to each other in v1 due to CRC checks;
* add an option to enable v1 CRC checks, because they are enabled in
  real world, and we might want to see the performance impact;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: track msg with tid in perf_async_msgr

fix perf_async_msgr by setting tid for perf_crimson_msgr to track
messages.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #28901 into master

* refs/pull/28901/head:
qa/standalone/scrub/osd-scrub-repair: fix 'scrub ok' grep
osd/osd_types: remove 'snap_context' from SnapSet::dump()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake/modules/Distutils.cmake: add setup as a dependency

so the extension will be rebuilt if setup.py is updated

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules/Distutils.cmake: avoid rebuilding cython ext

use the combination of add_custom_command() and add_custom_target() to
build cython extensions to avoid rebuilding the cython extension
everytime we run "make"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules/Distutils.cmake: add byproduct

so add_custom_target() has better idea what is generated. if any target
will use it, it can be used as a dependency.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add an arg for name of cython extension

so we can deduce the path of cython extension

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix closing tag in app.component.html (#28899)

mgr/dashboard: fix closing tag in app.component.html

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Upgrade ceph-iscsi config to version 10 (#28720)

mgr/dashboard: Upgrade ceph-iscsi config to version 10

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: check embedded Grafana dashboard references (#28234)

mgr/dashboard: check embedded Grafana dashboard references

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28859 from tchaikov/wip-40646

ceph.spec.in: pin to gcc-c++-8.2.1

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28838 from tchaikov/wip-rgw-printf-size_t

rgw/rgw_client_io_filters.h: print size_t the portable way

Merge pull request #28617 from zhangsw/rgw-fix-lifecycle-tag

rgw: Object tags shouldn't work with deletemarker or multipart expiration

Merge pull request #28256 from theanalyst/rgw-lc-placement-target-fixes

rgw: lc: check for valid placement target before processing transitions

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #28905 from adamemerson/wip-fix-crimson-test-build

rgw: Fix narrowing conversion error

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28192 from tianshan/fix_39989

rgw: fix list bucket with start maker and delimiter '/' will miss next object…

Merge pull request #28902 from tchaikov/wip-fix-crimson-test-build

test/crimson: fix FTBFS

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

test/crimson: fix FTBFS

Connection::send_message(Message*)

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: Fix narrowing conversion error

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph.spec.in,install-dep.sh: pin to gcc-c++-8.2.1

as an intermediate solution before
https://bugzilla.redhat.com/show_bug.cgi?id=1726630 gets fixed.

Fixes: https://tracker.ceph.com/issues/40646
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix npm vulnerabilities

Currently npm audit reports we have 12 vulnerabilities (4 moderate, 8 high).
This will fix most of them, leaving only 1 moderate that still doesn't have a
fix.

Fixes: http://tracker.ceph.com/issues/40677
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28890 from cyx1231st/wip-seastar-msgr-msgreuse

crimson/net: prevent reusing the sent messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: change to use make_message()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: check front_msg correctly during sweep

In order to check whether the front_msg is unchanged, we need to make sure:
* The sent message is not reused;
* The message to be checked is not freed;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: prevent reusing the sent messages

We are tracking sent messages by seq, so reusing messages will cause
undefined behavior because seq should not be shared between different
messages.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/dashboard: fix closing tag

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #28858 from callithea/wip-issue-40395

mgr/dashboard: Extend E2E test section

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: use mds_mem.dn for fs dentries (#28505)

mgr/dashboard: use mds_mem.dn for fs dentries

Reviewed-by: Volker Theile <vtheile@suse.com>

Packaging: Drop systemd BuildRequires in case of building for SUSE

There is already pkgconfig(systemd) present, which, in SUSE's packaging,
translates to the same package (systemd.pc is shipped as part of the main
systemd package). Not explicitly mentioning 'systemd' as package name allows
the openSUSE Build Service though to find shortcuts by using the
bootstrap packages, i.e. systemd-mini (ABI/API complete).

Signed-off-by: Dominique Leuenberger <dimstar@opensuse.org>

Merge pull request #28876 from tchaikov/wip-cmake-vstart-with-cython_cephfs

cmake: add cython_cephfs to vstart target

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28845 from ifed01/wip-ifed-fix-alloc-dump

os/bluestore: more smart allocator dump when lacking space for bluefs.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

doc: fix typo

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #28651 from ifed01/wip-ifed-remove-assert-bs-tool

os/bluestore/bluestore-tool: minor fixes around migrate

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #28732 from tchaikov/wip-32bit-size_t

osdc/Striper: specialize std::min<>

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28829 from tchaikov/wip-dmclock-uint

dmclock: pick up fix to replace uint

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #27834 from rzarzynski/wip-nss-drop-cms

rgw, common, build: drop NSS support

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/standalone/scrub/osd-scrub-repair: fix 'scrub ok' grep

The log now also has a 'purged_snaps scrub ok' message that (generally)
precedes the first scrubbed PG.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove 'snap_context' from SnapSet::dump()

We no longer have a snaps field with real values, so dumping this as a
"snap_context" is silly.  Instead, just dump the seq.

Adjust qa/standalone/scrub/osd-scrub-repair.sh accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore-tool: omit device from both source and taget
lists when migrating.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore/bluestore-tool: fix error output

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore/bluestore-tool: do not assert when migrate command fails.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #28835 from croit/fix-24420

debian/control: add python-routes dependency for dashboard

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Time diff service

The time diff service can easily calculate the difference between two
dates and print it out in a human readable duration format. It also can
calculate a end or start date based on a date and a duration string.

Fixes: https://tracker.ceph.com/issues/36722
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Allow test selection overwrite

Currently it's not easily possible to mock a selection during table
action tests, that's why I added a method that is used to create the
selection that is used through out the table action scenario tests.
You can simply overwrite it to get a different base object as selection.

Fixes: https://tracker.ceph.com/issues/36722
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Handle unset settings

It's now easy to deal with unset settings.

Fixes: https://tracker.ceph.com/issues/36722
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Silence Alertmanager alerts

Now you can silence alerts through the dashboard. You can now create,
recreate, edit and expire a silence. You can create a silence based on a
selected alert.

The silence form will help you create a silence that silences an alert.
It is provided with functionality to check if the silences, that
you are about to create, will or will not match an active alert or even
a rule.

It also provides help choosing the right values for the right chosen
matcher attribute name, through the use of type ahead values.

The dashboard will now use the Prometheus and the Alertmanager API

Fixes: https://tracker.ceph.com/issues/36722
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge PR #28877 into master

* refs/pull/28877/head:
doc/dev/corpus.rst: minor tweaks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28856 from tchaikov/wip-rbd-nbd-basename

tools/rbd_nbd: use POSIX basename()

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #28875 from tchaikov/wip-mgr-volumes-cleanup

mgr/volumes: do not import unused module

doc/dev/corpus.rst: minor tweaks

the executables are located in build/bin after we switched to cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Rename iSCSI gateways name to FQDN

iSCSI gateways name will be automatically renamed from host
shot name to FQDN.

Fixes: https://tracker.ceph.com/issues/40566
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Upgrade ceph-iscsi config to version 10

Fixes: https://tracker.ceph.com/issues/40566
Signed-off-by: Ricardo Marques <rimarques@suse.com>

cmake: add cython_cephfs to vstart target

in vstart.sh, if MDS is enabled, `ceph fs volume create` is used to
create cephfs volume. and `fs volume create` command is implemented by
`src/pybind/mgr/volumes/module.py`, which in turn uses `cephfs` python
binding indirectly. so we need to add `cephfs` to `vstart` target to
facilidate the cephfs development using vstart.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/volumes: do not import unused module

cephfs is not used in this module, so drop its `import`

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28753 from tchaikov/wip-doc-conf

doc/rados/configuration: update to be in sync with ConfUtils changes

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #28803 from badone/wip-ceph_ansible_upgrade-ansible-2.8

qa/ceph-ansible: Move to ansible 2.8

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28807 from badone/wip-ceph_ansible-cephfs_pools-pg_num-fix

qa/ceph-ansible: Replace pgs with pg_num

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: check embedded Grafana dashboard references

Add a script to check a cd-grafana component always refers to an
existing Grafana dashboard. The check is added as a new tox command.

Fixes: http://tracker.ceph.com/issues/40008
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #28867 into master

* refs/pull/28867/head:
qa/standalone/ceph-helpers: more osd debug

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #28865 into master

* refs/pull/28865/head:
mon/OSDMonitor: fix _lookup_snap to verify the pool matches
ceph_test_rados_api_*: make failing to clean up namespace non-fatal
osd: store purged_snaps history under separate object

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #28864 into master

* refs/pull/28864/head:
doc/releases/releases.yaml: set target EOL to Jun 1 for L and M
doc/releases: update schedule to describe a 12-month cycle

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #28334 from dzafman/wip-40073

osd: Fix the way that auto repair triggers after regular scrub

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/standalone/ceph-helpers: more osd debug

debug_ms=1
debug_monc=20

Hunting down http://tracker.ceph.com/issues/40666

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: lvm.activate: Return an error if WAL/DB devices absent

Fixes: https://tracker.ceph.com/issues/40100
Signed-off-by: David Casier <david.casier@aevoo.fr>

mon/OSDMonitor: fix _lookup_snap to verify the pool matches

We don't want to get false positives from keys for other pools.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28866 from alfredodeza/wip-rm40665

ceph-volume broken assertion errors after pytest changes

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume api.lvm catch IndexError when parsing dmmapper output

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests update to use error.value instead of str(error)

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph_test_rados_api_*: make failing to clean up namespace non-fatal

If we leak snaps, failing to clean up a namespace can happen and won't
get fixed until we scrub purged_snaps.  Make this a non-fatal condition.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: store purged_snaps history under separate object

We can't put this in the snapmapper object because filestore does not
allow multiple concurrent omap iterators on the same object.  (This is a
limitation that could be fixed with some read/write locking, but not
without some significant changes to DBObjectMap; since that is old crufty
legacy code let's avoid touching it!)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28788 from hjwsm1989/cleanup-debian

debian: remove dup ceph-fuse line

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/releases/releases.yaml: set target EOL to Jun 1 for L and M

Don't list an actual Luminous EOL yet.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases: update schedule to describe a 12-month cycle

Signed-off-by: Sage Weil <sage@redhat.com>

debian/control: add python-routes dependency

the dashboard requires python-routes via cherrypy/_cpdispatch.py during runtime
but the cherrypy debian package only recommends it and doesn't depend on it

Fixes: https://tracker.ceph.com/issues/24420
Signed-off-by: Paul Emmerich <paul.emmerich@croit.io>

Merge PR #28785 into master

* refs/pull/28785/head:
osd/PrimaryLogPG: do_op - do not create head object twice

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #28782 into master

* refs/pull/28782/head:
common/options.cc: Lower the default value of osd_deep_scrub_large_omap_object_key_threshold

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>