qa/tests - added all supported distro

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #22629 from dmick/master

doc/mgr/plugins.rst: explain more about the plugin command protocol

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22528 from rjfd/wip-dashboard-fix-rest-controller

mgr/dashboard: restcontroller: minor improvements and bug fixes

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22609 from qwren/qat-decompress

compressor: building error for QAT decompress

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/mgr/plugins.rst: explain more about the plugin command protocol

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #21635 from yangdongsheng/qos_read_write

librbd: support bps throttle and throttle read and write seperately.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22587 from trociny/wip-24545

librbd: deep_copy: don't hide parent if zero overlap for snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22627 from ceph/rm-24456

lvm: when osd creation fails log the exception

Reviewed-by: Alfredo Deza <adeza@redhat.com>

lvm: when osd creation fails log the exception

Fixes: http://tracker.ceph.com/issues/24456
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22509 from taodd/osdc-optimize

osdc: optimize the code doing the BufferHead mapping

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #22596 into master

* refs/pull/22596/head:
os/bluestore: use vector instead of set for zombies
os/bluestore: reuse zombie OpSequencers by collection id
qa/suites/rados/objecstore/backends/objectstore: capture coredumps
os/bluestore: more debug output
os/bluestore: print cnode from _open_collections
os/bluestore: print cnode on fsck
qa/suites/rados/objecstore: preserve data dir for ceph_test_objecstore

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: use vector instead of set for zombies

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: reuse zombie OpSequencers by collection id

We can get a sequence that deletes and then recreates a collection where
the transaction removing the collection is delayed (due to pending IO on
its sequencer) but colleciton create is not (new sequencer).

Avoid any such reordering by recycling the old collection's sequencer if
the zombie_osr has not been reaped yet.

Fixes: http://tracker.ceph.com/issues/24550
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/objecstore/backends/objectstore: capture coredumps

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22606 from yuriw/wip-yuriw-crontab

qa/tests - added powercycle suite to run on weekly basis on master and mimic

Merge pull request #22495 from taodd/osdc-bugfix

osdc: Fix the wrong BufferHead offset

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22570 from zy751713126/test_for_set_snap

pybind/rbd: add assert_raise in test set_snap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22566 from liewegas/wip-rbd-table

tools/rbd/action: align column headers left

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #22569 into master

* refs/pull/22569/head:
qa/cephfs: add test for "dump inode"
mds: add dump inode command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: remove unused header

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: fix gmock warnings introduced after 9e5b87c

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: deep_copy: update end_size only if zero interval caused truncate

The problem shown up when stripping was used, and assembling a
nonexistent destination object from source objects, and one of these
objects existed and snap diff returned a zero interval. The non-zero
end_size in that case triggered invalid object map update.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: deep_copy: don't hide parent if zero overlap for snapshot

Fixes: http://tracker.ceph.com/issues/24545
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #17414 from fangyuxiangGL/system

rgw: admin/user ops dump user 'system' flag

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #22604 from alfredodeza/wip-bluestore-config-ref

doc/bluestore: data doesn't use two partitions (ceph-disk era)

Reviewed-by: Sage Weil sage@redhat.com

Merge PR #22595 into master

* refs/pull/22595/head:
common/options: dump flags

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #22582 into master

* refs/pull/22582/head:
mon: destroy-new -> purge-new
mon/OSDMonitor: fix NEW flag adjustment in destroy[-new]

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge PR #22602 into master

* refs/pull/22602/head:
doc/releases/mimic: note lack of debian builds for mimic

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22489 into master

* refs/pull/22489/head:
ceph-volume lvm.common use destroy-new, doesn't need admin keyring

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22343 into master

* refs/pull/22343/head:
qa/standalone remove ceph-disk from activate_osd helper
cmake: remove subman.sh tests
test remove ceph-disk directory
debian: remove ceph_detect_init python files from base
qa/standalone remove virtualenv paths for ceph-disk and ceph-detect-init
debian: remove ceph-disk ceph-detect-init python files
rpm: remove ceph-disk ceph-detect-init python files
alpine: remove ceph-disk ceph-detect-init python files
alpine: remove ceph-osd and parttypeuuid udev rules
debian: remove ceph-osd and parttypeuuid udev rules
rpm: remove ceph-osd and parttypeuuid udev rules
ceph-helpers.sh: remove ceph-disk, set up osds directly
CMakeLists.txt: add back CEPH_BUILD_VIRTUALENV
alpine: remove ceph-disk, add ceph-volume in APKBUILD.in
upstart: remove ceph-disk activation call
doc/install add anchor for manual osd deployment in freebsd guide
doc/dev remove ceph-disk from freebsd guide, link to manual reference
doc/dev/config-key remove ceph-disk references
doc/dev remove ceph-disk.rst
doc/dev: change ceph-disk suite examples for ceph-deploy
doc/man_index: remove ceph-disk, ceph-detect-init refs
doc/install: remove ceph-disk from freebsd examples
doc/rados remove ceph-disk from man references
doc/man remove ceph-disk ref from ceph-volume-systemd
doc/man: update reference from ceph-disk to ceph-volume
doc/man: remove ceph-disk, ceph-detect-init from cmake
doc/man/ceph-volume remove doc reference to ceph-disk
doc/man: remove ceph-disk, ceph-detect-init
qa/suites: remove ceph-disk
qa/run-standalone.sh: remove requirement for ceph-detect-init virtualenv
qa/workunits: remove ceph-detect-init from rbdmapfile test
qa/workunits: remove ceph-detect-init from ceph-helpers-root.sh
qa/workunits: remove ceph-disk
build: remove ceph-disk from freebsd script
cmake: remove ceph-disk, ceph-detect-init tox tests
init-ceph: remove ceph-disk
cmake: remove top-level entries for ceph-disk, ceph-detect-init
debian: remove ceph-detect-init references
debian: remove ceph-disk references
src: remove ceph-detect-init tool
rpm: remove ceph-disk, ceph-detect-init from spec file
test: remove subman script
script: remove subman script
udev: remove parttypeuuid rules for ceph-disk
tool remove ceph-disk from ps-ceph.pl
upstart: remove ceph-disk conf file
systemd: remove ceph-disk from CMakeLists
systemd: remove ceph-disk service
udev: remove ceph-disk rules
src: remove ceph-disk tool

Merge pull request #22622 from jcsp/wip-doc-mgr

doc/mgr: note need for module.py file in plugins

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #22600 from votdev/issue_24525_improvements

mgr/dashboard: Fix more layout issues in UI

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

doc/mgr: note need for module.py file in plugins

This is kind of a kludgy thing but let's at least
document it!

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: Fix layout issues in UI

Improve the CSS3 selectors and use the margin-left values from 'btn-toolbar' Bootstrap.

Signed-off-by: Volker Theile <vtheile@suse.com>

compressor: building error for QAT decompress

The parameter of decompress changes from 'bufferlist::iterator' to
'bufferlist::const_iterator', but chis change miss class QatAccel,
and so the building could not be successful for QAT environment.

This patch fix this error with '-DWITH_QATZIP=ON'.

Signed-off-by: Qiaowei Ren <qiaowei.ren@intel.com>

Merge pull request #22480 from dzafman/wip-standalone

Fixes for standalone tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: run-standalone move aside cores found before testing starts

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fail immediately if some operations fail

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix standalone main usage

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix some function desciptions

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Put files in private test directory

Signed-off-by: David Zafman <dzafman@redhat.com>

doc/releases/mimic: note lack of debian builds for mimic

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tests - added powercycle suite to run on weekly basis on master and mimic

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc/bluestore: data doesn't use two partitions (ceph-disk era)

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22597 from votdev/bug_24525

mgr/dashboard: Fix layout issues in UI

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #22416 from votdev/get_rgw_api_user_via_rest

mgr/dashboard: Get user ID via RGW Admin Ops API

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

ceph-volume lvm.common use destroy-new, doesn't need admin keyring

Signed-off-by: Alfredo Deza <adeza@redhat.com>

common/options: dump flags

Also document.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Add missing blanks to fix layout issue of various texts.

This behaviour starts to occur after switching to Angular 6.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #22473 from Devp00l/wip-static-testbed-changes

mgr/dashboard: Removed unnecessary fake services from unit tests

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21850 from jan--f/add-monitoring-grafana-dash

add monitoring subdir and Grafana cluster dashboard

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #22521 from tchaikov/wip-cmake-seastar-cleanup

cmake: only build extra boost libraries only if WITH_SEASTAR

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

qa/cephfs: add test for "dump inode"

Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Merge PR #22583 into master

* refs/pull/22583/head:
test/objectstore/CMakefiles: fix unittest_alloc_bench build

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #21494 from theanalyst/rgw-reorder-auth

rgw: have a configurable authentication order

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/objectstore/CMakefiles: fix unittest_alloc_bench build

Broken by 434589a3206aafe94de5a3b95b67eddb2cfc3bdb.  The add_ceph_unittest
helper does more than just add this to the list of tests--it also adjusts
linking and build options.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: more debug output

Signed-off-by: Sage Weil <sage@redhat.com>

mon: destroy-new -> purge-new

What we actually want is a purge, not a destroy.  Destroy leaves the OSD
ID in used and allows it to be recreated.  What ceph-volume wants is to
purge all trace of the failed OSD setup.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: fix NEW flag adjustment in destroy[-new]

Only set the NEW flag if it isn't already set.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: print cnode from _open_collections

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: print cnode on fsck

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22577 from ErwanAliasr1/evelu-parallel

src/test: Using gtest-parallel to speedup unittests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21998 from p-na/osd-flags-wip

mgr/dashboard: Add ability to list,set and unset cluster-wide OSD flags to the backend

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #22340 into master

* refs/pull/22340/head:
doc: update description of mount options for cephfs

Reviewed-by: Zheng Yan <zyan@redhat.com>

mgr/dashboard: Get user ID via RGW Admin Ops API.

The RGW API user id (set via 'ceph dashboard set-rgw-api-user-id <xxx>') is optional but the user ID is required internally for some situations. Because of that the user ID is requested via a RGW Admin Ops API call if it is not configured via CLI.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #22464 into master

* refs/pull/22464/head:
mds: print dir decay counters on hit
DecayCounter: removed unused velocity
DecayCounter: remove unnecessary delta member
mds: use monotonic time for DecayCounter

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22512 into master

* refs/pull/22512/head:
client: fix bug #24491 _ll_drop_pins may access invalid iterator

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

mds: print dir decay counters on hit

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

DecayCounter: removed unused velocity

It's not actually used anywhere and it's not accurate.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

DecayCounter: remove unnecessary delta member

We can just add/sub to val directly (especially now that decay can be called on
a const DecayCounter).

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use monotonic time for DecayCounter

This commit has a few side-effects:
- Decaying the DecayCounter is more accurate, we do not need to "skip" decaying
  the counter if it's been less than a second since the last decay. The time
  delta is now at the granularity of the monotonic clock.
- Any check of the DecayCounter results in updating the current value, even
  const checks.
- DecayRate is now established when the DecayCounter is created. There's no longer
  a need to pass the DecayRate whenever interacting with the counter.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22357 into master

* refs/pull/22357/head:
common/DecayCounter: set last_decay to current time when decoding decay counter

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22289 into master

* refs/pull/22289/head:
mds: put Capability in map container

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22266 into master

* refs/pull/22266/head:
client:  update inode fields according to issued caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22234 into master

* refs/pull/22234/head:
mds: add magic to header of open file table

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ctest: Removing unittest_alloc_bench

unittest_alloc_bench is very cpu consuming and can take up to 20mn to
run. As per a discussion with the original author of this code, this
test is only about testing the performance while there is no validation
code in it.

To save time from people doing make check often or even the CI itself,
removing this test will save a lot of time while not reducing the test
coverage.

This commit is only removing the test from the make check but keep the
binary being compiled for those who want to run it manually.

Signed-off-by: Erwan Velu <erwan@redhat.com>

run-make-check: enable WITH_GTEST_PARALLEL

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add WITH_GTEST_PARALLEL option

and remove src/test/gtest-parallel submodule, because gtest-parallel is
only useful for running tests. and not all end-users are interested in
running test not to mention running them in parallel. so, to avoid
including gtest-parallel scripts in the dist tarball. it'd be better to
make it optional, and an external project.

Signed-off-by: Kefu Chai <kchai@redhat.com>

src/test: Using gtest-parallel to speedup unittests

Unittests are run sequentially and could take a long while to run.

This commit is about using gtest-parallel on some of them which are
known to be very slow due to this sequentiality.

To enable the parallel features, the 'parallel' argument just have to be
added to the add_ceph_unittest() call like in :
    -add_ceph_unittest(unittest_throttle)
    +add_ceph_unittest(unittest_throttle parallel)

This commit impact the following tests :

Test name                          Before   After (in seconds)
unittest_erasure_code_shec_all:       212      43
unittest_throttle                      15       5
unittest_crush                          9       6
unittest_rbd_mirror                    79      21

Total                                 315      75

This commit saves 240 seconds (4 minutes) per build.

Note it exist several other long tests but can't be parallelized since
there is explicit dependencies in the order to run the subtests.
Those stay sequential.

Signed-off-by: Erwan Velu <erwan@redhat.com>

mgr/dashboard: Remove unnecessary fake services

Removed unnecessary fake services.

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #22390 from votdev/rgw_metadata_self

rgw: Get the user metadata of the user used to sign the request

mgr/dashboard: Extend dev scripts for API testing

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Make noise when AuthRequired used incorrectly

Don't fail silently when AuthRequired is used incorrectly, namely
without parentheses.

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Extend `RestController.bulk_set` test

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard/backend: Enable get/set of cluster-wide OSD settings

Add ability to list, set and unset cluster-wide OSD flags.

Flags can be listed and changed through the `/api/osd/flags` API
resource. By using a GET request, the list is retrieved. By using a PUT
request, the flags are updated (all at once). Flags not contained in the
data of the PUT are removed, additional once are added. Note that the
PUT requests require a JSON body with the data contained as value of the
'flags' key like so:

    {"flags": ["flag1", "flag2", ...]}

Fixes: http://tracker.ceph.com/issues/24056
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: .pylintrc: ignore too-many-statements check

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #22459 from p-na/grafana-token-auth

mgr/dashboard: Add token authentication to Grafana proxy

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22014 from Devp00l/wip-task-wrapper

mgr/dashboard: Task wrapper service

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

pybind/rbd: add assert_raise in test set_snap

Signed-off-by: Zheng Yin <zhengyin@cmss.chinamobile.com>

mds: add dump inode command

dump inode with an specific inode number
this is useful when we only known the inode number.

Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Merge pull request #22524 from tchaikov/wip-history-les-bound

osd/PG: unset history_les_bound if local-les is used

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #22472 from majianpeng/bluestore-misc-cleanup

os/bluestore: misc cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22538 from alimaredia/wip-vstart-cmake-cls-opt

cmake: Add cls_opt for vstart target

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

tools/rbd/action: align column headers left

Even if the column content is right-aligned, we should align the column
header left.  That way intead of

  SNAPID NAME  SIZE TIMESTAMP                NAMESPACE
       4 snap 1 GiB Mon Jun  4 16:27:17 2018 user

which has makes it hard to visually parse what that 1 belongs to (NAME or
SIZE?), we get

  SNAPID NAME SIZE  TIMESTAMP                NAMESPACE
       4 snap 1 GiB Mon Jun  4 16:27:17 2018 user

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: Add cls_opt for vstart target

Adding a dependency on cls_opt for the radosgw so
that when the vstart target is made, `radosgw-admin
mfa` commands work.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge PR #22554 into master

* refs/pull/22554/head:
qa/standalone/ceph-helpers.sh: Fixing comment for wait_for_health()
tests: Protecting rados bench against endless loop
qa/standalone/ceph-helpers.sh: Defining custom timeout for wait_for_clean()

Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #22491 from tchaikov/wip-seastar-msgr

crimson: port messenger to seastar

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22547 from tchaikov/wip-spdk

spdk: update to latest spdk-18.05 branch

Tested-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

mgr/dashboard: Executing messages for tasks

Signed-off-by: Stephan Müller <smueller@suse.com>