qa: create common conf for all cephfs suites

This will be followed by removing common CephFS configurations in the
ceph.conf.template in teuthology.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove wrongly created random distro conf

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22719 into master

* refs/pull/22719/head:
qa: fix symlinks indirectly pointing at qa to .qa
qa: add .qa helper link

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22631 from tianshan/fix_notify_thread

rgw: fix meta and data notify thread miss stop cr manager

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22632 from tianshan/fix_index_zones_trace

rgw: fix index complete miss zones_trace set

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22660 from cfanz/wip-rgw-fix-24603

rgw-multisite: fix endless loop in RGWBucketShardIncrementalSyncCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #22725 into master

* refs/pull/22725/head:
qa/workunits/suites/blogbench.sh: use correct dir name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #22656 into master

* refs/pull/22656/head:
doc: update experimental features - snapshots
doc: highlight the flags

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22698 from cbodley/wip-async-shared-mutex

common: reintroduce async SharedMutex

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #22724 from ceph/rm-23941

ceph-volume: error on commands that need ceph.conf to operate

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22710 from tchaikov/wip-config-without-lock

common: add lockless `md_config_t`

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22712 from tspmelo/wip-package-version

mgr/dashboard: fix the version of all frontend dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22734 from s0nea/wip-update-map

githubmap, organizationmap: cleanup and add/update contributors/affiliation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #20297 from tanghaodong25/rdma-cm

msg/async/rdma: add iWARP RDMA protocol support

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: error on commands that need ceph.conf to operate

We had been ignoring the failure to load ceph.conf for all subcommands
but most of them require that a ceph.conf be present. This changes that
so only commands that do not need ceph.conf ignore the failure to find
it.

Fixes: http://tracker.ceph.com/issues/23941
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

organizationmap: add openATTIC team affiliation

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

organizationmap: fix typo

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

githubmap: update contributors

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

githubmap: cleanup alphabetical order

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

doc: update experimental features - snapshots

* drop the experimental warning of multi-MDS
* update snapshotting blocked off version
* update Multiple filesystems availability in Jewel

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #22673 from dillaman/wip-24558

rbd: basic support for images within namespaces

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22701 from falcon78921/wip-docs-falcon78921

doc: remove documentation for installing google-perftools on Debian systems

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: link librbd_test against Boost::thread

otherwise we will have:

[ 83%] Building CXX object src/tools/rbd/CMakeFiles/rbd.dir/Utils.cc.o
../../../lib/librbd_test.a(test_ImageWatcher.cc.o): In function
`boost::thread::do_try_join_until(timespec const&)':
/home/kchai/ceph/build/boost/include/boost/thread/detail/thread.hpp:789:
undefined reference to
`boost::thread::do_try_join_until_noexcept(timespec const&, bool&)'
collect2: error: ld returned 1 exit status
src/test/librbd/CMakeFiles/ceph_test_librbd.dir/build.make:137: recipe
for target 'bin/ceph_test_librbd' failed
make[2]: *** [bin/ceph_test_librbd] Error 1

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: add LockPolicy to md_config_t

before this change, we guard all read/write access to md_config_t with a
lock. after this change, this policy is optional. and is controled by a
enum named `LockPolicy`. we will use `md_config_impl<LockPolicy::SINGLE>`
to implement the lockless config used by crimson/osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22418 from wido/mgr-telegraf-generator

mgr/telegraf: Use Python generator and catch OSError

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22434 from wido/zabbix-pgs

mgr/zabbix: Send more PG information to Zabbix

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22436 from wido/mgr-telegraf-pg-stats

mgr/telegraf: Send more PG status information to Telegraf

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22556 from majianpeng/bluestore-misc-cleanup2

os/bluestore: cleanups

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #22728 from majianpeng/fio-compiler-error

test/fio: fix compiler failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/fio: fix compiler error.

Compile  with -DWITH_FIO=ON, met the following bug.
/home/ceph/src/test/fio/fio_ceph_objectstore.cc:148:9: error: ‘unique_ptr’ in namespace ‘ceph’ does not name a template type
   ceph::unique_ptr<std::mutex> lock;
         ^~~~~~~~~~
/home/ceph/src/test/fio/fio_ceph_objectstore.cc: In constructor ‘{anonymous}::Collection::Collection(const spg_t&, ObjectStore::CollectionHandle)’:
/home/ceph/src/test/fio/fio_ceph_objectstore.cc:158:9: error: class ‘{anonymous}::Collection’ does not have any field named ‘lock’
         lock(new std::mutex) {
         ^~~~
/home/ceph/src/test/fio/fio_ceph_objectstore.cc: In function ‘fio_q_status {anonymous}::fio_ceph_os_queue(thread_data*, io_u*)’:
/home/ceph/src/test/fio/fio_ceph_objectstore.cc:604:38: error: ‘struct {anonymous}::Collection’ has no member named ‘lock’
  std::lock_guard<std::mutex> l(*coll.lock);

This bug introduct by commit da5d156b6bdffdd31e4edbadc1cdfc69197bef49.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

doc: updated documentation for installing google-perftools

Made updated changes per @tchaikov's request

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

qa/workunits/suites/blogbench.sh: use correct dir name

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #22294 into master

* refs/pull/22294/head:
common/performance: Now hobject_t's hash no longer depends directly on oid.
common/performance: improved hash calculation for hobject_t and ghobject_t

Reviewed-by: Paul Emmerich <paul.emmerich@croit.io>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22696 into master

* refs/pull/22696/head:
mgr/DaemonServer: avoid dereferencing end() iterator

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #21544 into master

* refs/pull/21544/head:
common/config: support to save and re-expand special metavariables

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22580 into master

* refs/pull/22580/head:
mds: improve error handling in PurgeQueue

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22611 into master

* refs/pull/22611/head:
client: fix use-after-free in Client::link()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common: SharedMutex uses ref-counted implementation

add a reference-counted SharedMutexImpl so that lock guards can outlive
the SharedMutex itself. this is required because the lock guards are
passed with async completions, and there is no guarantee that the
executor will process those completions before the SharedMutex
destructs. this case is exercised by the async_destruct unit test

Fixes: http://tracker.ceph.com/issues/24124
Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: fix symlinks indirectly pointing at qa to .qa

Building on the previous commit.

Command used:

$ find suites/ -type l -and -not -name .qa -execdir ~/fix.sh {} \;

fix.sh:
    #!/bin/bash

    link="$(readlink "$1")"

    echo $link
    dirlink="$(dirname "$link")"
    baselink="$(basename "$link")"

    while true; do
        echo $dirlink
        if [ "$dirlink" -ef ~/ceph/qa ]; then
            ln -nsf ".qa/$baselink" "$1"
            exit
        else
            baselink="$(basename "$dirlink")/$baselink"
            dirlink="$(dirname "$dirlink")"
            if [ "$dirlink" -ef . ]; then
                break
            fi
        fi
    done

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add .qa helper link

This utilizes the recent feature in teuthology [1] to skip hidden files in
suites when building the job matrix.

Idea of this change is to enable referring to the top-level qa directory in a
position-independent way such that copies of a suite to another location do not
break any symlinks.

[1] https://github.com/ceph/teuthology/pull/1185

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20734 from tchaikov/wip-s-libs-lib

test,qa: s/.libs/lib/

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22690 from tchaikov/wip-no-memory.h

include/memory.h: remove memory.h

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22672 from tspmelo/wip-encode-uri

mgr/dashboard: Add support for URI encode

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

test,qa: s/.libs/lib/

after switching to cmake, the libraries are put in build/lib, instead of
.libs. so point the default settings to ".lib".

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22312 from ZVampirEM77/wip-default-zonegroup-apiname

rgw: fix CreateBucket with BucketLocation parameter failed under default zonegroup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Add support for URI encode

Created a decorator and pipe to help encode special URI components in the
frontend.

Modified the backend request handler to decode all the string args.

fixes: http://tracker.ceph.com/issues/24621

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Prettify some frontend files

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22715 from LenzGr/mgr-dashboard-24623

mgr/dashboard: Replaced "Pool" with "Pools" in navigation bar

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22283 from rjfd/wip-dashboard-user-roles

mgr/dashboard: role based authentication/authorization system

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22663 from joscollin/wip-doc-dashboard

doc: updated get-involved.rst for ceph-dashboard

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: User permissions control (frontend)

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: controllers: fix endpoint return type when method returns `bytes`

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: controllers: fix controllers endpoints initialization order

Signed-off-by: Ricardo Dias <rdias@suse.com>

qa/tasks/mgr/dashboard: test_cluster_conf: restore original value of mon_allow_pool_delete

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: hacking.rst: update controller access control instructions

Signed-off-by: Ricardo Dias <rdias@suse.com>

doc/mgr/dashboard: user and role management documentation

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: auth: return user permissions on login

Signed-off-by: Ricardo Dias <rdias@suse.com>

qa/tasks/mgr/dashboard: Adapted tests to work with new authentication system

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: configured security info for each controller

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: removed `@AuthRequired` decorator

Now all controllers require authentication by default.
No disable authentication, a boolean parameter can be passed
in the `@ApiController` decorator.

Fixes: http://tracker.ceph.com/issues/23796
Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: auth: remove single user authentication

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: pylint: updated list of files to verify

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: tests: local role-based authorization

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: load and cofigure authorization system

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: authmanager: authorization handling cherrypy tool

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: local role-based authorization system implementation

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: security: static scopes and permissions

Signed-off-by: Ricardo Dias <rdias@suse.com>

doc: updated get-involved.rst for ceph-dashboard

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #22706 from joscollin/wip-doc-dashboardgrafana

doc/dashboard: fix formatting on Grafana instructions-2

mgr/dashboard: fix the version of all frontend dependencies

This will reduce unexpected errors when building dashboard frontend.

Signed-off-by: Tiago Melo <tmelo@suse.com>

include/memory.h: remove memory.h

memory.h was introduced back when the shared_ptrs were still in TR1, but
we've moved to C++17 now. and the C++ clients should be compiled with
a C++11 compatible compiler. so there is no need to have this file anymore.

also replace all references of ceph::shared_ptr and ceph::weak_ptr with
std::shared_ptr and std::weak_ptr accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22694 from dillaman/wip-test-rbd-mirror

test/rbd-mirror: disable use of gtest-parallel

Reviewed-by: Erwan Velu <erwan@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22591 from tchaikov/wip-gcc-8.1

cmake: disable -Werror-stringop-truncation for rocksdb

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/dashboard: fix formatting on Grafana instructions

The whole list was in a code block, instead of just the snippets.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #22691 into master

* refs/pull/22691/head:
os/bluestore: fix minor issues in bitmap logging output
os/bluestore: fix overlappedd allocation returned by bitmap allocator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22682 from wwformat/fix-centos-install-deps

install-deps: check the exit status for the $builddepcmd

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: updated documentation for installing google-perftools on Debian systems

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge PR #22532 into master

* refs/pull/22532/head:
qa/suites: Add supported-random-distro$ links.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/DaemonServer: avoid dereferencing end() iterator

Fixes: http://tracker.ceph.com/issues/23249
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22626 into master

* refs/pull/22626/head:
scripts: add ceph-common-dbg to debug packages

Reviewed-by: David Zafman <dzafman@redhat.com>

scripts: add ceph-common-dbg to debug packages

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add an async SharedMutex

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rbd-mirror: disable use of gtest-parallel

This test repeatedly deadlocks when run under in parallel.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22693 from tspmelo/fix-lint-codelyzer

mgr/dashboard: fix lint error caused by codelyzer update

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix lint error caused by codelyzer update

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22544 from ifed01/wip-ifed-alloc-bench-fix

test/objectstore: fix random generator in allocator_bench.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluestore: fix minor issues in bitmap logging output

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: fix overlappedd allocation returned by bitmap allocator

Fixes: https://tracker.ceph.com/issues/24598
Signed-off-by: Igor Fedotov <ifedotov@suse.som>

mgr/dashboard: Replaced "Pool" with "Pools" in navigation bar

Replaced the string "Pool" with "Pools" in the navigation component.

Fixes: https://tracker.ceph.com/issues/24623
Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

install-deps: check the exit status for the $builddepcmd

in some case, the $builddepcmd will failed without any "error:" output.
so we should check the exit status to handle it.

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

Merge PR #22479 into master

* refs/pull/22479/head:
mgr/devicehealth: add self-test
mgr/devicehealth: make config options instance attributes
mgr/devicehealth: active -> enable_monitoring
mgr/mgr_module: update docstring for get()
mgr/devicehealth: fix 'device get-health-metrics' when no metrics are stored
mgr/devicehealth: add some scraping infrastructure
osd: optionally query smart for a single devid
mgr/ActivePyModules: add get for 'device <devid>'
osd: key smart results by unique device id
mgr: enable devicehealth by default
mgr/devicehealth: rename old smart module to devicehealth

Merge PR #22572 into master

* refs/pull/22572/head:
osd: update clone_overlap even the clone have been evicted

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22576 into master

* refs/pull/22576/head:
common/obj_bencher.cc: faster object name generation

Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge PR #22588 into master

* refs/pull/22588/head:
mgr/iostat: column_width should be int not float

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Mohamad Gebai <mgebai@suse.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22615 into master

* refs/pull/22615/head:
common: fix enum redeclaration

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22620 into master

* refs/pull/22620/head:
common/WorkQueue: Before set_ioprio it must do create/try_create.

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #22634 into master

* refs/pull/22634/head:
mgr, pybind/mgr: pass inbuf (ceph -i <file>) to modules

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/devicehealth: add self-test

Signed-off-by: Sage Weil <sage@redhat.com>