Merge pull request #22756 from wjwithagen/wjw-fix-missing-vector

mgr: `#include <vector>` for clang

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22752 from tchaikov/wip-freebsd-w/o-rdma

cmake,do_freebsd.sh: disable rdma features

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #22743 from rzarzynski/wip-common-drop-static_assert.h

common: drop static_assert.h as it looks unused.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22720 from rzarzynski/wip-build-static-ceph-common-in-osd

cmake: link ceph-osd with common statically

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22727 from dzafman/wip-21664

qa/standalone/scrub: When possible show side-by-side diff in addition to regular diff

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22763 from dzafman/wip-remove-sudo

qa: Don't use sudo when moving logs

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22441 from ErwanAliasr1/evelu-makecheck

Improving make check reliability

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

qa: Don't use sudo when moving logs

Caused by: f0964beac5b8b54bc44a11bfb50bdea1f2ae8825

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #22565 from tchaikov/wip-seastar-thread-pool

crimson/thread: add thread pool

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: add crimson static library

libcrimson consolidates the object library of crimson_{net,thread} and
seastar_{buffer,net,thread}_objs. by offering a static library glueing
seastar,  we can link against these libraries and its dependencies in a
simpler way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/thread: add a thread pool impl

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22007 from dalgaaf/wip-da-SCA-20180514

rgw, common: Fixes SCA issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: extract the RDMA libraries interface properties out

Signed-off-by: Kefu Chai <kchai@redhat.com>

do_freebsd.sh: disable rdma features

FreeBSD does have libibiverb and librdmacm, but we disable them anyway
for minimal dependency.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: consolidate WITH_{VERBS,RDMACM}

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22759 from ricardoasmarques/wip-fix-rbd-features-style

mgr/dashboard: Fix RBD features style

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22750 from joscollin/wip-dashboard-intro

doc: updated the overview and glossary for dashboard

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Fix RBD features style

Signed-off-by: Ricardo Marques <rimarques@suse.com>

doc: Updated dashboard doc references

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: updated the overview and glossary for dashboard

* Updated the overview of dashboard to avoid many paragraphs,
some outdated statements, etc.
* Updated glossary, glossary references and urls
* Dropped unnecessary term *module* in possible places
* Unified the new dashboard name as *Ceph Manager Dashboard*

Signed-off-by: Jos Collin <jcollin@redhat.com>

mgr: fix Clangs/FreeBSD missing vector

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #22702 from dzafman/wip-standalone-log

qa: For teuthology copy logs to teuthology expected location

Reviewed-by: Neha Ojha <nojha@redhat.com>

cmake: remove stale comments

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22726 from yuriw/wip-yuriw-crontab

qa/tests - added mimic-p2p suite

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #22560 into master

* refs/pull/22560/head:
qa/fs/upgrade/snaps: whitelist inconsistent rstat
mds: track snapshot count in rstat
mds: add MAY_SNAPSHOT in MDSAuthCaps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22719 into master

* refs/pull/22719/head:
qa: fix symlinks indirectly pointing at qa to .qa
qa: add .qa helper link

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22631 from tianshan/fix_notify_thread

rgw: fix meta and data notify thread miss stop cr manager

Reviewed-by: Casey Bodley <cbodley@redhat.com>

common: drop static_assert.h as it looks unused.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #22632 from tianshan/fix_index_zones_trace

rgw: fix index complete miss zones_trace set

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22660 from cfanz/wip-rgw-fix-24603

rgw-multisite: fix endless loop in RGWBucketShardIncrementalSyncCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #22725 into master

* refs/pull/22725/head:
qa/workunits/suites/blogbench.sh: use correct dir name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #22656 into master

* refs/pull/22656/head:
doc: update experimental features - snapshots
doc: highlight the flags

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22698 from cbodley/wip-async-shared-mutex

common: reintroduce async SharedMutex

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #22724 from ceph/rm-23941

ceph-volume: error on commands that need ceph.conf to operate

Reviewed-by: Alfredo Deza <adeza@redhat.com>

build: link ceph-osd with common statically. Enforce IE TLS model.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #22710 from tchaikov/wip-config-without-lock

common: add lockless `md_config_t`

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22712 from tspmelo/wip-package-version

mgr/dashboard: fix the version of all frontend dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22734 from s0nea/wip-update-map

githubmap, organizationmap: cleanup and add/update contributors/affiliation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #20297 from tanghaodong25/rdma-cm

msg/async/rdma: add iWARP RDMA protocol support

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: error on commands that need ceph.conf to operate

We had been ignoring the failure to load ceph.conf for all subcommands
but most of them require that a ceph.conf be present. This changes that
so only commands that do not need ceph.conf ignore the failure to find
it.

Fixes: http://tracker.ceph.com/issues/23941
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

organizationmap: add openATTIC team affiliation

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

organizationmap: fix typo

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

githubmap: update contributors

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

githubmap: cleanup alphabetical order

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

doc: update experimental features - snapshots

* drop the experimental warning of multi-MDS
* update snapshotting blocked off version
* update Multiple filesystems availability in Jewel

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #22673 from dillaman/wip-24558

rbd: basic support for images within namespaces

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22701 from falcon78921/wip-docs-falcon78921

doc: remove documentation for installing google-perftools on Debian systems

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: link librbd_test against Boost::thread

otherwise we will have:

[ 83%] Building CXX object src/tools/rbd/CMakeFiles/rbd.dir/Utils.cc.o
../../../lib/librbd_test.a(test_ImageWatcher.cc.o): In function
`boost::thread::do_try_join_until(timespec const&)':
/home/kchai/ceph/build/boost/include/boost/thread/detail/thread.hpp:789:
undefined reference to
`boost::thread::do_try_join_until_noexcept(timespec const&, bool&)'
collect2: error: ld returned 1 exit status
src/test/librbd/CMakeFiles/ceph_test_librbd.dir/build.make:137: recipe
for target 'bin/ceph_test_librbd' failed
make[2]: *** [bin/ceph_test_librbd] Error 1

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: add LockPolicy to md_config_t

before this change, we guard all read/write access to md_config_t with a
lock. after this change, this policy is optional. and is controled by a
enum named `LockPolicy`. we will use `md_config_impl<LockPolicy::SINGLE>`
to implement the lockless config used by crimson/osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22418 from wido/mgr-telegraf-generator

mgr/telegraf: Use Python generator and catch OSError

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22434 from wido/zabbix-pgs

mgr/zabbix: Send more PG information to Zabbix

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22436 from wido/mgr-telegraf-pg-stats

mgr/telegraf: Send more PG status information to Telegraf

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22556 from majianpeng/bluestore-misc-cleanup2

os/bluestore: cleanups

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #22728 from majianpeng/fio-compiler-error

test/fio: fix compiler failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/fio: fix compiler error.

Compile  with -DWITH_FIO=ON, met the following bug.
/home/ceph/src/test/fio/fio_ceph_objectstore.cc:148:9: error: ‘unique_ptr’ in namespace ‘ceph’ does not name a template type
   ceph::unique_ptr<std::mutex> lock;
         ^~~~~~~~~~
/home/ceph/src/test/fio/fio_ceph_objectstore.cc: In constructor ‘{anonymous}::Collection::Collection(const spg_t&, ObjectStore::CollectionHandle)’:
/home/ceph/src/test/fio/fio_ceph_objectstore.cc:158:9: error: class ‘{anonymous}::Collection’ does not have any field named ‘lock’
         lock(new std::mutex) {
         ^~~~
/home/ceph/src/test/fio/fio_ceph_objectstore.cc: In function ‘fio_q_status {anonymous}::fio_ceph_os_queue(thread_data*, io_u*)’:
/home/ceph/src/test/fio/fio_ceph_objectstore.cc:604:38: error: ‘struct {anonymous}::Collection’ has no member named ‘lock’
  std::lock_guard<std::mutex> l(*coll.lock);

This bug introduct by commit da5d156b6bdffdd31e4edbadc1cdfc69197bef49.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

qa/fs/upgrade/snaps: whitelist inconsistent rstat

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

doc: updated documentation for installing google-perftools

Made updated changes per @tchaikov's request

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

test: When possible show side-by-side diff in addition to regular diff

Fixes: https://tracker.ceph.com/issues/21664
Signed-off-by: David Zafman <dzafman@redhat.com>

qa/tests - added mimic-p2p suite

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/workunits/suites/blogbench.sh: use correct dir name

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #22294 into master

* refs/pull/22294/head:
common/performance: Now hobject_t's hash no longer depends directly on oid.
common/performance: improved hash calculation for hobject_t and ghobject_t

Reviewed-by: Paul Emmerich <paul.emmerich@croit.io>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22696 into master

* refs/pull/22696/head:
mgr/DaemonServer: avoid dereferencing end() iterator

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #21544 into master

* refs/pull/21544/head:
common/config: support to save and re-expand special metavariables

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22580 into master

* refs/pull/22580/head:
mds: improve error handling in PurgeQueue

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22611 into master

* refs/pull/22611/head:
client: fix use-after-free in Client::link()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common: SharedMutex uses ref-counted implementation

add a reference-counted SharedMutexImpl so that lock guards can outlive
the SharedMutex itself. this is required because the lock guards are
passed with async completions, and there is no guarantee that the
executor will process those completions before the SharedMutex
destructs. this case is exercised by the async_destruct unit test

Fixes: http://tracker.ceph.com/issues/24124
Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: fix symlinks indirectly pointing at qa to .qa

Building on the previous commit.

Command used:

$ find suites/ -type l -and -not -name .qa -execdir ~/fix.sh {} \;

fix.sh:
    #!/bin/bash

    link="$(readlink "$1")"

    echo $link
    dirlink="$(dirname "$link")"
    baselink="$(basename "$link")"

    while true; do
        echo $dirlink
        if [ "$dirlink" -ef ~/ceph/qa ]; then
            ln -nsf ".qa/$baselink" "$1"
            exit
        else
            baselink="$(basename "$dirlink")/$baselink"
            dirlink="$(dirname "$dirlink")"
            if [ "$dirlink" -ef . ]; then
                break
            fi
        fi
    done

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add .qa helper link

This utilizes the recent feature in teuthology [1] to skip hidden files in
suites when building the job matrix.

Idea of this change is to enable referring to the top-level qa directory in a
position-independent way such that copies of a suite to another location do not
break any symlinks.

[1] https://github.com/ceph/teuthology/pull/1185

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20734 from tchaikov/wip-s-libs-lib

test,qa: s/.libs/lib/

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22690 from tchaikov/wip-no-memory.h

include/memory.h: remove memory.h

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22672 from tspmelo/wip-encode-uri

mgr/dashboard: Add support for URI encode

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

test,qa: s/.libs/lib/

after switching to cmake, the libraries are put in build/lib, instead of
.libs. so point the default settings to ".lib".

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22312 from ZVampirEM77/wip-default-zonegroup-apiname

rgw: fix CreateBucket with BucketLocation parameter failed under default zonegroup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Add support for URI encode

Created a decorator and pipe to help encode special URI components in the
frontend.

Modified the backend request handler to decode all the string args.

fixes: http://tracker.ceph.com/issues/24621

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Prettify some frontend files

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22715 from LenzGr/mgr-dashboard-24623

mgr/dashboard: Replaced "Pool" with "Pools" in navigation bar

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22283 from rjfd/wip-dashboard-user-roles

mgr/dashboard: role based authentication/authorization system

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22663 from joscollin/wip-doc-dashboard

doc: updated get-involved.rst for ceph-dashboard

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: User permissions control (frontend)

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: controllers: fix endpoint return type when method returns `bytes`

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: controllers: fix controllers endpoints initialization order

Signed-off-by: Ricardo Dias <rdias@suse.com>

qa/tasks/mgr/dashboard: test_cluster_conf: restore original value of mon_allow_pool_delete

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: hacking.rst: update controller access control instructions

Signed-off-by: Ricardo Dias <rdias@suse.com>

doc/mgr/dashboard: user and role management documentation

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: auth: return user permissions on login

Signed-off-by: Ricardo Dias <rdias@suse.com>

qa/tasks/mgr/dashboard: Adapted tests to work with new authentication system

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: configured security info for each controller

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: removed `@AuthRequired` decorator

Now all controllers require authentication by default.
No disable authentication, a boolean parameter can be passed
in the `@ApiController` decorator.

Fixes: http://tracker.ceph.com/issues/23796
Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: auth: remove single user authentication

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: pylint: updated list of files to verify

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: tests: local role-based authorization

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: load and cofigure authorization system

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: authmanager: authorization handling cherrypy tool

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: local role-based authorization system implementation

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: security: static scopes and permissions

Signed-off-by: Ricardo Dias <rdias@suse.com>

doc: updated get-involved.rst for ceph-dashboard

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #22706 from joscollin/wip-doc-dashboardgrafana

doc/dashboard: fix formatting on Grafana instructions-2

mgr/dashboard: fix the version of all frontend dependencies

This will reduce unexpected errors when building dashboard frontend.

Signed-off-by: Tiago Melo <tmelo@suse.com>

include/memory.h: remove memory.h

memory.h was introduced back when the shared_ptrs were still in TR1, but
we've moved to C++17 now. and the C++ clients should be compiled with
a C++11 compatible compiler. so there is no need to have this file anymore.

also replace all references of ceph::shared_ptr and ceph::weak_ptr with
std::shared_ptr and std::weak_ptr accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>