Merge pull request #22764 from ceph/wip-rm24504

ceph-volume: refuse to zap mapper devices

Reviewed-by: Alfredo Deza

Merge pull request #22808 from jcsp/wip-mgr-doc-initial-modules

doc/dashboard: don't advise mgr_initial_modules

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

doc/dashboard: don't advise mgr_initial_modules

This setting tends to confuse people, as it's only respected
on the very first startup of the cluster.  Instead, mention
it (with appropriate caveats) on the general mgr admin
page.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #22802 from bradfitz/its

doc: fix some it's -> its typos

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: fix some it's -> its typos

Signed-off-by: Brad Fitzpatrick <brad@danga.com>

Merge pull request #22761 from fullerdj/wip-djf-24686

osd/filestore: Change default filestore_merge_threshold to -10

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21772 from oritwas/wip-rgw-test-bi-list

rgw: test bi list

Merge pull request #22769 from idryomov/wip-rbdmap-remote-fs-pre

systemd/rbdmap.service: order us before remote-fs-pre.target

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #22760 from s0nea/wip-dashboard-config-option-flags

mgr/dashboard: add supported flag information to config options documentation

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22772 from joscollin/wip-doc-dashboard-ref

doc: s/doc/ref for dashboard urls

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22776 from tchaikov/wip-thread-setaffinity

crimson/thread: pin thread pool to given CPU

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/thread: pin thread pool to given CPU

to take the full advantage of seastar reactor, we need to confine the
alien threads to given CPU core(s). in current setting, it's assumed
that alien threads do not perform CPU-bound tasks, so a single core
would suffice. we can always change the interface to pass a cpu_set_t
or a std::bitset if one CPU core is not enough.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #22111 into master

* refs/pull/22111/head:
cephfs-journal-tool: check fsid when import purge queue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22718 into master

* refs/pull/22718/head:
qa/workunits/suites/ffsb.sh: fix strncpy bug in ffsb

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd/filestore: Change default filestore_merge_threshold to -1

Performance evaluations of medium to large size Ceph clusters have
demonstrated negligible performance impact from unnecessarily deep
directory hierarchies but significant performance impact from filestore
split and merge activity. Disable merges by default.

Fixes: http://tracker.ceph.com/issues/24686
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #22736 from tchaikov/wip-seastar-lockless-shared-lru

common/shared_cache: add lockless SharedLRU

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: s/doc/ref for dashboard urls

Replaced :doc: with :ref: as per https://github.com/ceph/ceph/pull/22750#discussion_r198895040.

Signed-off-by: Jos Collin <jcollin@redhat.com>

systemd/rbdmap.service: order us before remote-fs-pre.target

If "/usr/bin/rbdmap unmap-all" notices a file system mounted on top of
an rbd device, it will call umount, interfering with systemd shutdown
logic.  Make sure we aren't invoked until all _netdev mounts are dealt
with by systemd.

Fixes: http://tracker.ceph.com/issues/24713
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #22771 from tchaikov/wip-test-librados-ns

test/librados: fix LibRadosList.ListObjectsNS

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

lvm zap: refuse to zap mapper devices

Fixes: http://tracker.ceph.com/issues/24504
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

adds the util.disk.is_mapper_device method

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22086 from ifed01/wip-ifed-new-bluefs-pcounter

os/bluestore: introduce new BlueFS perf counter to track the amount of

Reviewed-by: Sage Weil <sage@redhat.com>

test/librados: fix LibRadosList.ListObjectsNS

rados_ioctx_get_namespace() returns the length of namespace string on
success, not 0.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21634 from smithfarm/wip-bci-complain-dupe

scripts: backport-create-issue: complain about duplicates and support mimic

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: add flag info to config options documentation page

Add information about the supported flags (introduced by
https://github.com/ceph/ceph/pull/22595) to the config options
documentation page.

Fixes: https://tracker.ceph.com/issues/24605
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #22756 from wjwithagen/wjw-fix-missing-vector

mgr: `#include <vector>` for clang

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22752 from tchaikov/wip-freebsd-w/o-rdma

cmake,do_freebsd.sh: disable rdma features

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #22743 from rzarzynski/wip-common-drop-static_assert.h

common: drop static_assert.h as it looks unused.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22720 from rzarzynski/wip-build-static-ceph-common-in-osd

cmake: link ceph-osd with common statically

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22727 from dzafman/wip-21664

qa/standalone/scrub: When possible show side-by-side diff in addition to regular diff

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22763 from dzafman/wip-remove-sudo

qa: Don't use sudo when moving logs

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22441 from ErwanAliasr1/evelu-makecheck

Improving make check reliability

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

qa: Don't use sudo when moving logs

Caused by: f0964beac5b8b54bc44a11bfb50bdea1f2ae8825

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #22565 from tchaikov/wip-seastar-thread-pool

crimson/thread: add thread pool

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: add crimson static library

libcrimson consolidates the object library of crimson_{net,thread} and
seastar_{buffer,net,thread}_objs. by offering a static library glueing
seastar,  we can link against these libraries and its dependencies in a
simpler way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/thread: add a thread pool impl

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22007 from dalgaaf/wip-da-SCA-20180514

rgw, common: Fixes SCA issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: extract the RDMA libraries interface properties out

Signed-off-by: Kefu Chai <kchai@redhat.com>

do_freebsd.sh: disable rdma features

FreeBSD does have libibiverb and librdmacm, but we disable them anyway
for minimal dependency.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: consolidate WITH_{VERBS,RDMACM}

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22759 from ricardoasmarques/wip-fix-rbd-features-style

mgr/dashboard: Fix RBD features style

Reviewed-by: Tiago Melo <tmelo@suse.com>

systemd/rbdmap.service: remove a dependency on local-fs.target

We don't require anything outside of rootfs.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #22750 from joscollin/wip-dashboard-intro

doc: updated the overview and glossary for dashboard

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Fix RBD features style

Signed-off-by: Ricardo Marques <rimarques@suse.com>

doc: Updated dashboard doc references

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: updated the overview and glossary for dashboard

* Updated the overview of dashboard to avoid many paragraphs,
some outdated statements, etc.
* Updated glossary, glossary references and urls
* Dropped unnecessary term *module* in possible places
* Unified the new dashboard name as *Ceph Manager Dashboard*

Signed-off-by: Jos Collin <jcollin@redhat.com>

mgr: fix Clangs/FreeBSD missing vector

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #22702 from dzafman/wip-standalone-log

qa: For teuthology copy logs to teuthology expected location

Reviewed-by: Neha Ojha <nojha@redhat.com>

cmake: remove stale comments

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22726 from yuriw/wip-yuriw-crontab

qa/tests - added mimic-p2p suite

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #22560 into master

* refs/pull/22560/head:
qa/fs/upgrade/snaps: whitelist inconsistent rstat
mds: track snapshot count in rstat
mds: add MAY_SNAPSHOT in MDSAuthCaps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22719 into master

* refs/pull/22719/head:
qa: fix symlinks indirectly pointing at qa to .qa
qa: add .qa helper link

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22631 from tianshan/fix_notify_thread

rgw: fix meta and data notify thread miss stop cr manager

Reviewed-by: Casey Bodley <cbodley@redhat.com>

common: drop static_assert.h as it looks unused.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #22632 from tianshan/fix_index_zones_trace

rgw: fix index complete miss zones_trace set

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22660 from cfanz/wip-rgw-fix-24603

rgw-multisite: fix endless loop in RGWBucketShardIncrementalSyncCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #22725 into master

* refs/pull/22725/head:
qa/workunits/suites/blogbench.sh: use correct dir name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #22656 into master

* refs/pull/22656/head:
doc: update experimental features - snapshots
doc: highlight the flags

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22698 from cbodley/wip-async-shared-mutex

common: reintroduce async SharedMutex

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #22724 from ceph/rm-23941

ceph-volume: error on commands that need ceph.conf to operate

Reviewed-by: Alfredo Deza <adeza@redhat.com>

build: link ceph-osd with common statically. Enforce IE TLS model.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/shared_cache: add lockless SharedLRU

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: extract LockMutex and LockPolicy out

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/shared_cache: remove less and hash template parameters

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22710 from tchaikov/wip-config-without-lock

common: add lockless `md_config_t`

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22712 from tspmelo/wip-package-version

mgr/dashboard: fix the version of all frontend dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22734 from s0nea/wip-update-map

githubmap, organizationmap: cleanup and add/update contributors/affiliation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #20297 from tanghaodong25/rdma-cm

msg/async/rdma: add iWARP RDMA protocol support

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: error on commands that need ceph.conf to operate

We had been ignoring the failure to load ceph.conf for all subcommands
but most of them require that a ceph.conf be present. This changes that
so only commands that do not need ceph.conf ignore the failure to find
it.

Fixes: http://tracker.ceph.com/issues/23941
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

organizationmap: add openATTIC team affiliation

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

organizationmap: fix typo

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

githubmap: update contributors

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

os/bluestore: introduce new BlueFS perf counter to track the amount of
data written to slow device.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

githubmap: cleanup alphabetical order

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

doc: update experimental features - snapshots

* drop the experimental warning of multi-MDS
* update snapshotting blocked off version
* update Multiple filesystems availability in Jewel

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #22673 from dillaman/wip-24558

rbd: basic support for images within namespaces

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22701 from falcon78921/wip-docs-falcon78921

doc: remove documentation for installing google-perftools on Debian systems

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: link librbd_test against Boost::thread

otherwise we will have:

[ 83%] Building CXX object src/tools/rbd/CMakeFiles/rbd.dir/Utils.cc.o
../../../lib/librbd_test.a(test_ImageWatcher.cc.o): In function
`boost::thread::do_try_join_until(timespec const&)':
/home/kchai/ceph/build/boost/include/boost/thread/detail/thread.hpp:789:
undefined reference to
`boost::thread::do_try_join_until_noexcept(timespec const&, bool&)'
collect2: error: ld returned 1 exit status
src/test/librbd/CMakeFiles/ceph_test_librbd.dir/build.make:137: recipe
for target 'bin/ceph_test_librbd' failed
make[2]: *** [bin/ceph_test_librbd] Error 1

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: add LockPolicy to md_config_t

before this change, we guard all read/write access to md_config_t with a
lock. after this change, this policy is optional. and is controled by a
enum named `LockPolicy`. we will use `md_config_impl<LockPolicy::SINGLE>`
to implement the lockless config used by crimson/osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22418 from wido/mgr-telegraf-generator

mgr/telegraf: Use Python generator and catch OSError

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22434 from wido/zabbix-pgs

mgr/zabbix: Send more PG information to Zabbix

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22436 from wido/mgr-telegraf-pg-stats

mgr/telegraf: Send more PG status information to Telegraf

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22556 from majianpeng/bluestore-misc-cleanup2

os/bluestore: cleanups

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #22728 from majianpeng/fio-compiler-error

test/fio: fix compiler failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/fio: fix compiler error.

Compile  with -DWITH_FIO=ON, met the following bug.
/home/ceph/src/test/fio/fio_ceph_objectstore.cc:148:9: error: ‘unique_ptr’ in namespace ‘ceph’ does not name a template type
   ceph::unique_ptr<std::mutex> lock;
         ^~~~~~~~~~
/home/ceph/src/test/fio/fio_ceph_objectstore.cc: In constructor ‘{anonymous}::Collection::Collection(const spg_t&, ObjectStore::CollectionHandle)’:
/home/ceph/src/test/fio/fio_ceph_objectstore.cc:158:9: error: class ‘{anonymous}::Collection’ does not have any field named ‘lock’
         lock(new std::mutex) {
         ^~~~
/home/ceph/src/test/fio/fio_ceph_objectstore.cc: In function ‘fio_q_status {anonymous}::fio_ceph_os_queue(thread_data*, io_u*)’:
/home/ceph/src/test/fio/fio_ceph_objectstore.cc:604:38: error: ‘struct {anonymous}::Collection’ has no member named ‘lock’
  std::lock_guard<std::mutex> l(*coll.lock);

This bug introduct by commit da5d156b6bdffdd31e4edbadc1cdfc69197bef49.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

qa/fs/upgrade/snaps: whitelist inconsistent rstat

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

doc: updated documentation for installing google-perftools

Made updated changes per @tchaikov's request

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

test: When possible show side-by-side diff in addition to regular diff

Fixes: https://tracker.ceph.com/issues/21664
Signed-off-by: David Zafman <dzafman@redhat.com>

qa/tests - added mimic-p2p suite

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/workunits/suites/blogbench.sh: use correct dir name

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #22294 into master

* refs/pull/22294/head:
common/performance: Now hobject_t's hash no longer depends directly on oid.
common/performance: improved hash calculation for hobject_t and ghobject_t

Reviewed-by: Paul Emmerich <paul.emmerich@croit.io>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22696 into master

* refs/pull/22696/head:
mgr/DaemonServer: avoid dereferencing end() iterator

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #21544 into master

* refs/pull/21544/head:
common/config: support to save and re-expand special metavariables

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22580 into master

* refs/pull/22580/head:
mds: improve error handling in PurgeQueue

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22611 into master

* refs/pull/22611/head:
client: fix use-after-free in Client::link()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common: SharedMutex uses ref-counted implementation

add a reference-counted SharedMutexImpl so that lock guards can outlive
the SharedMutex itself. this is required because the lock guards are
passed with async completions, and there is no guarantee that the
executor will process those completions before the SharedMutex
destructs. this case is exercised by the async_destruct unit test

Fixes: http://tracker.ceph.com/issues/24124
Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: fix symlinks indirectly pointing at qa to .qa

Building on the previous commit.

Command used:

$ find suites/ -type l -and -not -name .qa -execdir ~/fix.sh {} \;

fix.sh:
    #!/bin/bash

    link="$(readlink "$1")"

    echo $link
    dirlink="$(dirname "$link")"
    baselink="$(basename "$link")"

    while true; do
        echo $dirlink
        if [ "$dirlink" -ef ~/ceph/qa ]; then
            ln -nsf ".qa/$baselink" "$1"
            exit
        else
            baselink="$(basename "$dirlink")/$baselink"
            dirlink="$(dirname "$dirlink")"
            if [ "$dirlink" -ef . ]; then
                break
            fi
        fi
    done

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add .qa helper link

This utilizes the recent feature in teuthology [1] to skip hidden files in
suites when building the job matrix.

Idea of this change is to enable referring to the top-level qa directory in a
position-independent way such that copies of a suite to another location do not
break any symlinks.

[1] https://github.com/ceph/teuthology/pull/1185

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20734 from tchaikov/wip-s-libs-lib

test,qa: s/.libs/lib/

Reviewed-by: Sage Weil <sage@redhat.com>