build/ops: rpm: sane packaging of %{_docdir}/ceph directory

91648044163 introduced a build error:

ceph-common-12.1.0+git.1498286248.2fcedc7b3d-1.1.x86_64.rpm: directories not
owned by a package:
 - /usr/share/doc/packages/ceph

The %docdir directive is a way of flagging anything in that directory as being
documentation. It does not actually package the directory. And we don't need
it because we're not dumping a large number of files into this directory.
For more information, see the "Directory-related Directives" section of
http://ftp.rpm.org/max-rpm/s1-rpm-inside-files-list-directives.html

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15876 from gmayyyha/config-unused-opt

common/config_opts: drop unused opt

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15108 from guihecheng/rgw_file-fix

rgw_file: fix misuse of make_key_name before make_fhk

Merge pull request #15893 from liewegas/wip-valgrind-everywhere

qa: do not restrict valgrind runs to centos

qa: do not restrict valgrind runs to centos

This reverts 693bd238510e69569cc3461f84b04c8667bc11da, which was
added in response to http://tracker.ceph.com/issues/18126. But
we updated the Ubuntu packages in sepia so it should be good to go.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15572 from rzarzynski/wip-rgw-17779

rgw: external auth engines of S3 honor rgw_keystone_implicit_tenants.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #14624 from ceph/wip-s3a-hadoop

qa/tasks: S3A hadoop task to test s3a with Ceph

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15665 from oritwas/wip-rgw-reshard-old-bucket

rgw: auto reshard old buckets

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15821 from jdurgin/wip-20302

qa/suites/powercycle/osd/tasks/radosbench: consume less space

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15154 from jcsp/wip-multimds-stable

Remove "experimental" warnings from multimds

doc: multimds is no longer experimental

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update cephtool test for multimds on by default

Signed-off-by: John Spray <john.spray@redhat.com>

qa: no longer need to explicitly enable multimds

Signed-off-by: John Spray <john.spray@redhat.com>

mds: enable multimds by default in new filesystems

Signed-off-by: John Spray <john.spray@redhat.com>

mon: remove experimental warning on multimds

Signed-off-by: John Spray <john.spray@redhat.com>

mds: don't call StrayManager::eval_stray() for undefined inode

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: drop locks before waiting for export targets

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: handle MDirUpdate race

mds may try discover several times for MDirUpdate, rename may kick
in and cause MDCache::path_traverse() to return error.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't forge replica dirfrag

MDCache::forge_replica_dir() set wrong dir_auth if the forged replica
dirfrag is subtree root.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: avoid submitting log entry while adjusting subtree map

MDCache::eval_subtree_root() may tigger scatter-gather process, which
submits log entry. Submitting log entry while adjusting subtree map is
bad, because subtree map in intermediate state may get used/logged.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't mark nestlock dirty on improper inode

If inode is replica and it has no auth subtree dirfrag, We should
not mark its nestlock dirty.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: create subtree root immediately after directory tree becomes frozen

When a directory tree become frozen, its WAIT_FROZEN contexts are
executed asynchronously. Before Migrator::export_frozen() set export
bounds, MDCache::try_subtree_merge_at() can merge newly imported
subtree into the frozen directory tree. This causes problem if there
are auth pins in newly imported subtree.

The fix is creating subtree root immediately after directory tree
becomes frozen. The new subtree root has dir_auth 'me, me', so it's
not meregeable.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix stray dentry replication in cache rejoin ack

To replicate s stray dentry, we need to replicate all its ancestors.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #15795 from myoungwon/wip-print-ignore_redirect

osd/osd_types: add flag name (IGNORE_REDIRECT)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15656 from aclamk/download_err_with_comp_followup

rgw: use 64-bit offsets for compression

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15848 from xiexingguo/wip-fix-rmcc

src/vstart.sh: kill dead upmap option

Merge pull request #15851 from liewegas/wip-luminous-notes

doc/release-notes: update luminous notes

doc/release-notes: 'osd crush class rename' is coming

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: ceph tell <foo> help

Signed-off-by: Sage Weil <sage@redhat.com>

doc/start/os-recommendations: update

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: note debian stretch addition

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: sleep settings

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: link to EC docs

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: update RGW metadata

Signed-off-by: Sage Weil <sage@redhat.com>

dev/release-notes: various updates from other PR

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: notes on new CLI commands

Signed-off-by: Sage Weil <sage@redhat.com>

mon: 'mon feature list' -> 'mon feature ls'

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: update luminous notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15764 from tchaikov/wip-20342

qa/suites/upgrade/hammer-jewel-x: add luminous.yaml

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15877 from wjwithagen/wip-wjw-vstart-ceph-mgr-restfull

vstart.sh: Work around mgr restfull not available

./src/vstart.sh: Work around mgr restfull not available

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

common/config_opts: drop unused opt

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

Merge pull request #15832 from Adirl/pd_assert

msg/async/rdma: remove assert from ibv_dealloc_pd in ProtectionDomain

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15283 from donglinpeng/copyPartUpload

rgw: multipart copy-part remove '/' for s3 java sdk request header.
(make check failures not related)

Merge pull request #15857 from theanalyst/doc/v12.1.0/release-notes-3

doc: v12.1.0 release notes notable changes addition again

doc: v12.1.0 release notes notable changes addition again

commmits from 9f259cb923e7474229bb2e5e54b59f3b7f068bc5..luminous && sort

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #15819 from linuxbox2/wip-clru-lockfix

rgw_file: avoid a recursive lane lock in LRU drain
(failing make check is unrelated)

rgw_file: avoid a recursive lane lock in LRU drain

While highly unlikely in normal fs operations, draining an
entire filesystem could induce a recursion on object delete/unref,
when provoked by cohort_lru::drain().

Adjusted to use intrusive slist rather than std:vector<T*>
after review.

Found running librgw_file_nfsns unit test.

Fixes: http://tracker.ceph.com/issues/20374
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

radosgw: fixed sizes of integers used to operate on offsets and lengths in compression,
         followup of http://tracker.ceph.com/issues/20231

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Merge pull request #15814 from liewegas/wip-everyone-hates-btrfs

qa/objectstore/filestore-btrfs: test btrfs on trusty only

src/vstart.sh: kill dead upmap option

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15820 from liewegas/wip-20303

mon/PGMap: slightly better debugging around pgmap updates

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15846 from xiexingguo/wip-remap-upmap

mon/OSDMonitor: _apply_remap -> _apply_upmap; less code redundancy

Reviewed-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: less code redundancy

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15805 from jcsp/wip-mds-flush-upgrade

client: check for luminous MDS before sending FLUSH_MDLOG

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15800 from ukernel/wip-20340

mds: save  projected path into inode_t::stray_prior_path

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15776 from ukernel/wip-20338

osdc/Journaler: fix memory leak in Journaler::_issue_read()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15430 from jcsp/wip-20165

mds: fix hangs involving re-entrant calls to journaler

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15548 from ukernel/wip-20196

mds: improvements for stray reintegration

Reviewed-by: John Spray <john.spray@redhat.com>

mon/MonCommand: fix description of "osd pg-upmap"

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: _apply_remap -> _apply_upmap

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15796 from lixiaoy1/fio_ceph_objectstore

fio_ceph_objectstore: Print db_statistics when rocksdb_perf is enabled

Merge pull request #15753 from pritha-srivastava/wip-rgw-s3tests-conf

rgw: Changes for s3test config file, to add user under a tenant.
Reviewed-by: Casey Bodely <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

msg/async/rdma: remove assert from ibv_dealloc_pd in ProtectionDomain
destructor

issue: 1044307

Change-Id: I2fe2099c5f4403f14877e0fbe5d4868dd34d4f9b
Signed-off-by: DanielBar-On <danielbo@mellanox.com>

rgw: check all users bucket for resharding, this will reshard bucket created in older versions

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15317 from liupan1111/wip-fix-list

rbd-nbd: display pool/image/snap information in list output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15803 from mikulely/cleanup-rgw_rados_doc

rgw: update function doc in rgw_rados.h  and rgw_rados.cc

Reviewed-by: Jos Collin <jcollin@redhat.com>

qa/workunits/rbd: do not test --export-format unless supported

in 55edd81, test for `--export-format` was added to exercise this
option. but this option is only supported on luminous, so we need to
check if it's avaialble before using it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

a/workunits/rados/test_envlibrados_for_rocksdb.sh: git clone --depth 1

no need to clone the whole history of rocksdb, we just need the HEAD of
master. so "--depth 1" is better and faster in this case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/upgrade/hammer-jewel-x: upgrade all mon to luminous before osd

luminous osd requires that monmap has REQUIRE_LUMINOUS before it boots.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/upgrade/hammer-jewel-x: replace kraken.yaml with luminous.yaml

* add mgr.x to roles
* to setup mgr and set the require-osd-release bit in osdmap
* do not restart an osd for waiting for healthy: the cluster is not
  healthy until the require-osd-release=luminous is set in osdmap, and a
  mgr is up and running.

Fixes: http://tracker.ceph.com/issues/20342
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15261 from mychoxin/yuanxin_use_func

os/filestore: when print log, use __func__ instead of hard code function name

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15354 from badone/wip-rados-ls-auth-fix

osd: Reverse order of op_has_sufficient_caps and do_pg_op

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15649 from liewegas/wip-20274

osd/PrimaryLogPG: record prior_version for DELETE events

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15601 from liewegas/wip-no-restful-cert

mgr: remove default cert; disable both restful and dashboard by default

Reviewed-by: Boris Ranto <branto@redhat.com>

rgw: update rgw_rados.h function doc

+ update system_obj_get_attr func comment
+ drop deprecated func comment for get_obj
+ update func comment for pool_iterate_begin

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Merge pull request #15755 from Adirl/wip-HUGEPAGES

msg/async/rdma: automatically set RDMAV_HUGEPAGES_SAFE according to conf

Reviewed-by: Haomai Wang <haomai@xsky.com>

qa/workunits/rest/test_mgr_rest_api: remove ruleset

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/rest/mgr-restful: configure cert

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: add 'restful restart' command

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15757 from runsisi/wip-image-ts

librbd: add create timestamp metadata for image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15444 from wjwithagen/wip-wjw-freebsd-termwidth

test: Make screencandy optional for FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PGLog: ignore ERROR entires in _merge_object_divergent_entries

Sometimes ERROR log entries do not have prior_version.  Also,
they aren't actually updates and don't affect the object.

Fixes: http://tracker.ceph.com/issues/20274
Signed-off-by: Sage Weil <sage@redhat.com>

qa/objectstore/bluestore: lower full thresholds

On NVMe we can write really fast... fast enough that we may fill up before
we realize we are nearly full.  It's a problem here only because our
devices are so small that the OSDs fill up proportionally quickly.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: slightly better debugging around pgmap updates

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/powercycle/osd/tasks/radosbench: consume less space

Fixes: http://tracker.ceph.com/issues/20302
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15783 from liewegas/wip-20358

os/bluestore: move sharedblob to new collection in same shard

Merge pull request #15785 from liewegas/wip-bluestore-fsck

qa/objectstore/bluestore*: fsck on mount

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/objectstore/filestore-btrfs: test btrfs on trusty only

Centos btrfs hits ENOSPC all the time :(, and now xenial is doing
it too.

See http://tracker.ceph.com/issues/20169
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15734 from dillaman/wip-librbd-image-remove

librbd: fix issues with image removal state machine

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15727 from ovh/bp-runtime-cfg-of-hb-inflator

osd, messages/MOSDPing: bunch of fixes related to ping inflation

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15723 from badone/wip-ceph-mon-startup-error-logging

mon: Log errors at startup

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15784 from liewegas/wip-thrash-cleanup

qa/tasks/ceph_manager: get osds all in after thrashing

Merge pull request #15787 from liewegas/wip-19943-fix

ceph-objectstore-tool: do not populate snapmapper with missing clones

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

s3a hadoop '2.8.0' tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

hadoop s3a '2.7.3' tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

s3a task to test radosgw compatibility with hadoop s3a interface

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #15813 from tchaikov/wip-cmake

cmake: do not add dependencies to INTERFACE library on cmake < 3.3

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

pybind/mgr/restful: remove default path names for the cert

I don't think these are useful.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/restful: do not start if no certificate is configured

This removes the default filename, by the way.  We also work around a
problem with make_server where it sets up the socket to listen before
checking for the cert, thereby making it problematic to rebind to the
port shortly thereafter when we do have a socket.  (SO_REUSEADDR would
be appropriate but there doesn't seem to be an easy way to make
make_server use it.)

Signed-off-by: Sage Weil <sage@redhat.com>