qa/suites/upgrade/kraken-x: limit fs matrix

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/kraken-x/stress-split-erasure-code: fix

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/kraken-x/parallel: fix

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x/parallel: remove stray kraken.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/kraken-x/stress-split: updates

Bring this in line with jewel-x (which now passes).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14831 from smithfarm/wip-stress-split

tests: fix broken links in upgrade/hammer-jewel-x/stress-split

Merge pull request #14751 from trociny/wip-19716

test/librbd/test_notify.py: don't disable feature in slave

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

tests: fix broken links in upgrade/hammer-jewel-x/stress-split

Fixes: http://tracker.ceph.com/issues/19793
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #14820 from tchaikov/wip-deb-maint-fun

debian: package crypto plugin only on amd64

Signed-off-by: Nathan Cutler <ncutler@suse.com>

debian: package crypto plugins only on amd64

currently, only plugin based on isa-l is installed. archs other than
amd64 will not have this directory or the plugin(s) residing in it.
hence dh_install will fail when trying to copy nonexistence file/dir.

* debian/ceph-common.install: chmod +x, and only install crypto on amd64
  so dh_install can filter the install list using dh-exec
* debian/control: depends on dh-exec now. dh-exec v0.13 introduces support
  for filtering based on architecture. see dh-exec's changelog for more
  details. but trusty only offers dh-exec v0.12. so do not require ">=
  0.13) at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: os should depend on crypto_plugins not a certain plugin

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14779 from smithfarm/wip-fix-arch-build

build/ops: rpm: package crypto on x86_64 only

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14499 from mdw-at-linuxbox/wip-rgw-acl

rgw: swift: ability to update swift read and write acls separately.

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #14714 from cbodley/wip-19446

rgw: dont spawn error_repo until lease is acquired

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12720 from kylinstorage/wip-loop-invariant-code-motion

osd/PG.cc: loop invariant code motion

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13053 from Ted-Chang/kstore-bloom-filter

os/kstore: Added rocksdb bloom filter settings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13070 from cbodley/wip-rgw-meta-sync-periods

rgw multisite: fixes for meta sync across periods

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13061 from guzhongyan/fix_typo

osd: fix typo in comment

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13824 from Jing-Scott/doc-update-swift-static-web

doc: update the support status of swift static website

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14715 from wangzhengyong/cmpext

osd,librados: cmpext support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14299 from rzarzynski/wip-rgw-deduplicate-rgw_make_bucket_entry_name

rgw: deduplicate variants of rgw_make_bucket_entry_name().

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14552 from shashalu/fix-sync-status

rgw: add 'state==SyncState::IncrementalSync' condition when add item …

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14808 from ceph/wip-hdd-throttle-cost

common/config_opts: Set the HDD throttle cost to 1.5M

Reviewed-by: Sage Weil <sage@redhat.com>

common/config_opts: Set the HDD throttle cost to 1.5M

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #14785 from idryomov/wip-krbd-exclusive-option

rbd: recognize exclusive option

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14804 from hrchu/rgwAdmin-apiDoc-getUsageInfo

doc: rgw: Get user usage needs to specify user

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14780 from liewegas/wip-osdmap-mempool

osd: put osdmap in mempool

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rbd: test exclusive map option

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #14781 from theanalyst/rgw/cleanup/ldap_err_msg

rgw_ldap: log the ldap err in case of bind failure

rgw: remove rgw_realm_reconfigure_delay

when the master zone is changed, this config variable was increasing the
window of time where the old master zone would continue to handle
requests to modify metadata. those changes would not be reflected by the
new metadata master zone, and would be lost to the cluster

it was an attempt to optimize for the unlikely case of multiple period
changes in a short period of time, but the logic in reload() handles this
case correctly as is

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: require --yes-i-really-mean-it to promote zone with stale metadata

if a zone is promoted to master before it has a chance to sync from the
previous master zone, any metadata entries after its sync position will
be lost

print an error if 'period commit' is trying to promote a zone that is
more than one period behind the current master, and only allow the
commit to proceed if the --yes-i-really-mean-it flag is provided

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period commit uses sync status markers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use RGWShardCollectCR for RGWReadSyncStatusCoroutine

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: change metadata read_sync_status interface

makes the same change to read_sync_status() in RGWMetaSyncStatusManager,
needed to support multiple concurrent readers for the rest interface

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: sync status ignores shard markers from previous periods

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: store realm epoch with sync status markers

sync status markers can't be compared between periods, so we need to
record the current period's realm epoch with its markers. when the
rgw_meta_sync_info.realm_epoch is more recent than the marker's
realm_epoch, we must treat the marker as empty

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWBackoffControlCR only retries until success

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: clean up RGWInitDataSyncStatusCoroutine

RGWInitDataSyncStatusCoroutine operates on a given rgw_data_sync_status
pointer, which saves us from having to read it back from rados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix marker comparison to detect end of mdlog period

Fixes: http://tracker.ceph.com/issues/18639
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add == and != operators for period history cursor

RGWMetaSyncCR was using operator== but it always returned true!

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13067 from cbodley/wip-rgw-test-multi-periods

test/rgw: fixes for test_multi_period_incremental_sync()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc: rgw: Get user usage needs to specify user

Signed-off-by: hrchu <petertc@gmail.com>

Merge pull request #14796 from tchaikov/wip-init-ceph

init-ceph: should have a space before "]"

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

init-ceph: should have a space before "]"

otherwise we will have

./bin/init-ceph: 269: [: missing ]

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12460 from umullan/bucketunlink

doc: Correcting the remove bucket example and adding bucket link/unlink examples

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Correcting the bucket unlink example with correct syntax
Signed-off-by: Uday Mullangi <umullangi@walmartlabs.com>

Merge pull request #14728 from dachary/wip-ceph-disk-lock

ceph-disk: implement prepare --no-locking

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/librados/c_operations: add cmpext tests
Dispatch compare-and-read and compare-and-write compound requests, and
confirm expected behaviour under compare and miscompare conditions.

Signed-off-by: Zhengyong Wang <wangzhengyong@cmss.chinamobile.com>
Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph_test_rados_api_aio: add cmpext tests
Write a buffer and compare it with a matching and non-matching buffer
via cmpext. Do this using rados_aio_cmpext(), ioctx.aio_cmpext() and
ioctx.aio_operate(op.cmpext())

Signed-off-by: Zhengyong Wang <wangzhengyong@cmss.chinamobile.com>
Signed-off-by: David Disseldorp <ddiss@suse.de>

librados: add cmpext API
The compare-extent (cmpext) operation allows callers to compare existing
object contents with an arbitrary buffer. cmpext requests can be
compounded with read and write operations, allowing for atomic object
content updates. return 0 on success, negative error code
on failure, (-MAX_ERRNO - mismatch_off) on mismatch

This commit is based on Mike Christie's initial C++ API, with the
addition of AIO support and a C API. Response marshalling was also
reworked, so that the miscompare offset is unmarshalled transparently to
the caller.

Signed-off-by: Zhengyong Wang <wangzhengyong@cmss.chinamobile.com>
Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph_test_rados_api_misc: add cmpext tests

Signed-off-by: Zhengyong Wang <wangzhengyong@cmss.chinamobile.com>
Signed-off-by: Mike Christie <mchristi@redhat.com>
[ddiss@suse.de: add rados_cmpext() test coverage]
Reviewed-by: David Disseldorp <ddiss@suse.de>

ceph osd: add support for new op cmpext
This adds support for a new op cmpext. The request will read
extent.length bytes and compare them to extent.length bytes at
extent.offset on disk. return 0 on success, negative error code
on failure, (-MAX_ERRNO - mismatch_off) on mismatch

rbd will use this in a multi op request to implement the
SCSI COMPARE_AND_WRITE request which is used by VMware for
its atomic test and set request.

Signed-off-by: Zhengyong Wang <wangzhengyong@cmss.chinamobile.com>
Signed-off-by: Mike Christie <mchristi@redhat.com>
[ddiss@suse.de: ReplicatedPG -> PrimaryLogPG]
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge pull request #14755 from cbodley/wip-msg-direct

core: introduce DirectMessenger

Reviewed-by: Greg Farnum gfarnum@redhat.com
Reviewed-by: Sage Weil sage@redhat.com

osd/OSDMap: put OSDMap[::Incremental] itself in mempool too

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14783 from dachary/wip-crush-classe-test

test: use 7130 for crush-classes.sh

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14651 from tchaikov/wip-silent-boost-build

cmake: pass -d0 to b2 if not CMAKE_VERBOSE_MAKEFILE

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14731 from ceph/wip-bluestore-type

os/bluestore: use correct bound encode size for unused

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14738 from liupan1111/wip-fix-initial

src/common: change last_work_queue to next_work_queue.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14757 from dmick/wip-einval

common: Better handling for missing/inaccessible ceph.conf files

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14764 from Liuchang0812/wip-strip-pglog-op-name

osd: strip pglog op name

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14772 from majianpeng/bluestore-remove-extentfreelist

os/bluestore: Remove ExtentFreeListManager.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14777 from tchaikov/wip-19398

osd/PGMonitor: always update pgmap with latest osdmap

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: Remove ExtentFreeListManager.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

test/rgw: wait for realm reload after set_master_zone

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: fixes for test_multi_period_incremental_sync()

test was only creating objects in subsequent periods, which wasn't
adding any entries to the mdlog. this wasn't correctly testing
incremental metadata sync across periods

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: meta checkpoint compares realm epoch

avoid marker comparisons between different periods

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rbd: recognize exclusive map option

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test: use 7130 for crush-classes.sh

7127 is already used by mon/misc.sh

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #14350 from batrick/maintainer-update

AUTHORS: update tech leads

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14352 from batrick/peoplemap-update

peoplemap: add pdonnell alias

Merge pull request #14494 from drunkard/master

doc: two minor fixes

Merge pull request #14493 from johnzzpcrystal/FileJournal-typo

os/filestore/FileJournal: Fix typo in the comment

Merge pull request #14503 from wjwithagen/wip-wjw-freebsd-doc-dev

doc/dev: add some info about FreeBSD

Merge pull request #14560 from wjwithagen/wip-wjw-freebsd-init-ceph

src/init-ceph.in:  allow one((re)?start|stop) as commands

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #14625 from theanalyst/doc/releases/12.0.2

doc: v12.0.2 (dev) release notes

Merge pull request #14713 from smithfarm/wip-19740

tests: set -x in suites/iozone.sh workunit

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw_ldap: log the ldap err in case of bind failure

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #14756 from yehudasa/wip-rgw-suite-fix

qa/suite: replace reference to fs/xfs.yaml

Merge pull request #14747 from Liuchang0812/wip-mailmap

mailmap: update organization info

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

build/ops: rpm: package crypto on x86_64 only

ca40e12845e78e507a0c3f45efa1689979029874 added the following lines to the spec
file:

%dir %{_libdir}/ceph/crypto
%{_libdir}/ceph/crypto/libceph_*.so*

and 350932979b377b292edd12dc8c612945cd793e7a made it so those build artifacts
are generated on x86_64 only.

The result is a build failure on non-x86_64 architectures:

error: Directory not found: /home/abuild/rpmbuild/BUILDROOT/ceph-12.0.2+git.1493119152.181baf6-1.1.ppc64le/usr/lib64/ceph/crypto
error: File not found by glob: /home/abuild/rpmbuild/BUILDROOT/ceph-12.0.2+git.1493119152.181baf6-1.1.ppc64le/usr/lib64/ceph/crypto/libceph_*.so*

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #14718 from liewegas/wip-bluestore-tool

ceph-kvstore-tool: allow 'bluestore-kv' as kvdb type; add escaping, compaction

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

core: introduce DirectMessenger

DirectMessenger provides an efficient mechanism to support
in-process embedding of Ceph components (e.g., embedding of the
Ceph OSD in storage targets such as NFSv4 or iSCSI targets).

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

osd/PGMonitor: always update pgmap with latest osdmap

without this change, we always update the pgmap with the latest osdmap in
PGMonitor::check_osd_map(), but assign pending_inc.osdmap_epoch with the
epoch passed to PGMonitor::check_osd_map(), even if the osdmap could be
be much newer than "epoch". for example, we could end up with an
MOSDPGCreate with an ancient epoch, but carrying some creating pgs whose
pools do not exist in osdmap of that ancient epoch yet. this happens
when RetryCheckOSDMap is involved.

Fixes: http://tracker.ceph.com/issues/19398
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14725 from tchaikov/wip-cmake

cmake: do not compile crush twice

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14730 from tchaikov/wip-19744

mon/OSDMonitor: update pg_creatings even the new acting set is empty

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14744 from ceph/wip-bluefs-alloc-size

os/bluestore: align reclaim size to bluefs_alloc_size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14749 from ifed01/wip-bluestore-nolock-alloc

os/bluestore: get rid off excessive lock at BitMapAllocator

Reviewed-by: Sage Weil <sage@redhat.com>

osd/OSDMap: move to 'osdmap' mempool

We leave a few things out:

- strings (to annoying, and they are small)
- erasure code profiles (ditto)

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: add osdmap mempool

Signed-off-by: Sage Weil <sage@redhat.com>

crush: templatize weights arg for do_rule

Signed-off-by: Sage Weil <sage@redhat.com>

include/encoding: use more auto

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: use auto where possible

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14762 from zy751713126/add-bench-notes

rbd/bench: add notes of default values, it's easy to use

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rbd/bench: add notes of default values, it's more easy to use

Signed-off-by: Zheng Yin <zhengyin@chinac.com>

src/common: change last_work_queue to next_work_queue.

Signed-off-by: Pan Liu <liupan1111@gmail.com>

mailmap: update organization info

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

squashme: review comments

Signed-off-by: Dan Mick <dan.mick@redhat.com>

test/cli/ceph-conf/env-vs-args.t: accept new error messages

Failing to load ceph.conf now dumps more errors for the user

Signed-off-by: Dan Mick <dan.mick@redhat.com>

test/librados_test_stub/LibradosTestStub.cc: accept ENOENT

Missing ceph.conf now returns ENOENT rather than EINVAL

Signed-off-by: Dan Mick <dan.mick@redhat.com>

global/global_init.cc global_pre_init: look for ENOENT

EINVAL for no conf file found changes to ENOENT

Signed-off-by: Dan Mick <dan.mick@redhat.com>