qa/rgw: fix assertions in radosgw_admin task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14816 from cbodley/wip-rgw-test-multi-list-fix

test/rgw: fix for empty lists as default arguments

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14473 from ShiqiCooperation/master

os/bluestore: assert blob map returns success

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14491 from liewegas/wip-bluestore-deferred-rewrite

os/bluestore: rewrite deferred write handling

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14695 from dachary/wip-choosleaf-stable

crush: builder: legacy has chooseleaf_stable = 0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14696 from dachary/wip-crush-workspace

crush: crush_init_workspace starts with struct crush_work

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14814 from neha-ojha/wip-osd-filestore-default

osd: switch filestore to default to rocksdb

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14795 from tchaikov/wip-so-priority-cleanup

msg/async: return right away in NetHandler::set_priority() if not supported

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14802 from tchaikov/wip-mgr-cleanup

mgr: various cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14807 from liewegas/wip-jewel-x

ceph_test_rados_api_tier: tolerate ENOENT from 'pg scrub'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14717 from liewegas/wip-bluestore-remove-max-ops

os/bluestore: rename/fix throttle options

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14819 from tchaikov/wip-objecter-more-constness

osdc/Objecter: more constness

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14773 from viva64/sizeof_pointer

src/msg/async/AsyncConnect.cc: Use of sizeof() on a Pointer Type

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14782 from yuyuyu101/wip-rdma-perf

msg/async/rdma: add inqueue rx chunks perf counter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14797 from wjwithagen/wip-wjw-freebsd-config-path

common/config: Add /usr/local/etc/ceph to default paths

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14801 from tchaikov/wip-crypto-cleanup

crypto: cleanup NSPR in main thread

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14805 from tchaikov/wip-mon-check-get

mon/OSDMonitor: check get()'s return value instead of bl's length

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14836 from idryomov/wip-krbd-wac-tests

qa: krbd write-after-checksum tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14803 from hrchu/rgwadmin-api-doc-usage-conf

doc: rgw: note rgw_enable_usage_log option in adminops guide

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14824 from renhwztetecs/renhw-wip-mgr-bootstrap

mgr: mkdir bootstrap-mgr

Reviewed-by: Sage Weil <sage@redhat.com>

qa: krbd write-after-checksum tests

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

common/config_opts.h: Allow for /usr/local/etc/ceph to hold keyrings

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #14831 from smithfarm/wip-stress-split

tests: fix broken links in upgrade/hammer-jewel-x/stress-split

Merge pull request #14751 from trociny/wip-19716

test/librbd/test_notify.py: don't disable feature in slave

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

tests: fix broken links in upgrade/hammer-jewel-x/stress-split

Fixes: http://tracker.ceph.com/issues/19793
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #14820 from tchaikov/wip-deb-maint-fun

debian: package crypto plugin only on amd64

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: mkdir bootstrap-mgr

ceph-deploy rely on bootstrap-mgr
as follows: https://github.com/ceph/ceph-deploy/blob/master/ceph_deploy/mgr.py#L37

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

debian: package crypto plugins only on amd64

currently, only plugin based on isa-l is installed. archs other than
amd64 will not have this directory or the plugin(s) residing in it.
hence dh_install will fail when trying to copy nonexistence file/dir.

* debian/ceph-common.install: chmod +x, and only install crypto on amd64
  so dh_install can filter the install list using dh-exec
* debian/control: depends on dh-exec now. dh-exec v0.13 introduces support
  for filtering based on architecture. see dh-exec's changelog for more
  details. but trusty only offers dh-exec v0.12. so do not require ">=
  0.13) at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: os should depend on crypto_plugins not a certain plugin

Signed-off-by: Kefu Chai <kchai@redhat.com>

crypto: cleanup NSPR in main thread

quote from nspr's header file

```
 * Perform a graceful shutdown of NSPR.  PR_Cleanup() may be called by
 * the primordial thread near the end of the main() function.
```

this helps to silence some warnings from valgrind. but it does not hurt
in practice, because the process is about to die. and the freed memory
chunks are only allocated once in NSPR.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osdc/Objecter: more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14779 from smithfarm/wip-fix-arch-build

build/ops: rpm: package crypto on x86_64 only

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: rgw: note rgw_enable_usage_log option in adminops guide

Signed-off-by: hrchu <petertc@gmail.com>

test/rgw: fix for empty lists as default arguments

i was incorrectly using [] as a default function argument, without
realizing that default values are mutable and shared across invocations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14499 from mdw-at-linuxbox/wip-rgw-acl

rgw: swift: ability to update swift read and write acls separately.

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

os/bluestore: narrow lock scope in _deferred_aio_finish

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use superclass for aio completions

This is less fragile and more explicit than using the lower bit
of the priv pointer.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: restructure deferred writes

Explicitly aggregate deferred writes into a batch.  When we
submit, take the opportunity to coalesce contiguous writes.
Handle aio completion independently from the original txcs.

Note that this paves the way for a few additional steps:

1- we could make deallocations cancel deferred writes.
2- we could drop the txc deferred states entirely and rely on
the explicit deferred write batch machinery instead... if we
build an alternative way to complete the SharedBlob writes
and ensure the lifecycle issue are dealt with.  (I'm not sure
it would be worth it, but it might be.)

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: handle non-block-aligned backing file

Fixes: http://tracker.ceph.com/issues/16644
Signed-off-by: Sage Weil <sage@redhat.com>

osd: switch filestore to default to rocksdb

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #14714 from cbodley/wip-19446

rgw: dont spawn error_repo until lease is acquired

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12720 from kylinstorage/wip-loop-invariant-code-motion

osd/PG.cc: loop invariant code motion

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13053 from Ted-Chang/kstore-bloom-filter

os/kstore: Added rocksdb bloom filter settings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13070 from cbodley/wip-rgw-meta-sync-periods

rgw multisite: fixes for meta sync across periods

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13061 from guzhongyan/fix_typo

osd: fix typo in comment

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13824 from Jing-Scott/doc-update-swift-static-web

doc: update the support status of swift static website

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14715 from wangzhengyong/cmpext

osd,librados: cmpext support

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: observe bluestore_throttle_cost_* options

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rename throttle configs: bluestore_throttle[_deferred]_bytes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove bluestore_max_ops

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14299 from rzarzynski/wip-rgw-deduplicate-rgw_make_bucket_entry_name

rgw: deduplicate variants of rgw_make_bucket_entry_name().

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14552 from shashalu/fix-sync-status

rgw: add 'state==SyncState::IncrementalSync' condition when add item …

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14808 from ceph/wip-hdd-throttle-cost

common/config_opts: Set the HDD throttle cost to 1.5M

Reviewed-by: Sage Weil <sage@redhat.com>

common/config_opts: Set the HDD throttle cost to 1.5M

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #14785 from idryomov/wip-krbd-exclusive-option

rbd: recognize exclusive option

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14804 from hrchu/rgwAdmin-apiDoc-getUsageInfo

doc: rgw: Get user usage needs to specify user

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_tier: tolerate ENOENT from 'pg scrub'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14780 from liewegas/wip-osdmap-mempool

osd: put osdmap in mempool

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rbd: test exclusive map option

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #14781 from theanalyst/rgw/cleanup/ldap_err_msg

rgw_ldap: log the ldap err in case of bind failure

rgw: remove rgw_realm_reconfigure_delay

when the master zone is changed, this config variable was increasing the
window of time where the old master zone would continue to handle
requests to modify metadata. those changes would not be reflected by the
new metadata master zone, and would be lost to the cluster

it was an attempt to optimize for the unlikely case of multiple period
changes in a short period of time, but the logic in reload() handles this
case correctly as is

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: require --yes-i-really-mean-it to promote zone with stale metadata

if a zone is promoted to master before it has a chance to sync from the
previous master zone, any metadata entries after its sync position will
be lost

print an error if 'period commit' is trying to promote a zone that is
more than one period behind the current master, and only allow the
commit to proceed if the --yes-i-really-mean-it flag is provided

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period commit uses sync status markers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use RGWShardCollectCR for RGWReadSyncStatusCoroutine

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: change metadata read_sync_status interface

makes the same change to read_sync_status() in RGWMetaSyncStatusManager,
needed to support multiple concurrent readers for the rest interface

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: sync status ignores shard markers from previous periods

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: store realm epoch with sync status markers

sync status markers can't be compared between periods, so we need to
record the current period's realm epoch with its markers. when the
rgw_meta_sync_info.realm_epoch is more recent than the marker's
realm_epoch, we must treat the marker as empty

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWBackoffControlCR only retries until success

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: clean up RGWInitDataSyncStatusCoroutine

RGWInitDataSyncStatusCoroutine operates on a given rgw_data_sync_status
pointer, which saves us from having to read it back from rados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix marker comparison to detect end of mdlog period

Fixes: http://tracker.ceph.com/issues/18639
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add == and != operators for period history cursor

RGWMetaSyncCR was using operator== but it always returned true!

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13067 from cbodley/wip-rgw-test-multi-periods

test/rgw: fixes for test_multi_period_incremental_sync()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mon/OSDMonitor: check get()'s return value instead of bl's length

instead of assuming how the bl is encoded, we'd better rely on the
return code of get().

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: rgw: Get user usage needs to specify user

Signed-off-by: hrchu <petertc@gmail.com>

mgr/MgrPyModule: no need to keep pClass and pModule around

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: pass python interpreter's path to embedded python

* also do prevent python from registering its own signal handler, it does
  not make sense in our embedded use case.
* pass python interpreter's path to embedded python before initializing
  it. python uses this path to look up the "site" modules, etc.

so it can use other interpreter if it is not intalled into $PATH,
otherwise the "python" in $PATH will always be used even the
PYTHON_EXECUTABLE in CMake's cache is pointing to another python
interpreter.

if one want to debug ceph-mgr with a customize build python, this would
be helpful.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: no need to link ceph-mgr against mds

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14796 from tchaikov/wip-init-ceph

init-ceph: should have a space before "]"

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

common/config: Add /usr/local/etc/ceph to default paths

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/msg/async/AsyncConnect.cc: PVS-Studio: CWE-467 Use of sizeof() on a Pointer Type

Signed-off-by: Svyatoslav <razmyslov@viva64.com>

init-ceph: should have a space before "]"

otherwise we will have

./bin/init-ceph: 269: [: missing ]

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: return right away in NetHandler::set_priority() if not supported

* SO_PRIORITY is linux specific, so no need to check __linux__
* early return if priority is less than 0 (maybe we should also return if
  it's higher than 6?), the less indent.
* store errno if fails to set SO_PRIORITY before printing log messages.
* guard the whole function with '#ifdef SO_PRIORITY' so on platforms
  where this option is not supported, this function will be a no-op.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12460 from umullan/bucketunlink

doc: Correcting the remove bucket example and adding bucket link/unlink examples

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Correcting the bucket unlink example with correct syntax
Signed-off-by: Uday Mullangi <umullangi@walmartlabs.com>

Merge pull request #14728 from dachary/wip-ceph-disk-lock

ceph-disk: implement prepare --no-locking

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/librados/c_operations: add cmpext tests
Dispatch compare-and-read and compare-and-write compound requests, and
confirm expected behaviour under compare and miscompare conditions.

Signed-off-by: Zhengyong Wang <wangzhengyong@cmss.chinamobile.com>
Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph_test_rados_api_aio: add cmpext tests
Write a buffer and compare it with a matching and non-matching buffer
via cmpext. Do this using rados_aio_cmpext(), ioctx.aio_cmpext() and
ioctx.aio_operate(op.cmpext())

Signed-off-by: Zhengyong Wang <wangzhengyong@cmss.chinamobile.com>
Signed-off-by: David Disseldorp <ddiss@suse.de>

librados: add cmpext API
The compare-extent (cmpext) operation allows callers to compare existing
object contents with an arbitrary buffer. cmpext requests can be
compounded with read and write operations, allowing for atomic object
content updates. return 0 on success, negative error code
on failure, (-MAX_ERRNO - mismatch_off) on mismatch

This commit is based on Mike Christie's initial C++ API, with the
addition of AIO support and a C API. Response marshalling was also
reworked, so that the miscompare offset is unmarshalled transparently to
the caller.

Signed-off-by: Zhengyong Wang <wangzhengyong@cmss.chinamobile.com>
Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph_test_rados_api_misc: add cmpext tests

Signed-off-by: Zhengyong Wang <wangzhengyong@cmss.chinamobile.com>
Signed-off-by: Mike Christie <mchristi@redhat.com>
[ddiss@suse.de: add rados_cmpext() test coverage]
Reviewed-by: David Disseldorp <ddiss@suse.de>

ceph osd: add support for new op cmpext
This adds support for a new op cmpext. The request will read
extent.length bytes and compare them to extent.length bytes at
extent.offset on disk. return 0 on success, negative error code
on failure, (-MAX_ERRNO - mismatch_off) on mismatch

rbd will use this in a multi op request to implement the
SCSI COMPARE_AND_WRITE request which is used by VMware for
its atomic test and set request.

Signed-off-by: Zhengyong Wang <wangzhengyong@cmss.chinamobile.com>
Signed-off-by: Mike Christie <mchristi@redhat.com>
[ddiss@suse.de: ReplicatedPG -> PrimaryLogPG]
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge pull request #14755 from cbodley/wip-msg-direct

core: introduce DirectMessenger

Reviewed-by: Greg Farnum gfarnum@redhat.com
Reviewed-by: Sage Weil sage@redhat.com

osd/OSDMap: put OSDMap[::Incremental] itself in mempool too

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14783 from dachary/wip-crush-classe-test

test: use 7130 for crush-classes.sh

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14651 from tchaikov/wip-silent-boost-build

cmake: pass -d0 to b2 if not CMAKE_VERBOSE_MAKEFILE

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14731 from ceph/wip-bluestore-type

os/bluestore: use correct bound encode size for unused

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14738 from liupan1111/wip-fix-initial

src/common: change last_work_queue to next_work_queue.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14757 from dmick/wip-einval

common: Better handling for missing/inaccessible ceph.conf files

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14764 from Liuchang0812/wip-strip-pglog-op-name

osd: strip pglog op name

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14772 from majianpeng/bluestore-remove-extentfreelist

os/bluestore: Remove ExtentFreeListManager.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14777 from tchaikov/wip-19398

osd/PGMonitor: always update pgmap with latest osdmap

Reviewed-by: Sage Weil <sage@redhat.com>