Merge pull request #15371 from liewegas/wip-mon-client-features

mon: track features from connect clients, and use it to gate set-require-min-compat-client

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15530 from tchaikov/wip-jq-without-first

qa/workunits/ceph-helpers.sh: use syntax understood by jq 1.3

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #15423 from liewegas/wip-fix-indent

mon/FSCommand: fix indentation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15497 from liewegas/wip-log-last

mon/LogMonitor: 'log last' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15522 from liewegas/wip-19901

ceph_test_rados_api_misc: fix LibRadosMiscConnectFailure.ConnectFailure retry

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: validate set-require-min-compat-client against connected clients

Prevent the user from setting a require_min_compat_client if a disallowed
client is currently connected to the mon cluster.  Instead, you'll get an
error like

Error EPERM: cannot set require_min_compat_client to luminous: 1 connected client(s) look like jewel (missing 0x800000000200000); add --yes-i-really-mean-it to do it anyway

Signed-off-by: Sage Weil <sage@redhat.com>

common: ceph_release_[from_]features

Map releases to client features (and back again).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add 'features' command to show connected client features

for whole mon cluster

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15422 from liewegas/wip-osd-rotational

osd: hdd vs ssd defaults for osd op thread pool

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15517 from liewegas/wip-20197

qa/suites/rados/upgrade: restart mds

Merge pull request #15523 from ceph/wip-16674

selinux: Allow read on var_run_t

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/ceph-helpers.sh: use syntax understood by jq 1.3

trusty still ships jq 1.3 which does not offer "first". see
https://stedolan.github.io/jq/manual/v1.3/ .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14745 from trociny/wip-18789

rbd-mirror: coordinate image syncs with leader

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15243 from yuyuyu101/wip-20049

msg: don't set msgr addr when disabing client bind

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14117 from zhangsw/fix-rgw-getobjectmetadata-bug

rgw: return the version id in get object and object metadata request.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15386 from ivancich/wip-add-readme-that-dmclock-is-a-subtree

doc: add README to dmclock subdir to inform developers it's a git subtree

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15392 from dillaman/wip-20049

msg: do not enable client-side binding by default

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15513 from Fabian-Gruenbichler/fixmgrdesc

debian: ceph-mgr: fix package description

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15375 from ivancich/wip-fix-dmclock-testing

osd/dmclock/testing: reorganize testing, building now optional

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15378 from Liuchang0812/print_ret_code_of_json_command

ceph.in: print return code when json_command failed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15496 from wjwithagen/wip-wjw-host-to-ceph-errno

core: include/types.h, introduce host_to_ceph_errno

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15500 from liewegas/wip-20135

pybind/ceph_argparse: fix empty string check

Merge pull request #15502 from xiexingguo/wip-fix-overload-extents

os/bluestore: avoid overloading extents during reshard; atomic deferred_batch_ops

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/upgrade: restart mds

Signed-off-by: Sage Weil <sage@redhat.com>

selinux: Allow read on var_run_t

Fixes: http://tracker.ceph.com/issues/16674
Signed-off-by: Boris Ranto <branto@redhat.com>

ceph_test_rados_api_misc: fix LibRadosMiscConnectFailure.ConnectFailure retry

Fixes: http://tracker.ceph.com/issues/19901
Signed-off-by: Sage Weil <sage@redhat.com>

osd/dmclock/testing: reorganize testing, building now optional

Modify cmake files to take advantage or re-organization of dmclock's
cmake set-up, specifically not having dmclock's cmake files call
add_test. Remove dmclock tests from being dependencies on ceph's
"test" target. Make change so dmclock tests are not built unless
WITH_TESTS and WITH_DMCLOCK_TESTS are both set. This is so openSUSE
Leap will build correctly.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge commit '5c612d7e32a07a01b3b97f9cf58f2a3cef0e2e73'

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Squashed 'src/dmclock/' changes from d6586d7..64bcc6e

64bcc6e Remove FindGTest and FindBoost, which already exist in cmake required version 2.8.11 and later. Also clean up.
ed6dc60 Do some clean-up and remove memory leaks from tests and simulations.
360172a Allow RunEvery object to be ended before destruction via "join" member function.
135b5d0 Fix two bugs discovered by valgrind. Do some other code clean-up as well.
a26d29e Modify cmake files to match naming of cmake's upstream gtest and ceph's boost.
e23e63f Reorganize CMakeLists.txt files, so that projects that incorporate dmclock can control whether add_test is called. Now, add_test is called only at the top-level. Projects that incorporate dmclock can use add_subdirectory on subdirectories beneath the top-level to selective exactly what is included.

git-subtree-dir: src/dmclock
git-subtree-split: 64bcc6ea697c7e790ec4a4229d1bb1d492f9e803

qa/workunits/cephtool/test.sh: test 'log last [n]'

Signed-off-by: Sage Weil <sage@redhat.com>

common/LogEntry: make LogEntryKey opaque and cache its hash value

This avoids recalculating.  We eliminate the default ctor at the same
time and make the type totally opaque (so that its fields cannot be
modified except by the ctor and decode).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15390 from liewegas/wip-osdmap-state

osd/OSDMap: make osd_state 32 bits wide

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15515 from tchaikov/wip-crc32-aarch64

crc32c: include acconfig.h to fix ceph_crc32c_aarch64()

Reviewed-by: Dan Mick <dmick@redhat.com>

crc32c: include acconfig.h to fix ceph_crc32c_aarch64()

HAVE_ARMV8_CRC is defined by "acconfig.h". so we need to include it for
the definition.

it's a regresssion introduced by 2a3382f. in that change,
crc32c_intel_baseline.h was removed from crc32.cc, and that header
includes accconfig.h indirectly. that's why we will have a dummy
ceph_crc32c_aarch64() which always returns 0. the dummy implementation
was intended to please the compiler. but we can probably just drop it in
future as a cleanup.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15511 from tchaikov/wip-doc-cartesian-product

doc: update sample explaning "%" operator in test suites

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #14074 from jecluis/wip-mon-osd-replacement

mon: replace osds with `osd destroy` and `osd new`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15487 from liewegas/wip-no-more-stat-spam

mgr/DaemonServer: stop spamming log with pg stats

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15494 from liewegas/wip-20187

qa/workunits/cephtool/test.sh: fix osd full health detail grep

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: fix no arg check

This fixes breakage from commit a1214a702cfd8465deb91847f4dc63c0e80fb586
that caused a [] to be appended to some json argument lists.

Fixes: http://tracker.ceph.com/issues/20135
Signed-off-by: Sage Weil <sage@redhat.com>

ceph-mgr: fix package description

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>

tool/ceph: print return code when json_command failed

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

doc: update sample explaning "%" operator in tes suites

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15499 from jcsp/wip-doc-whitespace

doc: fix syntax on code snippets in cephfs/multimds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15492 from dillaman/wip-20185

rbd: properly decode features when using image name optional

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15483 from dillaman/wip-20182

librbd: filter expected error codes from is_exclusive_lock_owner

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror A/A: coordinate image syncs with leader

Fixes: http://tracker.ceph.com/issues/18789
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15490 from ceph/wip-fix-selinux

selinux: Install ceph-base before ceph-selinux

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: resolve potential recursive lock

Would have been possible when release_image had lead to cancel
sync (i.e. cancel_sync_request).

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: make sync throttler per pool

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test/rbd_mirror: TestMockImageReplayer cleanup

Remove unnecessary ImageSyncThrottler dependency.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15415 from dillaman/wip-rbd-valgrind

librbd: fix valgrind errors and ensure tests detect future leaks

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15140 from dillaman/wip-18963

rbd-mirror: permit release of local image exclusive lock after force promotion

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: reduce overhead when trying to unshard blobs

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: atomic deferred_batch_ops

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: avoid overloading of extents during reshard

This is potentially big stuff...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15484 from Adirl/wip

msg/async/rdma: Add DSCP support

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15493 from yehudasa/wip-rgw-dynamic-sharding

rgw: dynamic resharding

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

cls/rgw: add missing members initialization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14239 from liewegas/wip-bluestore-ec-clone

os/bluestore: try to unshare blobs for EC overwrite workload

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15299 from joscollin/wip-cleanup-redundant-headers-6

common: Remove redundant includes - 6

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15429 from jcsp/wip-nuke-rest

pybind/mgr: Delete `rest` module

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix a race

We referred to complete_op_data after aio call that could have released
it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/cli/radosgw-admin: update reference help.t

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix index completion

Was missing bucket complete op, only had the guard (bad merge?).
Pass zones_trace as pointer and treat it as optional param (which
it is).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: increase max shard prime, consolidate logic

Handle larger num of shards (max ~64k now). Use the same hash logic for shard ids
everywhere.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: new num shards proportional to number of objects per shard

and also limit number of shards to not exceed the max.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle zones_trace in bi async completion handler

needed following a rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove debug helper code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: ignore -ENOENT on bi_list()

just treat it as success with empty result

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: run resharding thread only on master zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add check to see if out is null

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw:  update entry in resharding log when resharding has started

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: start resharding theard

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: replace reshard blocking sleep with interruptible condition wait

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add bucket to resharding queue if needs resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw-admin: radosgw-admin reshard execute -> reshard process

to be in line with other radosgw-admin commands.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update bucket instance with resharding details

can be useful info for later in a cross zone sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: reshard execute output adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: remove unused field in reshard entry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reshard: renew lease and handle marker when listing reshard repo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: fix reshard_list truncation identification

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: expose api to generate reshard indexing key

So that we could use it as a marker when doing reshard_list.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unneeded and duplicate code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialize bucket structure

after reading bucket info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: revive reshard execute command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket_id does not include bucket name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bi_list() initialize is_truncated if -ENOENT

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: internal api cleanup

rename method, minor change

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: reshard add command doesn't read existing status

No need to read existing status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: fix remove_reshard objclass op

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: modify logshard key

The logshard key shouldn't include bucket id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multiple fixes and adjustments related to resharding scheduler

still wip, but:
 - get rid of some unneeded rados locking
 - rename shards to logshards where relevant to avoid confusion
 - remove unused code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix compilation following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: replace reshard blocking sleep with interruptible condition wait

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: actually reshard an entry from the resharding queue

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: check if buckets needs resharding on all ops that add new objects

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add bucket to resharding queue if needs resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: check for bucket resharding  when adding an object

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add a function to add a bucket to the resharding queue

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add a check_bucket_shards to the quota

Signed-off-by: Orit Wasserman <owasserm@redhat.com>