qa/suites/krbd: enable generic/050 and generic/448

generic/448 has been failing on ext4 and is now fixed upstream.
The fix for generic/050 is in testing, scheduled for 4.15-rc1.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #16238 from trociny/wip-copy-deep

librbd: generalized deep copy function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rbd: enable fsx deep_copy test case

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

test/librbd: tweak fsx to optionally use deep-copy instead of clone

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

librbd: add deep_copy function to API

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

rbd-mirror: use generalized deep copy for image sync

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: generalized deep copy function

(based on rbd-mirror image sync)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: export ImageOptions print operator

(so it could be used for logging image options)

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

test/librbd: set rand seed for all tests

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #18635 from majianpeng/bluestore-omap-flush

os/bluestore: remove unused parameters.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove unused paramerter.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #18541 from trociny/wip-mstart-bashizm

mstart.sh: remove bashizm in /bin/sh script

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18729 from iliul/fix-vstart-typo

vstart.sh: fix a typo

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18693 from XinzeChi/wip-dup-async-read-stat

osd/PrimaryLogPG: fix dup stat for async read

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-By: Kefu Chai <kchai@redhat.com>

vstart.sh: fix a spelling mistake

Signed-off-by: iliul <liul124@chinaunicom.cn>

Merge pull request #18724 from iliul/add-missing-blank-line

doc: add missing blank line

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18708 from dillaman/wip-22036

librbd: added preprocessor macro for detecting compare-and-write support

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18704 from dillaman/wip-rbd-iscsi-win

doc/rbd: tweaks to the Windows iSCSI initiator directions

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18712 from liu-chunmei/rm_redundant_dpdksource

dpdk: remove redundant dpdk submodule

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18711 from adamemerson/wip-system-includes

cmake: System Includes to silence warnings from submodules and libraries!

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18701 from tchaikov/wip-denc

denc: support enum with underlying type

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18597 from tchaikov/wip-cmake-cleanup

cmake: cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

doc: Add missing blank line

Signed-off-by: iliul <liul124@chinaunicom.cn>

build: Merge wip-system-includes from upstream

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Squashed 'src/dmclock/' changes from 1bbd42b8fe..d1ff03b667

d1ff03b667 Merge pull request #44 from adamemerson/wip-system-includes
4cc4b949ca build: Mark dependency includes as SYSTEM in dmclock
05096c1756 Merge pull request #43 from TaewoongKim/anticipation_conf
f356c45461 Add missing anticipation_timeout argument for PullPriorityQueue constructor
9896448ec5 Merge pull request #42 from tchaikov/wip-cmake
979899ef86 add travis CI on gnu/linux
8a3dabdbee cmake: the built archives are located in ${binary_dir}
ee15ced3e9 cmake: check for include in /usr/include also

git-subtree-dir: src/dmclock
git-subtree-split: d1ff03b667d9551478b2803ea533fc356ca441a9

build: Mark dependency includes as SYSTEM

It is not really our business to debug python, boost, or our other
dependencies. Mark them as system includes.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #18672 from jcsp/wip-21999

mgr: emit cluster log message on serve() exception

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18691 from liewegas/wip-fix-balancer

mgr/balancer: fix upmap; default balancer module enabled

Merge PR #18538 into master

* refs/pull/18538/head:
mds: clean up clog damage message when repairing
mds: flush after scrub repairs
mds: don't report repaired backtraces in damagetable
mds: cleaner scrub complete log msg

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18555 into master

* refs/pull/18555/head:
mds: fix inode count check in MDCache::check_memory_usage()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18612 into master

* refs/pull/18612/head:
mds: fix race in PurgeQueue::wait_for_recovery()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18624 into master

* refs/pull/18624/head:
mds: trim 'N' log segments according to how many log segments are there

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18646 into master

* refs/pull/18646/head:
mds: fix MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 definition
mds: remove useless incompat feature definination in FSMap.h

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18654 into master

* refs/pull/18654/head:
ceph.in: pass RADOS inst to LibCephFS

Reviewed-by: Kefu Chai <kchai@redhat.com>
Tested-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18686 from xiexingguo/wip-bf-retry-interval

osd/PG: use osd_backfill_retry_interval for schedule_backfill_retry()

Reviewed-by: Sage Weil <sage@redhat.com>

remove redundant dpdk source folder, this one is not maintained since last Nov.2016,
the other one is located in src/spdk/dpdk, which is alwasy maintained and used by spdk.
so remove this src/dpdk folder, we only need maintain one copy.

Signed-off-by: chunmei <chunmei.liu@intel.com>

librbd: added preprocessor macro for detecting compare-and-write support

Fixes: http://tracker.ceph.com/issues/22036
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17923 from cbodley/wip-rgw-beast-update

rgw: pull up beast submodule and update frontend

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc/rbd: tweaks to the Windows iSCSI initiator directions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18522 from C2python/cleanup_to_string

rgw: Delete to_string functions. stringify defined in include/stringify.h can provide the same feature.

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18536 from ZVampirEM77/wip-fix-loadgen

rgw: loadgen fix generate random object name rgw crash issue

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18526 from tchaikov/wip-17454

mgr: In plugins 'module' classes need not to be called "Module" anymore.

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18678 from liu-chunmei/fix_dpdk_coremask

msg/async: fix ms_dpdk_coremask and ms_dpdk_coremask conflict

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18670 from liewegas/wip-min-in-4

qa/tasks/thrashosds: set min_in default to 4

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18671 from ifed01/wip-ifed-bluefs-empty-reclaiming-fix

os/bluestore: do not release empty bluefs_extents_reclaiming.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18653 from dillaman/wip-21391

mgr/dashboard: added iSCSI IOPS/throughput metrics

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

denc: support enum with underlying type

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake,deb: remove the last trace of atomic_ops

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix indent

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove unused definitions

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link ${XIO_LIBRARY} if not necessary

Signed-off-by: Kefu Chai <kchai@redhat.com>

ms_dpdk_coremask and ms_async_op_threads has some relationship,
dpdk library will create threasds as many as bit 1 number in
ms_dpdk_coremask and the thredas num should be equal ms_async_op_threads+1,
by ceph default confiuration,ms_async_op_threads =3,ms_dpdk_coremask =1,
so dpdk library can't create enough threads for dpdk workers. the sytem
will stuck at NetworkStack::start().so set ms_dpdk_coremask default 0xF.
and add assert to comapre the two configurations.

Signed-off-by: chunmei <chunmei.liu@intel.com>

Merge pull request #18695 from cooboos/wip-doc-wrongspell

doc/release-notes: fix typo 'psd' to 'osd'

Reviewed-By: Kefu Chai <kchai@redhat.com>

doc: correct wrong spell 'psd' to 'osd'

Signed-off-by: wangsongbo wangsongbo@cloudin.cn

osd/PrimaryLogPG: fix dup stat for async read

The async read would call do_osd_ops twice.

Signed-off-by: Xinze Chi <xinze@xsky.com>

pybind/mgr/mgr_module: fix calc_pg_upmaps

This was fixed in the object cleanup.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: enable module by default

It will still be "off".

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: loadgen fix rgw crash issue

When generating random bucket name and object name during
loadgen processing, RGW will crash. The reason is calling
"gen_rand_alphanumeric" with "cct = NULL".

Fixes: http://tracker.ceph.com/issues/22006
Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Merge pull request #13992 from dreamhost/wip-19264

rgw: fix Swift container naming rules.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18606 from adamemerson/wip-op-perm-refactor

rgw: Small refactor and two bug fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #14955 from aclamk/fix-civetweb-cryptopp

radosgw: Make compilation with CryptoPP possible.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18664 from oritwas/wip-rgw-21996

rgw: init oldest period after setting run_sync_thread
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18680 from tchaikov/wip-doc-mgr

doc/mgr: document facilities methods using `automethod` directive

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18685 from jcsp/wip-mgr-load-module

qa: fix mgr _load_module helper

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PG: use osd_backfill_retry_interval for schedule_backfill_retry()

These two options currently share the same value and I have no particular
preference, but using osd_backfill_retry_interval for the specific scenario
gives us more flexible choices.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa: fix mgr _load_module helper

I inadvertently broke this with the latest change
to the module ls output.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #18656 from ceph/ceph-volume-bluestore-tests

ceph-volume: adds functional testing for bluestore

Merge pull request #18399 from jcsp/wip-20950

mon: don't blow away bootstrap-mgr on upgrades

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18681 from tchaikov/wip-msg-async-popcnt

msg/async: use bitset<> to do the popcnt

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18652 from idryomov/wip-qa-krbd-blkroset

qa/suites/krbd: add krbd BLKROSET test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/async: use bitset<> to do the popcnt

libstdc++ uses popcnt to count the bits. it's faster and simpler.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/mgr: document mgr/plugin using automethod

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: sync the docstring with the doc/mgr

also update the docstring to confirm to
http://www.sphinx-doc.org/en/stable/domains.html#info-field-lists.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: In plugins 'module' classes need not to be called "Module" anymore.

Fixes: http://tracker.ceph.com/issues/17454
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: bhavishyagopesh <bhavishyagopesh@gmail.com>

Merge pull request #18619 from trociny/wip-qa-mirror-remove

qa/workunits/rbd: unnecessary sleep after failed remove

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17983 from liewegas/wip-balancer

mgr/balancer: make crush-compat mode work

Reviewed-by: John Spray <john.spray@redhat.com>

mgr: emit cluster log message on serve() exception

Fixes: http://tracker.ceph.com/issues/21999
Signed-off-by: John Spray <john.spray@redhat.com>

rgw: add missing current_history initialization

Fixes: http://tracker.ceph.com/issues/21996
Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: init oldest period after setting run_sync_thread

Fixes: http://tracker.ceph.com/issues/21996
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18657 from cbodley/wip-21990

rgw: remove placement_rule from rgw_link_bucket()

Merge pull request #18534 from ZVampirEM77/wip-emlc-contentmd5

rgw: lc support Content-MD5 request header and fix a rgw crash bug

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #18610 from cbodley/wip-osd-subop-num

osd: add processed_subop_count for cls_cxx_subop_version()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18641 from joke-lee/rgw_modify_subuser_access_error

rgw: modify s3 type subuser access permissions fail through admin rest api

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18659 from tchaikov/wip-21993

qa: stop testing deprecated "ceph osd create"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15193 from mynaramana/arraycode

osd: introduce sub-chunks to erasure code plugin interface

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: do not release empty bluefs_extents_reclaiming.
This saves us one mutex lock attempt in Stupid Allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #18637 from yaozongyou/fix-dashboard-bug

osd: make stat_bytes and stat_bytes_used counters PRIO_USEFUL

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18347 from Synesis-LLC/fix-21773

mgr: skip first non-zero incremental in PGMap::apply_incremental()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18453 from liewegas/wip-osd-devs

osd: enumerate device names in a simple way

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18509 from tchaikov/wip-osd-cleanups

osd,mgrclient: pass daemon_status by rvalue ref and other cleanups

Reviewed-by: John Spray <john.spray@redhat.com>

qa/tasks/thrashosds: set min_in default to 4

We have EC tests with k=2,m=2, so we need a min of 4.

Fixes: http://tracker.ceph.com/issues/21997
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: remove placement_rule from cls_user_bucket_entry

placement_rule is no longer needed in cls_user_bucket_entry, because the
only time that it's needed, we can read it from the bucket instance in
RGWRados::update_containers_stats()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18504 from badone/wip-GroupSnapshotNamespace-group_pool-uninitialised

cls/rbd: Silence gcc7 maybe-uninitialized warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: remove placement_rule from rgw_link_bucket()

with the fallback in RGWRados::update_containers_stats(), we no longer
need to pass placement_rule into the cls_user_bucket_entry

this removes the dependency between bucket entrypoint metadata and its
bucket instance metadata during multisite sync

Fixes: http://tracker.ceph.com/issues/21990
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: take placement_rule from bucket info in update_containers_stats

in case the cls_user_bucket_entry doesn't contain a placement_rule, take
it from the bucket instance info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/balancer: simplify pool_info tracking

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: less verbose on 'eval' by default; add 'eval-verbose'

The verbose output is helpful for debugging and understanding what is
being measured, but most of the time all that you care about is the score.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: fix pg vs object terminology

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: restrict to time of day

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/module: adjust osd_weight min step to .005

That should be ~1 PG or less on average; no real sense it taking a
step smaller than that!

Signed-off-by: Sage Weil <sage@redhat.com>