Merge pull request #17605 from jcsp/wip-21253

mgr: fix py calls for dne service perf counters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17568 from amitkumar50/cov-msg-2

msg: Initializing class members in module msg

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16699 from jcsp/wip-mgr-perf-threshold

mgr: apply a threshold to perf counter prios

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17361 from kungf/ulimit_setup

run-make-check.sh: run ulimit without sudo

Reviewed-by: Dan Mick <dmick@redhat.com>

run-make-check.sh: run ulimit without sudo

ulimit is a shell builtin like cd, not a separate program.
sudo looks for a binary to run, but there is no ulimit binary,
then you will get an error message like:
          "sudo: ulimit: command not found"
ulimit can only be run with root if you want raise the ulimit -n

Change-Id: I59ab09240cb15cabe5b0a7342c3f562cd50c75e9
Signed-off-by: yang.wang <yang.wang@easystack.cn>

Merge pull request #17713 from tchaikov/wip-ceph.in-python

ceph.in: execv using the same python

Merge pull request #17583 from neha-ojha/wip-cbt-teuthology-integration

qa: add cbt task for performance testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17680 from ceph/wip-smoke-whitelist

qa/tests: Various whitelists for smoke suite

Various whitelists for tests to pass

See tracker http://tracker.ceph.com/issues/21376 for details

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #17733 from yaozongyou/fix-doc-boolean-type-zero-to-false

doc: change boolean option default value from zero to false.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17610 from liewegas/wip-bluestore-fm-mas

os/bluestore: set bitmap freelist resolution to min_alloc_size

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17617 from liewegas/wip-drop-disk-log

mon/PGMap: drop DISK LOG column

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: change boolean option default value from zero to false.

Signed-off-by: Yao Zongyou yaozongyou@vip.qq.com

Merge pull request #17628 from majianpeng/bluestore-narrow-deferred-lock

os/bluestore/BlueStore: narrow deferred_lock in _deferred_submit_unlock.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17627 from majianpeng/bluefs-exce-debug-code

os/bluestore/BlueFS: Don't call debug related code under any condition.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17371 from xiexingguo/wip-per-pool-full-control

mon, osd: per pool space-full flag support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17368 from xiexingguo/wip-nice-data

mon/PGMap: nice numbers for 'data' section of 'ceph df' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.in: execv using the same python

otherwise we could be using another python when respawing the ceph cli.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #17657 into master

* refs/remotes/upstream/pull/17657/head:
mds: optimize MDCache::rejoin_scour_survivor_replicas()
mds: fix MDSCacheObject::clear_replica_map
mds: support limiting cache by memory
common: refactor of lru
mds: resolve unsigned coercion compiler warning
common: use safer uint64_t for list size
common: add bytes2str pretty print function
mds: check if waiting is allocated before use
mds: go back to compact_map for replicas
mds: use mempool for cache objects
mds: cleanup replica_map access
common: add alloc_ptr smart pointer
common: add warning on base class use of mempool
common: use atomic uin64_t for counter

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17689 into master

* refs/remotes/upstream/pull/17689/head:
ceph: fix "stopping mds can't export subtree"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17545 into master

* refs/remotes/upstream/pull/17545/head:
client: fix request reference leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #17700 from cbodley/wip-qa-smoke-s3tests

qa/smoke: add rgw crypto config for s3tests

Merge PR #17608 into master

* refs/remotes/upstream/pull/17608/head:
doc/cephfs/posix: put posix notes in perspective

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17641 from amitkumar50/cov-messages-12

messages: Initialization of variable beat

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #10460 from weiqiaomiao/wqm-wip-rgw-dellc

rgw: add tenant to shard_id in RGWDeleteLC::execute()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #17061 from shashalu/fix-rgw-lc

rgw: fix lc process only schdule the first item of lc objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: fang.yuxiang <fang.yuxiang@eisoo.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

qa/smoke: add rgw crypto config for s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17631 from joke-lee/RGW-CORS-Access-Control-Allow-Origin-HEADER

rgw: return 'Access-Control-Allow-Origin' header when the set bucket versioning through XMLHttpRequest

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17632 from joke-lee/RGW-CORS-Access-Control-Allow-Origin-HEADER-WEBSITE

rgw: return 'Access-Control-Allow-Origin' header when the set and delete bucket website through XMLHttpRequest

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17663 from shashalu/fix-rgw-LCFilter

rgw: Fix closing tag for Prefix

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17543 from iliul/fix-dup-global-section

vstart.sh: Remove duplicate global section

Reviewed-by: Sage Weil <sage@redhat.com>

vstart.sh: Remove duplicate global section

Signed-off-by: iliul <liul124@chinaunicom.cn>

Merge pull request #17544 from joelin/wip-delete-forgot-mark

msg: remove the ),it's redundant.

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17515 from xiexingguo/wip-data-digest

osd/PrimaryLogPG: do not set data digest for bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mds: optimize MDCache::rejoin_scour_survivor_replicas()

avoid iterating dentries if dirfrag is non-auth

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix MDSCacheObject::clear_replica_map

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17366 from smithfarm/wip-21176

build/ops: python-numpy-devel build dependency for SUSE

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Ewaznazari <Nazari@keyinternational.academy>

Merge pull request #17685 from ygtzf/HashIndex-clerical-error

os/filestore/HashIndex.h: fixed a typo in comment

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17688 from TsaiJin/wip-fix-assert-bug

msg/asyc/rmda: fix the bug of assert when Infiniband::recv_msg receives disconnect message

Reviewed-by: Haomai Wang <haomai@xsky.com>

osd/osd_types.h: default to no data/omap digest for new object

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PrimaryLogPG: do not generate data digest for BlueStore by default

BlueStore enables CRC by default, so this is a dup and gains
no more benefits.

Turn this off by default, which is good for performance.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph: fix "stopping mds can't export subtree"

caused by commit
790c34138f "Make standby-replay mds avoid initiating subtree export"

Fixes: http://tracker.ceph.com/issues/21378
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17687 from yonghengdexin735/wip-zzz-error

common: fix typo deamon in comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

mds: support limiting cache by memory

This introduces two config parameters:

    mds_cache_memory_limit: Sets the soft maximum of the cache to the given
    byte count. (Like mds_cache_size, this doesn't actually limit the maximum
    size of the cache. It just dictates the steady-state size.)

    mds_cache_reservation: This replaces mds_health_cache_threshold everywhere
    except the Beacon heartbeat sent to the mons. The idea here is to specify a
    reservation of memory (5% by default) for operations and the MDS tries to
    always maintain that reservation. So, the MDS will recall caps from clients
    when it begins dipping into its reservation of memory.

mds_cache_size still limits the cache by Inode count but is now by-default 0
(i.e. unlimited). The new preferred way of specifying cache limits is by memory
size. The default is 1GB.

Fixes: http://tracker.ceph.com/issues/20594
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1464976
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/asyc/rmda: fix the bug of assert when Infiniband::recv_msg receives
disconnect message

Signed-off-by: Jin Cai <caijin.caij@alibaba-inc.com>

common:fix error word

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

Merge pull request #17613 from smithfarm/wip-12056

doc: document include/ipaddr.h

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17644 from Vogtinator/readme-vstart

README: Improve vstart.sh usage

os/filestore/HashIndex.h: fixed a clerical error

Signed-off-by: yaoguotao <yaoguot@gmail.com>

Merge pull request #17654 from yuyuyu101/wip-spdk-doc

doc/bluestore: add SPDK usage for bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17658 from ukernel/wip-cephfs-seekdir

doc/cephfs/posix: remove stale information for seekdir

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17669 from ygtzf/parameter-default-value-error

doc: update default value of parameter mon_subscribe_interval

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17672 from FrankYu/master

doc: the client inputs the pool name instead of pool ID

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge PR #17679 into master

* refs/remotes/upstream/pull/17679/head:
qa: get asok path from ceph.conf
qa: use config_path property instead of literal

Reviewed-by: John Spray <john.spray@redhat.com>

qa: get asok path from ceph.conf

The asok path was recently moved to /tmp to resolve unix path too long errors.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use config_path property instead of literal

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: refactor of lru

Avoids an unnecessary "max" size of the LRU which was used to calculate the
midpoint. Instead, just dynamically move the LRUObjects between top and bottom
on-the-fly.

This change is necessary for a cache which which does not limit by the number
of objects but by some other metric. (In this case, memory.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: resolve unsigned coercion compiler warning

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: use safer uint64_t for list size

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add bytes2str pretty print function

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: check if waiting is allocated before use

This prevents accidental allocation of the map.

Also, privatize the variable to protect from this in child classes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: go back to compact_map for replicas

Zheng observed that an alloc_ptr doesn't really work in this case since any
call to get_replicas() will cause the map to be allocated, nullifying the
benefit. Use a compact_map until a better solution can be written. (This means
that the map will be allocated outside the mempool.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use mempool for cache objects

The purpose of this is to allow us to track memory usage by cached objects so
we can limit cache size based on memory available/allocated to the MDS.

This commit is a first step: it adds CInode, CDir, and CDentry to the mempool
but not all of the containers in these classes (e.g. std::map). However,
MDSCacheObject has been changed to allocate its containers through the mempool
by converting compact_* containers to the std versions offered through mempool
via the new alloc_ptr.

(A compact_* class simply wraps a pointer to the std:: version to reduce memory
usage of an object when the container is only occasionally used. The alloc_ptr
allows us to achieve the same thing explicitly with only a little handholding:
when all entries in the wrapped container are deleted, the caller must call
alloc_ptr.release().)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: cleanup replica_map access

The gymnastics protecting the map failed as the code evolved. Just expose it
normally with a getter.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add alloc_ptr smart pointer

This ptr is like a unique_ptr except it allocates the underlying object on
access. The idea being that we can save memory if the object is only needed
sometimes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17650 from ceph/wip-ceph-deploy-fix

qa/tests: Fix get_system_type failure due to invalid remote name

Merge pull request #17636 from dillaman/wip-21319

rbd: clean up warnings when mirror commands used on non-setup pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

qa: avoid using make install for fio

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: the client inputs the pool name instead of pool ID

Signed-off-by: Frank Yu <flyxiaoyu@gmail.com>

Merge pull request #13383 from wonzhq/wip-update-calc-stats

osd: change the conditional in _update_calc_stats

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17640 from tchaikov/wip-diagram-py3

doc/scripts: py3 compatible

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: update default value of parameter mon_subscribe_interval

Signed-off-by: yaoguotao <yaoguot@gmail.com>

Merge pull request #17639 from tchaikov/wip-doc-ceph-disk

doc: update ceph-disk with a state-transition diagram

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: LCFilter_S3::to_xml the second "prefix" missed "/"

Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

Merge pull request #17538 from dzafman/wip-21272

Add export and remove ceph-objectstore-tool command option

Reviewed-by: Sage Weil <sage@redhat.com>

doc/bluestore: add SPDK usage for bluestore

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #17537 from mychoxin/yuanxin_remove_submit

os/bluestore: remove useless function submit

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/cephfs/posix: remove stale information for seekdir

Current cephfs can support seekdir efficiently. The diverge was
fixed by https://github.com/ceph/ceph/pull/14317

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17606 from ceph/wip-bz1478598

ceph-volume allow filtering by `uuid`, do not require osd id

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

messages: Initialization of variable beat

Fixes the coverity issue:

** 717278 Uninitialized scalar field

CID 717278 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member beat is not
initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

ceph-volume lvm.trigger just split the osd_id once

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests catch improper parsing of osd uuid

When the osd ID existed in the UUID in the same format (e.g. '1-') the
parsing would break returning a bogus UUID that is impossible to find

Signed-off-by: Alfredo Deza <adeza@redhat.com>

tests: remove 'ceph-' prefix from test node hostnames

This updates our Vagrantfile to match the changes in ceph-ansible
introduced by https://github.com/ceph/ceph-ansible/commit/298a63c43

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.api PVolumes should filter by all matching tags, not just one

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests pv API should also match all pv_tags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests prove all volume tags have to match in vgs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17377 from cbodley/wip-rgw-test-multi-encryption

test/rgw: fix test_encrypted_object_sync for 3+ zones

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #17102 from mychoxin/yuanxin_print_log_if_add_event_fail

msg/async: print error log if add_event fail

Reviewed-by: Pan Liu <liupan1111@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16492 from yehudasa/wip-rgw-multi-log

rgw: multisite log tracing

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17400 from theanalyst/fix/rgw-set-attrs-mtime

rgw: rgw_rados: set_attrs now sets the same time for BI & object

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

admon/build-doc: be compatible with py3

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados: fix the method names

otherwise, sphinx fails to run with py3

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: switch to upstream sphinx-ditaa

it works with setuptools and is now compatible with py3

the py3 branch is created to track the upstream's master branch

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17638 from idryomov/wip-rbd-already-mapped-fixes

krbd: rewrite "already mapped" code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

README: Improve vstart.sh usage

Using long options where possible to make it easier to understand directly
and adding bluestore by default as it is the default storage backend.

Signed-off-by: Fabian Vogt <fvogt@suse.com>

os/bluestore/BlueStore: narrow deferred_lock in _deferred_submit_unlock.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BlueFS: Don't call debug related code under any condition.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ceph-volume lvm.api all vg tags should match, not just one

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests prove all tags have to match in lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api all lv tags should match, not just one

Signed-off-by: Alfredo Deza <adeza@redhat.com>