qa/suites/rados/perf: add optimized settings

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #17706 from liewegas/wip-test-listsnaps

ceph_test_rados_api_tier: add ListSnap test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17705 from dzafman/wip-21328

osd: Only scan for omap corruption once

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17703 from dzafman/wip-misc

Erasure code read test and code cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17699 from tchaikov/wip-lookout-for-large-elen

msg/msg_types: fix the entity_addr_t's decoder

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17667 from scienceluo/wip-luo-fix-cli-output-branch

pybind/ceph_argparse: fix cli output info

Reviewed-by: Chang Liu <liuchang0812@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17664 from Yan-waller/wip-walle-0912moninjectargs

common: fix daemon abnormal exit at parsing invalid arguments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17655 from optimistyzy/911_2

src/bluestore/NVMEDevice: make all read use aio_submit

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #17611 from chardan/jfw-wip-atomic_t-cleanup

follow-up fixups for atomic_t spinlocks

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17605 from jcsp/wip-21253

mgr: fix py calls for dne service perf counters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17568 from amitkumar50/cov-msg-2

msg: Initializing class members in module msg

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16699 from jcsp/wip-mgr-perf-threshold

mgr: apply a threshold to perf counter prios

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: fix the entity_addr_t's decoder

the daemon is vulnerable to malicious client, which is able to send
large elen, and corrupt the stack, etc.

* throw at seeing corrupted entity_addr_t where its elen exceeds
  the length of sockaddr
* handle the exception thrown when decoding entity_addr_t in messenger
  layer.
* if a malicious client manages to send a corrutped entity_addr_t to
  daemon, daemon will crash because decode fails and the exception is
  not handled. it's better than continuing working with the bogus
  message.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17361 from kungf/ulimit_setup

run-make-check.sh: run ulimit without sudo

Reviewed-by: Dan Mick <dmick@redhat.com>

run-make-check.sh: run ulimit without sudo

ulimit is a shell builtin like cd, not a separate program.
sudo looks for a binary to run, but there is no ulimit binary,
then you will get an error message like:
          "sudo: ulimit: command not found"
ulimit can only be run with root if you want raise the ulimit -n

Change-Id: I59ab09240cb15cabe5b0a7342c3f562cd50c75e9
Signed-off-by: yang.wang <yang.wang@easystack.cn>

Merge pull request #17713 from tchaikov/wip-ceph.in-python

ceph.in: execv using the same python

Merge pull request #17583 from neha-ojha/wip-cbt-teuthology-integration

qa: add cbt task for performance testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17680 from ceph/wip-smoke-whitelist

qa/tests: Various whitelists for smoke suite

Various whitelists for tests to pass

See tracker http://tracker.ceph.com/issues/21376 for details

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #17733 from yaozongyou/fix-doc-boolean-type-zero-to-false

doc: change boolean option default value from zero to false.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17610 from liewegas/wip-bluestore-fm-mas

os/bluestore: set bitmap freelist resolution to min_alloc_size

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17617 from liewegas/wip-drop-disk-log

mon/PGMap: drop DISK LOG column

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: change boolean option default value from zero to false.

Signed-off-by: Yao Zongyou yaozongyou@vip.qq.com

Merge pull request #17628 from majianpeng/bluestore-narrow-deferred-lock

os/bluestore/BlueStore: narrow deferred_lock in _deferred_submit_unlock.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17627 from majianpeng/bluefs-exce-debug-code

os/bluestore/BlueFS: Don't call debug related code under any condition.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17371 from xiexingguo/wip-per-pool-full-control

mon, osd: per pool space-full flag support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17368 from xiexingguo/wip-nice-data

mon/PGMap: nice numbers for 'data' section of 'ceph df' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.in: execv using the same python

otherwise we could be using another python when respawing the ceph cli.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #17657 into master

* refs/remotes/upstream/pull/17657/head:
mds: optimize MDCache::rejoin_scour_survivor_replicas()
mds: fix MDSCacheObject::clear_replica_map
mds: support limiting cache by memory
common: refactor of lru
mds: resolve unsigned coercion compiler warning
common: use safer uint64_t for list size
common: add bytes2str pretty print function
mds: check if waiting is allocated before use
mds: go back to compact_map for replicas
mds: use mempool for cache objects
mds: cleanup replica_map access
common: add alloc_ptr smart pointer
common: add warning on base class use of mempool
common: use atomic uin64_t for counter

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17689 into master

* refs/remotes/upstream/pull/17689/head:
ceph: fix "stopping mds can't export subtree"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17545 into master

* refs/remotes/upstream/pull/17545/head:
client: fix request reference leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #17700 from cbodley/wip-qa-smoke-s3tests

qa/smoke: add rgw crypto config for s3tests

ceph_test_rados_api_tier: add ListSnap test

Signed-off-by: Xuehan Xu <xuxuehan@360.cn>

osd: Only scan for omap corruption once

Before
  state 2: Can have complete tables (some may be bad)
state 3: Never had complete tables
After
  state 2: Can have complete tables (some may be bad)
state 3 with legacy: Can have complete tables (bad ones are cleared)
state 3: Never had complete tables

Once OSDs boot with this change you can't downgrade to a previous release.
If someone does downgrade they could have unstable OSDs that hit assert(state.v < 3).
The following command run after shutting down the cluster but before downgrading
ceph packages would be a way to fix this.

ceph-osdomap-tool --omap-path ... --command resetv2

Fixes: http://tracker.ceph.com/issues/21328
Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add the ability to reset state to v2

Available for testing and user downgrade.

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Show DB state information

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add --backend option to ceph-osdomap-tool default to rocksdb

Fix hard-coded "leveldb" backend.  The command is broken in Luminous
now that "rocksdb" is the default.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #17608 into master

* refs/remotes/upstream/pull/17608/head:
doc/cephfs/posix: put posix notes in perspective

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd: CLEANUP: Change getattr_maybe_cache() to reflect how it is being used

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add a removal test for erasure code read

Test feature: http://tracker.ceph.com/issues/14513

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #17641 from amitkumar50/cov-messages-12

messages: Initialization of variable beat

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #10460 from weiqiaomiao/wqm-wip-rgw-dellc

rgw: add tenant to shard_id in RGWDeleteLC::execute()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #17061 from shashalu/fix-rgw-lc

rgw: fix lc process only schdule the first item of lc objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: fang.yuxiang <fang.yuxiang@eisoo.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

qa/smoke: add rgw crypto config for s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17631 from joke-lee/RGW-CORS-Access-Control-Allow-Origin-HEADER

rgw: return 'Access-Control-Allow-Origin' header when the set bucket versioning through XMLHttpRequest

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17632 from joke-lee/RGW-CORS-Access-Control-Allow-Origin-HEADER-WEBSITE

rgw: return 'Access-Control-Allow-Origin' header when the set and delete bucket website through XMLHttpRequest

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17663 from shashalu/fix-rgw-LCFilter

rgw: Fix closing tag for Prefix

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17543 from iliul/fix-dup-global-section

vstart.sh: Remove duplicate global section

Reviewed-by: Sage Weil <sage@redhat.com>

vstart.sh: Remove duplicate global section

Signed-off-by: iliul <liul124@chinaunicom.cn>

Merge pull request #17544 from joelin/wip-delete-forgot-mark

msg: remove the ),it's redundant.

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17515 from xiexingguo/wip-data-digest

osd/PrimaryLogPG: do not set data digest for bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

test/config: regulate parameter order of ASSERT_EQ

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

common: fix daemon abnormal exit at parsing invalid arguments

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

mds: optimize MDCache::rejoin_scour_survivor_replicas()

avoid iterating dentries if dirfrag is non-auth

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix MDSCacheObject::clear_replica_map

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17366 from smithfarm/wip-21176

build/ops: python-numpy-devel build dependency for SUSE

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Ewaznazari <Nazari@keyinternational.academy>

Merge pull request #17685 from ygtzf/HashIndex-clerical-error

os/filestore/HashIndex.h: fixed a typo in comment

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17688 from TsaiJin/wip-fix-assert-bug

msg/asyc/rmda: fix the bug of assert when Infiniband::recv_msg receives disconnect message

Reviewed-by: Haomai Wang <haomai@xsky.com>

osd/osd_types.h: default to no data/omap digest for new object

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PrimaryLogPG: do not generate data digest for BlueStore by default

BlueStore enables CRC by default, so this is a dup and gains
no more benefits.

Turn this off by default, which is good for performance.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph: fix "stopping mds can't export subtree"

caused by commit
790c34138f "Make standby-replay mds avoid initiating subtree export"

Fixes: http://tracker.ceph.com/issues/21378
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17687 from yonghengdexin735/wip-zzz-error

common: fix typo deamon in comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

mds: support limiting cache by memory

This introduces two config parameters:

    mds_cache_memory_limit: Sets the soft maximum of the cache to the given
    byte count. (Like mds_cache_size, this doesn't actually limit the maximum
    size of the cache. It just dictates the steady-state size.)

    mds_cache_reservation: This replaces mds_health_cache_threshold everywhere
    except the Beacon heartbeat sent to the mons. The idea here is to specify a
    reservation of memory (5% by default) for operations and the MDS tries to
    always maintain that reservation. So, the MDS will recall caps from clients
    when it begins dipping into its reservation of memory.

mds_cache_size still limits the cache by Inode count but is now by-default 0
(i.e. unlimited). The new preferred way of specifying cache limits is by memory
size. The default is 1GB.

Fixes: http://tracker.ceph.com/issues/20594
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1464976
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/asyc/rmda: fix the bug of assert when Infiniband::recv_msg receives
disconnect message

Signed-off-by: Jin Cai <caijin.caij@alibaba-inc.com>

common:fix error word

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

Merge pull request #17613 from smithfarm/wip-12056

doc: document include/ipaddr.h

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17644 from Vogtinator/readme-vstart

README: Improve vstart.sh usage

os/filestore/HashIndex.h: fixed a clerical error

Signed-off-by: yaoguotao <yaoguot@gmail.com>

pybind/ceph_argparse: fix cli output info

Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>

Merge pull request #17654 from yuyuyu101/wip-spdk-doc

doc/bluestore: add SPDK usage for bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17658 from ukernel/wip-cephfs-seekdir

doc/cephfs/posix: remove stale information for seekdir

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17669 from ygtzf/parameter-default-value-error

doc: update default value of parameter mon_subscribe_interval

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17672 from FrankYu/master

doc: the client inputs the pool name instead of pool ID

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge PR #17679 into master

* refs/remotes/upstream/pull/17679/head:
qa: get asok path from ceph.conf
qa: use config_path property instead of literal

Reviewed-by: John Spray <john.spray@redhat.com>

qa: get asok path from ceph.conf

The asok path was recently moved to /tmp to resolve unix path too long errors.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use config_path property instead of literal

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: refactor of lru

Avoids an unnecessary "max" size of the LRU which was used to calculate the
midpoint. Instead, just dynamically move the LRUObjects between top and bottom
on-the-fly.

This change is necessary for a cache which which does not limit by the number
of objects but by some other metric. (In this case, memory.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: resolve unsigned coercion compiler warning

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: use safer uint64_t for list size

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add bytes2str pretty print function

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: check if waiting is allocated before use

This prevents accidental allocation of the map.

Also, privatize the variable to protect from this in child classes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: go back to compact_map for replicas

Zheng observed that an alloc_ptr doesn't really work in this case since any
call to get_replicas() will cause the map to be allocated, nullifying the
benefit. Use a compact_map until a better solution can be written. (This means
that the map will be allocated outside the mempool.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use mempool for cache objects

The purpose of this is to allow us to track memory usage by cached objects so
we can limit cache size based on memory available/allocated to the MDS.

This commit is a first step: it adds CInode, CDir, and CDentry to the mempool
but not all of the containers in these classes (e.g. std::map). However,
MDSCacheObject has been changed to allocate its containers through the mempool
by converting compact_* containers to the std versions offered through mempool
via the new alloc_ptr.

(A compact_* class simply wraps a pointer to the std:: version to reduce memory
usage of an object when the container is only occasionally used. The alloc_ptr
allows us to achieve the same thing explicitly with only a little handholding:
when all entries in the wrapped container are deleted, the caller must call
alloc_ptr.release().)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: cleanup replica_map access

The gymnastics protecting the map failed as the code evolved. Just expose it
normally with a getter.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add alloc_ptr smart pointer

This ptr is like a unique_ptr except it allocates the underlying object on
access. The idea being that we can save memory if the object is only needed
sometimes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17650 from ceph/wip-ceph-deploy-fix

qa/tests: Fix get_system_type failure due to invalid remote name

Merge pull request #17636 from dillaman/wip-21319

rbd: clean up warnings when mirror commands used on non-setup pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

qa: avoid using make install for fio

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: the client inputs the pool name instead of pool ID

Signed-off-by: Frank Yu <flyxiaoyu@gmail.com>

Merge pull request #13383 from wonzhq/wip-update-calc-stats

osd: change the conditional in _update_calc_stats

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17640 from tchaikov/wip-diagram-py3

doc/scripts: py3 compatible

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: update default value of parameter mon_subscribe_interval

Signed-off-by: yaoguotao <yaoguot@gmail.com>

Merge pull request #17639 from tchaikov/wip-doc-ceph-disk

doc: update ceph-disk with a state-transition diagram

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: LCFilter_S3::to_xml the second "prefix" missed "/"

Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

Merge pull request #17538 from dzafman/wip-21272

Add export and remove ceph-objectstore-tool command option

Reviewed-by: Sage Weil <sage@redhat.com>

doc/bluestore: add SPDK usage for bluestore

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #17537 from mychoxin/yuanxin_remove_submit

os/bluestore: remove useless function submit

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/cephfs/posix: remove stale information for seekdir

Current cephfs can support seekdir efficiently. The diverge was
fixed by https://github.com/ceph/ceph/pull/14317

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

src/bluestore/NVMEDevice: make all read use aio_submit

Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

os/bluestore/NVMEDevice: fix the large write I/O issue.

When there is large I/O, we need to split it into multiple
tasks. With this patch, we can avoid no buffer allocation
issue for the single write task.

Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>