cmake: build portable rocksdb

* so rocksdb is not built with -march=native, because
  - this is not portable
  - -march=native is not supported by older version of GCC on aarch64.
* and drop the sse42 specific settings for rocksdb. since rocksdb uses
  "target" attribute to build sse42 optimized crc32, as long as the
  compiler on building host is able to emit sse42 instructions.
  see https://github.com/facebook/rocksdb/pull/2807

Fixes: http://tracker.ceph.com/issues/20529
Signed-off-by: Kefu Chai <kchai@redhat.com>

kv/RocksDBStore: update to accomodate the change in rocksdb 5.6.0

per rocksdb's HISTORY.md, it

> replace global variable `PerfContext perf_context` with `PerfContext*
> get_perf_context()`.

hence the change.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rocksdb: sync with upstream

to pickup the fix of https://github.com/facebook/rocksdb/pull/2692.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17583 from neha-ojha/wip-cbt-teuthology-integration

qa: add cbt task for performance testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17680 from ceph/wip-smoke-whitelist

qa/tests: Various whitelists for smoke suite

Various whitelists for tests to pass

See tracker http://tracker.ceph.com/issues/21376 for details

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #17733 from yaozongyou/fix-doc-boolean-type-zero-to-false

doc: change boolean option default value from zero to false.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17610 from liewegas/wip-bluestore-fm-mas

os/bluestore: set bitmap freelist resolution to min_alloc_size

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17617 from liewegas/wip-drop-disk-log

mon/PGMap: drop DISK LOG column

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: change boolean option default value from zero to false.

Signed-off-by: Yao Zongyou yaozongyou@vip.qq.com

Merge pull request #17628 from majianpeng/bluestore-narrow-deferred-lock

os/bluestore/BlueStore: narrow deferred_lock in _deferred_submit_unlock.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17627 from majianpeng/bluefs-exce-debug-code

os/bluestore/BlueFS: Don't call debug related code under any condition.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17371 from xiexingguo/wip-per-pool-full-control

mon, osd: per pool space-full flag support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17368 from xiexingguo/wip-nice-data

mon/PGMap: nice numbers for 'data' section of 'ceph df' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #17657 into master

* refs/remotes/upstream/pull/17657/head:
mds: optimize MDCache::rejoin_scour_survivor_replicas()
mds: fix MDSCacheObject::clear_replica_map
mds: support limiting cache by memory
common: refactor of lru
mds: resolve unsigned coercion compiler warning
common: use safer uint64_t for list size
common: add bytes2str pretty print function
mds: check if waiting is allocated before use
mds: go back to compact_map for replicas
mds: use mempool for cache objects
mds: cleanup replica_map access
common: add alloc_ptr smart pointer
common: add warning on base class use of mempool
common: use atomic uin64_t for counter

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17689 into master

* refs/remotes/upstream/pull/17689/head:
ceph: fix "stopping mds can't export subtree"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17545 into master

* refs/remotes/upstream/pull/17545/head:
client: fix request reference leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #17700 from cbodley/wip-qa-smoke-s3tests

qa/smoke: add rgw crypto config for s3tests

Merge PR #17608 into master

* refs/remotes/upstream/pull/17608/head:
doc/cephfs/posix: put posix notes in perspective

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17641 from amitkumar50/cov-messages-12

messages: Initialization of variable beat

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #10460 from weiqiaomiao/wqm-wip-rgw-dellc

rgw: add tenant to shard_id in RGWDeleteLC::execute()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #17061 from shashalu/fix-rgw-lc

rgw: fix lc process only schdule the first item of lc objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: fang.yuxiang <fang.yuxiang@eisoo.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

qa/smoke: add rgw crypto config for s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17631 from joke-lee/RGW-CORS-Access-Control-Allow-Origin-HEADER

rgw: return 'Access-Control-Allow-Origin' header when the set bucket versioning through XMLHttpRequest

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17632 from joke-lee/RGW-CORS-Access-Control-Allow-Origin-HEADER-WEBSITE

rgw: return 'Access-Control-Allow-Origin' header when the set and delete bucket website through XMLHttpRequest

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17663 from shashalu/fix-rgw-LCFilter

rgw: Fix closing tag for Prefix

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17543 from iliul/fix-dup-global-section

vstart.sh: Remove duplicate global section

Reviewed-by: Sage Weil <sage@redhat.com>

vstart.sh: Remove duplicate global section

Signed-off-by: iliul <liul124@chinaunicom.cn>

Merge pull request #17544 from joelin/wip-delete-forgot-mark

msg: remove the ),it's redundant.

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17515 from xiexingguo/wip-data-digest

osd/PrimaryLogPG: do not set data digest for bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mds: optimize MDCache::rejoin_scour_survivor_replicas()

avoid iterating dentries if dirfrag is non-auth

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix MDSCacheObject::clear_replica_map

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17366 from smithfarm/wip-21176

build/ops: python-numpy-devel build dependency for SUSE

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Ewaznazari <Nazari@keyinternational.academy>

Merge pull request #17685 from ygtzf/HashIndex-clerical-error

os/filestore/HashIndex.h: fixed a typo in comment

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17688 from TsaiJin/wip-fix-assert-bug

msg/asyc/rmda: fix the bug of assert when Infiniband::recv_msg receives disconnect message

Reviewed-by: Haomai Wang <haomai@xsky.com>

osd/osd_types.h: default to no data/omap digest for new object

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PrimaryLogPG: do not generate data digest for BlueStore by default

BlueStore enables CRC by default, so this is a dup and gains
no more benefits.

Turn this off by default, which is good for performance.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph: fix "stopping mds can't export subtree"

caused by commit
790c34138f "Make standby-replay mds avoid initiating subtree export"

Fixes: http://tracker.ceph.com/issues/21378
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17687 from yonghengdexin735/wip-zzz-error

common: fix typo deamon in comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

mds: support limiting cache by memory

This introduces two config parameters:

    mds_cache_memory_limit: Sets the soft maximum of the cache to the given
    byte count. (Like mds_cache_size, this doesn't actually limit the maximum
    size of the cache. It just dictates the steady-state size.)

    mds_cache_reservation: This replaces mds_health_cache_threshold everywhere
    except the Beacon heartbeat sent to the mons. The idea here is to specify a
    reservation of memory (5% by default) for operations and the MDS tries to
    always maintain that reservation. So, the MDS will recall caps from clients
    when it begins dipping into its reservation of memory.

mds_cache_size still limits the cache by Inode count but is now by-default 0
(i.e. unlimited). The new preferred way of specifying cache limits is by memory
size. The default is 1GB.

Fixes: http://tracker.ceph.com/issues/20594
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1464976
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/asyc/rmda: fix the bug of assert when Infiniband::recv_msg receives
disconnect message

Signed-off-by: Jin Cai <caijin.caij@alibaba-inc.com>

common:fix error word

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

Merge pull request #17613 from smithfarm/wip-12056

doc: document include/ipaddr.h

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17644 from Vogtinator/readme-vstart

README: Improve vstart.sh usage

os/filestore/HashIndex.h: fixed a clerical error

Signed-off-by: yaoguotao <yaoguot@gmail.com>

Merge pull request #17654 from yuyuyu101/wip-spdk-doc

doc/bluestore: add SPDK usage for bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17658 from ukernel/wip-cephfs-seekdir

doc/cephfs/posix: remove stale information for seekdir

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17669 from ygtzf/parameter-default-value-error

doc: update default value of parameter mon_subscribe_interval

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17672 from FrankYu/master

doc: the client inputs the pool name instead of pool ID

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge PR #17679 into master

* refs/remotes/upstream/pull/17679/head:
qa: get asok path from ceph.conf
qa: use config_path property instead of literal

Reviewed-by: John Spray <john.spray@redhat.com>

qa: get asok path from ceph.conf

The asok path was recently moved to /tmp to resolve unix path too long errors.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use config_path property instead of literal

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: refactor of lru

Avoids an unnecessary "max" size of the LRU which was used to calculate the
midpoint. Instead, just dynamically move the LRUObjects between top and bottom
on-the-fly.

This change is necessary for a cache which which does not limit by the number
of objects but by some other metric. (In this case, memory.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: resolve unsigned coercion compiler warning

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: use safer uint64_t for list size

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add bytes2str pretty print function

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: check if waiting is allocated before use

This prevents accidental allocation of the map.

Also, privatize the variable to protect from this in child classes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: go back to compact_map for replicas

Zheng observed that an alloc_ptr doesn't really work in this case since any
call to get_replicas() will cause the map to be allocated, nullifying the
benefit. Use a compact_map until a better solution can be written. (This means
that the map will be allocated outside the mempool.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use mempool for cache objects

The purpose of this is to allow us to track memory usage by cached objects so
we can limit cache size based on memory available/allocated to the MDS.

This commit is a first step: it adds CInode, CDir, and CDentry to the mempool
but not all of the containers in these classes (e.g. std::map). However,
MDSCacheObject has been changed to allocate its containers through the mempool
by converting compact_* containers to the std versions offered through mempool
via the new alloc_ptr.

(A compact_* class simply wraps a pointer to the std:: version to reduce memory
usage of an object when the container is only occasionally used. The alloc_ptr
allows us to achieve the same thing explicitly with only a little handholding:
when all entries in the wrapped container are deleted, the caller must call
alloc_ptr.release().)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: cleanup replica_map access

The gymnastics protecting the map failed as the code evolved. Just expose it
normally with a getter.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add alloc_ptr smart pointer

This ptr is like a unique_ptr except it allocates the underlying object on
access. The idea being that we can save memory if the object is only needed
sometimes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17650 from ceph/wip-ceph-deploy-fix

qa/tests: Fix get_system_type failure due to invalid remote name

Merge pull request #17636 from dillaman/wip-21319

rbd: clean up warnings when mirror commands used on non-setup pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

qa: avoid using make install for fio

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: the client inputs the pool name instead of pool ID

Signed-off-by: Frank Yu <flyxiaoyu@gmail.com>

Merge pull request #13383 from wonzhq/wip-update-calc-stats

osd: change the conditional in _update_calc_stats

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17640 from tchaikov/wip-diagram-py3

doc/scripts: py3 compatible

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: update default value of parameter mon_subscribe_interval

Signed-off-by: yaoguotao <yaoguot@gmail.com>

Merge pull request #17639 from tchaikov/wip-doc-ceph-disk

doc: update ceph-disk with a state-transition diagram

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: LCFilter_S3::to_xml the second "prefix" missed "/"

Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

Merge pull request #17538 from dzafman/wip-21272

Add export and remove ceph-objectstore-tool command option

Reviewed-by: Sage Weil <sage@redhat.com>

doc/bluestore: add SPDK usage for bluestore

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #17537 from mychoxin/yuanxin_remove_submit

os/bluestore: remove useless function submit

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/cephfs/posix: remove stale information for seekdir

Current cephfs can support seekdir efficiently. The diverge was
fixed by https://github.com/ceph/ceph/pull/14317

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17606 from ceph/wip-bz1478598

ceph-volume allow filtering by `uuid`, do not require osd id

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

messages: Initialization of variable beat

Fixes the coverity issue:

** 717278 Uninitialized scalar field

CID 717278 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member beat is not
initialized in this constructor nor in any functions that it calls.

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

ceph-volume lvm.trigger just split the osd_id once

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests catch improper parsing of osd uuid

When the osd ID existed in the UUID in the same format (e.g. '1-') the
parsing would break returning a bogus UUID that is impossible to find

Signed-off-by: Alfredo Deza <adeza@redhat.com>

tests: remove 'ceph-' prefix from test node hostnames

This updates our Vagrantfile to match the changes in ceph-ansible
introduced by https://github.com/ceph/ceph-ansible/commit/298a63c43

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.api PVolumes should filter by all matching tags, not just one

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests pv API should also match all pv_tags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests prove all volume tags have to match in vgs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17377 from cbodley/wip-rgw-test-multi-encryption

test/rgw: fix test_encrypted_object_sync for 3+ zones

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #17102 from mychoxin/yuanxin_print_log_if_add_event_fail

msg/async: print error log if add_event fail

Reviewed-by: Pan Liu <liupan1111@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16492 from yehudasa/wip-rgw-multi-log

rgw: multisite log tracing

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17400 from theanalyst/fix/rgw-set-attrs-mtime

rgw: rgw_rados: set_attrs now sets the same time for BI & object

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

admon/build-doc: be compatible with py3

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados: fix the method names

otherwise, sphinx fails to run with py3

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: switch to upstream sphinx-ditaa

it works with setuptools and is now compatible with py3

the py3 branch is created to track the upstream's master branch

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17638 from idryomov/wip-rbd-already-mapped-fixes

krbd: rewrite "already mapped" code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

README: Improve vstart.sh usage

Using long options where possible to make it easier to understand directly
and adding bluestore by default as it is the default storage backend.

Signed-off-by: Fabian Vogt <fvogt@suse.com>

os/bluestore/BlueStore: narrow deferred_lock in _deferred_submit_unlock.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BlueFS: Don't call debug related code under any condition.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ceph-volume lvm.api all vg tags should match, not just one

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests prove all tags have to match in lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api all lv tags should match, not just one

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.activate allow not using osd_id

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests prove missing args.osd_id is fine

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add is_root and lv fixtures to conftest

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/scripts/gen_state_diagram.py: port to py3

Signed-off-by: Kefu Chai <kchai@redhat.com>