doc: rgw: add a note for resharding in 12.2.1 docs

Since we're now enabling dynamic resharding, mention  this in release
notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #17709 from liewegas/wip-bluestore-repair

ceph-bluestore-tool: better fsck/repair, bluefs-bdev-{expand,sizes}

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17388 from tchaikov/wip-rocksdb

rocksdb: sync with upstream

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17706 from liewegas/wip-test-listsnaps

ceph_test_rados_api_tier: add ListSnap test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17705 from dzafman/wip-21328

osd: Only scan for omap corruption once

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17703 from dzafman/wip-misc

Erasure code read test and code cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17699 from tchaikov/wip-lookout-for-large-elen

msg/msg_types: fix the entity_addr_t's decoder

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17667 from scienceluo/wip-luo-fix-cli-output-branch

pybind/ceph_argparse: fix cli output info

Reviewed-by: Chang Liu <liuchang0812@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17664 from Yan-waller/wip-walle-0912moninjectargs

common: fix daemon abnormal exit at parsing invalid arguments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17655 from optimistyzy/911_2

src/bluestore/NVMEDevice: make all read use aio_submit

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #17611 from chardan/jfw-wip-atomic_t-cleanup

follow-up fixups for atomic_t spinlocks

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17605 from jcsp/wip-21253

mgr: fix py calls for dne service perf counters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17568 from amitkumar50/cov-msg-2

msg: Initializing class members in module msg

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16699 from jcsp/wip-mgr-perf-threshold

mgr: apply a threshold to perf counter prios

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: fix the entity_addr_t's decoder

the daemon is vulnerable to malicious client, which is able to send
large elen, and corrupt the stack, etc.

* throw at seeing corrupted entity_addr_t where its elen exceeds
  the length of sockaddr
* handle the exception thrown when decoding entity_addr_t in messenger
  layer.
* if a malicious client manages to send a corrutped entity_addr_t to
  daemon, daemon will crash because decode fails and the exception is
  not handled. it's better than continuing working with the bogus
  message.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build portable rocksdb

* so rocksdb is not built with -march=native, because
  - this is not portable
  - -march=native is not supported by older version of GCC on aarch64.
* and drop the sse42 specific settings for rocksdb. since rocksdb uses
  "target" attribute to build sse42 optimized crc32, as long as the
  compiler on building host is able to emit sse42 instructions.
  see https://github.com/facebook/rocksdb/pull/2807

Fixes: http://tracker.ceph.com/issues/20529
Signed-off-by: Kefu Chai <kchai@redhat.com>

kv/RocksDBStore: update to accomodate the change in rocksdb 5.6.0

per rocksdb's HISTORY.md, it

> replace global variable `PerfContext perf_context` with `PerfContext*
> get_perf_context()`.

hence the change.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rocksdb: sync with upstream

to pickup the fix of https://github.com/facebook/rocksdb/pull/2692.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17361 from kungf/ulimit_setup

run-make-check.sh: run ulimit without sudo

Reviewed-by: Dan Mick <dmick@redhat.com>

run-make-check.sh: run ulimit without sudo

ulimit is a shell builtin like cd, not a separate program.
sudo looks for a binary to run, but there is no ulimit binary,
then you will get an error message like:
          "sudo: ulimit: command not found"
ulimit can only be run with root if you want raise the ulimit -n

Change-Id: I59ab09240cb15cabe5b0a7342c3f562cd50c75e9
Signed-off-by: yang.wang <yang.wang@easystack.cn>

Merge pull request #17713 from tchaikov/wip-ceph.in-python

ceph.in: execv using the same python

Merge pull request #17583 from neha-ojha/wip-cbt-teuthology-integration

qa: add cbt task for performance testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17680 from ceph/wip-smoke-whitelist

qa/tests: Various whitelists for smoke suite

Various whitelists for tests to pass

See tracker http://tracker.ceph.com/issues/21376 for details

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

ceph-bluestore-tool: better default logging; --log-file and --log-level options

- keep derr going to stderr (so we see 'fsck error' messages)
- hide the rest
- add friendly --log-file option

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: fix error prefixes

__func__ is now _fsck instead of fsck

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluestore-tool: add 'bluefs-bdev-expand' to expand wal or db usage

If you are using the wal or db devices, this will expand bluefs's usage to
include the entire block device.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluestore-tool: add 'bluefs-bdev-sizes' command

Show bdev sizes vs owned extents.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluestore-tool: factor out bluefs mount

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: allow repair invocation

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add repair interface

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17733 from yaozongyou/fix-doc-boolean-type-zero-to-false

doc: change boolean option default value from zero to false.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17610 from liewegas/wip-bluestore-fm-mas

os/bluestore: set bitmap freelist resolution to min_alloc_size

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17617 from liewegas/wip-drop-disk-log

mon/PGMap: drop DISK LOG column

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: change boolean option default value from zero to false.

Signed-off-by: Yao Zongyou yaozongyou@vip.qq.com

Merge pull request #17628 from majianpeng/bluestore-narrow-deferred-lock

os/bluestore/BlueStore: narrow deferred_lock in _deferred_submit_unlock.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17627 from majianpeng/bluefs-exce-debug-code

os/bluestore/BlueFS: Don't call debug related code under any condition.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17371 from xiexingguo/wip-per-pool-full-control

mon, osd: per pool space-full flag support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17368 from xiexingguo/wip-nice-data

mon/PGMap: nice numbers for 'data' section of 'ceph df' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.in: execv using the same python

otherwise we could be using another python when respawing the ceph cli.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #17657 into master

* refs/remotes/upstream/pull/17657/head:
mds: optimize MDCache::rejoin_scour_survivor_replicas()
mds: fix MDSCacheObject::clear_replica_map
mds: support limiting cache by memory
common: refactor of lru
mds: resolve unsigned coercion compiler warning
common: use safer uint64_t for list size
common: add bytes2str pretty print function
mds: check if waiting is allocated before use
mds: go back to compact_map for replicas
mds: use mempool for cache objects
mds: cleanup replica_map access
common: add alloc_ptr smart pointer
common: add warning on base class use of mempool
common: use atomic uin64_t for counter

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17689 into master

* refs/remotes/upstream/pull/17689/head:
ceph: fix "stopping mds can't export subtree"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17545 into master

* refs/remotes/upstream/pull/17545/head:
client: fix request reference leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #17700 from cbodley/wip-qa-smoke-s3tests

qa/smoke: add rgw crypto config for s3tests

ceph_test_rados_api_tier: add ListSnap test

Signed-off-by: Xuehan Xu <xuxuehan@360.cn>

osd: Only scan for omap corruption once

Before
  state 2: Can have complete tables (some may be bad)
state 3: Never had complete tables
After
  state 2: Can have complete tables (some may be bad)
state 3 with legacy: Can have complete tables (bad ones are cleared)
state 3: Never had complete tables

Once OSDs boot with this change you can't downgrade to a previous release.
If someone does downgrade they could have unstable OSDs that hit assert(state.v < 3).
The following command run after shutting down the cluster but before downgrading
ceph packages would be a way to fix this.

ceph-osdomap-tool --omap-path ... --command resetv2

Fixes: http://tracker.ceph.com/issues/21328
Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add the ability to reset state to v2

Available for testing and user downgrade.

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Show DB state information

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add --backend option to ceph-osdomap-tool default to rocksdb

Fix hard-coded "leveldb" backend.  The command is broken in Luminous
now that "rocksdb" is the default.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #17608 into master

* refs/remotes/upstream/pull/17608/head:
doc/cephfs/posix: put posix notes in perspective

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd: CLEANUP: Change getattr_maybe_cache() to reflect how it is being used

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add a removal test for erasure code read

Test feature: http://tracker.ceph.com/issues/14513

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #17641 from amitkumar50/cov-messages-12

messages: Initialization of variable beat

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #10460 from weiqiaomiao/wqm-wip-rgw-dellc

rgw: add tenant to shard_id in RGWDeleteLC::execute()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #17061 from shashalu/fix-rgw-lc

rgw: fix lc process only schdule the first item of lc objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: fang.yuxiang <fang.yuxiang@eisoo.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

qa/smoke: add rgw crypto config for s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17631 from joke-lee/RGW-CORS-Access-Control-Allow-Origin-HEADER

rgw: return 'Access-Control-Allow-Origin' header when the set bucket versioning through XMLHttpRequest

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17632 from joke-lee/RGW-CORS-Access-Control-Allow-Origin-HEADER-WEBSITE

rgw: return 'Access-Control-Allow-Origin' header when the set and delete bucket website through XMLHttpRequest

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17663 from shashalu/fix-rgw-LCFilter

rgw: Fix closing tag for Prefix

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17543 from iliul/fix-dup-global-section

vstart.sh: Remove duplicate global section

Reviewed-by: Sage Weil <sage@redhat.com>

vstart.sh: Remove duplicate global section

Signed-off-by: iliul <liul124@chinaunicom.cn>

Merge pull request #17544 from joelin/wip-delete-forgot-mark

msg: remove the ),it's redundant.

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17515 from xiexingguo/wip-data-digest

osd/PrimaryLogPG: do not set data digest for bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

test/config: regulate parameter order of ASSERT_EQ

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

common: fix daemon abnormal exit at parsing invalid arguments

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

mds: optimize MDCache::rejoin_scour_survivor_replicas()

avoid iterating dentries if dirfrag is non-auth

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix MDSCacheObject::clear_replica_map

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17366 from smithfarm/wip-21176

build/ops: python-numpy-devel build dependency for SUSE

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Ewaznazari <Nazari@keyinternational.academy>

Merge pull request #17685 from ygtzf/HashIndex-clerical-error

os/filestore/HashIndex.h: fixed a typo in comment

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17688 from TsaiJin/wip-fix-assert-bug

msg/asyc/rmda: fix the bug of assert when Infiniband::recv_msg receives disconnect message

Reviewed-by: Haomai Wang <haomai@xsky.com>

osd/osd_types.h: default to no data/omap digest for new object

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PrimaryLogPG: do not generate data digest for BlueStore by default

BlueStore enables CRC by default, so this is a dup and gains
no more benefits.

Turn this off by default, which is good for performance.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph: fix "stopping mds can't export subtree"

caused by commit
790c34138f "Make standby-replay mds avoid initiating subtree export"

Fixes: http://tracker.ceph.com/issues/21378
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17687 from yonghengdexin735/wip-zzz-error

common: fix typo deamon in comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

mds: support limiting cache by memory

This introduces two config parameters:

    mds_cache_memory_limit: Sets the soft maximum of the cache to the given
    byte count. (Like mds_cache_size, this doesn't actually limit the maximum
    size of the cache. It just dictates the steady-state size.)

    mds_cache_reservation: This replaces mds_health_cache_threshold everywhere
    except the Beacon heartbeat sent to the mons. The idea here is to specify a
    reservation of memory (5% by default) for operations and the MDS tries to
    always maintain that reservation. So, the MDS will recall caps from clients
    when it begins dipping into its reservation of memory.

mds_cache_size still limits the cache by Inode count but is now by-default 0
(i.e. unlimited). The new preferred way of specifying cache limits is by memory
size. The default is 1GB.

Fixes: http://tracker.ceph.com/issues/20594
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1464976
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/asyc/rmda: fix the bug of assert when Infiniband::recv_msg receives
disconnect message

Signed-off-by: Jin Cai <caijin.caij@alibaba-inc.com>

common:fix error word

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

Merge pull request #17613 from smithfarm/wip-12056

doc: document include/ipaddr.h

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17644 from Vogtinator/readme-vstart

README: Improve vstart.sh usage

os/filestore/HashIndex.h: fixed a clerical error

Signed-off-by: yaoguotao <yaoguot@gmail.com>

pybind/ceph_argparse: fix cli output info

Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>

Merge pull request #17654 from yuyuyu101/wip-spdk-doc

doc/bluestore: add SPDK usage for bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17658 from ukernel/wip-cephfs-seekdir

doc/cephfs/posix: remove stale information for seekdir

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17669 from ygtzf/parameter-default-value-error

doc: update default value of parameter mon_subscribe_interval

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17672 from FrankYu/master

doc: the client inputs the pool name instead of pool ID

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge PR #17679 into master

* refs/remotes/upstream/pull/17679/head:
qa: get asok path from ceph.conf
qa: use config_path property instead of literal

Reviewed-by: John Spray <john.spray@redhat.com>

qa: get asok path from ceph.conf

The asok path was recently moved to /tmp to resolve unix path too long errors.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use config_path property instead of literal

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: refactor of lru

Avoids an unnecessary "max" size of the LRU which was used to calculate the
midpoint. Instead, just dynamically move the LRUObjects between top and bottom
on-the-fly.

This change is necessary for a cache which which does not limit by the number
of objects but by some other metric. (In this case, memory.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: resolve unsigned coercion compiler warning

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: use safer uint64_t for list size

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add bytes2str pretty print function

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: check if waiting is allocated before use

This prevents accidental allocation of the map.

Also, privatize the variable to protect from this in child classes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: go back to compact_map for replicas

Zheng observed that an alloc_ptr doesn't really work in this case since any
call to get_replicas() will cause the map to be allocated, nullifying the
benefit. Use a compact_map until a better solution can be written. (This means
that the map will be allocated outside the mempool.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use mempool for cache objects

The purpose of this is to allow us to track memory usage by cached objects so
we can limit cache size based on memory available/allocated to the MDS.

This commit is a first step: it adds CInode, CDir, and CDentry to the mempool
but not all of the containers in these classes (e.g. std::map). However,
MDSCacheObject has been changed to allocate its containers through the mempool
by converting compact_* containers to the std versions offered through mempool
via the new alloc_ptr.

(A compact_* class simply wraps a pointer to the std:: version to reduce memory
usage of an object when the container is only occasionally used. The alloc_ptr
allows us to achieve the same thing explicitly with only a little handholding:
when all entries in the wrapped container are deleted, the caller must call
alloc_ptr.release().)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: cleanup replica_map access

The gymnastics protecting the map failed as the code evolved. Just expose it
normally with a getter.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add alloc_ptr smart pointer

This ptr is like a unique_ptr except it allocates the underlying object on
access. The idea being that we can save memory if the object is only needed
sometimes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17650 from ceph/wip-ceph-deploy-fix

qa/tests: Fix get_system_type failure due to invalid remote name

Merge pull request #17636 from dillaman/wip-21319

rbd: clean up warnings when mirror commands used on non-setup pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

qa: avoid using make install for fio

Signed-off-by: Neha Ojha <nojha@redhat.com>