Merge pull request #17748 from tchaikov/wip-mon-constness

mon: more constness

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17727 from tchaikov/wip-pybind-cleanup

pybind: remove unused get_ceph_version()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17749 from tchaikov/wip-vstart

vstart.sh: simplify the objectstore related logic

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #17418 from Songweibin/wip-osdmap-full-try

rados: support python API of "set_osdmap_full_try"

Reviewed-by: Pan Liu <wanjun.lp@alibaba-inc.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17295 from david-z/wip-compress-scrub-warning

mon: Compress the warnings of pgs not scrubbed or deep-scrubbed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17732 from Songweibin/wip-fix-comments

doc,os,osdc: drop and modify comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17675 from theanalyst/doc-rgw-reshard

doc: rgw: add a note for resharding in 12.2.1 docs

doc: rgw: add a note for resharding in 12.2.1 docs

Since we're now enabling dynamic resharding, mention  this in release
notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #17701 into master

* refs/remotes/upstream/pull/17701/head:
qa/cephfs: Fix error in test_filtered_df

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17694 into master

* refs/remotes/upstream/pull/17694/head:
qa/cephfs: kill mount if it gets evicted by mds
qa/cephfs: fix test_evict_client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17681 into master

* refs/remotes/upstream/pull/17681/head:
mon: remove mdsmap-derived perf counters

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17676 into master

* refs/remotes/upstream/pull/17676/head:
qa/tasks/cephfs: Whitelist POOL_APP_NOT_ENABLED for test_misc

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17671 into master

* refs/remotes/upstream/pull/17671/head:
messages: Initilization of left memebers

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #17670 into master

* refs/remotes/upstream/pull/17670/head:
messages: Initialization of member variables

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #17652 into master

* refs/remotes/upstream/pull/17652/head:
client: use unsigned trim_caps count

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #17598 into master

* refs/remotes/upstream/pull/17598/head:
messages: Initialization of members MMDSCacheRejion

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #17595 into master

* refs/remotes/upstream/pull/17595/head:
ceph-fuse: should free array using "delete[]"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17542 into master

* refs/remotes/upstream/pull/17542/head:
client: Delete onsafe

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17709 from liewegas/wip-bluestore-repair

ceph-bluestore-tool: better fsck/repair, bluefs-bdev-{expand,sizes}

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17388 from tchaikov/wip-rocksdb

rocksdb: sync with upstream

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17706 from liewegas/wip-test-listsnaps

ceph_test_rados_api_tier: add ListSnap test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17705 from dzafman/wip-21328

osd: Only scan for omap corruption once

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17703 from dzafman/wip-misc

Erasure code read test and code cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17699 from tchaikov/wip-lookout-for-large-elen

msg/msg_types: fix the entity_addr_t's decoder

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17667 from scienceluo/wip-luo-fix-cli-output-branch

pybind/ceph_argparse: fix cli output info

Reviewed-by: Chang Liu <liuchang0812@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17664 from Yan-waller/wip-walle-0912moninjectargs

common: fix daemon abnormal exit at parsing invalid arguments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17655 from optimistyzy/911_2

src/bluestore/NVMEDevice: make all read use aio_submit

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #17611 from chardan/jfw-wip-atomic_t-cleanup

follow-up fixups for atomic_t spinlocks

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17605 from jcsp/wip-21253

mgr: fix py calls for dne service perf counters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17568 from amitkumar50/cov-msg-2

msg: Initializing class members in module msg

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16699 from jcsp/wip-mgr-perf-threshold

mgr: apply a threshold to perf counter prios

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: fix the entity_addr_t's decoder

the daemon is vulnerable to malicious client, which is able to send
large elen, and corrupt the stack, etc.

* throw at seeing corrupted entity_addr_t where its elen exceeds
  the length of sockaddr
* handle the exception thrown when decoding entity_addr_t in messenger
  layer.
* if a malicious client manages to send a corrutped entity_addr_t to
  daemon, daemon will crash because decode fails and the exception is
  not handled. it's better than continuing working with the bogus
  message.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build portable rocksdb

* so rocksdb is not built with -march=native, because
  - this is not portable
  - -march=native is not supported by older version of GCC on aarch64.
* and drop the sse42 specific settings for rocksdb. since rocksdb uses
  "target" attribute to build sse42 optimized crc32, as long as the
  compiler on building host is able to emit sse42 instructions.
  see https://github.com/facebook/rocksdb/pull/2807

Fixes: http://tracker.ceph.com/issues/20529
Signed-off-by: Kefu Chai <kchai@redhat.com>

kv/RocksDBStore: update to accomodate the change in rocksdb 5.6.0

per rocksdb's HISTORY.md, it

> replace global variable `PerfContext perf_context` with `PerfContext*
> get_perf_context()`.

hence the change.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rocksdb: sync with upstream

to pickup the fix of https://github.com/facebook/rocksdb/pull/2692.

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: simplify the objectstore related logic

also avoid overriding --memstore specified by command line with
the default "bluestore"

Reported-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17361 from kungf/ulimit_setup

run-make-check.sh: run ulimit without sudo

Reviewed-by: Dan Mick <dmick@redhat.com>

run-make-check.sh: run ulimit without sudo

ulimit is a shell builtin like cd, not a separate program.
sudo looks for a binary to run, but there is no ulimit binary,
then you will get an error message like:
          "sudo: ulimit: command not found"
ulimit can only be run with root if you want raise the ulimit -n

Change-Id: I59ab09240cb15cabe5b0a7342c3f562cd50c75e9
Signed-off-by: yang.wang <yang.wang@easystack.cn>

Merge pull request #17713 from tchaikov/wip-ceph.in-python

ceph.in: execv using the same python

Merge pull request #17583 from neha-ojha/wip-cbt-teuthology-integration

qa: add cbt task for performance testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17680 from ceph/wip-smoke-whitelist

qa/tests: Various whitelists for smoke suite

Various whitelists for tests to pass

See tracker http://tracker.ceph.com/issues/21376 for details

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

ceph-bluestore-tool: better default logging; --log-file and --log-level options

- keep derr going to stderr (so we see 'fsck error' messages)
- hide the rest
- add friendly --log-file option

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: fix error prefixes

__func__ is now _fsck instead of fsck

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluestore-tool: add 'bluefs-bdev-expand' to expand wal or db usage

If you are using the wal or db devices, this will expand bluefs's usage to
include the entire block device.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluestore-tool: add 'bluefs-bdev-sizes' command

Show bdev sizes vs owned extents.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluestore-tool: factor out bluefs mount

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: allow repair invocation

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add repair interface

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17733 from yaozongyou/fix-doc-boolean-type-zero-to-false

doc: change boolean option default value from zero to false.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17610 from liewegas/wip-bluestore-fm-mas

os/bluestore: set bitmap freelist resolution to min_alloc_size

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17617 from liewegas/wip-drop-disk-log

mon/PGMap: drop DISK LOG column

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: change boolean option default value from zero to false.

Signed-off-by: Yao Zongyou yaozongyou@vip.qq.com

Merge pull request #17628 from majianpeng/bluestore-narrow-deferred-lock

os/bluestore/BlueStore: narrow deferred_lock in _deferred_submit_unlock.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc,os,osdc: drop and modify some comments

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #17627 from majianpeng/bluefs-exce-debug-code

os/bluestore/BlueFS: Don't call debug related code under any condition.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17371 from xiexingguo/wip-per-pool-full-control

mon, osd: per pool space-full flag support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17368 from xiexingguo/wip-nice-data

mon/PGMap: nice numbers for 'data' section of 'ceph df' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind: remove unused get_ceph_version()

get_ceph_version() was get_version() before 7b04e7ff. and after that
change, the version number of the python extension is hardwired to
`__version__`. because pybind is a wrapper around the public APIs
exposed by librados, librbd and libcephfs, which are versioned
separately from ceph daemons and tools, it's reasonable not to use the
version number in ceph_ver.h. thus, it's safe to drop this function.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in: execv using the same python

otherwise we could be using another python when respawing the ceph cli.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #17657 into master

* refs/remotes/upstream/pull/17657/head:
mds: optimize MDCache::rejoin_scour_survivor_replicas()
mds: fix MDSCacheObject::clear_replica_map
mds: support limiting cache by memory
common: refactor of lru
mds: resolve unsigned coercion compiler warning
common: use safer uint64_t for list size
common: add bytes2str pretty print function
mds: check if waiting is allocated before use
mds: go back to compact_map for replicas
mds: use mempool for cache objects
mds: cleanup replica_map access
common: add alloc_ptr smart pointer
common: add warning on base class use of mempool
common: use atomic uin64_t for counter

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17689 into master

* refs/remotes/upstream/pull/17689/head:
ceph: fix "stopping mds can't export subtree"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17545 into master

* refs/remotes/upstream/pull/17545/head:
client: fix request reference leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #17700 from cbodley/wip-qa-smoke-s3tests

qa/smoke: add rgw crypto config for s3tests

ceph_test_rados_api_tier: add ListSnap test

Signed-off-by: Xuehan Xu <xuxuehan@360.cn>

osd: Only scan for omap corruption once

Before
  state 2: Can have complete tables (some may be bad)
state 3: Never had complete tables
After
  state 2: Can have complete tables (some may be bad)
state 3 with legacy: Can have complete tables (bad ones are cleared)
state 3: Never had complete tables

Once OSDs boot with this change you can't downgrade to a previous release.
If someone does downgrade they could have unstable OSDs that hit assert(state.v < 3).
The following command run after shutting down the cluster but before downgrading
ceph packages would be a way to fix this.

ceph-osdomap-tool --omap-path ... --command resetv2

Fixes: http://tracker.ceph.com/issues/21328
Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add the ability to reset state to v2

Available for testing and user downgrade.

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Show DB state information

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add --backend option to ceph-osdomap-tool default to rocksdb

Fix hard-coded "leveldb" backend.  The command is broken in Luminous
now that "rocksdb" is the default.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #17608 into master

* refs/remotes/upstream/pull/17608/head:
doc/cephfs/posix: put posix notes in perspective

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd: CLEANUP: Change getattr_maybe_cache() to reflect how it is being used

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add a removal test for erasure code read

Test feature: http://tracker.ceph.com/issues/14513

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #17641 from amitkumar50/cov-messages-12

messages: Initialization of variable beat

Reviewed-by: Jos Collin <jcollin@redhat.com>

qa/cephfs: Fix error in test_filtered_df

ceph df accounts for pool size, so there is no need to do it in the test.

Fixes: http://tracker.ceph.com/issues/21381
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #10460 from weiqiaomiao/wqm-wip-rgw-dellc

rgw: add tenant to shard_id in RGWDeleteLC::execute()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #17061 from shashalu/fix-rgw-lc

rgw: fix lc process only schdule the first item of lc objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: fang.yuxiang <fang.yuxiang@eisoo.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

qa/smoke: add rgw crypto config for s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17631 from joke-lee/RGW-CORS-Access-Control-Allow-Origin-HEADER

rgw: return 'Access-Control-Allow-Origin' header when the set bucket versioning through XMLHttpRequest

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17632 from joke-lee/RGW-CORS-Access-Control-Allow-Origin-HEADER-WEBSITE

rgw: return 'Access-Control-Allow-Origin' header when the set and delete bucket website through XMLHttpRequest

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17663 from shashalu/fix-rgw-LCFilter

rgw: Fix closing tag for Prefix

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17543 from iliul/fix-dup-global-section

vstart.sh: Remove duplicate global section

Reviewed-by: Sage Weil <sage@redhat.com>

vstart.sh: Remove duplicate global section

Signed-off-by: iliul <liul124@chinaunicom.cn>

qa/cephfs: kill mount if it gets evicted by mds

otherwise, teardown() hange at umount

Fixes: http://tracker.ceph.com/issues/21275
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17544 from joelin/wip-delete-forgot-mark

msg: remove the ),it's redundant.

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17515 from xiexingguo/wip-data-digest

osd/PrimaryLogPG: do not set data digest for bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

test/config: regulate parameter order of ASSERT_EQ

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

common: fix daemon abnormal exit at parsing invalid arguments

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

qa/cephfs: fix test_evict_client

executing mount_a.kill() twice, then executing mount_b.kill_cleanup()
twice do not make sense.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: optimize MDCache::rejoin_scour_survivor_replicas()

avoid iterating dentries if dirfrag is non-auth

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix MDSCacheObject::clear_replica_map

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17366 from smithfarm/wip-21176

build/ops: python-numpy-devel build dependency for SUSE

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Ewaznazari <Nazari@keyinternational.academy>

Merge pull request #17685 from ygtzf/HashIndex-clerical-error

os/filestore/HashIndex.h: fixed a typo in comment

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17688 from TsaiJin/wip-fix-assert-bug

msg/asyc/rmda: fix the bug of assert when Infiniband::recv_msg receives disconnect message

Reviewed-by: Haomai Wang <haomai@xsky.com>

osd/osd_types.h: default to no data/omap digest for new object

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PrimaryLogPG: do not generate data digest for BlueStore by default

BlueStore enables CRC by default, so this is a dup and gains
no more benefits.

Turn this off by default, which is good for performance.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph: fix "stopping mds can't export subtree"

caused by commit
790c34138f "Make standby-replay mds avoid initiating subtree export"

Fixes: http://tracker.ceph.com/issues/21378
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #17687 from yonghengdexin735/wip-zzz-error

common: fix typo deamon in comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

mds: support limiting cache by memory

This introduces two config parameters:

    mds_cache_memory_limit: Sets the soft maximum of the cache to the given
    byte count. (Like mds_cache_size, this doesn't actually limit the maximum
    size of the cache. It just dictates the steady-state size.)

    mds_cache_reservation: This replaces mds_health_cache_threshold everywhere
    except the Beacon heartbeat sent to the mons. The idea here is to specify a
    reservation of memory (5% by default) for operations and the MDS tries to
    always maintain that reservation. So, the MDS will recall caps from clients
    when it begins dipping into its reservation of memory.

mds_cache_size still limits the cache by Inode count but is now by-default 0
(i.e. unlimited). The new preferred way of specifying cache limits is by memory
size. The default is 1GB.

Fixes: http://tracker.ceph.com/issues/20594
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1464976
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>