os/bluestore/NVMEDevice: Remove using dpdk thread

Do not use DPDK's thread but use the run in complete mode,
thus we do not need to bind two cores for SPDK.

Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

Merge pull request #17737 from ivancich/wip-pull-updated-dmclock

osd: update dmclock library w git subtree pull

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17762 from yaozongyou/fix-typo-in-php-rst

doc: fix typo in php.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17763 from xiexingguo/wip-21409

qa: fix pool-quota related tests

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fix typo in php.rst

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

qa: fix pool-quota related tests

https://github.com/ceph/ceph/pull/17371 introduces support of
per-pool space-full flag, which turns out to set both
full and full_no_quota flags now if a pool is currently running out
of quota.

Actually this test is fragile as long as we keep appending new flags
at pool granularity, but let's not bother with that complexity now.

Fixes: http://tracker.ceph.com/issues/21409
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17615 from tchaikov/wip-osx

cmake,common: changes to port part of ceph to osx

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

client: use st_mtimespec instea of st_mtime on osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: fix the dencoder of entity_addr_t

improve the interoperability between freebsd/osx and GNU/Linux, because
the their layouts of sockaddr_storage are different, and we use the
one of linux as the wire format. so need to convert it on freebsd/osx
side.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17698 from liewegas/wip-bluestore-big-allocations

os/bluestore: allocate entire write in one go

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

crc32c: pass integer param by uint64_t

the assembly is expecting 64bits input, so make this explicit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crc32c: mach-o 64 follows the same calling convention as elf64 does

so reuse the register-parameter/variable mappings

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/encoding/readable.sh: use sysctl -n hw.ncpu instead of nproc on osx also

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for fdatasync() on non osx

on osx, fdatasync() is a syscall without userspace wrapper.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: use the best-effort if procfs is not available

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: allows undefined symbol in plugin on osx

clang on osx erros out when linking a dynamic library if any symbols is
missing by default. so disable this behavior.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: disable plugins using isa-l on osx

clang on osx emits functions with leading underscore, but the isa-l
assembly's functions have no leading underscore. we could label the
function declaration like `int foo asm("foo")` to remove the leading
underscore, but isa-l is a git submodule, let do this later. and in the
meanwhile, disable this plugin on osx.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crc32: label assembler functions without leading underscore

clang onder osx adds leading undescore to the function names to be
ABI compatible. but the assembly code does not do so. so we need to
control the name using gcc/clang extension. see
https://gcc.gnu.org/onlinedocs/gcc-4.4.0/gcc/Asm-Labels.html#Asm-Labels

Signed-off-by: Kefu Chai <kchai@redhat.com>

compat: consolidate definitions of osx and freebsd

on osx, ENODATA = 96, so we need to fix it. also define
CLOCK_MONOTONIC_COARSE and CLOCK_REALTIME_COARSE for osx, ceph_time.h
defines this also, but i don't want to include compat.h in ceph_time.h
at this moment.

and silence the warning of
 #warning ENODATA already defined to a value different from 87 (ENOATRR), refining to fix
because it is fired everywhere on osx when "compat.h" is included.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: fix the denc of sockaddr_storage on freebsd/osx

the layout of sockaddr_in and sockaddr_in6 are different on
GNU/Linux and FreeBSD/OSX:
 - on GNU/Linux, sockaddr does not have sa_len,
 - on GNU/Linux, sockaddr* use a 16 bit integer for sa_family, but
   on FreeBSD, a 32bit integer is used.
so we need to be more care when memcpy() between sockaddr_storage()
and ceph_sockaddr_storage().

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: consolidate sockaddr_storage dencoders

utilize ceph_sockaddr_storage on FreeBSD and osx to perform the
dencoding, to improve the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not pass --exclude-libs on osx

it is not supported by osx linker

Signed-off-by: Kefu Chai <kchai@redhat.com>

crc32c: only add qualifiers for elf64 format

they are not understood by non-elf formats

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: compile Mach-O format instead of ELF on OSX

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against libresolv on OSX

Signed-off-by: Kefu Chai <kchai@redhat.com>

time: use mach_absolute_time() for monotonic time

* use mach_absolute_time() for monotonic time
  mach_absolute_time() is faster and monotonic, see
  https://developer.apple.com/library/content/qa/qa1398/_index.html

  for its implementation, see
  https://opensource.apple.com/source/xnu/xnu-3248.60.10/libsyscall/wrappers/mach_absolute_time.s

  it's using rdtsc.
* and remove unnecessary headers from ceph_time.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove unnecessary linkage of librt

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link against librt on osx

as osx does not offer librt

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/util: port collect_sys_info() to osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/simple: port to osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/fd: port dump_open_fds() to osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move MemoryModel.cc to $mds_srcs

as it is only used by mds.

Signed-off-by: Kefu Chai <kchai@redhat.com>

global/signal_handler: port get_name_by_pid() to osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for osx using `APPLE` not `DARWIN`

Signed-off-by: Kefu Chai <kchai@redhat.com>

os,common: check __APPLE__ for osx not DARWIN

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/io_priority: include <errno.h> on osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/-Wl,-export-dynamic/$CMAKE_EXE_EXPORTS_C_FLAG/

* s/-Wl,-export-dynamic/$CMAKE_EXE_EXPORTS_C_FLAG/
  this address the issue on osx:

    ld: unknown option: --export-dynamic

  because ld on osx does not support `-export-dynamic`, it supports
  `-export_dynamic` though.

  CMAKE_EXE_EXPORTS_C_FLAG is set to -Wl,-export-dynamic on FreeBSD
  and GNU/Linux.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17718 from kungf/scrub_loadavg

osd: get loadavg per cpu for scrub load threshold check

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17704 from liewegas/wip-osd-deq

osd: minor optimizations for op wq

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #17666 from david-z/wip-objectore-tool

tools/ceph-objectstore-tool: split filestore directories offline to target hash level

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17748 from tchaikov/wip-mon-constness

mon: more constness

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17727 from tchaikov/wip-pybind-cleanup

pybind: remove unused get_ceph_version()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17749 from tchaikov/wip-vstart

vstart.sh: simplify the objectstore related logic

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #17418 from Songweibin/wip-osdmap-full-try

rados: support python API of "set_osdmap_full_try"

Reviewed-by: Pan Liu <wanjun.lp@alibaba-inc.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17295 from david-z/wip-compress-scrub-warning

mon: Compress the warnings of pgs not scrubbed or deep-scrubbed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17732 from Songweibin/wip-fix-comments

doc,os,osdc: drop and modify comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17675 from theanalyst/doc-rgw-reshard

doc: rgw: add a note for resharding in 12.2.1 docs

doc: rgw: add a note for resharding in 12.2.1 docs

Since we're now enabling dynamic resharding, mention  this in release
notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #17701 into master

* refs/remotes/upstream/pull/17701/head:
qa/cephfs: Fix error in test_filtered_df

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17694 into master

* refs/remotes/upstream/pull/17694/head:
qa/cephfs: kill mount if it gets evicted by mds
qa/cephfs: fix test_evict_client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17681 into master

* refs/remotes/upstream/pull/17681/head:
mon: remove mdsmap-derived perf counters

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17676 into master

* refs/remotes/upstream/pull/17676/head:
qa/tasks/cephfs: Whitelist POOL_APP_NOT_ENABLED for test_misc

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17671 into master

* refs/remotes/upstream/pull/17671/head:
messages: Initilization of left memebers

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #17670 into master

* refs/remotes/upstream/pull/17670/head:
messages: Initialization of member variables

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #17652 into master

* refs/remotes/upstream/pull/17652/head:
client: use unsigned trim_caps count

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #17598 into master

* refs/remotes/upstream/pull/17598/head:
messages: Initialization of members MMDSCacheRejion

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #17595 into master

* refs/remotes/upstream/pull/17595/head:
ceph-fuse: should free array using "delete[]"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17542 into master

* refs/remotes/upstream/pull/17542/head:
client: Delete onsafe

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17709 from liewegas/wip-bluestore-repair

ceph-bluestore-tool: better fsck/repair, bluefs-bdev-{expand,sizes}

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17388 from tchaikov/wip-rocksdb

rocksdb: sync with upstream

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17706 from liewegas/wip-test-listsnaps

ceph_test_rados_api_tier: add ListSnap test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17705 from dzafman/wip-21328

osd: Only scan for omap corruption once

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17703 from dzafman/wip-misc

Erasure code read test and code cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17699 from tchaikov/wip-lookout-for-large-elen

msg/msg_types: fix the entity_addr_t's decoder

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17667 from scienceluo/wip-luo-fix-cli-output-branch

pybind/ceph_argparse: fix cli output info

Reviewed-by: Chang Liu <liuchang0812@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17664 from Yan-waller/wip-walle-0912moninjectargs

common: fix daemon abnormal exit at parsing invalid arguments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17655 from optimistyzy/911_2

src/bluestore/NVMEDevice: make all read use aio_submit

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #17611 from chardan/jfw-wip-atomic_t-cleanup

follow-up fixups for atomic_t spinlocks

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17605 from jcsp/wip-21253

mgr: fix py calls for dne service perf counters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17568 from amitkumar50/cov-msg-2

msg: Initializing class members in module msg

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16699 from jcsp/wip-mgr-perf-threshold

mgr: apply a threshold to perf counter prios

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: fix the entity_addr_t's decoder

the daemon is vulnerable to malicious client, which is able to send
large elen, and corrupt the stack, etc.

* throw at seeing corrupted entity_addr_t where its elen exceeds
  the length of sockaddr
* handle the exception thrown when decoding entity_addr_t in messenger
  layer.
* if a malicious client manages to send a corrutped entity_addr_t to
  daemon, daemon will crash because decode fails and the exception is
  not handled. it's better than continuing working with the bogus
  message.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build portable rocksdb

* so rocksdb is not built with -march=native, because
  - this is not portable
  - -march=native is not supported by older version of GCC on aarch64.
* and drop the sse42 specific settings for rocksdb. since rocksdb uses
  "target" attribute to build sse42 optimized crc32, as long as the
  compiler on building host is able to emit sse42 instructions.
  see https://github.com/facebook/rocksdb/pull/2807

Fixes: http://tracker.ceph.com/issues/20529
Signed-off-by: Kefu Chai <kchai@redhat.com>

kv/RocksDBStore: update to accomodate the change in rocksdb 5.6.0

per rocksdb's HISTORY.md, it

> replace global variable `PerfContext perf_context` with `PerfContext*
> get_perf_context()`.

hence the change.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rocksdb: sync with upstream

to pickup the fix of https://github.com/facebook/rocksdb/pull/2692.

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: simplify the objectstore related logic

also avoid overriding --memstore specified by command line with
the default "bluestore"

Reported-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17361 from kungf/ulimit_setup

run-make-check.sh: run ulimit without sudo

Reviewed-by: Dan Mick <dmick@redhat.com>

run-make-check.sh: run ulimit without sudo

ulimit is a shell builtin like cd, not a separate program.
sudo looks for a binary to run, but there is no ulimit binary,
then you will get an error message like:
          "sudo: ulimit: command not found"
ulimit can only be run with root if you want raise the ulimit -n

Change-Id: I59ab09240cb15cabe5b0a7342c3f562cd50c75e9
Signed-off-by: yang.wang <yang.wang@easystack.cn>

Merge pull request #17713 from tchaikov/wip-ceph.in-python

ceph.in: execv using the same python

Merge pull request #17583 from neha-ojha/wip-cbt-teuthology-integration

qa: add cbt task for performance testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17680 from ceph/wip-smoke-whitelist

qa/tests: Various whitelists for smoke suite

Various whitelists for tests to pass

See tracker http://tracker.ceph.com/issues/21376 for details

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

ceph-bluestore-tool: better default logging; --log-file and --log-level options

- keep derr going to stderr (so we see 'fsck error' messages)
- hide the rest
- add friendly --log-file option

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: fix error prefixes

__func__ is now _fsck instead of fsck

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluestore-tool: add 'bluefs-bdev-expand' to expand wal or db usage

If you are using the wal or db devices, this will expand bluefs's usage to
include the entire block device.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluestore-tool: add 'bluefs-bdev-sizes' command

Show bdev sizes vs owned extents.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluestore-tool: factor out bluefs mount

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: allow repair invocation

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add repair interface

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17733 from yaozongyou/fix-doc-boolean-type-zero-to-false

doc: change boolean option default value from zero to false.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17610 from liewegas/wip-bluestore-fm-mas

os/bluestore: set bitmap freelist resolution to min_alloc_size

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17617 from liewegas/wip-drop-disk-log

mon/PGMap: drop DISK LOG column

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: change boolean option default value from zero to false.

Signed-off-by: Yao Zongyou yaozongyou@vip.qq.com

Merge pull request #17628 from majianpeng/bluestore-narrow-deferred-lock

os/bluestore/BlueStore: narrow deferred_lock in _deferred_submit_unlock.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc,os,osdc: drop and modify some comments

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #17627 from majianpeng/bluefs-exce-debug-code

os/bluestore/BlueFS: Don't call debug related code under any condition.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17371 from xiexingguo/wip-per-pool-full-control

mon, osd: per pool space-full flag support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17368 from xiexingguo/wip-nice-data

mon/PGMap: nice numbers for 'data' section of 'ceph df' command

Reviewed-by: Kefu Chai <kchai@redhat.com>