osd: For recovery get all possible shards to read on errors

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Allow recovery to send additional reads

For now it doesn't include non-acting OSDs
Added test for this case

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Allow modified options to existing setup functions

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use feature to get last array element

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #17795 from jdurgin/wip-21428

osd: request new map from PG when needed

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

osd: request new map from PG when needed

The fast dispatch refactor in 3cc48278bf0ee5c9535d04b60a661f988c50063b
eliminated the osdmap subscription in the ms_fast_dispatch path, which
meant ops could reach a PG without having the latest map. In a cluster
with few osdmap updates, where the monitor fails to send a new map to
an osd (it tries one random osd), this can result in indefinitely
blocked requests.

Fix this by adding an OSDService mechanism for scheduling a new osdmap
subscription request.

Fixes: http://tracker.ceph.com/issues/21428
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17607 from liewegas/wip-post-luminous-more-mon

mon: post-luminous cleanup (part 3 of ?)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16913 from liewegas/wip-20944

os/filestore: fix device/partition metadata detection

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17427 from liewegas/wip-pg-num-limits

mon/OSDMonitor: implement cluster pg limit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17760 from liewegas/wip-21410-b

Revert "osd/OSDMap: allow bidirectional swap of pg-upmap-items"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15369 from rzarzynski/wip-rgw-custom-errmsgs-2

rgw: improve handling of Swift's error messages and limits

Merge pull request #12704 from rzarzynski/wip-rgw-per-storage-policy-stats

rgw: add support for Swift's per storage policy statistics

Merge pull request #17786 from neha-ojha/wip-cbt-teuthology-integration-cntd

qa/suites/rados/perf: add optimized settings

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: rename the configurables for metadata limits to start with rgw_.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: calculate and print Swift's X-Account-Storage-Policy-* headers.

Fixes: http://tracker.ceph.com/issues/17932
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: bucket linking stores also the info about a placement rule.

Fixes: http://tracker.ceph.com/issues/17932
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: convey placement rule in RGWBucketEnt and cls_user_bucket_entry.

Fixes: http://tracker.ceph.com/issues/17932
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: clean-up around and implement the move semantics in RGWBucketEnt.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #17508 from liewegas/wip-crushrule-convert

mon: more aggressively convert crush rulesets -> distinct rules

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17802 from ygtzf/option-default-value-error

doc: update default value of option mon_sync_timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

doc: update default value of option mon_sync_timeout

Signed-off-by: Yao Guotao <yaoguot@gmail.com>

qa/suites/rados/perf: add optimized settings

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #15273 from rzarzynski/wip-rgw-18977

rgw: list_objects() honors end_marker regardless of namespace.

Merge pull request #17320 from rzarzynski/wip-rgw-21148

rgw: add support for Swift's reversed account listings

Merge pull request #17777 from cbodley/wip-qa-hadoop

qa/rgw: hadoop-s3a suite targets centos_latest

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/rgw: hadoop suite targets centos_latest

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: use overrides for hadoop version

instead of having two separate tasks, add v27.yaml and v28.yaml with
overrides for hadoop_version

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17776 from smithfarm/wip-rh-74

tests: CentOS 7.4 is now the latest

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

tests: CentOS 7.4 is now the latest

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #17372 from ceph/wip-doc-cephfs-pools

doc: add example of setting pool in cephfs layout

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17318 from jcsp/wip-20899

mgr/prometheus: Fix for MDS metrics

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16762 from jcsp/wip-dashboard-trim

mgr: remove unused static files from dashboard module

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: re-arrange static files

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: don't include remote js/css

This will help anyone running away from the internet.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: remove un-minified datatables source

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: remove non-minified bootstrap

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: remove unneeded bits of AdminLTE

AdminLTE is delivered in a handy all-in format,
but we don't need all this stuff at runtime.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #17737 from ivancich/wip-pull-updated-dmclock

osd: update dmclock library w git subtree pull

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17762 from yaozongyou/fix-typo-in-php-rst

doc: fix typo in php.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17763 from xiexingguo/wip-21409

qa: fix pool-quota related tests

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fix typo in php.rst

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

qa: fix pool-quota related tests

https://github.com/ceph/ceph/pull/17371 introduces support of
per-pool space-full flag, which turns out to set both
full and full_no_quota flags now if a pool is currently running out
of quota.

Actually this test is fragile as long as we keep appending new flags
at pool granularity, but let's not bother with that complexity now.

Fixes: http://tracker.ceph.com/issues/21409
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17615 from tchaikov/wip-osx

cmake,common: changes to port part of ceph to osx

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

client: use st_mtimespec instea of st_mtime on osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: fix the dencoder of entity_addr_t

improve the interoperability between freebsd/osx and GNU/Linux, because
the their layouts of sockaddr_storage are different, and we use the
one of linux as the wire format. so need to convert it on freebsd/osx
side.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17698 from liewegas/wip-bluestore-big-allocations

os/bluestore: allocate entire write in one go

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

crc32c: pass integer param by uint64_t

the assembly is expecting 64bits input, so make this explicit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crc32c: mach-o 64 follows the same calling convention as elf64 does

so reuse the register-parameter/variable mappings

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/encoding/readable.sh: use sysctl -n hw.ncpu instead of nproc on osx also

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for fdatasync() on non osx

on osx, fdatasync() is a syscall without userspace wrapper.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: use the best-effort if procfs is not available

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: allows undefined symbol in plugin on osx

clang on osx erros out when linking a dynamic library if any symbols is
missing by default. so disable this behavior.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: disable plugins using isa-l on osx

clang on osx emits functions with leading underscore, but the isa-l
assembly's functions have no leading underscore. we could label the
function declaration like `int foo asm("foo")` to remove the leading
underscore, but isa-l is a git submodule, let do this later. and in the
meanwhile, disable this plugin on osx.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crc32: label assembler functions without leading underscore

clang onder osx adds leading undescore to the function names to be
ABI compatible. but the assembly code does not do so. so we need to
control the name using gcc/clang extension. see
https://gcc.gnu.org/onlinedocs/gcc-4.4.0/gcc/Asm-Labels.html#Asm-Labels

Signed-off-by: Kefu Chai <kchai@redhat.com>

compat: consolidate definitions of osx and freebsd

on osx, ENODATA = 96, so we need to fix it. also define
CLOCK_MONOTONIC_COARSE and CLOCK_REALTIME_COARSE for osx, ceph_time.h
defines this also, but i don't want to include compat.h in ceph_time.h
at this moment.

and silence the warning of
 #warning ENODATA already defined to a value different from 87 (ENOATRR), refining to fix
because it is fired everywhere on osx when "compat.h" is included.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: fix the denc of sockaddr_storage on freebsd/osx

the layout of sockaddr_in and sockaddr_in6 are different on
GNU/Linux and FreeBSD/OSX:
 - on GNU/Linux, sockaddr does not have sa_len,
 - on GNU/Linux, sockaddr* use a 16 bit integer for sa_family, but
   on FreeBSD, a 32bit integer is used.
so we need to be more care when memcpy() between sockaddr_storage()
and ceph_sockaddr_storage().

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: consolidate sockaddr_storage dencoders

utilize ceph_sockaddr_storage on FreeBSD and osx to perform the
dencoding, to improve the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not pass --exclude-libs on osx

it is not supported by osx linker

Signed-off-by: Kefu Chai <kchai@redhat.com>

crc32c: only add qualifiers for elf64 format

they are not understood by non-elf formats

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: compile Mach-O format instead of ELF on OSX

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against libresolv on OSX

Signed-off-by: Kefu Chai <kchai@redhat.com>

time: use mach_absolute_time() for monotonic time

* use mach_absolute_time() for monotonic time
  mach_absolute_time() is faster and monotonic, see
  https://developer.apple.com/library/content/qa/qa1398/_index.html

  for its implementation, see
  https://opensource.apple.com/source/xnu/xnu-3248.60.10/libsyscall/wrappers/mach_absolute_time.s

  it's using rdtsc.
* and remove unnecessary headers from ceph_time.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove unnecessary linkage of librt

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link against librt on osx

as osx does not offer librt

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/util: port collect_sys_info() to osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/simple: port to osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/fd: port dump_open_fds() to osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move MemoryModel.cc to $mds_srcs

as it is only used by mds.

Signed-off-by: Kefu Chai <kchai@redhat.com>

global/signal_handler: port get_name_by_pid() to osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for osx using `APPLE` not `DARWIN`

Signed-off-by: Kefu Chai <kchai@redhat.com>

os,common: check __APPLE__ for osx not DARWIN

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/io_priority: include <errno.h> on osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/-Wl,-export-dynamic/$CMAKE_EXE_EXPORTS_C_FLAG/

* s/-Wl,-export-dynamic/$CMAKE_EXE_EXPORTS_C_FLAG/
  this address the issue on osx:

    ld: unknown option: --export-dynamic

  because ld on osx does not support `-export-dynamic`, it supports
  `-export_dynamic` though.

  CMAKE_EXE_EXPORTS_C_FLAG is set to -Wl,-export-dynamic on FreeBSD
  and GNU/Linux.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17718 from kungf/scrub_loadavg

osd: get loadavg per cpu for scrub load threshold check

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17704 from liewegas/wip-osd-deq

osd: minor optimizations for op wq

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #17666 from david-z/wip-objectore-tool

tools/ceph-objectstore-tool: split filestore directories offline to target hash level

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17748 from tchaikov/wip-mon-constness

mon: more constness

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17727 from tchaikov/wip-pybind-cleanup

pybind: remove unused get_ceph_version()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17749 from tchaikov/wip-vstart

vstart.sh: simplify the objectstore related logic

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #17418 from Songweibin/wip-osdmap-full-try

rados: support python API of "set_osdmap_full_try"

Reviewed-by: Pan Liu <wanjun.lp@alibaba-inc.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17295 from david-z/wip-compress-scrub-warning

mon: Compress the warnings of pgs not scrubbed or deep-scrubbed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17732 from Songweibin/wip-fix-comments

doc,os,osdc: drop and modify comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17675 from theanalyst/doc-rgw-reshard

doc: rgw: add a note for resharding in 12.2.1 docs

doc: rgw: add a note for resharding in 12.2.1 docs

Since we're now enabling dynamic resharding, mention  this in release
notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #17701 into master

* refs/remotes/upstream/pull/17701/head:
qa/cephfs: Fix error in test_filtered_df

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17694 into master

* refs/remotes/upstream/pull/17694/head:
qa/cephfs: kill mount if it gets evicted by mds
qa/cephfs: fix test_evict_client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17681 into master

* refs/remotes/upstream/pull/17681/head:
mon: remove mdsmap-derived perf counters

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17676 into master

* refs/remotes/upstream/pull/17676/head:
qa/tasks/cephfs: Whitelist POOL_APP_NOT_ENABLED for test_misc

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17671 into master

* refs/remotes/upstream/pull/17671/head:
messages: Initilization of left memebers

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #17670 into master

* refs/remotes/upstream/pull/17670/head:
messages: Initialization of member variables

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #17652 into master

* refs/remotes/upstream/pull/17652/head:
client: use unsigned trim_caps count

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #17598 into master

* refs/remotes/upstream/pull/17598/head:
messages: Initialization of members MMDSCacheRejion

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #17595 into master

* refs/remotes/upstream/pull/17595/head:
ceph-fuse: should free array using "delete[]"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17542 into master

* refs/remotes/upstream/pull/17542/head:
client: Delete onsafe

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Revert "osd/OSDMap: allow bidirectional swap of pg-upmap-items"

This reverts commit 09af9b8afb40cc8aa629501582a75e03edf0bf2e.

We need to prevent duplicates in the final result.  For example, we
can currently take
 [1,2,3] and apply [(1,2)] and get [2,2,3]
or
 [1,2,3] and apply [(3,2)] and get [1,2,2]

The rest of the system is not prepared to handle duplicates in the
result set like this.

The reverted commit was intended to allow

 [1,2,3] and [(1,2),(2,1)] to get [2,1,3]

to reorder primaries.  First, this bidirectional swap is hard to implement
in a way that also prevents dups.  For example,
 [1,2,3] and [(1,4),(2,3),(3,4)] would give [4,3,4]
but would we just drop the last step we'd have [4,3,3] which
is also invalid, etc.  Simpler to just not handle bidirectional
swaps.  In practice, they are not needed: if you just want to choose
a different primary then use primary_affinity, or pg_upmap
(not pg_upmap_items).

Fixes: http://tracker.ceph.com/issues/21410
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17709 from liewegas/wip-bluestore-repair

ceph-bluestore-tool: better fsck/repair, bluefs-bdev-{expand,sizes}

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: drop old upgrade guards; make upgrades stop at O

We may not end up doing this but it keeps a template for how the
guard worked before in the code for future reference.

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: define SERVER_O

Signed-off-by: Sage Weil <sage@redhat.com>

mon: app labels unconditional

Signed-off-by: Sage Weil <sage@redhat.com>