qa/suites/rados/singleton/all/recovery-preemption: add test

This mirrors what I was testing locally.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: handle racy preemption

If we finish recovery/backfill and go active, but also get
preempted at the same time, we can ignore the event.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: allow local recovery reservations to be preempted

If a PG has a higher recovery priority and a lower-priority item is in
progress, allow it to be preempted.  This triggers the RecoveryCancel
or BackfillCancel event with a 0 delay, which means it will immediately
re-request a reservation (and presumably wait).

Signed-off-by: Sage Weil <sage@redhat.com>

common/AsyncReserver: support preemption

If an (optional) preemption context is provided, use that to preempt
and existing reservation and grant a higher-priority one.

Signed-off-by: Sage Weil <sage@redhat.com>

common/AsyncReserver: get a cct

Signed-off-by: Sage Weil <sage@redhat.com>

osd: PG_STATE_BACKFILL -> PG_STATE_BACKFILLING

Match user-facing string

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: make BACKFILL <-> "backfilling" for parser

We render BACKFILL as "backfilling"; make sure parse works that
way too.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove weird BACKFILL state hack

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: Cancel{Recovery,Backfill} -> Defer{Recovery,Backfill}

"Defer" is more accurate here; we aren't canceling anything, just
rescheduling the work.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: specify delay in Cancel{Recovery,Backfill}

For now it is always the retry interval, but later perhaps not!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make some trivial events TrivialEvent

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: set {backfill,recovery}_wait when canceling backfill/recovery

The only caller currently is when we get as far as we can with backfill
or recovery but still have unfound objects.  In this case, go back into
the *_wait state instead of appearing as though we are still doing
something.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17747 from tchaikov/wip-qa

qa/standalone: respect $TEMPDIR

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #17828 from jdurgin/wip-21428

osd/PrimaryLogPG: request osdmap update in the right block

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: request osdmap update in the right block

Fixes: http://tracker.ceph.com/issues/21428
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17785 from dzafman/wip-add-repair

test: Fix ceph-objectstore-tool usage check

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17138 from Liuchang0812/wip-updata-osd-hostname-in-metaupdater

mgr: preventing blank hostname in DaemonState

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17738 from jtlayton/wip-jlayton-20988

common: lockdep fixes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17753 from tchaikov/wip-osd-cleanup

osd: cleanups

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17787 from tchaikov/wip-os-release-wo-version_id

common/util: do not print error if VERSION_ID is missing

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #17788 from tchaikov/wip-cmake-rocksdb-tcmalloc

cmake: identify the possible incompatibility of rocksdb and tcmalloc

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

cmake: error out if rocksdb is incompatible w/ tcmalloc

the commit d406f228 in gperf implements a c11 feature used by a
recent change in rocksdb: 16e03882, which uses aligned_alloc().
and 16e03882 in rocksdb was merged after v5.7 was tagged, while
16e03882 in gperf was merged after v2.6.1 was tagged.

because aligned_alloc() is not implemented by tcmalloc until the
not-yet-released 2.6.2, if we call aligned_alloc() in an application
linked against tcmalloc, what gets called will be the glibc's
aligned_alloc(). but if we free() the memory chunk allocated by
aligned_alloc(), the tcmalloc's implementation kicks in, then
InvalidFree() is called, because the memory chunk being freed was
allocated by tcmalloc. in short, "mixing allocators", quote from
Dan Mick.

in rocksdb, aligned_alloc() is used if _ISOC11_SOURCE is defined, this
makes sense, because aligned_alloc() is a C11 function. we could avoid
using it by not defining _ISOC11_SOURCE. but as long as _GNU_SOURCE is
defined, glibc defines _ISOC11_SOURCE. and libstdc++ requires
_GNU_SOURCE, because it uses a fair amount of GNU extensions.

Fixes: http://tracker.ceph.com/issues/21422
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17795 from jdurgin/wip-21428

osd: request new map from PG when needed

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

osd: request new map from PG when needed

The fast dispatch refactor in 3cc48278bf0ee5c9535d04b60a661f988c50063b
eliminated the osdmap subscription in the ms_fast_dispatch path, which
meant ops could reach a PG without having the latest map. In a cluster
with few osdmap updates, where the monitor fails to send a new map to
an osd (it tries one random osd), this can result in indefinitely
blocked requests.

Fix this by adding an OSDService mechanism for scheduling a new osdmap
subscription request.

Fixes: http://tracker.ceph.com/issues/21428
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17607 from liewegas/wip-post-luminous-more-mon

mon: post-luminous cleanup (part 3 of ?)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16913 from liewegas/wip-20944

os/filestore: fix device/partition metadata detection

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17427 from liewegas/wip-pg-num-limits

mon/OSDMonitor: implement cluster pg limit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17760 from liewegas/wip-21410-b

Revert "osd/OSDMap: allow bidirectional swap of pg-upmap-items"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15369 from rzarzynski/wip-rgw-custom-errmsgs-2

rgw: improve handling of Swift's error messages and limits

Merge pull request #12704 from rzarzynski/wip-rgw-per-storage-policy-stats

rgw: add support for Swift's per storage policy statistics

Merge pull request #17786 from neha-ojha/wip-cbt-teuthology-integration-cntd

qa/suites/rados/perf: add optimized settings

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: rename the configurables for metadata limits to start with rgw_.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: calculate and print Swift's X-Account-Storage-Policy-* headers.

Fixes: http://tracker.ceph.com/issues/17932
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: bucket linking stores also the info about a placement rule.

Fixes: http://tracker.ceph.com/issues/17932
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: convey placement rule in RGWBucketEnt and cls_user_bucket_entry.

Fixes: http://tracker.ceph.com/issues/17932
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: clean-up around and implement the move semantics in RGWBucketEnt.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #17508 from liewegas/wip-crushrule-convert

mon: more aggressively convert crush rulesets -> distinct rules

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17802 from ygtzf/option-default-value-error

doc: update default value of option mon_sync_timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

doc: update default value of option mon_sync_timeout

Signed-off-by: Yao Guotao <yaoguot@gmail.com>

cmake: requires rockdb version 5.8

we've updated the rockdb wrapper on ceph side to be compatible with
the latest version of rocksdb upstream. so ceph is not compatible with
older version of rocksdb.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: detect rocksdb's version

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: detect gperf-tools' version

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: extract BuildRocksDB into BuildRocksDB.cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: enable find_package(perftools) to check non-default dir

so it checks $ENV{GPERF_ROOT} first.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/perf: add optimized settings

Signed-off-by: Neha Ojha <nojha@redhat.com>

test: Fix ceph-objectstore-tool usage check

Caused by: c7b7a1f04f78fa62890c567d0ca53874c8d75eb7

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #15273 from rzarzynski/wip-rgw-18977

rgw: list_objects() honors end_marker regardless of namespace.

Merge pull request #17320 from rzarzynski/wip-rgw-21148

rgw: add support for Swift's reversed account listings

Merge pull request #17777 from cbodley/wip-qa-hadoop

qa/rgw: hadoop-s3a suite targets centos_latest

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/rgw: hadoop suite targets centos_latest

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: use overrides for hadoop version

instead of having two separate tasks, add v27.yaml and v28.yaml with
overrides for hadoop_version

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17776 from smithfarm/wip-rh-74

tests: CentOS 7.4 is now the latest

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

tests: CentOS 7.4 is now the latest

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #17372 from ceph/wip-doc-cephfs-pools

doc: add example of setting pool in cephfs layout

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17318 from jcsp/wip-20899

mgr/prometheus: Fix for MDS metrics

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16762 from jcsp/wip-dashboard-trim

mgr: remove unused static files from dashboard module

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: re-arrange static files

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: don't include remote js/css

This will help anyone running away from the internet.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: remove un-minified datatables source

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: remove non-minified bootstrap

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: remove unneeded bits of AdminLTE

AdminLTE is delivered in a handy all-in format,
but we don't need all this stuff at runtime.

Signed-off-by: John Spray <john.spray@redhat.com>

common/util: do not print error if VERSION_ID is missing

per os-release(5), VERSION_ID is optional.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17737 from ivancich/wip-pull-updated-dmclock

osd: update dmclock library w git subtree pull

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17762 from yaozongyou/fix-typo-in-php-rst

doc: fix typo in php.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17763 from xiexingguo/wip-21409

qa: fix pool-quota related tests

Reviewed-by: Sage Weil <sage@redhat.com>

client: test shutdown race

Spawn threads that bring up a bunch of ceph_mounts with individual
CephContext objects, and then tear them down in parallel.

Tracker: http://tracker.ceph.com/issues/20988
Signed-off-by: Jeff Layton <jlayton@redhat.com>

lockdep: fix Mutex tests to disable lockdep properly

...and make g_lockdep a bool.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

lockdep: fix races with concurrent lockdep teardown

If the cct is unregistered while other threads are flogging mutexes,
then we can hit all sorts of bugs. Ensure that we handle that
situation sanely, by checking that g_lockdep is still set after
we take the lockdep_mutex.

Also, remove an assertion from lockdep_unregister, and just turn it into
an immediate return. It's possible to have a call to
lockdep_unregister_ceph_context, and then a call to
lockdep_register_ceph_context while a mutex is being held by another
task.

In that case, it's possible the lock does not exist in the map
when we go to unregister it. That's not a bug though, just a natural
consequence of that series of actions.

Tracker: http://tracker.ceph.com/issues/20988
Signed-off-by: Jeff Layton <jlayton@redhat.com>

lockdep: don't take lockdep_mutex twice for new lock registrations

We can do it under the same mutex, which should be more efficient.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: fix signed/unsigned comparison compiler warning

The build says:

src/client/Client.cc: In member function ‘void Client::trim_caps(MetaSession*, int)’:
src/client/Client.cc:4121:22: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (s->caps.size() > max)
       ~~~~~~~~~~~~~~~^~~~~

Signed-off-by: Jeff Layton <jlayton@redhat.com>

doc: fix typo in php.rst

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

qa: fix pool-quota related tests

https://github.com/ceph/ceph/pull/17371 introduces support of
per-pool space-full flag, which turns out to set both
full and full_no_quota flags now if a pool is currently running out
of quota.

Actually this test is fragile as long as we keep appending new flags
at pool granularity, but let's not bother with that complexity now.

Fixes: http://tracker.ceph.com/issues/21409
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17615 from tchaikov/wip-osx

cmake,common: changes to port part of ceph to osx

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

client: use st_mtimespec instea of st_mtime on osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: fix the dencoder of entity_addr_t

improve the interoperability between freebsd/osx and GNU/Linux, because
the their layouts of sockaddr_storage are different, and we use the
one of linux as the wire format. so need to convert it on freebsd/osx
side.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17698 from liewegas/wip-bluestore-big-allocations

os/bluestore: allocate entire write in one go

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

crc32c: pass integer param by uint64_t

the assembly is expecting 64bits input, so make this explicit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crc32c: mach-o 64 follows the same calling convention as elf64 does

so reuse the register-parameter/variable mappings

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/encoding/readable.sh: use sysctl -n hw.ncpu instead of nproc on osx also

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for fdatasync() on non osx

on osx, fdatasync() is a syscall without userspace wrapper.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: use the best-effort if procfs is not available

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: allows undefined symbol in plugin on osx

clang on osx erros out when linking a dynamic library if any symbols is
missing by default. so disable this behavior.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: disable plugins using isa-l on osx

clang on osx emits functions with leading underscore, but the isa-l
assembly's functions have no leading underscore. we could label the
function declaration like `int foo asm("foo")` to remove the leading
underscore, but isa-l is a git submodule, let do this later. and in the
meanwhile, disable this plugin on osx.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crc32: label assembler functions without leading underscore

clang onder osx adds leading undescore to the function names to be
ABI compatible. but the assembly code does not do so. so we need to
control the name using gcc/clang extension. see
https://gcc.gnu.org/onlinedocs/gcc-4.4.0/gcc/Asm-Labels.html#Asm-Labels

Signed-off-by: Kefu Chai <kchai@redhat.com>

compat: consolidate definitions of osx and freebsd

on osx, ENODATA = 96, so we need to fix it. also define
CLOCK_MONOTONIC_COARSE and CLOCK_REALTIME_COARSE for osx, ceph_time.h
defines this also, but i don't want to include compat.h in ceph_time.h
at this moment.

and silence the warning of
 #warning ENODATA already defined to a value different from 87 (ENOATRR), refining to fix
because it is fired everywhere on osx when "compat.h" is included.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: fix the denc of sockaddr_storage on freebsd/osx

the layout of sockaddr_in and sockaddr_in6 are different on
GNU/Linux and FreeBSD/OSX:
 - on GNU/Linux, sockaddr does not have sa_len,
 - on GNU/Linux, sockaddr* use a 16 bit integer for sa_family, but
   on FreeBSD, a 32bit integer is used.
so we need to be more care when memcpy() between sockaddr_storage()
and ceph_sockaddr_storage().

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: consolidate sockaddr_storage dencoders

utilize ceph_sockaddr_storage on FreeBSD and osx to perform the
dencoding, to improve the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not pass --exclude-libs on osx

it is not supported by osx linker

Signed-off-by: Kefu Chai <kchai@redhat.com>

crc32c: only add qualifiers for elf64 format

they are not understood by non-elf formats

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: compile Mach-O format instead of ELF on OSX

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against libresolv on OSX

Signed-off-by: Kefu Chai <kchai@redhat.com>

time: use mach_absolute_time() for monotonic time

* use mach_absolute_time() for monotonic time
  mach_absolute_time() is faster and monotonic, see
  https://developer.apple.com/library/content/qa/qa1398/_index.html

  for its implementation, see
  https://opensource.apple.com/source/xnu/xnu-3248.60.10/libsyscall/wrappers/mach_absolute_time.s

  it's using rdtsc.
* and remove unnecessary headers from ceph_time.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove unnecessary linkage of librt

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link against librt on osx

as osx does not offer librt

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/util: port collect_sys_info() to osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/simple: port to osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/fd: port dump_open_fds() to osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move MemoryModel.cc to $mds_srcs

as it is only used by mds.

Signed-off-by: Kefu Chai <kchai@redhat.com>

global/signal_handler: port get_name_by_pid() to osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for osx using `APPLE` not `DARWIN`

Signed-off-by: Kefu Chai <kchai@redhat.com>