osd: Handle objects on non-actingbackfill nodes during recovery

With this fix we can see more misplaced and no degraded when
doing recovery to a new up/acting set with node(s) containing
objects that are not longer part of up/acting.

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Adjust misplaced calculation

When there are more backfill ndoes then misplaced nodes the
object calculation would cause misplace to drop too quickly.
Assuming that all backfill nodes are at approximate the same
place, we compute backfill per node and only adjust by
the number of misplaced nodes.

Fixes: http://tracker.ceph.com/issues/21898
Signed-off-by: David Zafman <dzafman@redhat.com>

osd/PG: _update_calc_stats: backfill targets do not always affect degraded

Our progress on a backfill target should only count toward degraded if
the target is needed in order for us to reach the target pool size.  If
we have remapped other complete copies, we can backfill and not be degraded
at all.  And we may be some combination of the two if there are multiple
backfill targets; use the target(s) with the most objects.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: Re-organize _update_calc_stats() to make the code clearer

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix object_copies calculation so degraded is accurate

Fixes: http://tracker.ceph.com/issues/21803
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #18229 from liewegas/wip-21753

ceph-objectstore-tool: fix import of post-split pg from pre-split ancestor

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #18224 from liewegas/wip-pglog-cf

os/bluestore: dynamic CF configuration; put pglog omap in separate CF

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #18186 from xiexingguo/wip-miss-loc

osd/PG: reduce some overhead on operating MissingLoc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16030 from myoungwon/wip-revamp-opwq

osd: generalize queueing and lock interface for OpWq

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17541 from mikulely/log-time-msecs

rgw: fix total_time to msec as per AWS S3

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18214 from shashalu/fix-null-data-mtime

rgw: don't change rados object's mtime when update olh

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18245 from yaozongyou/wip-radosgw-admin-bi-fix

rgw: udpate radosgw-admin usage with bi purge

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18248 from ivancich/wip-rgw-unify-bi-hash-calc

rgw: consolidate code that implements hashing algorithms

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18272 from cbodley/wip-compressor-random

compressor: use generate_random_number() for type="random"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18240 from tchaikov/wip-21716

test: fix misc fiemap testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18221 from tchaikov/wip-kvstore-tool-cleanup

ceph-kvstore-tool: use unique_ptr<> to manage the lifecycle of bluestore

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #18285 from tchaikov/wip-cmake

cmake: bump up required cmake version to 2.8.12

Merge pull request #18273 from cbodley/wip-21775

rgw: 'zone placement' commands validate compression type

cmake: fix gperftools version detection

there is chance that its "patch" version is empty. in that case, do not
include it in its version string. otherwise, we will have something
like:

Found gperftools:
GPERFTOOLS_TCMALLOC_LIBRARY;GPERFTOOLS_TCMALLOC_MINIMAL_LIBRARY;GPERFTOOLS_PROFILER_LIBRARY
(found version "2.4.#define TC_VERSION_PATCH  """)

when running cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18286 from tchaikov/wip-kill-bluestore-warning

os/bluestore: silence -Wreturn-type warning

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18277 from dillaman/wip-librbd-unused-variables

librbd: remove unused variables from ReadResult refactor

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

cmake: use CXX_VISIBILITY_PRESET and VISIBILITY_INLINES_HIDDEN

instead of using "-fvisibility=hidden -fvisibility-inlines-hidden", more
cmake-ish this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/LINK_PRIVATE/PRIVATE/

as 2.8.12 is required not, no need to be compatible with 2.8.11 anymore.

this reverts 457f023.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: refactor -fsanitize=... options

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: silence -Wreturn-type warning

this silences following warning:

ceph/src/os/bluestore/bluestore_types.h: In member function ‘bool bluestore_blob_t::_validate_range(uint64_t, uint64_t, bool) const’:
ceph/src/os/bluestore/bluestore_types.h:664:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use CMAKE_POSITION_INDEPENDENT_CODE not -fPIC

we use the global setting of CMAKE_POSITION_INDEPENDENT_CODE for
defining this flag.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: bump up the required version to 2.8.12

as both trusty and centos 7.3 offer 2.8.12

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17793 from amitkumar50/cov-misc-6

librbd,os: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17825 from amitkumar50/cov-misc-9

librbd,test: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17972 from cbodley/wip-21401

auth: keep /dev/urandom open for get_random_bytes

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18204 from ownedu/wip-fix-async-rdma-coredump

msg/async/rdma: fix a coredump introduced by PR #18053

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18217 from liewegas/wip-pg-clang

osd/PG: fix clang build vs private state events

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18259 from xiexingguo/wip-blue-reclaim

os/bluestore/BlueFS: sanity check that alloc->allocate() won't return 0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18262 from tchaikov/wip-rocksdb

rocksdb: sync with upstream

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

os/bluestore: dynamically configure CFs and options via bluestore_rocksdb_cfs

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: put pglog omap in a separate CF

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: remove unused variables from ReadResult refactor

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18210 from shangfufei/wip-async-fix

msg/async: fix bug of data type conversion when uint64_t -> int -> uint64_t

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Amit Kumar amitkuma@redhat.com

Merge pull request #18246 from liewegas/wip-seesaw-warning

qa/suites/rados/singleton/all/mon-seesaw: whitelist MON_DOWN

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18270 from dillaman/wip-21391

cls/journal: possible infinite loop within tag_list class method

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

compressor: use generate_random_number for type="random"

use an existing thread-local random engine instead of constructing,
seeding, and locking a separate engine

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: 'zone placement' commands validate compression type

Fixes: http://tracker.ceph.com/issues/21775
Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/journal: fixed possible infinite loop which could kill the OSD

Fixes: http://tracker.ceph.com/issues/21771
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: ceph_test_cls_journal was dropped when converting to cmake

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18258 from shinobu-x/sk-osdmonitor_move_bufferlist_before_use

mon/OSDMonitor: cleanup: move bufferlist before use

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #18235 from tchaikov/wip-qa-backfilling

qa: s/backfill/backfilling/

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18222 from adamemerson/wip-don't-fix-what-ain't-broke

config: Remove _get_val

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rocksdb: sync with upstream

to pickup two changes

- prefetched skiplist
- enable SSE42 CRC32 on EL7

Fixes: http://tracker.ceph.com/issues/21603
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore/BlueFS: sanity check that alloc->allocate() won't return 0

If alloc[id]->allocate() returns 0, no space is actually reclaimed.
But BlueStore treats returning 0 as success and will keep reclaiming
space from BlueFS until hit the given threshold, which turns out to
be deadloop-prone.

For now, Bitmap and Stupid never return 0-length allocated space
(which is instead converted to -ENOSPC to suggest an error to caller),
so sanity checking against result code 0 will suffice.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: Move bufferlist before use

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

osd: fix reference a deleted item

move() delete to_process' entries. This commit will prevent
referencing a deleted item

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #17824 from shashalu/fix-lc-expiration

rgw:lc fix expiration time

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #18102 from MI-OSiRIS/mgr_influx_dbcheck

Modify mgr-influx module database check to not require admin privileges

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18104 from linuxbox2/rgw-rel-lock-earlyx

rgw: release cls lock if taken in RGWCompleteMultipart

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18175 from yaozongyou/fix-radosgw-admin-command-arg-bug

rgw: fix command argument error for radosgw-admin

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18193 from cbodley/wip-rgw-pool-application-jewel

qa/rgw: ignore errors from 'pool application enable'

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #18225 from adamemerson/wip-retract-assertions

rgw: Remove assertions in IAM Policy

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

rgw: udpate radosgw-admin usage with bi purge.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #17420 from tchaikov/wip-19705

arch/arm: set ceph_arch_aarch64_crc32 only if the build host supports crc32cx

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18242 from tchaikov/wip-21758

crushtool: print error message to stderr not dout(1)

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton/all/mon-seesaw: whitelist MON_DOWN

Mgr can get marked down when mon weirdness is happening.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18137 from cbodley/wip-21685

rgw: RGWUser::init no longer overwrites user_id

Merge pull request #18233 from gmayyyha/ceph-fs-status-21752

mgr/status: fix ceph fs status returns error

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crushtool: print error message to stderr not dout(1)

in hope to fix the mysterious test failure where cli/crushtool/build.t
prints nothing when error message is expected.

Fixes: http://tracker.ceph.com/issues/21758
Signed-off-by: Kefu Chai <kchai@redhat.com>

test: fix misc fiemap testing

1) Different filesystem will have different behavior to
allocate extents. Therefore, even if write 4000 extents,
the filesystem may not really allocate 4000 extents.

2) kstore always return [0, xxx] even if offset ~= 0. Therefore,
the whole non-zero offset FiemapHoles test should be skipped

3) enable fiemap test for filestore, bluestore, memstore again

Fixes: http://tracker.ceph.com/issues/21716
Signed-off-by: Ning Yao <yaoning@unitedstack.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: fix bug of data type conversion in do_request()

Signed-off-by: shangfufei <shangfufei@inspur.com>

msg/async/rdma: fix a coredump bug which is introduced by PR #18053,
where the iterator is not working properly after erase().

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Merge pull request #18230 from ceph/liewegas-patch-1

.githubmap: Update

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: s/backfill/backfilling/

it's renamed "backfilling" in 4015343f .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18203 from majianpeng/bluestore-wake-kvfinathread

os/bluestore/BlueStore: ASAP wake up _kv_finalize_thread.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #18201 from gregsfortytwo/wip-21737-shutdown-assert

common: by default, do not assert on leaks in the shared_cache code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18218 from tchaikov/wip-21609

mon/PGMap: let pg_string_state() return boost::optional<>

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18213 from tchaikov/wip-log-on-osx

log: fix build on osx

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #18216 from tchaikov/wip-cmake-cleanup

cmake: set supported language the right way

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

mgr/status: fix ceph fs status returns error

Fixes: http://tracker.ceph.com/issues/21752
Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

Merge pull request #18228 from ktdreyer/doc-mds-blacklist-interval-manual

doc: rm stray ")" character from mds config ref

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17900 from markhpc/wip-filestore-rocksdb-compaction-readahead-master

common/options.cc: Set Filestore rocksdb compaction readahead option.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Update .githubmap

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: fix import of post-split pg from pre-split ancestor

If we export a PG, it later splits, and then we try to import a
post-split portion of it, we need to generate the PastIntervals
properly (instead of crashing).

Fixes: http://tracker.ceph.com/issues/21753
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #18223 into master

* refs/pull/18223/head:
githubmap: define merge conflict resolution

Reviewed-by: Sage Weil <sage@redhat.com>

doc: rm stray ")" character from mds config ref

I accidentally introduced this in
4fb89a63172d2dc6ce8f4702fe90904f9a8fd7bd

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rgw: Remove assertions in IAM Policy

A couple of them could be triggered by user input.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

githubmap: define merge conflict resolution

Use "union" merge to automatically resolve trivial append merge conflicts from
using ptl-tool.py.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17882 from ZVampirEM77/wip-rgw-encryption

rgw: clean up and fix some bugs for encryption

config: Remove _get_val

I added this originally to prevent a deadlock when reading a new-style
configuration variable from a called observer. I didn't realize at the
time that the config mutex was recursive, so this change was
superfluous.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17759 from liewegas/wip-21407

osd: fix waiting_for_peered vs flushing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18208 from jan--f/mgr-prometheus-fix-undef-metrics

pybind/mgr/prometheus: fix metric type undef -> untyped

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>

Merge pull request #18136 from dillaman/wip-21561

rbd-mirror: primary image should register in remote, non-primary image's journal

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18158 from dillaman/wip-21711

librbd: refresh image after applying new metadata

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

ceph-kvstore-tool: use unique_ptr<> manage the lifecycle of bluestore and db

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: generalize PGSrub, PGRecovery, PGSnapTrim

This commit will allow PGScrub, PGRecovery, PGSnapTrim
to be created anywhere (e.g. TestMClockClientQueue.cc)

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: use get_pgid() instead of info.pgid

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: use global id instead of entity_inst_t

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: move pgid into OpQueueItem

this pave the road to generalize OpWQ

We're going to want to be able to queue things that are not ordered by
the PG lock.  To that end, this patch genearlizes OSD::ShardedOpWQ to
use a type which can specify an ordering token and locking structure
other than a PG.

There is a lot of collateral damage which I didn't feel was worth
separating out into other commits.  The code in ShardedOpWQ itself got
some superficial cleanup.  Also, the item being queued has been switched
to not use a boost::variant.  It was a cute way before to make the type
easily copyable, but adding more visitors for the locking support would
have been annoying.  Instead, the variant is a unique_ptr to an
interface.  This makes the queue item type no longer copyable, which is
just as well since we don't really want to be copying queue items anyway
(duplicates would be most likely a bug)

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd/PG: fix clang build vs private state events

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: refresh image after applying new/removing old metadata

Fixes: http://tracker.ceph.com/issues/21711
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/PGMap: let pg_string_state() return boost::optional<>

better to be specific, so we don't run into the same problem even we
are using the MSB of uint64_t for a pg state bit in future. we could,
however use uint64_t(-1) to indicate the pg_string_state()'s failure to
parse the state string, because pg_string_state() only translate a
single state a time. but it's always better to be explicit than
implicit.

Fixes: http://tracker.ceph.com/issues/21609
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/PGMap: use uint64_t for pg state

Signed-off-by: Kefu Chai <kchai@redhat.com>

src/test: fix move semantics

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

common,osd: enforce move semantics in WorkQueue

enable the WorkQueue to support move-only template types.

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Myoungwon Oh <omwmw@sk.com>