test: Cleanup test-erasure-eio.sh code

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add replicated recovery/backfill test

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Better recovery/backfill restart for mark_unfound_lost delete/revert

Signed-off-by: David Zafman <dzafman@redhat.com>

osd, mon: Add new pg states recovery_unfound and backfill_unfound

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add new UnfoundBackfill and UnfoundRecovery pg transitions

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test case that reproduces tracker 18162

recover_replicas: object added to missing set for backfill, but is not in recovering, error!

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Better handle failure to get enough EC shards to backfill

Fixes: http://tracker.ceph.com/issues/18162
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #18129 from dragonylffly/wip-fix-coverity-warning-for-uninialized-member

librbd: fix coverity warning for uninitialized member

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18314 from MahatiC/example_librbd

examples: add new librbd example

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18265 from majianpeng/lirbd-cleanup

librbd: drop redundant check for null ImageCtx

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18326 from joscollin/wip-log-sign-compare-warning

log: silence warning from -Wsign-compare

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18288 from trociny/wip-metadata

test/librbd: test metadata_set/remove is applied

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15941 from oritwas/wip-rgw-dynamic-resharding-doc

rgw: Add dynamic resharding documentation

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Alfredo Deza <adeza@redhat.com>

Merge pull request #18209 from tchaikov/wip-new-style-opt-mon

mon/OSDMonitor: use new style options

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18093 from jcsp/wip-mgr-gil-safety

mgr: safety checks on pyThreadState usage

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

log: silence warning from -Wsign-compare

This warning appears in jenkins build:

/home/jenkins-build/build/workspace/ceph-pull-requests/src/log/test.cc:266:232:   required from here
/home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {

Signed-off-by: Jos Collin <jcollin@redhat.com>

mgr: safety checks on pyThreadState usage

Previously relied on the caller of Gil() to
pass new_thread=true if they would be
calling from a different thread.

Enforce this with an assertion, by wrapping
PyThreadState in a SafeThreadState class
that remembers which POSIX thread
it's meant to be used in.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: move Gil implementation into .cc

The inclusion of Python.h in the .h was awkward
for other files including Gil.h.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: reduce Gil verbosity at level 20

Even at 20, it's pretty heavy to be logging
every lock acquire/release.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #18309 into master

* refs/pull/18309/head:
mds: prevent trim count from underflowing

Reviewed-by: Zheng Yan <zyan@redhat.com>

examples: add new librbd example

Adds a librbd example

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

rgw: Add dynamic resharding documentation

Fixes: http://tracker.ceph.com/issues/21553
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18292 from xiexingguo/wip-recovery-order

osd/PrimaryLogPG: arrange recovery order by number of missing objects

Reviewed-by: Sage Weil <sage@redhat.com>

mds: prevent trim count from underflowing

Fixes: http://tracker.ceph.com/issues/21807
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18295 from adamwg/awg/update-orgmap

organizationmap: Add Adam Wolfe Gordon's affiliation

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #18139 into master

* refs/pull/18139/head:
test: make the LibCephFS.ShutdownRacer test even more thrashy
lockdep: free_ids and lock_ref hashes must be truly global
common: add a clear_g_str_vec() function to clear g_str_vec
common: make it safe to call env_to_vec multiple times

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #18183 into master

* refs/pull/18183/head:
osdc/ObjectCacher: limit memory usage of BufferHead

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18215 into master

* refs/pull/18215/head:
mds: update client metadata for already open session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18261 into master

* refs/pull/18261/head:
mds: keep CInode::STATE_QUEUEDEXPORTPIN state when exporting inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18244 into master

* refs/pull/18244/head:
mds: additional damage handling case in EImportStart

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18229 from liewegas/wip-21753

ceph-objectstore-tool: fix import of post-split pg from pre-split ancestor

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #18224 from liewegas/wip-pglog-cf

os/bluestore: dynamic CF configuration; put pglog omap in separate CF

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #18186 from xiexingguo/wip-miss-loc

osd/PG: reduce some overhead on operating MissingLoc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16030 from myoungwon/wip-revamp-opwq

osd: generalize queueing and lock interface for OpWq

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17541 from mikulely/log-time-msecs

rgw: fix total_time to msec as per AWS S3

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18214 from shashalu/fix-null-data-mtime

rgw: don't change rados object's mtime when update olh

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18245 from yaozongyou/wip-radosgw-admin-bi-fix

rgw: udpate radosgw-admin usage with bi purge

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18248 from ivancich/wip-rgw-unify-bi-hash-calc

rgw: consolidate code that implements hashing algorithms

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18272 from cbodley/wip-compressor-random

compressor: use generate_random_number() for type="random"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18240 from tchaikov/wip-21716

test: fix misc fiemap testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18221 from tchaikov/wip-kvstore-tool-cleanup

ceph-kvstore-tool: use unique_ptr<> to manage the lifecycle of bluestore

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Chang Liu <liuchang0812@gmail.com>

organizationmap: Add Adam Wolfe Gordon's affiliation (DigitalOcean)

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>

Merge pull request #18285 from tchaikov/wip-cmake

cmake: bump up required cmake version to 2.8.12

Merge pull request #18273 from cbodley/wip-21775

rgw: 'zone placement' commands validate compression type

mon/OSDMonitor: use new style options

* also mark "mon_osd_reporter_subtree_level" safe.
* change "mon_osd_min_down_reporters" to unsigned
* change "osd_pool_default_size" to unsigned
* change "osd_pool_default_min_size" to unsigned
* change "osd_pool_default_pg_num" to unsigned
* change "osd_pool_default_pgp_num" to unsigned

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: arrange recovery order by number of missing objects

Replica which has the shortest missing list (smallest amount of
objects to recover) should go recovering first, since it can be
technically brought back to normal sooner than others.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/librbd: test metadata_set/remove is applied

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

cmake: fix gperftools version detection

there is chance that its "patch" version is empty. in that case, do not
include it in its version string. otherwise, we will have something
like:

Found gperftools:
GPERFTOOLS_TCMALLOC_LIBRARY;GPERFTOOLS_TCMALLOC_MINIMAL_LIBRARY;GPERFTOOLS_PROFILER_LIBRARY
(found version "2.4.#define TC_VERSION_PATCH  """)

when running cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18286 from tchaikov/wip-kill-bluestore-warning

os/bluestore: silence -Wreturn-type warning

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

cls/rbd: fix log message

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18277 from dillaman/wip-librbd-unused-variables

librbd: remove unused variables from ReadResult refactor

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

cmake: use CXX_VISIBILITY_PRESET and VISIBILITY_INLINES_HIDDEN

instead of using "-fvisibility=hidden -fvisibility-inlines-hidden", more
cmake-ish this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/LINK_PRIVATE/PRIVATE/

as 2.8.12 is required not, no need to be compatible with 2.8.11 anymore.

this reverts 457f023.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: refactor -fsanitize=... options

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: silence -Wreturn-type warning

this silences following warning:

ceph/src/os/bluestore/bluestore_types.h: In member function ‘bool bluestore_blob_t::_validate_range(uint64_t, uint64_t, bool) const’:
ceph/src/os/bluestore/bluestore_types.h:664:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use CMAKE_POSITION_INDEPENDENT_CODE not -fPIC

we use the global setting of CMAKE_POSITION_INDEPENDENT_CODE for
defining this flag.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: bump up the required version to 2.8.12

as both trusty and centos 7.3 offer 2.8.12

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17793 from amitkumar50/cov-misc-6

librbd,os: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17825 from amitkumar50/cov-misc-9

librbd,test: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17972 from cbodley/wip-21401

auth: keep /dev/urandom open for get_random_bytes

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18204 from ownedu/wip-fix-async-rdma-coredump

msg/async/rdma: fix a coredump introduced by PR #18053

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18217 from liewegas/wip-pg-clang

osd/PG: fix clang build vs private state events

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18259 from xiexingguo/wip-blue-reclaim

os/bluestore/BlueFS: sanity check that alloc->allocate() won't return 0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18262 from tchaikov/wip-rocksdb

rocksdb: sync with upstream

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

os/bluestore: dynamically configure CFs and options via bluestore_rocksdb_cfs

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: put pglog omap in a separate CF

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: remove unused variables from ReadResult refactor

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18210 from shangfufei/wip-async-fix

msg/async: fix bug of data type conversion when uint64_t -> int -> uint64_t

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Amit Kumar amitkuma@redhat.com

Merge pull request #18246 from liewegas/wip-seesaw-warning

qa/suites/rados/singleton/all/mon-seesaw: whitelist MON_DOWN

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18270 from dillaman/wip-21391

cls/journal: possible infinite loop within tag_list class method

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

compressor: use generate_random_number for type="random"

use an existing thread-local random engine instead of constructing,
seeding, and locking a separate engine

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: 'zone placement' commands validate compression type

Fixes: http://tracker.ceph.com/issues/21775
Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: drop redundant check for ictx.

ictx alreay checked.It must not null.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cls/journal: fixed possible infinite loop which could kill the OSD

Fixes: http://tracker.ceph.com/issues/21771
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: ceph_test_cls_journal was dropped when converting to cmake

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18258 from shinobu-x/sk-osdmonitor_move_bufferlist_before_use

mon/OSDMonitor: cleanup: move bufferlist before use

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #18235 from tchaikov/wip-qa-backfilling

qa: s/backfill/backfilling/

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18222 from adamemerson/wip-don't-fix-what-ain't-broke

config: Remove _get_val

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: keep CInode::STATE_QUEUEDEXPORTPIN state when exporting inode

Fixes: http://tracker.ceph.com/issues/21768
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

rocksdb: sync with upstream

to pickup two changes

- prefetched skiplist
- enable SSE42 CRC32 on EL7

Fixes: http://tracker.ceph.com/issues/21603
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore/BlueFS: sanity check that alloc->allocate() won't return 0

If alloc[id]->allocate() returns 0, no space is actually reclaimed.
But BlueStore treats returning 0 as success and will keep reclaiming
space from BlueFS until hit the given threshold, which turns out to
be deadloop-prone.

For now, Bitmap and Stupid never return 0-length allocated space
(which is instead converted to -ENOSPC to suggest an error to caller),
so sanity checking against result code 0 will suffice.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: Move bufferlist before use

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

osd: fix reference a deleted item

move() delete to_process' entries. This commit will prevent
referencing a deleted item

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #17824 from shashalu/fix-lc-expiration

rgw:lc fix expiration time

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #18102 from MI-OSiRIS/mgr_influx_dbcheck

Modify mgr-influx module database check to not require admin privileges

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18104 from linuxbox2/rgw-rel-lock-earlyx

rgw: release cls lock if taken in RGWCompleteMultipart

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18175 from yaozongyou/fix-radosgw-admin-command-arg-bug

rgw: fix command argument error for radosgw-admin

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18193 from cbodley/wip-rgw-pool-application-jewel

qa/rgw: ignore errors from 'pool application enable'

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #18225 from adamemerson/wip-retract-assertions

rgw: Remove assertions in IAM Policy

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

test: make the LibCephFS.ShutdownRacer test even more thrashy

Have each thread do the startup and shutdown in a loop for a specified
number of times.

Tracker: http://tracker.ceph.com/issues/21512
Signed-off-by: Jeff Layton <jlayton@redhat.com>

lockdep: free_ids and lock_ref hashes must be truly global

It's possible for the teardown of g_lockdep_ceph_ctx to occur, followed
by a new context being registered as the lockdep context. When that
occurs, we can end up reusing lock id's that were previously handed out
to consumers. We need for those IDs to be persistent across lockdep
enablement and disablement.

Make both the free_ids table, and the lock_refs map persistent across
lockdep_unregister_ceph_context and lockdep_register_ceph_context cycles.
Entries in those tables will only be deleted by the destruction of the
associated mutex.

When lockdep_unregister is called, do the refcounting like we normally
would, but only clear out the state when the lockid is registered
in the lock_names hash.

Finally, we do still need to handle the case where g_lockdep has gone
false even when there are outstanding references after the decrement.
Only log the message if that's not the case.

With this, we can deal with the case of multiple clients enabling and
disabling lockdep in an unsynchronized way.

Tracker: http://tracker.ceph.com/issues/21512
Signed-off-by: Jeff Layton <jlayton@redhat.com>

common: add a clear_g_str_vec() function to clear g_str_vec

Prefix str_vec and str_vec_lock with "g_" to make it clear that they are
truly global values. Add a new clear_g_str_vec function to allow it to
be explicitly cleaned out by callers that need that functionality
(mostly testcase for now).

Tracker: http://tracker.ceph.com/issues/21512
Signed-off-by: Jeff Layton <jlayton@redhat.com>

common: make it safe to call env_to_vec multiple times

After it has been called once and we have outstanding CephContexts with
pointers into str_vec, we can't call get_str_vec on it again.

Add a static local mutex to protect access to str_vec.

Tracker: http://tracker.ceph.com/issues/21512
Signed-off-by: Jeff Layton <jlayton@redhat.com>

rgw: udpate radosgw-admin usage with bi purge.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #17420 from tchaikov/wip-19705

arch/arm: set ceph_arch_aarch64_crc32 only if the build host supports crc32cx

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18242 from tchaikov/wip-21758

crushtool: print error message to stderr not dout(1)

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton/all/mon-seesaw: whitelist MON_DOWN

Mgr can get marked down when mon weirdness is happening.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18137 from cbodley/wip-21685

rgw: RGWUser::init no longer overwrites user_id

mds: additional damage handling case in EImportStart

Fixes: http://tracker.ceph.com/issues/21759
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #18233 from gmayyyha/ceph-fs-status-21752

mgr/status: fix ceph fs status returns error

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>