os/bluestore/BlueStore: Fix wrong usage of std::unique_lock for OpSequenecer

When set bluestore_sync_wal_apply = false, the osd must be crashed.
I add some debug message in WALWQ::_dequeue():
// preserve wal ordering for this sequencer by taking the lock
/ while still holding the queue lock
-      i->osr->wal_apply_lock.lock();
+      try {
+       i->osr->wal_apply_lock.lock();
+      } catch (const system_error& e) {
+       if (e.code() == std::errc::resource_deadlock_would_occur) {
+         assert(0 == "would deadlock");
+       } else
+         assert(0 == "other system_error");
+      }
       return i;
     }

2016-05-24 00:02:29.107165 7f0f32ea6700 -1 os/bluestore/BlueStore.h: In
function 'virtual BlueStore::TransContext* BlueStore::WALWQ::_dequeue()'
thread 7f0f32ea6700 time 2016-05-24 00:02:29.103360
os/bluestore/BlueStore.h: 489: FAILED assert(0 == "would deadlock")

 ceph version 10.2.0-1232-g21c7599 (21c759936ddcbfec50cf177a577206a2c443c212)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x560196215ac0]
 2: (ThreadPool::WorkQueue<BlueStore::TransContext>::_void_dequeue()+0x234)[0x560195e618b4]
 3: (ThreadPool::worker(ThreadPool::WorkThread*)+0x654) [0x560196207324]
 4: (ThreadPool::WorkThread::entry()+0x10) [0x560196208d70]
 5: (()+0x760a) [0x7f0f411d260a]
 6: (clone()+0x6d) [0x7f0f3f17a59d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

This because the wron usage of std::unique_lock.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge branch 'xio-rename_XioRsp' of https://github.com/Adirl/ceph into Adirl-xio-rename_XioRsp

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #9235 from athanatos/wip-throttle

BackoffThrottle: fix for freebsd and avoid spinning when full

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #9276 from linuxbox2/xio-enqueue-2

xio: follow 03d33e59 (enqueue_for_send()->enqueue())

xio: follow 03d33e59 (enqueue_for_send()->enqueue())

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #9275 from ktdreyer/wip-doc-cephfs-jcsp

doc: John Spray is now CephFS lead

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc: John Spray is now CephFS lead

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #8856 from avnerbh/fixes_4_keepalive_to_xio

xio: provide keepalive(s) and their ack(s)

Merge pull request #9171 from weiqiaomiao/wqm-wip-rgw-mem

rgw: fix some memory leak

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8873 from roidayan/xio-use_xio_session_event_str

xio: use accelio api to get event str

(this is a decision for the Accelio API, we need to follow it)

Merge pull request #9205 from Adirl/fix_compiltaion_entity_addr_t3

Fix compiltaion entity addr t

Merge pull request #9229 from oritwas/wip-15937

rgw: add missing metadata_heap pool to old zones

Reviewed-by: Casey Bodley <cbodley@redhat.com>

BackoffThrottle: wait() if past max

Otherwise, we risk spinning on wait_for() with a small
delay.

Fixes: http://tracker.ceph.com/issues/15953
Signed-off-by: Samuel Just <sjust@redhat.com>

BackoffThrottle: use wait_for instead of wait_until

On some platforms, wait_until won't surrender the lock
with a negative argument.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #9249 from tchaikov/wip-cmake-ocf

cmake: add ocf support

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9227 from ceph/rpm-autosetup-kdreyer

ceph.spec.in: use %autosetup macro

Reviewed-by: Nathan Cuthler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #9147 from flaf/master

doc: update required packages list to build doc

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9247 from cxwshawn/wip-tc4-fix

global: optimize header file dependency.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9251 from cxwshawn/wip-tc5-fix

client: optimize header file dependency.

Reviewed-by: Sage Weil <sage@redhat.com>

osd/HitSet.h: add 'override' to virtual functionsReviewed-by: Kefu Chai <kchai@redhat.com>

osd/HitSet.h: add 'override' to virtual functions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9255 from yuyuyu101/wip-test-msgr-fix

test_msgr: add verbose info for pending connections

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8929 from ajarr/wip-15614

ceph_volume_client: allow read-only authorization for volumes

Reviewed-by: John Spray <john.spray@redhat.com>

ceph_volume_client: allow read-only authorization for volumes

Allow clients to be restricted to read-only mount of the volume
by restricting their ceph auth ID's MDS and OSD caps to read-only.

Fixes: http://tracker.ceph.com/issues/15614
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #8473 from rjfd/wip-14421

rbd-mirror: Delete local image mirror when remote image mirroring is disabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: test: added image-deleter test case when mirroring is disabled

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd-mirror: Unregister clients from non-primary images journal

A non-primary image may have registered clients on its journal
(for instance a primary image that was later demoted). We must
unregister the clients when disabling image mirroring with the
force option.

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd-mirror: test: Added unit test for testing image-deleter thread

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd-mirror: test: Fixed test_PoolWatcher to support the new changes in PoolWatcher class

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd-mirror: Replayer: bootstrap existing mirrored images

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd-mirror: replayer: Added image-deleter thread to replayer

Fixes: http://tracker.ceph.com/issues/14421
Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd-mirror: Added image-deleter thread to rbd-mirror

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd-mirror: Added implementation of image-deleter thread

image-deleter thread will handle the deletion of local images that have
been disabled for mirroring by the primary site.

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #9179 from vshankar/rbd-min-commit-position

cls_journal: initialize empty commit position upon client register

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7836 from yuyuyu101/wip-13938

TaskFinisher: use singleton SafeTimer and Finisher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test_msgr: add verbose info for pending connections

Signed-off-by: Haomai Wang <haomai@xsky.com>

client: optimize header file dependency.

same work as https://github.com/ceph/ceph/pull/9247

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

osd/HitSet.h: add 'override' to virtual functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

osdc/Objecter: move C_TwoContexts class to cc fileReviewed-by: Sage Weil <sage@redhat.com>

osdc/Objecter: move C_TwoContexts class to cc file

Reviewed-by: Sage Weil <sage@redhat.com>

osdc/Objecter: move C_TwoContexts class to cc file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

cmake: add ocf support

a new option is added: WITH_OCF. it is OFF by default.

Signed-off-by: Kefu Chai <kchai@redhat.com>

global: optimize header file dependency.

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

Merge pull request #9241 from ceph/wip-fix-q-tests

common/: fix tests for prioritized_queue and wpq

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9102 from yehudasa/wip-rgw-leak-4

rgw: don't unregister request if request is not connected to manager

Reviewed-by: Casey Bodley <cbodley@redhat.com>

common/: fix tests for prioritized_queue and wpq

Introduced: 27b6ec40b93c05fbaa228783e11e2fac8f1a1ee1
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #9216 from dzafman/wip-test-fix

test: Disable tests due to recovery race

test: Disable tests due to recovery race

Disable:
osd-scrub-repair.sh
test-erasure-eio.sh
test-erasure-code.sh

These test are failing due to race conditions that is exposing
bug http://tracker.ceph.com/issues/13937.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #9211 from dillaman/wip-15938

librbd: write-after-write might result in an inconsistent replicated image

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9231 from athanatos/wip-recovery-wq

osd: Put background recovery tasks in the main OpWQ

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6208 from liewegas/wip-alloc-hints

librados,osd: rados hints for future access pattern

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7070 from majianpeng/headerbeatmap-check

common/HeartbeatMap: print stack info of unhealth thread.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7843 from branch-predictor/bp-smart-async-wp

msg/async: Implement smarter worker thread selection

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8141 from songbaisen/song14

crush: fix the problem can not find the define item below the bucket

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8840 from tchaikov/wip-opt-u32

common/config: cast OPT_U32 options using uint32_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9072 from liewegas/wip-15840

osd: fix crush location update on startup

Merge pull request #9116 from ddiss/librados_sparse_read_use_snapid

librados: use caller provided snapid for aio_sparse_read()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #9002 from rzarzynski/wip-rgw-swift-acct_cont_md-fix

rgw: fix updating account/container metadata of Swift API.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: add missing metadata_heap pool to old zones

Fixes: http://tracker.ceph.com/issues/15937
Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cls_journal: Select min commit position for new clients

When a new client registers, its commit position is initialized
to the least commit position of all other clients.

Fixes: http://tracker.ceph.com/issues/15757
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #9225 from dachary/wip-run-make-check

tests: rm -fr /tmp/*virtualenv*

Reviewed-by: Erwan Velu <erwan@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: use %autosetup macro

New in rpm 4.11,
http://www.rpm.org/wiki/PackagerDocs/Autosetup

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #9165 from mikulely/fix-rgw-multisite-test

rgw: misc fix for test_multi.py

Reviewed-by: Casey Bodley <cbodley@redhat.com>

tests: rm -fr /tmp/*virtualenv*

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #9139 from majianpeng/bluestore-statfs

os/bluestore/BlueStore: free space of bluestore should reduce spaces …

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9161 from cxwshawn/wip-tc1-fix

osd: optimize header file dependency to decreace compile time when modify any file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9193 from cxwshawn/wip-tc2-fix

mon: optimize header file dependency.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9068 from trociny/wip-pybind-rbd-mirror

librbd: initial python APIs to support mirroring

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9088 from trociny/wip-15500

qa: dynamic_features.sh: return error only if it failed on alive QEMU

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9140 from zhouyuan/fix_disable_journaling

librbd: close journal before removing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9177 from trociny/wip-15916

rbd-mirror: it might got stuck in calculating "behind master" position

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

journal: replay position might change after pruning stale tags

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: delay commit of overwritten journal event

With the cache enabled and write-after-write IOs to the same
object extents, it was possible for the overwritten journal event
to be committed before the overwriter journal event was written
to disk.  If a client crash occurs before the event is written,
the image will be inconsistent on replay.

Fixes: http://tracker.ceph.com/issues/15938
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9196 from tchaikov/wip-cmake-vstart

cmake: add cython_rados to vstart target

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8164 from rzarzynski/wip-rgw-bulkdel-slashfix

rgw: handle initial slashes properly in BulkDelete of Swift API.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: handle initial slashes properly in BulkDelete of Swift API.

Fixes: http://tracker.ceph.com/issues/15948
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #9220 from cxwshawn/wip-tc3-fix

auth: optimize header file dependency.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9212 from liewegas/wip-bit-alloc-test

tests: disable unittest_bit_alloc for now

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #9219 from renhwztetecs/renhw-wip-logmonitor-cleanup

mon/log: clean unused function and comments

Reviewed-by: Sage Weil <sage@redhat.com>

test: file change affected by auth file header optimization.

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

AUTH: optimize header file dependency.
same work as PR: https://github.com/ceph/ceph/pull/9193, https://github.com/ceph/ceph/pull/9161

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

mon/log: using standard macros instead of -1

using standard macros instead of -1

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mon/log: clean unused function and comments

clean unused function and comments

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #8656 from kmroz/wip-rgw-handler-cleanup-1

rgw-rest: remove redundant virtual functions
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8886 from cbodley/wip-rgw-zone-short-ids

rgw: check for short_zone_id collisions on period update
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9148 from rzarzynski/wip-rgw-15902

rgw: camelcase names of custom attributes in Swift's responses.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9042 from rzarzynski/wip-rgw-15812

rgw: handle errors properly during GET on Swift's DLO.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9149 from cbodley/wip-rgw-zone-modify-master

rgw: add_zone only clears master_zone if --master=false
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9154 from yehudasa/wip-15886

rgw: keep track of written_objs correctly
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9190 from dillaman/wip-15928

librbd: metadata retrieval added to open image state machine

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9191 from dillaman/wip-15930

rbd-mirror: disable librbd caching for replicated images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9173 from dillaman/wip-15909

rbd-mirror: potential crash during image status update

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8983 from dzafman/wip-li-rados

tools/rados: Improve list-inconsistent json format

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: misc fix for test_multi.p

remove redundent meta_checkpoint, cause realm.create_user(user) will
call meta_checkpoint by default(via wait_meta = True).

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

TaskFinisher: use singleton SafeTimer and Finisher

Fix #13938
Signed-off-by: Haomai Wang <haomai@xsky.com>

OSD: optimize header file dependency to decreace compile time when modify any file
    OSD's left header dependency optimize.
Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

PG: optimize header file dependency to decreace compile time when modify any file
PG Layer's header dependency optimize.
Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

PGBackend: optimize header file dependency to decreace compile time when modify any file

hmm, everytime modify any file, then compile, then wait&wait&wait, so trying to optimize
this part, wish to decrease header dependency, try to improve compile speed.

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

tool: change affected file include relation by mon file header optimize

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

MON: optimize header file dependency.

same work as PR: https://github.com/ceph/ceph/pull/9161

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

crush: fix the problem that can not find the define item below the bucket

Fixes: http://tracker.ceph.com/issues/15154
Signed-off-by: song baisen <song.baisen@zte.com.cn>

Merge pull request #9108 from tchaikov/wip-13990

osd: reset session->osdmap if waiting_for_pg is empty

Reviewed-by: Samuel Just <sjust@redhat.com>