librbd: removal of partially deleted image needs id lookup

Several operations depend on the image id but if the image cannot be
opened to retrieve the id, these cleanup operations cannot be executed.

Fixes: http://tracker.ceph.com/issues/16471
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9945 from rjfd/wip-16488

rbd-mirror: fix potential race condition in resync listener removal

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9962 from liewegas/wip-bluestore-unused

os/bluestore: replace interval_set with bitmap for unused

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: replace bluestore_blob_t::unused from interval to bitmap

Signed-off-by: Igor Fedotov <ifedotov@miranitis.com>

os/bluestore: move add_unused call after csum initialization as it should depend on chunk size.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: introduce bluestore_blob_t::get_chunk_size method

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: cleanup _pad_zeros to eliminate redundant length param

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9959 from xiexingguo/xxg-wip-bluestore-cache-clean

os/bluestore: add  some sanity checks

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: audit buffer bytes before applying delta

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix typo

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check when try to trim onode from cache

We have verified that the onode size to be trimmed is positive ahead,
so "p" shall never point to onode_lru.begin() below here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9951 from Yan-waller/yj-wip-rgwtestcompilingwarning

rgw/test: use variable type 'size_t' instead of 'int'

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9839 from renhwztetecs/renhw-wip-cdir-cleanup

mds: unused classes and cleanup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9677 from stiopaa1/mds_removeUnneededLineAndAddConst

mds/SessionMap.h: remove duplicated line and add const to member functions

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9877 from batrick/mds-comm-locks

mds: add missing mds_lock locks

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9942 from jcsp/wip-doc-netdev

doc: add _netdev to cephfs fstab lines

Merge pull request #9949 from trociny/fix-use-after-free

librbd: potential use after free on refresh error

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: potential use after free on refresh error

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rgw/test: use variable type 'size_t' instead of 'int' seems to be more reasonable

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

rbd-mirror: image-replayer: Fix bug in resync listener remotion

Fixes: http://tracker.ceph.com/issues/16488
Signed-off-by: Ricardo Dias <rdias@suse.com>

doc: add _netdev to cephfs fstab lines

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #9742 from xiexingguo/xxg-wip-server_cmode

mds/server: fix rare race when waitting for osdmap

mds/server: fix rare race when waitting for osdmap

If wait_for_map() returns true, we have got the requested osdmap
at the specified epoch, so we shall try again to do further
verification whether we have the specific pool or not.

The above case can happen because we drop the objecter internal
rwlock during the switch between the objecter->with_osdmap()
and objecter->wait_for_map() methods.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9922 from dillaman/wip-16470

librbd: do not propagate mirror status notification failures

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9913 from xiexingguo/xxg-wip-fix-bitallocatorut

os/bluestore: fix ut failure when switching to bitmap-allocator

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

os/bluestore: fix memory leak during bit_alloc ut

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: use std limits for int16_t/int32_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: update comment

We have decreased the default concurrent scans to 1,
so the comment is misleading.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: allow total blocks fall into one single zone

So the blusfs ut will be happy with this.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: update output of offset/length pair to hex version

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9905 from dillaman/wip-16450

librbd: mark exclusive lock as released after journal is closed

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9899 from dillaman/wip-16350

librbd: ignore missing object map during snap remove

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9896 from dillaman/wip-16446

librbd: delete ExclusiveLock instance when switching to snapshot

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9895 from dillaman/wip-16445

librbd: ignore snap unprotect -EBUSY errors during journal replay

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9746 from xiexingguo/xxg-wip-mdcache_hoir

mds/MDCache: fix leak of MMDSOpenInoReply

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

librbd: do not propagate mirror status notification failures

These should be treated as a broadcast since no ACK is required. The
peer will eventually see the notification or its watch will have timed
out and it will re-sync.

Fixes: http://tracker.ceph.com/issues/16470
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9921 from liewegas/wip-bluestore-fix-discard

os/bluestore: fix discard

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: un-inline some 2Q methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix buffer discard accounting for 2Q

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simplify and fix blob invalidation

We can't invalidate based on any particular lextent we drop, as there may
be other lextents (in this onode or in another one) that reference the
same data.  This can break even a single multi-write transaction that
relies on the buffer cache to fill out blocks.

Fix by just doing the unallocated version unconditionally.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9920 from zhouyuan/librd_remove

librbd: remove unused code block

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9916 from xiexingguo/xxg-wip-fix-decode-bnode

os/bluestore: fix decoding pool from bnode key

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9910 from tchaikov/wip-cmake-drop-ceph-ra

cmake: ocf: drop ceph ra

librbd: memory leak possible if journal op event failed

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: trivial fix to remove unused code block

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

os/bluestore: literal tidy-ups

There error codes are improperly arranged, which looks weird.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make fsck can catch bad bnode key too

The case may be rare but the check shall do no harm.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix decoding pool from bnode key

The current result is raw and not properly translated.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: ocf: drop ceph ra

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: default to bitmap allocator

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9906 from stiopaa1/msg_asyncmess_workepool_deleteCopycontrAndAss

msg/AsyncMessenger:delete cpy constr and assign op

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9273 from stiopaa1/osd_removeClosingBrace_

osd/OSD.h: remove closing brace } from comment

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #9842 from jcsp/wip-vstart-len

vstart: set osd_check_max_object_name_len_on_startup

Merge pull request #8784 from ddiss/vstart_ifconfig

vstart: replace ifconfig usage with ip addr

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #9456 from ukernel/wip-msg-seq

msg: don't truncate message sequence to 32-bits

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9248 from xiexingguo/xxg-wip-expose-requirejewel

mon: expose require_jewel_osds flag to user

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9305 from xiexingguo/xxg-wip-replicatedpg-fixup

osd/ReplicatedPG: minor fixes and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9904 from cbodley/wip-cmake-vstart

cmake: add radosgw to 'make vstart' target

Reviewed-by: Ali Maredia <amaredia@redhat.com>

msg/AsyncMessenger:delete cpy constr and assign op

Delete copy constructor and assignment operator, so the checking can
be moved from linking to compile time - as it was when the above
where only declared, but not defined.

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

librbd: mark exclusive lock as released after journal is closed

Fixes: http://tracker.ceph.com/issues/16450
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9882 from dillaman/wip-16364

librbd: fix crash while using advisory locks with R/O image

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9881 from dillaman/wip-16433

librbd: journal::Replay no longer holds lock while completing callback

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9867 from dillaman/wip-16411

rbd-mirror: disallow proxied maintenance ops to non-primary images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

cmake: add cls libraries as radosgw dependencies

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9880 from stiopaa1/msg_asyncConn_move_workerpool

msg/async: mv WorkerPool class to cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8818 from SUSE/wip-drop-ceph-ra

ocf: drop ceph Resource Agent

Reviewed-by: Tim Serong <tserong@suse.com>

cmake: add ceph-mds to 'make vstart' target only if WITH_MDS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: add radosgw, radosgw-admin to 'make vstart' target

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9902 from liewegas/wip-cmake-tests

cmake: do not exclude tests from 'make all'

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: ignore missing object map during snap remove

Fixes: http://tracker.ceph.com/issues/16350
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cmake: do not exclude tests from 'make all'

This prevent us from doing 'make' and then 'ctest ...'.
It is also super annoying in general to not verify the
tests compile with a simple 'make'.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9870 from liewegas/wip-cmake

do_cmake.sh: simple cmake helper

Reviewed-by: Kefu Chai <kchai@redhat.com>

README.md: note about builds (they are for development!)

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: delete ExclusiveLock instance when switching to snapshot

Fixes: http://tracker.ceph.com/issues/16446
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9887 from ceph/wip-cmake

cmake: fix linking issues

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9890 from xiexingguo/xxg-wip-fix-bfs-alloc

os/bluestore: shutdown allocator of bluefs more gracefully

Reviewed-by: Sage Weil <sage@redhat.com>

librbd: ignore snap unprotect -EBUSY errors during journal replay

Fixes: http://tracker.ceph.com/issues/16445
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: remove duplicated reset of buf

buf->reset() will do the clear up effectively.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check for underflow

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: use hex version for offset/length

We update bluestore this way a couple of days ago.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: kill dead code

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: shutdown allocator of bluefs more gracefully

The new BitAllocator requires we call shutdown() explicitly,
thus we shall fix it here before we apply to the new BitAllocator.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: add gtest to ceph_test_snap_mapper

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: test_rbd_mirror_image_replay missing ContextCompletion.cc

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: link libradostriper with librados_objs

create an OBJECT target for libradostriper, so it is able to access the
internal symbols of librados.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove visibility flags from libcephfs,libradosstriper

because libradosstriper does not put the `__attribute__ ((visibility
("default")))` specifier on the exported class/methods. we cannot
add the visibility to this library yet. the same applies to libcephfs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move rados_snap_set_diff_obj into librbd_internal

ceph_test_librbd use it and links directly against librbd_internal
instead of against librbd

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: fix crash while using advisory locks with R/O image

Fixes: http://tracker.ceph.com/issues/16364
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: journal::Replay no longer holds lock while completing callback

Fixes: http://tracker.ceph.com/issues/16433
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9207 from rjfd/wip-15670

rbd-mirror: image resync

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/AsyncMessenger: mv WorkerPool class to cc file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

README.md: drop autotools instructions

Never look back!

Signed-off-by: Sage Weil <sage@redhat.com>

README.md: simplify cmake instructions around do_cmake.sh

Be prescriptive about the 'build' subdir, at least for the purposes of
this document (developer quick start).

Signed-off-by: Sage Weil <sage@redhat.com>

README.md: drop dependencies section

This is autotools-specific, and I'm not sure we need to document it here.

Signed-off-by: Sage Weil <sage@redhat.com>

do_cmake.sh: simple cmake helper

Signed-off-by: Sage Weil <sage@redhat.com>

mds: add missing mds_lock locks

These commands traverse the mdcache and/or subtrees and so must have the
mds_lock.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: move mds_lock to command subfuncs

Again, to make the lock management consistent with the other command
sub-functions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use Locker to manage mds_lock

This makes these blocks consistent with the others for trivial management of
mds_lock.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #9626 from liewegas/wip-bluestore-2q

os/bluestore: implement 2Q buffer cache policy

ceph_test_objectstore: define matrixes over config options and write sizes

These replace a few of the ad hoc synthetic tests.

Signed-off-by: Sage Weil <sage@redhat.com>