librbd: flush journal commit positions before starting op

Ensure all IO has been properly flushed and committed to the
journal before starting an op that could affect the IO path.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: leaked AioCompletion if IO issued after image shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: resize and snap create can hang on journal replay failure

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: missing journal state for flushing/restarting replay

Fixes: http://tracker.ceph.com/issues/16077
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track IO in the order issued

Fixes: http://tracker.ceph.com/issues/15034
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: AioCompletion should retrieve CephContext from ImageCtx

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Journal no longer requires AioCompletion for IO events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: bubble journal error up via existing AioObjectRequests

The journal cannot directly fail the AioCompletion because it
might have one-or-more outstanding requests associated with
it.

Fixes: http://tracker.ceph.com/issues/15791
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9296 from tchaikov/wip-test-chain_xattr

test: Fix the order of ASSERT_EQ() in chain_xattr tests

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #9361 from stiopaa1/osd_osd_override

osd/OSD.h: add override to virtual functions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9527 from linuxbox2/rgw-ldap

rgw ldap: fix ldap bindpw parsing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw ldap: fix ldap bindpw parsing

Also add additional LDAP debugging output at 0, 10, and 15 to make
troubleshooting easier.

Fixes DN search issue using QE configuration of MS AD.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #9525 from liewegas/wip-wq

unittest_workqueue: fix wq test for 0 threads

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9517 from idryomov/wip-rbd-concurrent

krbd: don't segfault if images are unmapped concurrently

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #9432 from ceph/wip-16113

Clean up hobject_t::max() values in decode, avoid comparing directly to get_max()

Reviewed-by: Sage Weil <sage@redhat.com>

unittest_workqueue: fix wq test for 0 threads

In 81517aea0696e4e62c2eb0830d7b0bd378c57230 we added
support for a wq with 0 worker threads.  Fix unit test
accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9522 from ceph/wip-rocksdb

Makefile-rocksdb: updates

hobject: enforce max canonical value

Signed-off-by: Samuel Just <sjust@redhat.com>

src/: remove all direct comparisons to get_max()

get_max() now returns a special singleton type from which hobject_t's
can be assigned and constructed, but which cannot be directly compared.

This patch also cleans up all such uses to use is_max() instead.

This should prevent some issues like 16113 by preventing us from
checking for max-ness by comparing against a sentinel value.  The more
complete fix will be to make all fields of hobject_t private and enforce
a canonical max() representation that way.  That patch will be hard to
backport, however, so we'll settle for this for now.

Fixes: http://tracker.ceph.com/issues/16113
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #9518 from cbodley/wip-mds-override

mds: fix MDRequestImpl::print override

Reviewed-by: John Spray <jspray@redhat.com>

Makefile-rocksdb: updates

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9506 from tchaikov/wip-cmake

cmake: s/BUILD_SHARED_LIBS/ENABLE_SHARED/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mds: fix MDRequestImpl::print override

Fixes compiler warnings:

In file included from /home/cbodley/ceph/src/mds/Locker.cc:22:0:
/home/cbodley/ceph/src/mds/Mutation.h:161:16: warning: ‘virtual void
MutationImpl::print(std::ostream&) const’ was hidden
[-Woverloaded-virtual]
   virtual void print(ostream &out) const {
                ^
/home/cbodley/ceph/src/mds/Mutation.h:340:8: warning:   by ‘void
MDRequestImpl::print(std::ostream&)’ [-Woverloaded-virtual]
   void print(ostream &out);

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9512 from Yan-waller/yj-wip-rgwdocs

doc/rgw: correct radosgw-admin command name

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9175 from majianpeng/bluestore-sync-wal-apply

os/bluestore: If using sync_wal_apply, no need start wal_tp thread.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9470 from liewegas/wip-bluestore

os/bluestore: shard cache to match osd op wq sharding

Merge pull request #9516 from liewegas/wip-bluestore-fixes

os/bluestore: misc fixes

Reviewed-by: Sage Weil <sage@redhat.com>

bluestore: add "0x" prefix for tracing messages

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

bluestore: simplify flag translation logic

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

bluestore: fix is_allocated() method of bluestore_blob_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9254 from branch-predictor/bp-lockdep-on-diet

lockdep: Lockdep on diet

Reviewed-by: Kefu Chai <kchai@redhat.com>

krbd: don't segfault if images are unmapped concurrently

"rbd map c" can die from a NULL dereference on any of this_pool,
this_image or this_snap in wait_for_udev_add():

    <image a is mapped>
    rbd map c
                                    rbd map b
                                    rbd unmap a
                                    rbd unmap b

However unlikely, this segfault is triggered by the rbd/concurrent.sh
workunit on a regular basis.

Similarly, "rbd showmapped" can die if an image to be listed is
unmapped.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #9384 from stiopaa1/osd_replicatedpg_override

osd/ReplicatedPG.h: add override to virtual funct

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9497 from Yan-waller/yj-wip-fixosdmaptools

tools/osdmaptool: fix some errors

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9490 from stiopaa1/osd_replicatedpg_addConstOutputOp

osd/ReplicatedPG.h: add const to operator<< param

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9499 from Yan-waller/yj-wip-fixrgwwarning

rgw/op: kill compiling warning

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9501 from stiopaa1/all_operatorOutput

all: add const to operator<< param

Reviewed-by: Sage Weil <sage@redhat.com>

rocksdb: pull up to master

This includes a few patches from Willem Jan Withagen
<wjw@digiware.nl> that fix the build on FreeBSD.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rgw: correct radosgw-admin command name

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #8598 from renhwsky/wip-renhw-ceph_mds-help

mds: fix usage() display

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8641 from renhwztetecs/renhw-wip-mds-cleanup

mds: MDSDaemon::init bug fix

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8753 from renhwztetecs/renhw-wip-fuse-fuse_file_info

client: fuse_file_info.fh type should be uint64_t

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8777 from ukernel/wip-allow-dirfrags

mds: fix 'allows_dirfrags' check for fragment_dir command

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8799 from renhwztetecs/renhw-wip-fusemain

client/fuse: object release order error

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #9459 from dillaman/wip-16123

test/rbd: fsx needs to shut down the journal

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

cmake: s/BUILD_SHARED_LIBS/ENABLE_SHARED/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9503 from stiopaa1/common_cephjson_duplicateDeclaration

common/ceph_json.h: remove duplicate declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/ceph_json.h: remove duplicate declaration

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

all: add const to operator<< param

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #9174 from majianpeng/osd-ms-dispatch

OSD: cleanup code for ms_dispatch

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9178 from wjwithagen/wip-wjw-freebsd-compile-2

build: freebsd changes (part 2)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9446 from ceph/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

PG::replica_scrub: don't adjust pool on max object

Signed-off-by: Samuel Just <sjust@redhat.com>

hobject: compensate for non-canonical hobject_t::get_max() encodings

This closes a loop-hole that could allow a non-canonical in memory
hobject_t::get_max() object which would return true for is_max(), but
false for *this == hobject_t::get_max().

Fixes: http://tracker.ceph.com/issues/16113
Signed-off-by: Samuel Just <sjust@redhat.com>

rgw/op: kill compiling warning

rgw/rgw_op.cc: In member function 'virtual int RGWCreateBucket::verify_permission()':
rgw/rgw_op.cc:1810:37: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (buckets.count() >= s->user->max_buckets) {

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

tools/osdmaptool: fix error message if we are unable to parse pgid

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

tools/osdmaptool: exit if we are unable to resolve pg_num

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

lockdep: Convert free_ids from list to bitmap

Convert "free_ids" list to bitmap, reducing its memory usage from
16 + MAX_LOCKS * 20 (~20KB) to MAX_LOCKS/8 (512b) and save some time
(re)populating the free id array.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

cmake: use ENABLE_SHARED instead of BUILD_SHARED_LIBS

we decided to drop the static libraries from the *-dev debian packages.
so no more tricky changes for supporting creating both libraries.

* partially revert 1dbfb26: s/BUILD_SHARED_LIBS/ENABLE_SHARED/
 - add_library() command use BUILD_SHARED_LIBS as the default library
   type, and all intermediate libraries should static ones. it would
   cause unnecessary confusion if some developer uses add_library() without
   specifying the library type, and ends up with a .so. so we use our
   own setting variables for specifying the type of user facing libraries.
* revert ac47440
 - the BUILD_SHARED_LIBS setting is also populated to the gmock libraries,
   which should be compiled as static ones. otherwise the unit tests will
   crash on exit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install binaries used by ceph-qa-suite

they are:
- ceph_test_rados: used by
  tasks/rados.py
- ceph_test_rados_delete_pools_parallel: used by
  suites/rados/monthrash/workloads/pool-create-delete.yaml
- ceph_test_filejournal
  suites/rados/objectstore/filejournal.yaml
- ceph_test_objectstore: used by
  suites/rados/objectstore/objectstore.yaml
- ceph_test_{async_driver,msgr}: used by
  suites/rados/singleton-nomsgr/all/msgr.yaml

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: compile more radosstriper tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link ceph_test_rgw_*, ceph-dencoder against cls_kvs

it is not used, and causes link errors.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link ceph-dencoder against os lib

os is a top level target which is linked into executables or user facing
libraries. so it is not included by any libraries linked by
ceph-dencoder so far.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link rbd cli against krbd_objs directly

rbd cli => Kernel.cc => krbd_create_from_context() <= krbd.cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix linkage of ceph_test_cls_${name}

they should link against cls_${name}_client, instead of against
cls_${name} directly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

makefile: install mount.fuse.ceph,mount.ceph into /usr/sbin

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install ceph-disk to /usr/sbin

* add keyword "INSTALL_SCRIPT" to distutils_install_module(), so we can
  override the install path of ceph-disk script.
* refactor the Distutils.cmake module a little bit, the ${option}
  variable out lives the CODE snipplet. so we need to reset it
  at the beginning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG.h: add const to operator<< param

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

os/bluestore: paranoid checks on onode_map vs lru

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: replace broken get_next with map_any

The get_next implementation assumed the lru was local to the OnodeSpace,
but it includes the whole cache now.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: better audit_lru debugging

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: shard the cache

Signed-off-by: Sage Weil <sage@redhat.com>

osd: communicate shard count to ObjectStore

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use spg_t method to hash pg to (wq) shard

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: unify lrus into single Cache class

This will be sharded soon, but for now there's just one instance.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9483 from ceph/wip-rocksdb

rocksdb: fix build

Merge pull request #9477 from tchaikov/wip-fix-cmake-unittests

cmake: do not create gtest/gmock shared libraries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: do not create gtest/gmock shared libraries

otherwise unit tests crash at exit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore/BlueStore: optimize WALWQ::flush()

In fact ThreadPool::drain will do
a)wait WorkQueue::empty() is true
b)wait processing thread to complete.
So it can remove the same work which wait WorkQueue::empty().

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BlueStore: Don't support dynamic set wal_apply mode.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/WorkQueue: Make decrease thread num to zero.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/config: make md_config_t::apply_changes recursive.

This make in md_config_obs_t::handle_conf_change to call apply_changes.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore: If using sync_wal_apply, no need start wal_tp thread.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #9381 from weiqiaomiao/wqm-wip-rgw-align

rgw:fix some indentation errors of "radosgw-admin --help"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Makefile-rocksdb: remove .d files

Not really sure why autotools is removing these... but
whatever.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9480 from ErwanAliasr1/evelu-cephtool

cephtool/test.sh: Adding iterations numbers in output log

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: unique type names or onode and buffer lrus

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: take collection lock when trimming collection cache

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: audit LRU at various points

This is purely for debugging.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: don't print bnode key

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix buffercache discard bug

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9465 from liewegas/wip-bluestore-zero

os/bluestore: remove aio_zero from bdev interface

Merge pull request #9417 from chhabaramesh/master

os/bluestore: Tree based bitmapalloc

Merge pull request #9466 from liewegas/wip-rocksdb

rocksdb: pull up to latest master

Makefile-rocksdb.am: update

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9457 from yehudasa/wip-14534

rgw: can set negative max_buckets on RGWUserInfo
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

unittest_bluefs: fix more

- wait for aio to finish
- shut down bdev, which stops the aio completion thread
- *then* deallocate the IOContexts

Signed-off-by: Sage Weil <sage@redhat.com>

cephtool/test.sh: Adding iterations numbers in output log

When this test is failing and reach the limits, reading the log doesn't make
obvious that we reach them.

This simple patch adds the iterations numbers inside the output log.

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #9399 from mikulely/update-ceph-deploy-link

doc: update broken link for deploy ceph by Chef

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9441 from xinxinsh/wip-16706

librbd: remove invalid object map flag when object map is disabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>