librbd: quickly check for duplicate snap name upon create

During journal replay, quickly check for a duplicate snap name
before allocating a snapshot sequence from the OSDs and
reverting due to the duplicate name.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: stop external replayer before closing remote journaler

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: stop journal recorder before starting external replay

Fixes: http://tracker.ceph.com/issues/16165
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8008 from cernceph/dvanders_minsize

mon: default ec min_size to k+1

Dan van der Ster <daniel.vanderster@cern.ch>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8789 from vuhuong/wip-msgr-flex-prop-list

msgr,xio: flexible Messenger::create options

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9184 from liewegas/wip-addr-work

msg: encode entity_addr_t with features

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9486 from ifed01/wip-bluestore-stats3

os/bluestore: add per-store allocation/compression statistics

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9314 from xiexingguo/xxg-wip-fix-radosclient

RadosClient: reduce cpu burning; fix message leak

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9413 from stiopaa1/common_simplecache_addMove

common/simple_cache.hpp: add std::move

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9414 from branch-predictor/bp-no-invalidate-new-ptr

buffer: don't invalidate crc cache of new ptr before zeroing it

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/BlueStore: Adds improved per-store statistics including compression ones.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9487 from ceph/wip_control_compiler_diagnostic_colors

Allows developers to turn on diagnostic colors for compiler output when stderr is not a terminal

Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/BlueStore: Fixes fake blob length after put_ref removes all the extents

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/BlueStore: Adding compressed_length field to blob and fixing corresponding issues at read path

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/ObjectStore: Refactor ObjectStore::statfs method to return extended statistics

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9412 from Yan-waller/yj-hwip-rgwfixglobalinit

rgw/init: add errno and remove some unnecessary temporary variables.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9331 from stiopaa1/common_workQueue_addOverride

common/WorkQueue.h: add override to virtual func

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9389 from badone/wip-releases-table-plus-jewel

doc: Add Jewel header to releases table

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9529 from gregsfortytwo/wip-cephfs-doc-link

doc: include link to CephFS' experimental features

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9551 from dillaman/wip-16178

cls_rbd: mirror image status summary should read full directory

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9563 from markhpc/wip-rocksdb-jemalloc

kv: explicitly set DISABLE_JEMALLOC for rocksdb in Makefile.am.

Reviewed-by: Sage Weil <sage@redhat.com>

explicitly set JEMALLOC for rocksdb.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Change implementation to use a documented CACHE variable, and allow
any of the three values supported by GCC to be used. Just as the GCC
default is 'auto', so is the default within the cmake file.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Add documentation to README.cmake.md. Have cmake produce messages
(status or warning) when diagnostics color is requested via
-DDIAG_COLOR_ALWAYS=yes.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

If compiler allows it and if -DDIAG_COLOR_ALWAYS=yes is provided to
cmake, turn on diagnostic colors always, even when compiler output
does not go directly to terminal (e.g., when piped to "less -R").

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #9370 from majianpeng/bitmap-allocator

os/bluestore/BitMapAllocator: fix free space beyond size of BitAlloca…

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #8657 from rzarzynski/wip-rgw-swift-auth

rgw: authentication subsystem rework

Passed teuthology rgw suite.

Merge pull request #9532 from xiexingguo/xxg-wip-fix-csumcheck

os/bluestore: make verify_csum catch unsupported csum type error

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9492 from xiexingguo/xxg-wip-fix-bitmapallocator

bluestore/BitAllocator: miscellaneous fixes

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #9528 from stiopaa1/common_ceph_json_addConsToMember

common/ceph_json.h: add const to member function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9530 from majianpeng/bluestore-cleanup

os/bluestore/BlueStore: remove the duplicated code.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9549 from ifed01/wip-bluestore-large-blob-fix

os/bluestore: Fix for blob length on large writes

Reviewed-by: Sage Weil <sage@redhat.com>

bluestore: end scope of std::hex properly

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

bluestore: make verify_csum can catch unsupported csum type error

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cls_rbd: mirror image status summary should read full directory

Previously only retrieved the status for the first 64 images in
the rbd_mirroring directory.

Fixes: http://tracker.ceph.com/issues/16178
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/BlueStore: Fixes improper blob length assignment when creating a new blob for large writes

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/BlueStore: Adds checkes for lextent range overlaps to _fsck

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9533 from jjhuo/no_pagecache

os/bluestore: make rocksdb not to use page cache

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8233 from majianpeng/bluestore-perfcount

os/bluestore: perf counters

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: remove the duplicated code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

bluestore/BitAllocator: literal cleanups

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9296 from tchaikov/wip-test-chain_xattr

test: Fix the order of ASSERT_EQ() in chain_xattr tests

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #9361 from stiopaa1/osd_osd_override

osd/OSD.h: add override to virtual functions

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore/BlueFS: convert all read/write IO to direct IO.

RocksDB is the sole user of BlueFS, after BlueFS uses direct IO,
RocksDB will have to use its own block cache to cache raw data
blocks.

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>

os/bluestore: convert read_buffered() to read_random() with buffered flag

New function will handle unaligned random read for both buffered or
direct read.

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>

os/bluestore/BitAllocator: fix compiler warning.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BitMapAllocator: fix free space beyond size of BitAllocator

When Using bitmap allocator, the osd crashed. The stack info as
follows:

ceph-osd: os/bluestore/BitAllocator.cc:910: bool
BitAllocator::is_allocated(int64_t, int64_t): Assertion `start_block >=
0 && (start_block + num_blocks <= size())' failed.
*** Caught signal (Aborted) **
in thread 7f795bbc48c0 thread_name:ceph-osd ceph version
10.2.0-1333-g3f4cf16
(3f4cf16)
1: (()+0x98e25e) [0x55e112a9625e]
2: (()+0x109f0) [0x7f795a6719f0]
3: (gsignal()+0x38) [0x7f7958542a28]
4: (abort()+0x16a) [0x7f795854462a]
5: (()+0x2d227) [0x7f795853b227]
6: (()+0x2d2d2) [0x7f795853b2d2]
7: (()+0x7eaa59) [0x55e1128f2a59]
8: (BitAllocator::free_blocks(long, long)+0x22) [0x55e1128f2e82]
9: (BitMapAllocator::insert_free(unsigned long, unsigned long)+0x22c)
[0x55e1128ef12c]
10: (BitMapAllocator::init_add_free(unsigned long, unsigned long)+0x22c)
[0x55e1128ef41c]
11: (BlueFS::_init_alloc()+0x2b9) [0x55e1128c27b9]
12: (BlueFS::mkfs(uuid_d)+0x45a) [0x55e1128d475a]
13: (BlueStore::_open_db(bool)+0xd76) [0x55e1127bbcb6]
14: (BlueStore::mkfs()+0x8b1) [0x55e1127e4f11]
15: (OSD::mkfs(CephContext, ObjectStore, std::__cxx11::basic_string,
std::allocator > const&, uuid_d, int)+0x117) [0x55e112484de7]
16: (main()+0x101f) [0x55e112425f2f]
17: (__libc_start_main()+0xf0) [0x7f795852e580]
18: (_start()+0x29) [0x55e1124684e9]

This because in BitAllocator::init will decrease size of blkdev which
make size align w/ zone-size.
The later add free extent will beyond the size.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BlueStore: remove the unused perfcounter:l_bluestore_state_wal_done_lat

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #9527 from linuxbox2/rgw-ldap

rgw ldap: fix ldap bindpw parsing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw ldap: fix ldap bindpw parsing

Also add additional LDAP debugging output at 0, 10, and 15 to make
troubleshooting easier.

Fixes DN search issue using QE configuration of MS AD.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

doc: include link to CephFS' experimental features

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

RocksDBStore: add option to configure number of block cache shards

rocksdb_cache_size will be splitted into number of shards evenly.
More shards will reduce lock contentions.

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>

common/ceph_json.h: add const to member function

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #9525 from liewegas/wip-wq

unittest_workqueue: fix wq test for 0 threads

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9517 from idryomov/wip-rbd-concurrent

krbd: don't segfault if images are unmapped concurrently

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #9432 from ceph/wip-16113

Clean up hobject_t::max() values in decode, avoid comparing directly to get_max()

Reviewed-by: Sage Weil <sage@redhat.com>

unittest_workqueue: fix wq test for 0 threads

In 81517aea0696e4e62c2eb0830d7b0bd378c57230 we added
support for a wq with 0 worker threads.  Fix unit test
accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9522 from ceph/wip-rocksdb

Makefile-rocksdb: updates

hobject: enforce max canonical value

Signed-off-by: Samuel Just <sjust@redhat.com>

src/: remove all direct comparisons to get_max()

get_max() now returns a special singleton type from which hobject_t's
can be assigned and constructed, but which cannot be directly compared.

This patch also cleans up all such uses to use is_max() instead.

This should prevent some issues like 16113 by preventing us from
checking for max-ness by comparing against a sentinel value.  The more
complete fix will be to make all fields of hobject_t private and enforce
a canonical max() representation that way.  That patch will be hard to
backport, however, so we'll settle for this for now.

Fixes: http://tracker.ceph.com/issues/16113
Signed-off-by: Samuel Just <sjust@redhat.com>

os/bluestore/BlueStore: add perfcoutner:l_bluestore_write_penalty_read_ops

For write, because the align requirement it maybe need read data.
Using this perfcounter to record the read ops.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #9518 from cbodley/wip-mds-override

mds: fix MDRequestImpl::print override

Reviewed-by: John Spray <jspray@redhat.com>

os/bluestore/BlueStore: add perfcounter: l_bluestore_wal_write_ops/bytes.

Using this record wal ops and wal bytes.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BlueStore: Add perfcounter::l_bluestore_write_pad_bytes.

For write, sometimes it need padd zero. Use this perfcounter to
record the padding bytes.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Makefile-rocksdb: updates

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9506 from tchaikov/wip-cmake

cmake: s/BUILD_SHARED_LIBS/ENABLE_SHARED/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mds: fix MDRequestImpl::print override

Fixes compiler warnings:

In file included from /home/cbodley/ceph/src/mds/Locker.cc:22:0:
/home/cbodley/ceph/src/mds/Mutation.h:161:16: warning: ‘virtual void
MutationImpl::print(std::ostream&) const’ was hidden
[-Woverloaded-virtual]
   virtual void print(ostream &out) const {
                ^
/home/cbodley/ceph/src/mds/Mutation.h:340:8: warning:   by ‘void
MDRequestImpl::print(std::ostream&)’ [-Woverloaded-virtual]
   void print(ostream &out);

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9512 from Yan-waller/yj-wip-rgwdocs

doc/rgw: correct radosgw-admin command name

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9175 from majianpeng/bluestore-sync-wal-apply

os/bluestore: If using sync_wal_apply, no need start wal_tp thread.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9470 from liewegas/wip-bluestore

os/bluestore: shard cache to match osd op wq sharding

Merge pull request #9516 from liewegas/wip-bluestore-fixes

os/bluestore: misc fixes

Reviewed-by: Sage Weil <sage@redhat.com>

bluestore: add "0x" prefix for tracing messages

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

bluestore: simplify flag translation logic

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

bluestore: fix is_allocated() method of bluestore_blob_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9254 from branch-predictor/bp-lockdep-on-diet

lockdep: Lockdep on diet

Reviewed-by: Kefu Chai <kchai@redhat.com>

krbd: don't segfault if images are unmapped concurrently

"rbd map c" can die from a NULL dereference on any of this_pool,
this_image or this_snap in wait_for_udev_add():

    <image a is mapped>
    rbd map c
                                    rbd map b
                                    rbd unmap a
                                    rbd unmap b

However unlikely, this segfault is triggered by the rbd/concurrent.sh
workunit on a regular basis.

Similarly, "rbd showmapped" can die if an image to be listed is
unmapped.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #9384 from stiopaa1/osd_replicatedpg_override

osd/ReplicatedPG.h: add override to virtual funct

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9497 from Yan-waller/yj-wip-fixosdmaptools

tools/osdmaptool: fix some errors

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9490 from stiopaa1/osd_replicatedpg_addConstOutputOp

osd/ReplicatedPG.h: add const to operator<< param

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9499 from Yan-waller/yj-wip-fixrgwwarning

rgw/op: kill compiling warning

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9501 from stiopaa1/all_operatorOutput

all: add const to operator<< param

Reviewed-by: Sage Weil <sage@redhat.com>

rocksdb: pull up to master

This includes a few patches from Willem Jan Withagen
<wjw@digiware.nl> that fix the build on FreeBSD.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rgw: correct radosgw-admin command name

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #8598 from renhwsky/wip-renhw-ceph_mds-help

mds: fix usage() display

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8641 from renhwztetecs/renhw-wip-mds-cleanup

mds: MDSDaemon::init bug fix

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8753 from renhwztetecs/renhw-wip-fuse-fuse_file_info

client: fuse_file_info.fh type should be uint64_t

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8777 from ukernel/wip-allow-dirfrags

mds: fix 'allows_dirfrags' check for fragment_dir command

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8799 from renhwztetecs/renhw-wip-fusemain

client/fuse: object release order error

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #9459 from dillaman/wip-16123

test/rbd: fsx needs to shut down the journal

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

cmake: s/BUILD_SHARED_LIBS/ENABLE_SHARED/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9503 from stiopaa1/common_cephjson_duplicateDeclaration

common/ceph_json.h: remove duplicate declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/ceph_json.h: remove duplicate declaration

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

all: add const to operator<< param

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #9174 from majianpeng/osd-ms-dispatch

OSD: cleanup code for ms_dispatch

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9178 from wjwithagen/wip-wjw-freebsd-compile-2

build: freebsd changes (part 2)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9446 from ceph/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

PG::replica_scrub: don't adjust pool on max object

Signed-off-by: Samuel Just <sjust@redhat.com>

hobject: compensate for non-canonical hobject_t::get_max() encodings

This closes a loop-hole that could allow a non-canonical in memory
hobject_t::get_max() object which would return true for is_max(), but
false for *this == hobject_t::get_max().

Fixes: http://tracker.ceph.com/issues/16113
Signed-off-by: Samuel Just <sjust@redhat.com>

rgw/op: kill compiling warning

rgw/rgw_op.cc: In member function 'virtual int RGWCreateBucket::verify_permission()':
rgw/rgw_op.cc:1810:37: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (buckets.count() >= s->user->max_buckets) {

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

tools/osdmaptool: fix error message if we are unable to parse pgid

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>