osdc/ObjectCacher.h: remove public access specifier

This is duplicated 3 times, so can be removed.

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #9538 from chhabaramesh/master

os/bluestore: Unaligned device length and Correct alloc size fix for BitMap allocator

Merge pull request #9600 from stiopaa1/osdc_objectcacher_moveCwritecommit

osdc/ObjectCacher.h: move classes to cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9596 from liewegas/wip-statfs

os: fix statfs for filestore and fusestore

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9597 from dillaman/wip-cls_rbd-encoding

cls_rbd: encode entity_addr_t with features

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8779 from ukernel/wip-cephfs-rstat

mds: fix rstat bugs

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8780 from ukernel/wip-renew-caps

client: renew caps for read/write if mds session got killed.

Reviewed-by: John Spray <john.spray@redhat.com>

correct allocation block size for bluestore and bluefs

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

remove truncation of unaligned blocks at init

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #9443 from vishalkanaujia/master

mds: Removed tmap fetch code from Cdir

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9536 from xiexingguo/xxg-wip-fix-prdwrv

client: fix sanity check of preadv/pwritev

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9523 from liewegas/wip-pg-to-osds

osd/OSDMap: pg_to_osds -> pg_to_raw_osds, fix libcephfs caller to get acting osds

Reviewed-by: Song Baisen <song.baisen@zte.com.cn>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9507 from xiexingguo/xxg-wip-fix-mdsutility

tools/cephfs: fix message leak; add flush for dump cmd

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8890 from ukernel/wip-15634

client: enable fuse_use_invalidate_cb by default

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9509 from ukernel/wip-16137

client: skip executing async invalidates while umounting

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9541 from renhwztetecs/renhw-wip-rbd

rbd: cleanup unnecessary switch breaks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osdc/ObjectCacher.h: move classes to cc file

Move C_WriteCommit and C_WaitForWrite to cc file.

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #9595 from tchaikov/wip-librados-do-not-return-local-ref

librados: get_pool_name() should not return ref of a local variable

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8830 from trociny/wip-rbd-mirror-bootstrap-cancel

rbd-mirror: support bootstrap canceling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: encode entity_addr_t with features

mirror_image_status_get method call currently fails with EIO

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd/osd_types: fix description of store_statfs_t::available field

It's bytes, not blocks.

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: fix statfs

Signed-off-by: Sage Weil <sage@redhat.com>

os/FuseStore: fix statfs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9386 from tchaikov/wip-bye-py2.6

ceph-detect-init,ceph-disk: drop support of py2.6

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9476 from ceph/wip-no-unittests-in-deb

packaging: do not package unused unittests in deb

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9475 from tchaikov/wip-debian

debian: package compressor plugins and silence dpkg-shlibdeps warnings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9315 from ShiqiCooperation/master

mon: remove unused local variable "proposal_bl"

Reviewed-by: Kefu Chai <kchai@redhat.com>

librados: get_pool_name() should not return ref of a local variable

was introduced by 1a44e6f

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9023 from dillaman/wip-15791

librbd: journal IO error results in failed assertion in AioCompletion

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9440 from dillaman/wip-16114

librbd: prevent error messages when journal externally disabled

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9573 from tchaikov/wip-cmake-rocksdb

cmake: fix rocksdb linkage with jemalloc installed but not enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8008 from cernceph/dvanders_minsize

mon: default ec min_size to k+1

Dan van der Ster <daniel.vanderster@cern.ch>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8789 from vuhuong/wip-msgr-flex-prop-list

msgr,xio: flexible Messenger::create options

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9184 from liewegas/wip-addr-work

msg: encode entity_addr_t with features

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9486 from ifed01/wip-bluestore-stats3

os/bluestore: add per-store allocation/compression statistics

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9314 from xiexingguo/xxg-wip-fix-radosclient

RadosClient: reduce cpu burning; fix message leak

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9413 from stiopaa1/common_simplecache_addMove

common/simple_cache.hpp: add std::move

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9414 from branch-predictor/bp-no-invalidate-new-ptr

buffer: don't invalidate crc cache of new ptr before zeroing it

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/BlueStore: Adds improved per-store statistics including compression ones.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9487 from ceph/wip_control_compiler_diagnostic_colors

Allows developers to turn on diagnostic colors for compiler output when stderr is not a terminal

Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/BlueStore: Fixes fake blob length after put_ref removes all the extents

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/BlueStore: Adding compressed_length field to blob and fixing corresponding issues at read path

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/ObjectStore: Refactor ObjectStore::statfs method to return extended statistics

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

osd/OSDMap: rename pg_to_osds -> pg_to_raw_osds

Signed-off-by: Sage Weil <sage@redhat.com>

client: return acting, not raw, osds

Reported-by: song baisen <song.baisen@zte.com.cn>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9412 from Yan-waller/yj-hwip-rgwfixglobalinit

rgw/init: add errno and remove some unnecessary temporary variables.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9331 from stiopaa1/common_workQueue_addOverride

common/WorkQueue.h: add override to virtual func

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9389 from badone/wip-releases-table-plus-jewel

doc: Add Jewel header to releases table

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: disable rocksdb from using jemalloc unless asked

rocksdb detects used libraries by itself. it we don't disable it
explicitly, it will pull in unresolved symbols at ceph's link
time. so we disable it unless jemalloc is enabled at configure-time.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: always set ALLOCATOR at configure time

so it would be easier to figure out what allocator we are using later,
instead of poking around ${Foo}_FOUND variables.

Signed-off-by: Kefu Chai <kchai@redhat.com>

kv: explicitly set DISABLE_JEMALLOC for rocksdb in Makefile.am.

we should disable rocksdb from using jemalloc even it is installed
in the building host. as it causes unresolved symbols if ceph is not
configured to use jemalloc and hence is not linked against it.

we can pass "DISABLE_JEMALLOC=0" to rocksdb's Makefile in hope to enable
jemalloc. because rocksdb checks for jemalloc using
```
ifndef DISABLE_JEMALLOC
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9529 from gregsfortytwo/wip-cephfs-doc-link

doc: include link to CephFS' experimental features

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9551 from dillaman/wip-16178

cls_rbd: mirror image status summary should read full directory

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9563 from markhpc/wip-rocksdb-jemalloc

kv: explicitly set DISABLE_JEMALLOC for rocksdb in Makefile.am.

Reviewed-by: Sage Weil <sage@redhat.com>

explicitly set JEMALLOC for rocksdb.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Change implementation to use a documented CACHE variable, and allow
any of the three values supported by GCC to be used. Just as the GCC
default is 'auto', so is the default within the cmake file.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Add documentation to README.cmake.md. Have cmake produce messages
(status or warning) when diagnostics color is requested via
-DDIAG_COLOR_ALWAYS=yes.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

If compiler allows it and if -DDIAG_COLOR_ALWAYS=yes is provided to
cmake, turn on diagnostic colors always, even when compiler output
does not go directly to terminal (e.g., when piped to "less -R").

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #9370 from majianpeng/bitmap-allocator

os/bluestore/BitMapAllocator: fix free space beyond size of BitAlloca…

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #8657 from rzarzynski/wip-rgw-swift-auth

rgw: authentication subsystem rework

Passed teuthology rgw suite.

Merge pull request #9532 from xiexingguo/xxg-wip-fix-csumcheck

os/bluestore: make verify_csum catch unsupported csum type error

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9492 from xiexingguo/xxg-wip-fix-bitmapallocator

bluestore/BitAllocator: miscellaneous fixes

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #9528 from stiopaa1/common_ceph_json_addConsToMember

common/ceph_json.h: add const to member function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9530 from majianpeng/bluestore-cleanup

os/bluestore/BlueStore: remove the duplicated code.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9549 from ifed01/wip-bluestore-large-blob-fix

os/bluestore: Fix for blob length on large writes

Reviewed-by: Sage Weil <sage@redhat.com>

bluestore: end scope of std::hex properly

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

bluestore: make verify_csum can catch unsupported csum type error

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cls_rbd: mirror image status summary should read full directory

Previously only retrieved the status for the first 64 images in
the rbd_mirroring directory.

Fixes: http://tracker.ceph.com/issues/16178
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/BlueStore: Fixes improper blob length assignment when creating a new blob for large writes

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/BlueStore: Adds checkes for lextent range overlaps to _fsck

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9533 from jjhuo/no_pagecache

os/bluestore: make rocksdb not to use page cache

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8233 from majianpeng/bluestore-perfcount

os/bluestore: perf counters

Reviewed-by: Sage Weil <sage@redhat.com>

tool/rbd: cleanup in utils.cc

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

librbd: flush journal commit positions before starting op

Ensure all IO has been properly flushed and committed to the
journal before starting an op that could affect the IO path.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: leaked AioCompletion if IO issued after image shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: resize and snap create can hang on journal replay failure

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: missing journal state for flushing/restarting replay

Fixes: http://tracker.ceph.com/issues/16077
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track IO in the order issued

Fixes: http://tracker.ceph.com/issues/15034
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: AioCompletion should retrieve CephContext from ImageCtx

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Journal no longer requires AioCompletion for IO events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: bubble journal error up via existing AioObjectRequests

The journal cannot directly fail the AioCompletion because it
might have one-or-more outstanding requests associated with
it.

Fixes: http://tracker.ceph.com/issues/15791
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore/BlueStore: remove the duplicated code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

client: fix sanity check of preadv/pwritev

The preadv/pwritev call is only supposed to return a postive
number indicating how many bytes we succeed in reading from
or writing into disk, which is obviously not what we want
under the following cases.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

configure.ac,debian,rpm: do not package .a files in rpm/deb packages

this change removes *.a from -dev or -devel packages.

* configure.ac:
  - use LT_INIT instead of AM_PROG_LIBTOOL. the later is deprecated.
  - disable static lib:
    .a static libraries not not useful, it's tricky to generate both
    static libraries and shared libraries in cmake.
    see http://www.spinics.net/lists/ceph-devel/msg30637.html
* ceph.spec.in: fedora packaging policy forbids this
  see
  https://fedoraproject.org/wiki/Packaging:Guidelines#Packaging_Static_Libraries
* debian/*-dev.install: debian does not requires this
  see
  https://www.debian.org/doc/debian-policy/ch-sharedlibs.html#s-sharedlibs-static

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: do not install *.la files

rationales:
* .la files are worthless
  see http://www.spinics.net/lists/ceph-devel/msg30637.html
* debian does not like it:
  see https://wiki.debian.org/ReleaseGoals/LAFileRemoval
* neither does fedora:
  https://fedoraproject.org/wiki/Packaging:Guidelines#Packaging_Static_Libraries
* also they are created by libtools, and cmake does not use libtools.

Signed-off-by: Kefu Chai <kchai@redhat.com>

makefile,debian,rpm: do not package librgw_file* anymore

they are unit tests for rgw.

Signed-off-by: Kefu Chai <kchai@redhat.com>

bluestore/BitAllocator: literal cleanups

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9296 from tchaikov/wip-test-chain_xattr

test: Fix the order of ASSERT_EQ() in chain_xattr tests

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #9361 from stiopaa1/osd_osd_override

osd/OSD.h: add override to virtual functions

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: potential duplicate snap removal can result in crash

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore/BlueFS: convert all read/write IO to direct IO.

RocksDB is the sole user of BlueFS, after BlueFS uses direct IO,
RocksDB will have to use its own block cache to cache raw data
blocks.

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>

os/bluestore: convert read_buffered() to read_random() with buffered flag

New function will handle unaligned random read for both buffered or
direct read.

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>

os/bluestore/BitAllocator: fix compiler warning.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BitMapAllocator: fix free space beyond size of BitAllocator

When Using bitmap allocator, the osd crashed. The stack info as
follows:

ceph-osd: os/bluestore/BitAllocator.cc:910: bool
BitAllocator::is_allocated(int64_t, int64_t): Assertion `start_block >=
0 && (start_block + num_blocks <= size())' failed.
*** Caught signal (Aborted) **
in thread 7f795bbc48c0 thread_name:ceph-osd ceph version
10.2.0-1333-g3f4cf16
(3f4cf16)
1: (()+0x98e25e) [0x55e112a9625e]
2: (()+0x109f0) [0x7f795a6719f0]
3: (gsignal()+0x38) [0x7f7958542a28]
4: (abort()+0x16a) [0x7f795854462a]
5: (()+0x2d227) [0x7f795853b227]
6: (()+0x2d2d2) [0x7f795853b2d2]
7: (()+0x7eaa59) [0x55e1128f2a59]
8: (BitAllocator::free_blocks(long, long)+0x22) [0x55e1128f2e82]
9: (BitMapAllocator::insert_free(unsigned long, unsigned long)+0x22c)
[0x55e1128ef12c]
10: (BitMapAllocator::init_add_free(unsigned long, unsigned long)+0x22c)
[0x55e1128ef41c]
11: (BlueFS::_init_alloc()+0x2b9) [0x55e1128c27b9]
12: (BlueFS::mkfs(uuid_d)+0x45a) [0x55e1128d475a]
13: (BlueStore::_open_db(bool)+0xd76) [0x55e1127bbcb6]
14: (BlueStore::mkfs()+0x8b1) [0x55e1127e4f11]
15: (OSD::mkfs(CephContext, ObjectStore, std::__cxx11::basic_string,
std::allocator > const&, uuid_d, int)+0x117) [0x55e112484de7]
16: (main()+0x101f) [0x55e112425f2f]
17: (__libc_start_main()+0xf0) [0x7f795852e580]
18: (_start()+0x29) [0x55e1124684e9]

This because in BitAllocator::init will decrease size of blkdev which
make size align w/ zone-size.
The later add free extent will beyond the size.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BlueStore: remove the unused perfcounter:l_bluestore_state_wal_done_lat

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #9527 from linuxbox2/rgw-ldap

rgw ldap: fix ldap bindpw parsing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw ldap: fix ldap bindpw parsing

Also add additional LDAP debugging output at 0, 10, and 15 to make
troubleshooting easier.

Fixes DN search issue using QE configuration of MS AD.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

doc: include link to CephFS' experimental features

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

RocksDBStore: add option to configure number of block cache shards

rocksdb_cache_size will be splitted into number of shards evenly.
More shards will reduce lock contentions.

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>

common/ceph_json.h: add const to member function

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>