osd/OSD.cc: remove unneeded searching of map

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10981 from SUSE/wip-fix-install-deps

build/ops: fix /etc/os-release parsing in install-deps.sh

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

build/ops: fix /etc/os-release parsing in install-deps.sh

85a370e35fc42031a7f7e24dea9e50a649c0f309 introduced the DISTRO variable whose
value was obtained by parsing /etc/os-release like this:

DISTRO=$(grep  "^ID=" /etc/os-release | sed "s/ID=//")

This unfortunately picks up the double-quotes, so on a CentOS system DISTRO
will be equal to '"centos"'.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10969 from xiexingguo/xxg-wip-fix-cache-logger

os/bluestore: replace store with logger in Cache

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9592 from trociny/wip-15632

librbd: API methods to directly acquire and release the exclusive lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9758 from SUSE/wip-btrfs-not-ideal

doc: remove btrfs contradiction

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10469 from wjwithagen/wip-wjw-compileflag

cmake: supress more warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: check iterator before using during fsck

Also drop redundant checking.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: replace store with logger in Cache

As what we really want is to log cache-related statistics.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10945 from dillaman/wip-17188

librbd: deadlock when replaying journal during image open

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10746 from ceph/wip-async-default

msg: make async backend default

Merge pull request #10938 from liewegas/wip-bluestore-csum-error

os/bluestore: fix/improve csum error message

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10961 from xiexingguo/xxg-wip-bluestore-09-02

os/bluestore: add cache-related stats

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10953 from liewegas/wip-bluestore-fixes

os/bluestore: misc fixes

Reviewed- by: Igor Fedotov ifedotov@mirantis.com

os/bluestore: trivial cleanups

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: don't overwrite type if it does already exist and at desired type

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

src/CMakeLists.txt: -Woverloaded-virtual is already included in -Wall on Clang

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10956 from ceph/wip-cmake

cmake: pass -DINTEL* to gf-complete cflags

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: conditional checking label of bluefs devices

As NVMe devices may not support to do so.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: pass -DINTEL* to gf-complete cflags

* in addition to the -march=foobar, we also need to pass -DFOOBAR to
  gf-complete sources' cflags, so that the SIMD instructions can be
  generated as expected.
* also extract the compiler support for instruction extensions detections
  into SIMDExt.cmake.
* and only check the compiler support if CMAKE_SYSTEM_PROCESSOR matches
  with the instruction-set/arch to be checked.

Signed-off-by: tone.zhang <tone.zhang@linaro.org>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10934 from tchaikov/wip-kill-warnings

Wip kill warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

os/bluestore: end scope of std::hex properly

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add onode/buffer cache statistics

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10517 from wjwithagen/wip-wjw-freebsd-cmake-execludes

cmake: FreeBSD specific excludes in CMakeLists.txt files

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph_test_objectstore: fix TryMoveRename test

Put the objects in the right pool.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: clean up Rename test a bit

Drop useless remove calls.

Leave in the no-op setattr call since it triggered a
bluestore bug :)

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: assert umount succeeds

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make object existence check robust

We may have the OnodeRef defined but the object may have
exists==false.  Move the check out of the lookup block
so that we get the right behavior.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use _setattr for SETATTR

This is a bit faster.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10843 from liewegas/wip-osdmap-reencode

messages/MOSDMap: mark as enlighten OSDMap encoder

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10950 from xiexingguo/xxg-wip-blue-09-01

os/bluestore: gifting bluefs more carefully

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: don't over-gift if bluefs free is smaller than bluestore_bluefs_min

- If free space of bluefs is smaller than bluestore_bluefs_min, then raise it
  exactly up to bluestore_bluefs_min.

- Pre-check if bluestore_bluefs_min will exceed bluestore_bluefs_max_ratio * bluefs_total,
  so if free space of bluefs is smaller than bluestore_bluefs_min_ratio * bluefs_total,
  we'll still be able to gift.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: end scope of std::hex properly

So it does not extend to the following output stream and
causes unexpected behaviour.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

librbd: deadlock when replaying journal during image open

Fixes: http://tracker.ceph.com/issues/17188
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: improve image state machine debug log messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10404 from ilc/build_fix

make-dist/ceph.spec.in: Fix srpm build breakage.

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #10846 from stiopaa1/mds_mdstable_addConstToMemFun

mds/MDSTable: add const to member functions

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10781 from stiopaa1/osdc_objectcacher_moveClassesToCC

osdc/ObjectCacher: move C_ReadFinish, C_RetryRead

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10744 from stiopaa1/osdc_journaler_moveCDelayFlushClass

osdc/Journaler: move C_DelayFlush class to .cc

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10668 from stiopaa1/mon_mdsmonitor_moveCUpdatedClass

mon/MDSMonitor: move C_Updated class to .cc file

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10628 from jcsp/wip-16909

mon: tear down standby replays on MDS rank stop

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10937 from ceph/dang-build

Allow compressor build without YASM

Reviewed-by: Ali Maredia <amaredia@redhat.com>

os/bluestore: improve failed csum debug output

Signed-off-by: Sage Weil <sage@redhat.com>

script/bdev_grep: update to use hex

Signed-off-by: Sage Weil <sage@redhat.com>

Allow compressor build without YASM

The isa-l compressors need YASM to build.  Conditionalize on that.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #10931 from xiexingguo/xxg-wip-fix-isa-l-assert

os/bluestore: get rid off "isa-l" type in ZLibCompressor ctor

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10923 from xiexingguo/xxg-wip-bluefs-08-30

os/bluestore/bluefs: use map to track dirty files

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: get rid off "isa-l" type in ZLibCompressor ctor

This is introduced by https://github.com/ceph/ceph/pull/10158 and causes
the following failure:

/home/xxg/ceph-dev/src/os/bluestore/bluestore_types.h: 284: FAILED assert(0 == "invalid compression algorithm")

 ceph version v11.0.0-1941-g47c8a21 (47c8a214356983f0539f24fe1c5724b3bbae0406)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f5668c1ad45]
 2: (BlueStore::_do_alloc_write(BlueStore::TransContext*, BlueStore::WriteContext*)+0x1a65) [0x7f5668b19d55]
 3: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0x4ac) [0x7f5668b3b3fc]
 4: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0x10f) [0x7f5668b3bf2f]
 5: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x1067) [0x7f5668b405f7]
 6: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, std::shared_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x382) [0x7f5668b42432]
 7: (SyntheticWorkloadState::write()+0x3a5) [0x7f5668a2ace5]
 8: (doSyntheticTest(boost::scoped_ptr<ObjectStore>&, unsigned long, unsigned long, unsigned long)+0xb9d) [0x7f56689dfa7d]
 9: (do_matrix_choose(char const* (*) [10], int, int, int, boost::scoped_ptr<ObjectStore>&)+0x29d) [0x7f56689dfffd]
 10: (do_matrix_choose(char const* (*) [10], int, int, int, boost::scoped_ptr<ObjectStore>&)+0x9e) [0x7f56689dfdfe]
 11: (do_matrix_choose(char const* (*) [10], int, int, int, boost::scoped_ptr<ObjectStore>&)+0x9e) [0x7f56689dfdfe]
 12: (do_matrix_choose(char const* (*) [10], int, int, int, boost::scoped_ptr<ObjectStore>&)+0x9e) [0x7f56689dfdfe]
 13: (do_matrix_choose(char const* (*) [10], int, int, int, boost::scoped_ptr<ObjectStore>&)+0x9e) [0x7f56689dfdfe]
 14: (do_matrix_choose(char const* (*) [10], int, int, int, boost::scoped_ptr<ObjectStore>&)+0x9e) [0x7f56689dfdfe]
 15: (do_matrix(char const* (*) [10], boost::scoped_ptr<ObjectStore>&)+0x317) [0x7f56689e0347]
 16: (()+0x297f3c) [0x7f56689e0f3c]
 17: (StoreTest_SyntheticMatrixCompressionAlgorithm_Test::TestBody()+0x118) [0x7f56689e1078]
 18: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x43) [0x7f5668d673e3]
 19: (testing::Test::Run()+0xb7) [0x7f5668d5a127]
 20: (testing::TestInfo::Run()+0x9e) [0x7f5668d5a1ce]
 21: (testing::TestCase::Run()+0xa5) [0x7f5668d5a2d5]
 22: (testing::internal::UnitTestImpl::RunAllTests()+0x278) [0x7f5668d5a5b8]
 23: (testing::UnitTest::Run()+0x68) [0x7f5668d5a888]
 24: (main()+0x2f4) [0x7f566895be84]
 25: (__libc_start_main()+0xf5) [0x7f5663e5ab15]
 26: (()+0x265ba5) [0x7f56689aeba5]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/perf_local: fix -Wmaybe-uninitialized warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

test,pybind: silence -Wsign-compare warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/CMakeLists.txt: undubble kv.cc usage.

 - kv.cc is both in bluestore and in kstore.
   But atm. both are the same
 - Bluestore is not build on FreeBSD (no AIO)

So use the kstore one to be the one to build against.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

os/bluestore/bluefs: drop redundant eof check for read_random()

Because we have done the sanity check and fixed the length
at the entrance, so we don't have to check it again,
especially in the while loop, which is less efficient.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/bluefs: update mtime of inode in unified path

To reduce code redundancy.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/bluefs: use map to track dirty files

So dirty files are arranged by dirty_seq and then
those of same dirty_seq are grouped into list and
as a result we can clean up dirty files much faster
during _flush_and_sync_log().

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10158 from Ved-vampir/zlib_isal_extension

Isa-l extention for zlib compression plugin

Merge pull request #10858 from atheism/rgw_getattr-return

rgw: remove duplicated calls to getattr

verified: f23

Merge pull request #10927 from liewegas/wip-0.94.9

doc/release-notes: v0.94.9

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc/release-notes: v0.94.9

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10745 from linuxbox2/rgw-nfs-31

rgw nfs v3 completions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10926 from ifed01/wip-bluestore-fix-statfs-test

os/bluestore: yet another statfs test fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10875 from athanatos/wip-16908

Revert "osd/ReplicatedPG: for sync-read it don't cacl l_osd_op_r_prep…

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Revert "osd/ReplicatedPG: for sync-read it don't cacl l_osd_op_r_prepare_lat."

Accessing the OpContext* after execute_ctx is invalid, on error, it
will have been deleted.

This reverts commit 52be772788d9d96accaa7af9eaf9f29a3792df49.

Fixes: http://tracker.ceph.com/issues/16908
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #10881 from ceph/wip-kill-warnings

Wip kill warnings

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: yet another statfs test fix

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10917 from ceph/wip-cmake

cmake changes related to LTTng-UST

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10768 from liewegas/wip-bluestore-blobbuffers

os/bluestore: avoid blob_t reencode when unchanged

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10910 from liewegas/wip-bluestore-statfs

os/bluestore: fix statfs tests

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10017 from jan--f/os-release-based-install-deps

build/ops: install-deps.sh based on /etc/os-release

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

cmake: let librados_api_obj depend on librados-tp

so we don't need to messing up with the target names.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: replace Findlttng-ust.cmake with FindLTTngUST.cmake

find_package(lttng-ust REQUIRED) fails to find the lttng library without
this change. because find_path(LTTNG_LIBRARY_DIR ...) does not search
in the default library paths. and the second mode of
FIND_PACKAGE_HANDLE_STANDARD_ARGS() does not stop the cmake with a fatal
message, even some of the required vars are missing. so use the
implemetantion from cmake upstream.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove unused librados_api target

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9930 from yuyuyu101/wip-16472

msg/simple: wait dispatch_queue until all pipes closed

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

messages/MOSDMap: mark as enlighten OSDMap encoder

Back in 70094407c184baaf963577cc0d98edde12e8e9da we require that anyone
encoding an OSDMap pass in a special feature bit indicating that they are
'enlightened' and understand the rules around encoding OSDMaps (basically,
only mons get to do it).  We forgot to update MOSDMap, which may have to
reencode an OSDMap without some features to talk to a really old client.
We haven't noticed until now because the old set of features we had to do
this for are really old.

We are about to introduce new features (for addr2 encoding) that change
that, and this fix is needed to prevent an assert when doing a reencode.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9121 from dillaman/wip-15688

librbd: initial hooks for client-side, image-extent cache in IO path

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph_test_objectstore: disable statfs checks on available space

These are too sensitive to other activity, like bluefs
files and bluefs vs bluestore freespace balancing.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix ObjectStore/StoreTest.BluestoreStatFSTest/2 on ssd

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10897 from batrick/caps-graphic

doc: add graphic for cap bit field

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

install-deps.sh: initial distro detection based on /etc/os-release

This avoids initial distribution detection (before lsb_release is available)
based on the presence of a specific package manager. This caused an
issue with openSUSE Tumbleweed before since both zypper and apt-get are
available.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

src/test/common/CMakeLists.txt: BLKID exclude

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/test/CMakeLists.txt: RADOSGW|RBD exclude

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/os/CMakeLists.txt: AIO|FUSE exclude

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/librbd/CMakeLists.txt: krbd and secret.c are LINUX dependant

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/CMakeLists.txt: RBD|KVS|FUSE exclude

 - krbd is the Linux kernel RBD driver, and that is not going to be
   ported to FreeBSD in its current state. If anything it will be
   a module based on a derivative  of gated.
 - The current secret.c depends on keyutils
   Which is an interesting concept, but still needs to be ported to
   FreeBSD one way or another.
 - ceph-dencoder can live without these modules on FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/test/bench/CMakeLists.txt: RBD exclude

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10863 from xiexingguo/xxg-wip-fix-bfs-08-25

os/bluestore/bluefs: add file refs check

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10280 from rzarzynski/wip-rgw-swift-at-root-features

rgw: add suport for Swift-at-root dependent features of Swift API

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #10727 from weiqiaomiao/wqm-wip-checkout-ret

rgw: fix RGWAccessControlPolicy_SWIFT::create return value check error

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #10902 from stiopaa1/remove_unneeded

common/Throttle.h: remove unneeded class

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10906 from ceph/wip-cmake

cmake: fix the tracing header dependencies

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9301 from objoo/YD-mailmap-updates-MAY3

mailmap updates for upcoming 11.0.0

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

cmake: fix the tracing header dependencies

in aa679df, we move librados.cc into a separated obj target. this broken
the dependency from librados.o to its tracing header. we need to update
it accordingly.

this change fixes the following compilation error:
```
/var/ceph/ceph/src/librados/librados.cc:47:30: fatal error:
tracing/librados.h: No such file or directory
 #include "tracing/librados.h"
                              ^
compilation terminated.
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10878 from stiopaa1/mds_snaprealm_addconstToMem

mds/SnapRealm.h: add const to member functions

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10901 from stiopaa1/mds_mdcache_accessspecifierU

mds/MDCache.h: remove unneeded access specifier

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10894 from smithfarm/wip-suse-build-pycache

build/ops: force Python 3 packages to build in SUSE

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #10903 from SUSE/wip-release-notes-correct-correction

doc: fix some nits in release notes and releases table

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add 11.0.0 to releases table

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: fix two broken links in releases.rst

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: release-notes.rst: move misplaced line

46264a3b6cb7f490680c79bf0957c36aeaea5f6c added the missing line to the wrong
section. This commit moves it to the right section (v0.94.8).

Signed-off-by: Nathan Cutler <ncutler@suse.com>

common/Throttle.h: remove unneeded class

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

librbd: integrate image cache hooks into IO path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>