os/bluestore/bluefs: drop redundant eof check for read_random()

Because we have done the sanity check and fixed the length
at the entrance, so we don't have to check it again,
especially in the while loop, which is less efficient.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/bluefs: update mtime of inode in unified path

To reduce code redundancy.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/bluefs: use map to track dirty files

So dirty files are arranged by dirty_seq and then
those of same dirty_seq are grouped into list and
as a result we can clean up dirty files much faster
during _flush_and_sync_log().

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10158 from Ved-vampir/zlib_isal_extension

Isa-l extention for zlib compression plugin

Merge pull request #10858 from atheism/rgw_getattr-return

rgw: remove duplicated calls to getattr

verified: f23

Merge pull request #10927 from liewegas/wip-0.94.9

doc/release-notes: v0.94.9

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc/release-notes: v0.94.9

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10745 from linuxbox2/rgw-nfs-31

rgw nfs v3 completions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10926 from ifed01/wip-bluestore-fix-statfs-test

os/bluestore: yet another statfs test fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10875 from athanatos/wip-16908

Revert "osd/ReplicatedPG: for sync-read it don't cacl l_osd_op_r_prep…

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Revert "osd/ReplicatedPG: for sync-read it don't cacl l_osd_op_r_prepare_lat."

Accessing the OpContext* after execute_ctx is invalid, on error, it
will have been deleted.

This reverts commit 52be772788d9d96accaa7af9eaf9f29a3792df49.

Fixes: http://tracker.ceph.com/issues/16908
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #10881 from ceph/wip-kill-warnings

Wip kill warnings

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: yet another statfs test fix

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10917 from ceph/wip-cmake

cmake changes related to LTTng-UST

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10768 from liewegas/wip-bluestore-blobbuffers

os/bluestore: avoid blob_t reencode when unchanged

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10910 from liewegas/wip-bluestore-statfs

os/bluestore: fix statfs tests

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10017 from jan--f/os-release-based-install-deps

build/ops: install-deps.sh based on /etc/os-release

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

cmake: let librados_api_obj depend on librados-tp

so we don't need to messing up with the target names.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: replace Findlttng-ust.cmake with FindLTTngUST.cmake

find_package(lttng-ust REQUIRED) fails to find the lttng library without
this change. because find_path(LTTNG_LIBRARY_DIR ...) does not search
in the default library paths. and the second mode of
FIND_PACKAGE_HANDLE_STANDARD_ARGS() does not stop the cmake with a fatal
message, even some of the required vars are missing. so use the
implemetantion from cmake upstream.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove unused librados_api target

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9930 from yuyuyu101/wip-16472

msg/simple: wait dispatch_queue until all pipes closed

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #9121 from dillaman/wip-15688

librbd: initial hooks for client-side, image-extent cache in IO path

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph_test_objectstore: disable statfs checks on available space

These are too sensitive to other activity, like bluefs
files and bluefs vs bluestore freespace balancing.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix ObjectStore/StoreTest.BluestoreStatFSTest/2 on ssd

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10897 from batrick/caps-graphic

doc: add graphic for cap bit field

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

install-deps.sh: initial distro detection based on /etc/os-release

This avoids initial distribution detection (before lsb_release is available)
based on the presence of a specific package manager. This caused an
issue with openSUSE Tumbleweed before since both zypper and apt-get are
available.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #10863 from xiexingguo/xxg-wip-fix-bfs-08-25

os/bluestore/bluefs: add file refs check

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10280 from rzarzynski/wip-rgw-swift-at-root-features

rgw: add suport for Swift-at-root dependent features of Swift API

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #10727 from weiqiaomiao/wqm-wip-checkout-ret

rgw: fix RGWAccessControlPolicy_SWIFT::create return value check error

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #10902 from stiopaa1/remove_unneeded

common/Throttle.h: remove unneeded class

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10906 from ceph/wip-cmake

cmake: fix the tracing header dependencies

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9301 from objoo/YD-mailmap-updates-MAY3

mailmap updates for upcoming 11.0.0

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

cmake: fix the tracing header dependencies

in aa679df, we move librados.cc into a separated obj target. this broken
the dependency from librados.o to its tracing header. we need to update
it accordingly.

this change fixes the following compilation error:
```
/var/ceph/ceph/src/librados/librados.cc:47:30: fatal error:
tracing/librados.h: No such file or directory
 #include "tracing/librados.h"
                              ^
compilation terminated.
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10878 from stiopaa1/mds_snaprealm_addconstToMem

mds/SnapRealm.h: add const to member functions

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10901 from stiopaa1/mds_mdcache_accessspecifierU

mds/MDCache.h: remove unneeded access specifier

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10894 from smithfarm/wip-suse-build-pycache

build/ops: force Python 3 packages to build in SUSE

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #10903 from SUSE/wip-release-notes-correct-correction

doc: fix some nits in release notes and releases table

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add 11.0.0 to releases table

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: fix two broken links in releases.rst

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: release-notes.rst: move misplaced line

46264a3b6cb7f490680c79bf0957c36aeaea5f6c added the missing line to the wrong
section. This commit moves it to the right section (v0.94.8).

Signed-off-by: Nathan Cutler <ncutler@suse.com>

common/Throttle.h: remove unneeded class

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

librbd: integrate image cache hooks into IO path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: C_ImageCacheRead bridge from Context to AioCompletion

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: differentiate between image and object cache

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: consolidate IO clipping to reusable method

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: initial integration of ImageWriteback to AioImageRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add internal support for scatter/gather IO

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support bufferlist payload for AioImageWrite

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Revert "osdc: ObjectCacher flusher might needs additional locks"

This reverts commit a38f9e5104a6e08e130dc4f15ad19a06d9e63719.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove use of owner_lock on IO path

IO is fully flushed before releasing the exclusive lock so holding
the owner_lock isn't required.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: blocking parent IO completion no longer required

The completion will directly invoke the callback with the
result code.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: skeleton implementation of client-side image cache

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10899 from SUSE/wip-retirement

doc: do not list all major versions in get-packages.rst

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10900 from SUSE/wip-hammer-release-notes-correction

doc: add missing PR to hammer 0.94.8 release notes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds/MDCache.h: remove unneeded access specifier

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

doc: add missing PR to hammer 0.94.8 release notes

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: do not list all major versions in get-packages.rst

The list of major versions is difficult to maintain. This commit drops it and
replaces it with a link to releases.rst plus some general language about how we
recommend that everyone keep their clusters up-to-date.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10895 from liewegas/wip-changelog

doc: fix/add changelog for 10.2.2, 0.94.7, 0.94.8

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10844 from liewegas/wip-bluestore-fsck-leak

os/bluestore: fix when block device is not a multiple of the block size

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add graphic for cap bit field

Thanks to Jeff for putting this doc together!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

unittest_bluefs: fix block device sizes

These should be a multiple of the block size (since we are feeding
the size directly into add_block_extent).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10779 from cbodley/wip-17073

rgw multisite: RGWDataSyncCR fails on errors from RGWListBucketIndexesCR

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10778 from cbodley/wip-17047

rgw multisite: RGWCoroutinesManager::run returns status of last cr

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10742 from cbodley/wip-rgw-test-multi-gateway

test/rgw: add --gateways-per-zone to test_multi.py

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10837 from ceph/wip-cmake

cmake: do not link against unused objects or libraries

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

doc/release-notes: fix v10.2.2 changelog link

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases: add v0.94.8 to timeline

Signed-off-by: Sage Weil <sage@redhat.com>

doc: add v0.94.8 changelog

Signed-off-by: Sage Weil <sage@redhat.com>

doc: add v0.94.7 changelog

Signed-off-by: Sage Weil <sage@redhat.com>

rpm: force creation of Python 3 __pycache__ on SUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Dominique Leuenberger <dimstar@opensuse.org>

Revert "ceph.spec.in: don't try to package __pycache__ for SUSE"

This reverts commit be39d6dc71af13eb3ee0d4482012a2091d121f66.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10547 from wjwithagen/wip-wjw-test-util

test/common/test_util.cc: FreeBSD does not have distro information

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9869 from yuyuyu101/wip-kqueue

msg/async/kqueue: avoid remove nonexist kqueue event

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10879 from dmick/wip-jerasure-aarch64

jerasure: include generic objects in neon jerasure lib (like sse3/4)

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/unittest_alloc: fix -Wsign-compare warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/ceph_test_msgr: fix -Wsign-compare warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: silence -Wsign-compare warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: should dh_python3 python3-* not python-*

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: remove duplicated calls to getattr

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

Merge pull request #9943 from ceph/wip-ceph-disk-systemd

ceph-disk: do not activate device that is not ready

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10828 from michelmno/perf_local_flags

cmake: use PERF_LOCAL_FLAGS only if defined

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/kstore: kill dead code

These are not applicable anymore.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/bluefs: assert refs of file don't underflow

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/bluefs: check against locked field during unlink

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/bluefs: avoid check "locked" field for newly created file

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds/SnapRealm.h: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #9757 from SUSE/wip-16353

rpm: drop obsolete libs-compat and python-ceph-compat metapackages

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #10730 from SUSE/wip-release-notes-draft

release-notes.rst: draft 0.94.8 release notes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10805 from SUSE/wip-dont-package-__pycache__

ceph.spec.in: don't try to package __pycache__ for SUSE

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Dirk Mueller <dmueller@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

jerasure: include generic objects in neon jerasure lib (like sse3/4)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #10577 from wjwithagen/wip-wjw-test-rados-striper

Rados-stripper: Flexible string matching for not found attributes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10770 from dillaman/wip-16773

librbd: interlock image refresh and exclusive lock operations

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10831 from cbodley/wip-17051

rgw: delete region map after upgrade to zonegroup map
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore: decode blobs on demand

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: avoid reencoding unchanged blobs

Keep a dirty flag on each blob, set by the accessor.

Cache the encoded blob_t and only reencode it if it was
modified.

It improves 4K random IOPS on my nvme vstart cluster
by about 9%.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use Blob const and non-const accessors

get_blob() returns a const ref, dirty_blob() returns a
non-const ref.  No change in behavior yet.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: make blob_t get_chunk_size const

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: -b for bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix some #includes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ensure block device size is a multiple of the block size

We might have a backing device that is an odd number of 512-byte sectors
but have the block_size configured to 4096.  Ensure the reported size
rounds down to avoid confusing other layers of the stack.

Signed-off-by: Sage Weil <sage@redhat.com>