os/bluestore/BlockDevice: disable readahead of fd_buffered

Readahead can break our mix of aio/dio and buffered io.  Although the
dio path will invalidate pages before and after initiating the IO, and we
do not issue racing buffered reads, the readahead code might.  That would
result in pages in the page cache that are not coherent with an aio write.

This is easy to reproduce with ceph_test_objectstore's Synthetic case
when bluestore_default_buffered_read=true.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_rocksdb_option_static: fix signed/unsigned warnings

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add flag to make reads default to buffered

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: fix fiemap

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: fix split_collection to persist cnode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix split_collection to persist cnode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simplify rebalance_freespace

- simplify tunables (min/max ratio of freespace between bluefs vs bluestore)
- add reclaim support

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: add reclaim_blocks interface

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: base initial bluefs allocation on min_alloc_ratio

Simplify config, and avoid immediate gift or reclaim when it doesn't match.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5882 from aiicore/add_thread_names

common: add thread names

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7053 from xiaoxichen/fix_preboot

osd: check health state before pre_booting

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7059 from xiexingguo/xxg-wip-14187

osd: misc FileStore fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7083 from cxwshawn/wip-pg_stat-fix

mon: PGMonitor: acting primary diff with cur_stat, should not set pg to stale

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7092 from kylinstorage/cleanup_PGLog

osd: PGLog: clean up read_log

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7096 from H3C/wip-yrf-message-leak

msg/simple: pipe: memory leak when signature check failed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7107 from xiexingguo/xxg-wip-14227

osd: small fixes to memstore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7125 from mslovy/wip-fix-bluestore-onode-attr-leak

osd: bluestore: fix bluestore onode_t attr leak

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7130 from liewegas/wip-bluestore

osd: bluestore: more fixes

Merge pull request #7168 from tchaikov/wip-fix-bluestore-test

osd: bluestore: fix bluestore_wal_transaction_t encoding test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7189 from dachary/wip-mailmap

mailmap updates

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #7021 from tchaikov/wip-rados-manpage

man: document listwatchers cmd in "rados" manpage

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Dongmao Zhang affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: sort files

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7173 from thesues/master

Change organization for Dongmao Zhang

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7132 from jcsp/wip-14144-jcsp

mds: fix standby replay thread creation

Merge pull request #7072 from dllgzcl/wip-zcl-ceph-s

mon/PGMap: show rd/wr iops separately in status reports

Reviewed-by: Kefu Chai <kchai@redhat.com>

Change organization for Dongmao Zhang

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>

os/bluestore: fix bluestore_wal_transaction_t encoding test

bluestore_wal_transaction_t::seq is encoded/dumped but it is not
initialized in its ctor. so set it to 0 in its ctor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7126 from tchaikov/wip-detect-bzip2-lz4

cmake: detect bzip2 and lz4

Merge pull request #7166 from kylinstorage/wip-rbdnbd-misc

rbd-nbd: add copyright

os/bluestore: fix allocation vs reservation for bluefs extents

- drop the MIN cruft
- unreserve what we didn't get

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bluestore_debug_prefill = <fill ratio>

Prefill the objectstore by fragmenting the freespace.  Max free allocations
are bluestore_debug_prefragment_max bytes (default 1M).  Prefill defaults
to 0, of course.  One would normally just set this to .8 to get something
resembling an aged disk.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7161 from liewegas/wip-port

test/mon/osd-erasure-code-profile: pick new mon port

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: pad trailing zeros for WAL write past eof

If we're past EOF, we need to zero the rest of the block.

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: make fiemap behave past EOF

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make fiemap return valid result beyond EOF

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: debug bluestore = 30

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/RadosModel: fix sparse read result check

The object may have trailing zeros.

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: implement minimal fiemap

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove #warning cruft

These are on my todo list elsewhere.

Signed-off-by: Sage Weil <sage@redhat.com>

os/fs: make aio EAGAIN backoff exponential

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: reduce aio queue size

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: clean up _do_read debug output

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix _compact_log seq jump

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: make pread() safe

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix fiemap

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unused TransContext mutex and cond

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: quieter default log level

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: make flush a no-op if there was no IO

fdatasync(2) does a cache flush on the device, which we want to avoid if
we didn't actually write anything.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix fiemap on object with no data

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: initiate bufferd writes at aio_write time

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: buffered writes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: no lock on read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: do not wait for IO on flush

Wait on fsync and _flush_log only.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: read_random

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: read_buffered

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: keep bool random in FileReader

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: reserve first block for label when bluefs = false

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix bdev label error message

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: send all old bluefs IOContexts for reap by bdev

Doing it in the aio thread ensures that there isn't a use-after-free.
Note that this is only bluefs written files.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix valgrind warning on fsid read

Signed-off-by: Sage Weil <sage@redhat.com>

os/keyvaluestore: fix valgrind warning on fsid read

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: fix valgrind warning on fsid read

Signed-off-by: Sage Weil <sage@redhat.com>

test/mon/osd-erasure-code-profile: pick new mon port

7108 is used by mark-down.sh

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6911 from stiopaa1/const_variable_length

tests: add const for ec test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6989 from stiopaa1/message_inline

msg: remove unneeded inline

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7160 from yehudasa/wip-rgw-warn

rgw: fix compilation warning

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix compilation warning

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7149 from fullerdj/wip-14280

cls_rbd: enable object map checksums for object_map_save

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7154 from liewegas/wip-make-check

unittest_ipaddr: fix segv

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

unittest_ipaddr: fix segv

Since b6d0fc9e0e515e50894c08217d688a8c94db7570 we need ifa_name to
be defined.

Signed-off-by: Sage Weil <sage@redhat.com>

Add copyright for rbd-nbd

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

Merge pull request #6002 from Sandy4999/wip-12890

rgw: add a method to purge all associate keys when removing a subuser

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6080 from clever215/wkm-master-139

rgw: modify documents and help infos' descriptions to the usage of option date when executing command "log show"

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4216 from rzarzynski/wip-10682

rgw: add support for "end_marker" parameter for GET on Swift account.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6652 from ifed01/wip-13824

rgw_admin: orphans finish segfaults

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7037 from mathslinux/rgw-add-missing-error-code

rgw: add missing error code for admin op API

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7093 from dwj192/dwj-bucket-index-fix

rgw: radosgw-admin bucket check --fix not work

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6817 from david-z/wip-fix-init-radosgw

rgw: fix a typo in init-radosgw

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7102 from dzafman/wip-cot-fixes

Fix for make check.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7153 from liewegas/wip-make-check

build: misc make check fixes

Merge pull request #7152 from liewegas/wip-hexdump

unittest_bufferlist: fix hexdump test

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

unittest_bluefs: call close_for_write()

Signed-off-by: Sage Weil <sage@redhat.com>

test/test_objectstore_memstore.sh: clear store_test_temp_dir

ceph_test_objectstore now expects this to not exist.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bufferlist: fix hexdump test

Signed-off-by: Sage Weil <sage@redhat.com>

cls_rbd: enable object map checksums for object_map_save

object_map_save disables CRCs when an object map footer isn't provided.
Unconditionally re-enable object map CRCs before re-encoding the new object
map.

Fixes: #14280
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #6955 from dachary/wip-dev-guide-ceph-qa-suite

doc: dev: document ceph-qa-suite

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

doc: dev: document ceph-qa-suite

Signed-off-by: Loic Dachary <loic@dachary.org>

cmake: detect bzip2 and lz4 for rocksdb

* this change is the cmake port of 911e7a0.
  the rocksdb's Makefile will detect the installed libbz2 and libz4 by
  its own. if the building env happens to have these libraries installed,
  a link time dependency is introduced. so we are forced to link against
  them.
* do not REQUIRE BZip2 in cmake anymore
* only link against bzip2 if it is detected

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add check for liblz4

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7131 from ceph/wip-cmake

cmake: made rocksdb an imported library

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7073 from dllgzcl/wip-zcl-show-version

tools: monstore: add 'show-versions' command.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

cmake: made rocksdb an imported library

Systems with the minimum required version of CMake
such as Ubuntu Trusty Tahr don't support
INTERFACE_LINK_LIBRARIES, so rocksdb was made into
an IMPORTED library. Added ROCKSDB_INCLUDE_DIR to
simplify target_include_directories rocksdb.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge remote-tracking branch 'origin/jewel'

Merge pull request #7129 from dillaman/wip-14167-jewel

librbd: disable copy-on-read when not exclusive lock owner

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6886 from ssssztes/renhw-wip-14047

rbd: rbd order will be place in 22, when set to 0 in the config_opt

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/jewel'

Merge branch 'wip-14241-jewel' of git://github.com/dillaman/ceph into jewel

librbd: don't use write_full optimization with clones

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7127 from trociny/wip-rbd-nbd-misc

rbd-nbd: small improvements in logging and forking

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6388 from ukernel/wip-13583

ceph-fuse: fix fsync()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>