cmake: udev linking and Findudev module changes

Linked udev into librbd, restored UDEV_I to
UDEV_INCLUDE_DIR in Findudev.cmake

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Findxio.cmake update Xio OFF by default

Option WITH_XIO now OFF but required by default,
Findxio.cmake now using
FIND_PACKAGE_HANDLE_STANDARD_ARGS.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Check for required Cython package

Added a FindCython.cmake, Cython is required.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Added cls_numops_client library

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: test/erasure-code CMake additions

Added missing executables and libraries.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: test CMake additions

Added missing tests, changed linking, among
other changes.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: src/erasure-code CMakeLists.txt's changes

Added ec_neon libraries in jerasure and shec dirs,
reorganized checks for flags, other minor
erasure-code directory changes.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Missing binaries from src/tools added

Added ceph-objecstore-tool, ceph-kvstore-tool,
ceph-psim, ceph-client-debug, ceph-scratchtool,
ceph-scratchtoolpp, ceph-radosacl,
ceph-osdomap-tool, ceph-monmap-tool from src/tools
Makefiles.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: CMAKE_DL_LIBS is not linked into osdmaptool

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: libradosstriper dir CMakeLists.txt added

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Added librados_api library

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Added common/Cycles.cc to libcommon

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Boost libraries cleanup

Added components to Boost find_package, changed
libraries from boost_foo to ${Boost_FOO_LIBRARY}

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: add detection for HAVE_CURL_MULTI_WAIT

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: rbd_replay CMakeLists.txt, rbd_replay fixes

moved rbd_replay targets into it's directory,
rbd-replay-prep added, removed "lib" in front of
all rbd_replay libraries, linked rbd_replay into
ceph-dencoder.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Babeltrace detection supported

Babeltrace defaults to ON, Findbabeltrace.cmake &
BABELTRACE variables in config-h.in.cmake added.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: LTTng functionality added

Set LTTNG option to ON by default, added
Findlttng.cmake, added CMakeLists.txt in
src/tracing that generates .h files using
lttng-gen-tp, added WITH_LTTNG to
config-h.in.cmake.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Updated allocator checking

Added a FindJeMalloc.cmake file, changed
allocator checking logic (tcmalloc is default),
replaced TCMALLOC_LIBS to ALLOC_LIBS, added
HAVE_LIB(ALLOCATOR) to config-h.in.cmake.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: kv dir and bluestore unittests

Added CMakeLists.txt in src/kv, replaced
os_mon_objs with kv_objs, linked libkv to os,
added HAVE_LIBROCKSDB to config-h.in.cmake,
tweaked rocksdb unittest, added bluestore
unittests, made rocksdb build with
add_custom_target instead of custom_command.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge remote-tracking branch 'origin/jewel'

Merge pull request #7114 from dillaman/wip-14234-jewel

librbd: reduce verbosity of common error condition logging

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'wip-14237-jewel' of git://github.com/dillaman/ceph into jewel

qa/workunits/rbd: import_export should use clean temp space

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7113 from dillaman/wip-14168-jewel

journal: disconnect watch after watch error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/rbd: import_export should use clean temp space

Fixes: #14237
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reduce verbosity of common error condition logging

Fixes: #14234
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7046 from clever215/merge_diff_1224

rbd: correct an output string for merge-diff

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6975 from yehudasa/wip-rgw-content-length

Wip rgw content length

journal: disconnect watch after watch error

Fixes: #14168
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: correct an output string for merge-diff

This modification correct an output string for rbd merge-diff, in which mistakenly output variables of first diff header rather than second diff header.

Signed-off-by: Kongming Wu <wu.kongming@h3c.com>

Merge remote-tracking branch 'origin/jewel'

Merge pull request #6669 from dzafman/omap-small-bugs

Omap small bugs adapted

Reviewed-by: Kefu Chai <kchai@redhat.com>

Revert "rgw: use smart pointer for C_Reinitwatch"

This commit introduced regressions, issues #14173, #14180.

The problem is that the context deletes itself after execution,
so using a shared_ptr to hold it is problematic, as we'll eventually
have a double free. Also, it doesn't solve the underlying leak issue.

This reverts commit e9daed221e7c6fd84aef78c3d5fc18470b916fcb.

Merge pull request #7101 from dillaman/wip-14221-jewel

qa/workunits/rbd: rbd-nbd test should use sudo for map/unmap ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7100 from dachary/wip-14220-make-check

tests: configure with rocksdb by default

Reviewed-by: Sage Weil <sage@redhat.com>

osd: Allow get_hash_info to use attrs map if available

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add stat_error for regular scrub handling

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix ScrubMap::object::decode version

Signed-off-by: David Zafman <dzafman@redhat.com>

osd/ReplicatedPG: for copy_get get omap, firstly check ob whether has omap.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/PGBackend: Don't change error type from DEEP_ERROR to SHALLOW_ERROR.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/ReplicatedBackend: Don't forget reset_tp_timeout for first read.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/ReplicatedPG: For omapclear/omaprmkeys, no need call touch.

The code already check object whether exist. So it don't need call
touch.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/ReplicatedPG: for omapclear/omaprmkeys no neet set FLAG_OMAP.

When set omapheader/omapvals, it alreade set FLAG_OMAP. So for remove
don't set again.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/ReplicatedPG: It should call clear_omap_digest.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7063 from mathslinux/rgw-dont-abort-for-bad-argumentment

rgw: do not abort radowgw server when using admin op API with bad parameters

Merge pull request #7099 from jcsp/wip-cmake-bluestore

Fix cmake build with bluestore

qa/workunits/rbd: rbd-nbd test should use sudo for map/unmap ops

Fixes: #14221
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: configure with rocksdb by default

Otherwise bluestore won't compile and run-make-check.sh will fail.

http://tracker.ceph.com/issues/14220 Fixes: #14220

Signed-off-by: Loic Dachary <loic@dachary.org>

cmake: build and link rocksdb etc for bluestore

Signed-off-by: John Spray <john.spray@redhat.com>

cmake: add rocksdb include path for bluestore

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6854 from jecluis/wip-monmapmon

mon: MonmapMonitor: don't expose uncommitted state to client

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mon: MonmapMonitor: don't expose uncommitted state to client

During prepare_command(), we were returning to the user based on
pending_map's state. Even though this weren't causing any issues we are
aware of, we really shouldn't do that.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #6760 from ddiss/bug13943_ceph_disk_validate_part_uuid

ceph-disk: warn for prepare partitions with bad GUIDs

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6992 from dachary/wip-mailmap

mailmap updates

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Chengyuan Li affiliation

Reviewed-by: Chengyuan Li <chengyli@ebay.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Hervé Rousseau name normalization

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Chris Holcombe name normalization

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Jianhui Yuan affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Joe Julian affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: David Coles affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Vasu Kulkarni affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Tobias Suckow affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Rahul Aggarwal affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Brian Felton affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7030 from ceph/wip-detect-bz2-lz4

configure: detect bz2 and lz4

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7016 from xiexingguo/xxg-wip-14151

osd: fix sparse-read result code checking logic

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7031 from wonzhq/log-monitor-2

mon: fix the can't change subscribe level bug in monitoring log

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7009 from majianpeng/osd-scrub-sleep-cause-thread-suicide

osd: avoid osd_op_thread suicide because osd_scrub_sleep

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7007 from tchaikov/wip-13990

osd: do not keep ref of old osdmap in pg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6977 from stiopaa1/msg_override

msg: add override to virutal methods

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #6916 from xiexingguo/xxg-wip-14072

osd: fix null pointer access and race condition

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7028 from yuyuyu101/wip-async-emplace

msg/async: don't use shared_ptr to manage EventCallback

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6896 from liewegas/wip-bluestore

bluestore: latest and greatest

qa/workunits/rados/test_rados_tool.sh: fix path

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_rados_tool: adapt to new buffer hexdump

Signed-off-by: Sage Weil <sage@redhat.com>

uniform the type format

Signed-off-by: Ning Yao <zay11022@gmail.com>

os/bluestore/BlueStore: fix overlay keys

Signed-off-by: YiQiang Chen <cyqsign@163.com>

fs: fix unclosed formatter session

Fixes: #14210
Signed-off-by: xie.xingguo <xie.xingguo@zte.com.cn>

qa/workunits/cephtool/test.sh: leave sortbitwise set

New backends don't work if it's off.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: remove deleted file from dirty list

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: Don't forget close fd in _setup_block_symlink_or_file.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore: write 'type' file

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: don't require journal path

Assume journal symlink is present.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: get store type from 'type' file, if present

Just like ceph-osd.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: wait for flush aios to complete before returning

This makes all the IOs effectively synchronous (while holding the lock),
which isn't great, but at least it's correct.  We can contemplate async
later..

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: clean up aio completion a bit

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: delay IOContext dtor until after io completes

It's possible for the IO to be in flight when the caller closes the
writer handle (although dangerous of them).  Queue the IOContext for
async cleanup when we sync everything to disk.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix a few leaks

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: close cct on shutdown

This lets us do leak checking.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueRocksEnv: fix several leaks

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix leak of iocs in writer

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: fix leak of custom Env

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: fix FS leak

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: no need for 'quit' to ceph cli

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: k=2 for ec

It didn't like k=1 with the default profile.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: be tolerant of collection_list bounds

In particular, we may get a shard specified along with hobject_t min or
max (from PGBackend::objects_list_range()).

Signed-off-by: Sage Weil <sage@redhat.com>

CMakeLists: update for os/ reorg

Signed-off-by: Sage Weil <sage@redhat.com>

os/Makefile: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: cache in-flight stripe updates in memory

The write process may do a read/modify/write on a stripe.  In order to
allow multiple writes to coexist within the same transaction, we need to
be able to "see" our writes.

Clear the "cached" stripe values when the last TransContext touching an
onode is finished.  In theory we could pin memory with a constant stream
of updates to an object; we may need to address that later.

Signed-off-by: Sage Weil <sage@redhat.com>

common/buffer: make hexdump look like 'hexdump -C ...'

...the better to diff them with.

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: debug nid

Signed-off-by: Sage Weil <sage@redhat.com>