ceph_test_objectstore: fix omap test cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use fdatasync instead of fsync

On XFS at least, fdatasync is sufficient to make data readable.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: update todo

Signed-off-by: Sage Weil <sage@redhat.com>

os/Newstore: Check onode.omap_head in valid() and next()

The db iter will be set to KeyValueDB::Iterator() if onode.omap_head
not present. In that case if we touch the db iter we will get a segmentation
fault.

Prevent to touch the db iter when onode.omap_head is invalid(equals to 0).

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Use .str() to output a stringstream.

a nit.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/Newstore: Allow gap in _do_write append mode

We can allow some gap so we only need to ensure
onode.size <= offset.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Implement get_omap_iterator

implemented get_omap_iterator

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/KeyValueDB: Add raw_key() interface for IteratorImpl

raw_key() is useful to split out the prefix.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

test/store_test Add get_omap_iterator test cases

omap iterator test cases include:
  iter aganist omap
  lower_bound
  upper_bound

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: drop sync()

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: drop sync()

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: drop flush

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: drop sync_and_flush

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use FS::zero()

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use O_DIRECT is write is page-aligned

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: pass flags to _{open,create}_fid

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: drop unused FragmentHandle

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: do not call completions from kv thread

Reads may call wait_wal() holding user locks, and so we cannot block
progress on WAL completion/flushing by calling callbacks that may take
user locks.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: let wal cleanup kv txn get batched

No need to trigger another sync kv commit here; just let the next KV
commit catch it.

We could possibly do a bit better here by not waking up the kv thread at
all...

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: fix off-by-one on overlay_max_length

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use lower_bound for finding overlay extents in map

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use overlay even if it is a new object or append

This avoids the fsync for small writes.

Signed-off-by: Sage Weil <sage@redhat.com>

os/Newstore:Change assert in get_onode

db->get will return negtive when key is not found.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: open by handle

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use fs abstaction layer

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: cap fid_max below newstore_max_dir_size

Prevent fid_max over the max_dir_size when preallocation.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: keep smallish overlay extents in kv db

If we have a small overwrite, keep the extent in the key/value database.
Only write it back to the file/fragment later, and when we do, write them
all at once.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: assigned unique nid to each new object

Use this as the key for omap (omap_head), but keep the omap_head field
so that we can tell when no omap data is present.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: consolite collection_list to a single implementation

Signed-off-by: Sage Weil <sage@redhat.com>

Clear removed_collections after reap

Previous code forgot to clear the removed_collections queues
after reaped the collections in _reap_collection.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: ref count OpSequencer

Our OpSequencer may live longer than the ObjectStore::Sequencer interface
object does.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: send complete overwrite to a new fid

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: clone omap

Signed-off-by: Sage Weil <sage@redhat.com>

newstore: initial version

This includes a bunch of new ceph_test_objectstore tests, and a ton of fixes
to existing tests so that objects actually live inside the collections they
are written to.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: debug newstore

Revert "os/Makefile.am: add os/fs/XFS.cc"

This reverts commit 32331ede41ef5b1dc4eb85304d2e86d7c027c75c.

Doh, this is in a conditional below.

Merge pull request #5173 from ceph/wip-12000-12200

Fast read for erasure coding pool and erasure code error handling

Error handling
Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Fast Read
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

os/Makefile.am: add os/fs/XFS.cc

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5695 from tchaikov/wip-12012

osd: translate sparse_read to read for ecpool

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_aio: add a test for aio_sparse_read

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph_test_rados_api_io: add tests for sparse_read

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph_test_rados: also send sparse_read in ReadOp

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: should use ec_pool() when checking for an ecpool

we were using pool.info.require_rollback() in do_osd_ops() when
handling OP_SPARSE_READ to tell if a pool is an ecpool. should
use pool.info.ec_pool() instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: translate sparse_read to read for ecpool

Fixes: #12012
Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: Fix the diagnostic logging mostly to dout(20)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5652 from dachary/wip-mailmap

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.spec: package cls_numops

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5684 from liewegas/wip-hammer-corpus

add hammer objects to ceph-object-corpus

Merge pull request #5451 from XinzeChi/wip-recovery-attr

bug fix: osd: do not cache unused buffer in attrs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5357 from ceph/wip-12325

mon: LogMonitor: handle boolean options consistently

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #5622 from ceph/wip-memstore

MemStore locking enhancements and bufferlist alternative

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5705 from dachary/wip-12786-centos-systemd

ceph-disk: {CentOS,RHEL} >= 7 && Fedora >= 22 are systemd

Reviewed-by: Travis Rhoden <trhoden@redhat.com>

Merge pull request #5699 from dachary/wip-11881-multipath

  ceph-disk: add multipath support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5655 from duduribeiro/patch-1

doc: update ruby doc with the aws-sdk gem usage

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: update ruby doc with the aws-sdk gem usage

Since I'm using ceph with the `aws-sdk` gem (https://github.com/aws/aws-sdk-ruby) instead `aws-s3` (https://github.com/marcel/aws-s3) because the aws-s3 have a trouble with the new active support (https://github.com/marcel/aws-s3/issues/98) (and the downgrade active-support wasn't a option), I proposed change the doc to receive the usage instructions with the aws-sdk gem.

I used ceph with aws-sdk gem with this commands.

Thanks so much

Signed-off-by: Carlos E Ribeiro <mail@carlosribeiro.me>

Merge pull request #4869 from joaquimrocha/cls_numops

CLS numops

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #5724 from ceph/wip-i386

perf_serialize: fix i386 build

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5713 from yuyuyu101/wip-12798

Fix keyvaluestore fails ceph_test_objectstore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5603 from wonzhq/doc-read-recency

doc: add the description for min_read_recency_for_promote

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5460 from ceph/port/new

OSX and BSD derivatives port

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4896 from xinxinsh/wip-monitor-dump-transaction

MonitorDBStore : make monitor transaction more readable

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

make: do not compile XFS.cc if --without-libxfs

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/fs: include <sys/mount.h> on osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

test_c_headers: don't use -Werror option for clang

When using clang to compile ceph, there are lots of warnings:
clang: warning: argument unused during compilation: '-Wp,-D_FORTIFY_SOURCE=2'

Signed-off-by: Yan, Zheng <zyan@redhat.com>

ceph: use 'sed -ie' to edit file in-place

On OSX, 'sed -i script' does not work, because it considers 'script'
as suffix of backup file. 'sed -ie script' works on both OSX and Linux.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: fix compile error on OSX

On OSX, type of the 3rd parameters of getgrouplist is 'int *'

Signed-off-by: Yan, Zheng <zyan@redhat.com>

common/blkdev: fix complie error on OSX/FreeBSD

Signed-off-by: Yan, Zheng <zyan@redhat.com>

common/admin_socket: fix compile error on OSX

Invalid operands to binary expression ('__bind<int &, sockaddr *,
unsigned long>' and 'int')

Without :: clang confuses C bind function and std::bind().

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tests: only use posix_fadvise on linux

Signed-off-by: Yan, Zheng <zyan@redhat.com>

os/chain_xattr: set CHAIN_XATTR_MAX_NAME_LEN according to max length of xattr name

Signed-off-by: Yan, Zheng <zyan@redhat.com>

test/admin_socket: check error message according to OS

Signed-off-by: Yan, Zheng <zyan@redhat.com>

compat: move definitions in porting.h into include/compat.h

Signed-off-by: Yan, Zheng <zyan@redhat.com>

TestLFNIndex.cc: don't use "cp --preserve=xattr"

--preserve option is not supported by OSX's cp. use 'cp -a' instead

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tests: replace std::tr1::shared_ptr with ceph::shared_ptr

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tests: disable unittest_blkdev on OSX

the test uses sysfs, which does not exist on OSX

Signed-off-by: Yan, Zheng <zyan@redhat.com>

test/librados: replace sem_init() with sem_open()

sem_init() always fails on OSX (it's no longer supported).

Signed-off-by: Yan, Zheng <zyan@redhat.com>

buffer: make buffer::exception classes undefined in dynamic objects

On OSX, if the an exception class is declared and defined in header file,
but it ends up being compiled as private symbols in different binaries.
The exception handling code will take the two compiled exception classes
as different types! In our case, the one in libcls_xxx.so and the one is
ceph-osd are considered as different classes, thus the try-catch statement
fails to work.

The fix is force buffer::exception classes undefined in libcls_xxx.so. The
ibcls_xxx.so are compiled with '-undefined dynamic_lookup' option. when
it is loaded into ceph-osd, buffer::exception classes in ceph-osd will be
used.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

test/librados: replace errno -125 with -ECANCELED

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: convert XATTR_{CREATE,REPLACE} to CEPH_XATTR_{CREATE,REPLACE}

Signed-off-by: Yan, Zheng <zyan@redhat.com>

xattr: convert ENOATTR to ENODATA on DARWIN

On Darwin, getxattr/removexattr return ENOATTR when extended
attribute does not exist. But lots of ceph codes expect ENODATA
is returned in that case. On Darwin, ENOATTR and ENODATA have
different values, so we convert ENOATTR to ENODATA.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

test_libcephfs: disable flock test on OSX

there is no sem_timedwait on OSX. For inter-threads communication,
we can replace sem_t with pthread_cond_t. But for inter-processes
communication, it's extremely hard to figure out how to do a timed
wait. So disable the test case on OSX

Signed-off-by: Yan, Zheng <zyan@redhat.com>

libradosstriper/striping.cc: include "include/types.h"

include definitions of __le{32,64)

Signed-off-by: Yan, Zheng <zyan@redhat.com>

test_c_headers: don't use -Wold-style-declaration option for clang

clang does not support this option

Signed-off-by: Yan, Zheng <zyan@redhat.com>

libcephfs: define loff_t as off_t on OSX

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tools: link ceph-client-debug to LIBCLIENT

Signed-off-by: Yan, Zheng <zyan@redhat.com>

init-ceph: check if /lib/lsb/init-functions exists

On OSX/FreeBSD, /lib/lsb/init-functions does not exist

Signed-off-by: Yan, Zheng <zyan@redhat.com>

init-ceph: don't use procfs to check if daemon is running

use ps(1) instead, which is portable

Signed-off-by: Yan, Zheng <zyan@redhat.com>

init-ceph: replace 'echo -n' with printf

'echo -n' is no portable. On OSX, output of 'echo -n' can be '-n'

Signed-off-by: Yan, Zheng <zyan@redhat.com>

vstart.sh: use portable way to get ip address

Signed-off-by: Yan, Zheng <zyan@redhat.com>

vstart.sh: append ceph library path to {LD,DYLD}_LIBRARY_PATH

Signed-off-by: Yan, Zheng <zyan@redhat.com>

On Darwin: subfix of dynamic library is dylib

Signed-off-by: Yan, Zheng <zyan@redhat.com>

porting.h: add TODO comment

Signed-off-by: Kefu Chai <kchai@redhat.com>

Makefile: add porting.h to dist tarball

Signed-off-by: Kefu Chai <kchai@redhat.com>

Don't use '--exclude-libs' linker option on DARWIN

Only GNU linker supports this option

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: don't try trimming kernel dcache on DARWIN/FreeBSD

still don't know if the methods used on linux also works on
DARWIN/FreeBSD

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: don't include unsupported mount options on DARWIN/FreeBSD

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Link ceph-fuse to fuse on DARWIN

Signed-off-by: Yan, Zheng <zyan@redhat.com>

msg: fix encoding/decoding sockaddr_storage on DARWIN/FreeBSD

sockaddr_storage on DARWIN/FreeBSD is different from sockaddr_storage
on Linux. sockaddr_storage on DARWIN/FreeBSD includes a ss_len field.
Besides, Its ss_len and ss_family fields are 'unsigned char'.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

librbd: Need to include errno.h on DARWIN.

The file is not really required

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>