os/newstore: assigned unique nid to each new object

Use this as the key for omap (omap_head), but keep the omap_head field
so that we can tell when no omap data is present.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: consolite collection_list to a single implementation

Signed-off-by: Sage Weil <sage@redhat.com>

Clear removed_collections after reap

Previous code forgot to clear the removed_collections queues
after reaped the collections in _reap_collection.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: ref count OpSequencer

Our OpSequencer may live longer than the ObjectStore::Sequencer interface
object does.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: send complete overwrite to a new fid

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: clone omap

Signed-off-by: Sage Weil <sage@redhat.com>

newstore: initial version

This includes a bunch of new ceph_test_objectstore tests, and a ton of fixes
to existing tests so that objects actually live inside the collections they
are written to.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: debug newstore

Revert "os/Makefile.am: add os/fs/XFS.cc"

This reverts commit 32331ede41ef5b1dc4eb85304d2e86d7c027c75c.

Doh, this is in a conditional below.

Merge pull request #5173 from ceph/wip-12000-12200

Fast read for erasure coding pool and erasure code error handling

Error handling
Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Fast Read
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

os/Makefile.am: add os/fs/XFS.cc

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5695 from tchaikov/wip-12012

osd: translate sparse_read to read for ecpool

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_aio: add a test for aio_sparse_read

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph_test_rados_api_io: add tests for sparse_read

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph_test_rados: also send sparse_read in ReadOp

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: should use ec_pool() when checking for an ecpool

we were using pool.info.require_rollback() in do_osd_ops() when
handling OP_SPARSE_READ to tell if a pool is an ecpool. should
use pool.info.ec_pool() instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: translate sparse_read to read for ecpool

Fixes: #12012
Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: Fix the diagnostic logging mostly to dout(20)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5652 from dachary/wip-mailmap

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.spec: package cls_numops

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5684 from liewegas/wip-hammer-corpus

add hammer objects to ceph-object-corpus

Merge pull request #5451 from XinzeChi/wip-recovery-attr

bug fix: osd: do not cache unused buffer in attrs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5357 from ceph/wip-12325

mon: LogMonitor: handle boolean options consistently

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #5622 from ceph/wip-memstore

MemStore locking enhancements and bufferlist alternative

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5705 from dachary/wip-12786-centos-systemd

ceph-disk: {CentOS,RHEL} >= 7 && Fedora >= 22 are systemd

Reviewed-by: Travis Rhoden <trhoden@redhat.com>

Merge pull request #5699 from dachary/wip-11881-multipath

  ceph-disk: add multipath support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5655 from duduribeiro/patch-1

doc: update ruby doc with the aws-sdk gem usage

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: update ruby doc with the aws-sdk gem usage

Since I'm using ceph with the `aws-sdk` gem (https://github.com/aws/aws-sdk-ruby) instead `aws-s3` (https://github.com/marcel/aws-s3) because the aws-s3 have a trouble with the new active support (https://github.com/marcel/aws-s3/issues/98) (and the downgrade active-support wasn't a option), I proposed change the doc to receive the usage instructions with the aws-sdk gem.

I used ceph with aws-sdk gem with this commands.

Thanks so much

Signed-off-by: Carlos E Ribeiro <mail@carlosribeiro.me>

Merge pull request #4869 from joaquimrocha/cls_numops

CLS numops

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #5724 from ceph/wip-i386

perf_serialize: fix i386 build

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5713 from yuyuyu101/wip-12798

Fix keyvaluestore fails ceph_test_objectstore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5603 from wonzhq/doc-read-recency

doc: add the description for min_read_recency_for_promote

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5460 from ceph/port/new

OSX and BSD derivatives port

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4896 from xinxinsh/wip-monitor-dump-transaction

MonitorDBStore : make monitor transaction more readable

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

make: do not compile XFS.cc if --without-libxfs

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/fs: include <sys/mount.h> on osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

test_c_headers: don't use -Werror option for clang

When using clang to compile ceph, there are lots of warnings:
clang: warning: argument unused during compilation: '-Wp,-D_FORTIFY_SOURCE=2'

Signed-off-by: Yan, Zheng <zyan@redhat.com>

ceph: use 'sed -ie' to edit file in-place

On OSX, 'sed -i script' does not work, because it considers 'script'
as suffix of backup file. 'sed -ie script' works on both OSX and Linux.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: fix compile error on OSX

On OSX, type of the 3rd parameters of getgrouplist is 'int *'

Signed-off-by: Yan, Zheng <zyan@redhat.com>

common/blkdev: fix complie error on OSX/FreeBSD

Signed-off-by: Yan, Zheng <zyan@redhat.com>

common/admin_socket: fix compile error on OSX

Invalid operands to binary expression ('__bind<int &, sockaddr *,
unsigned long>' and 'int')

Without :: clang confuses C bind function and std::bind().

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tests: only use posix_fadvise on linux

Signed-off-by: Yan, Zheng <zyan@redhat.com>

os/chain_xattr: set CHAIN_XATTR_MAX_NAME_LEN according to max length of xattr name

Signed-off-by: Yan, Zheng <zyan@redhat.com>

test/admin_socket: check error message according to OS

Signed-off-by: Yan, Zheng <zyan@redhat.com>

compat: move definitions in porting.h into include/compat.h

Signed-off-by: Yan, Zheng <zyan@redhat.com>

TestLFNIndex.cc: don't use "cp --preserve=xattr"

--preserve option is not supported by OSX's cp. use 'cp -a' instead

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tests: replace std::tr1::shared_ptr with ceph::shared_ptr

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tests: disable unittest_blkdev on OSX

the test uses sysfs, which does not exist on OSX

Signed-off-by: Yan, Zheng <zyan@redhat.com>

test/librados: replace sem_init() with sem_open()

sem_init() always fails on OSX (it's no longer supported).

Signed-off-by: Yan, Zheng <zyan@redhat.com>

buffer: make buffer::exception classes undefined in dynamic objects

On OSX, if the an exception class is declared and defined in header file,
but it ends up being compiled as private symbols in different binaries.
The exception handling code will take the two compiled exception classes
as different types! In our case, the one in libcls_xxx.so and the one is
ceph-osd are considered as different classes, thus the try-catch statement
fails to work.

The fix is force buffer::exception classes undefined in libcls_xxx.so. The
ibcls_xxx.so are compiled with '-undefined dynamic_lookup' option. when
it is loaded into ceph-osd, buffer::exception classes in ceph-osd will be
used.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

test/librados: replace errno -125 with -ECANCELED

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: convert XATTR_{CREATE,REPLACE} to CEPH_XATTR_{CREATE,REPLACE}

Signed-off-by: Yan, Zheng <zyan@redhat.com>

xattr: convert ENOATTR to ENODATA on DARWIN

On Darwin, getxattr/removexattr return ENOATTR when extended
attribute does not exist. But lots of ceph codes expect ENODATA
is returned in that case. On Darwin, ENOATTR and ENODATA have
different values, so we convert ENOATTR to ENODATA.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

test_libcephfs: disable flock test on OSX

there is no sem_timedwait on OSX. For inter-threads communication,
we can replace sem_t with pthread_cond_t. But for inter-processes
communication, it's extremely hard to figure out how to do a timed
wait. So disable the test case on OSX

Signed-off-by: Yan, Zheng <zyan@redhat.com>

libradosstriper/striping.cc: include "include/types.h"

include definitions of __le{32,64)

Signed-off-by: Yan, Zheng <zyan@redhat.com>

test_c_headers: don't use -Wold-style-declaration option for clang

clang does not support this option

Signed-off-by: Yan, Zheng <zyan@redhat.com>

libcephfs: define loff_t as off_t on OSX

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tools: link ceph-client-debug to LIBCLIENT

Signed-off-by: Yan, Zheng <zyan@redhat.com>

init-ceph: check if /lib/lsb/init-functions exists

On OSX/FreeBSD, /lib/lsb/init-functions does not exist

Signed-off-by: Yan, Zheng <zyan@redhat.com>

init-ceph: don't use procfs to check if daemon is running

use ps(1) instead, which is portable

Signed-off-by: Yan, Zheng <zyan@redhat.com>

init-ceph: replace 'echo -n' with printf

'echo -n' is no portable. On OSX, output of 'echo -n' can be '-n'

Signed-off-by: Yan, Zheng <zyan@redhat.com>

vstart.sh: use portable way to get ip address

Signed-off-by: Yan, Zheng <zyan@redhat.com>

vstart.sh: append ceph library path to {LD,DYLD}_LIBRARY_PATH

Signed-off-by: Yan, Zheng <zyan@redhat.com>

On Darwin: subfix of dynamic library is dylib

Signed-off-by: Yan, Zheng <zyan@redhat.com>

porting.h: add TODO comment

Signed-off-by: Kefu Chai <kchai@redhat.com>

Makefile: add porting.h to dist tarball

Signed-off-by: Kefu Chai <kchai@redhat.com>

Don't use '--exclude-libs' linker option on DARWIN

Only GNU linker supports this option

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: don't try trimming kernel dcache on DARWIN/FreeBSD

still don't know if the methods used on linux also works on
DARWIN/FreeBSD

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: don't include unsupported mount options on DARWIN/FreeBSD

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Link ceph-fuse to fuse on DARWIN

Signed-off-by: Yan, Zheng <zyan@redhat.com>

msg: fix encoding/decoding sockaddr_storage on DARWIN/FreeBSD

sockaddr_storage on DARWIN/FreeBSD is different from sockaddr_storage
on Linux. sockaddr_storage on DARWIN/FreeBSD includes a ss_len field.
Besides, Its ss_len and ss_family fields are 'unsigned char'.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

librbd: Need to include errno.h on DARWIN.

The file is not really required

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

porting.h: add porting.h for porting/compatibility on DARWIN

Collection of porting/compatibility defines to compile on DARWIN

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

tools/ceph_objectstore_tool: Missing O_LARGEFILE on DARWIN

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

test: Fix error with clang on DARWIN (LLVM 3.6.0svn)

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

rbd-replay: Different location of endian.h on DARWIN

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

rbd-fuse: Add position to set/get xattr on DARWIN

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

os/FileStore: fail with ENOTSUP if using sparse files on DARWIN

For now, fail with Not supported (ENOTSUP) if configured using sparse files on DARWIN.

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

librbd: Include porting header

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

common/xattr: Split out xattr on Linux and DARWIN.

DARWIN requires position parameter

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

common/util: include sys/param.h and mount.h on DARWIN

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

common/SubProcess: fix build on DARWIN

include <signal.h> to avoid errors/warnings on missing functions

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

ceph_fuse: Implement set/get xattr with position parameter on OSX

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

msg/async: Include porting.h for MSG_*

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

msg/simple: Move MSG_ and SO_NOSIGPIPE into porting.h

Only enable SO_PRIORITY on Linux

Signed-off-by: Dennis Schafroth <dennis@schafroth.com>

Merge pull request #5024 from xinxinsh/wip-12018

make rbd and pool quota work

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5448 from kylinstorage/wip-optimize-agent_choose_mode

osd: avoid unnecessary calculation in agent_choose_mode()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5418 from branch-predictor/bp-remove-str-concat

osd/osd_types.cc: get rid of str concat when making hash key

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5570 from dachary/wip-12673-proxy-read

osd: trigger the cache agent after a promotion

Reviewed-by: Kefu Chai <kchai@redhat.com>

KeyValueStore: Fix broken assert statement

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Fix getattrs nonexist object need return -ENOENT

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5710 from tchaikov/wip-check-boost_random-mt

configure.ac: check for libboost_random-mt also

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5711 from dachary/wip-12781-dmcrypt

ceph-disk: fix dmcrypt typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: fix dmcrypt typo

Fix the typo introduced by 29431944c77adbc3464a8faeb7e052b24f821780

http://tracker.ceph.com/issues/12781 Fixes: #12781

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4731 from stiopaa1/cephdetectinitfix

ceph-detect-init/debian/__init__: improved syntax

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5503 from zhouyuan/isal_2.14_update

erasure-code: Update ISA-L to 2.14

Reviewed-by: Loic Dachary <ldachary@redhat.com>

configure.ac: check for libboost_random-mt also

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5467 from dachary/wip-ceph-release-notes

tools: ceph-release-notes unicode handling

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

mailmap: Jevon Qiao affiliation

Signed-off-by: Loic Dachary <ldachary@redhat.com>