os/Newstore: flush_commit return true on STATE_KV_DONE

There is a racing condition here, if the flush_commit() call
happened after _txc_finish_kv and before next state, the context
was pushed to on_commits but no one will handle the context since
we already pass _txc_finish_kv. This bug can be easily reproduce
by putting a sleep(5) after _txc_finish_kv, and trigger the bug by
ceph-osd -i 0 --mkfs.

Fix this bug by return true directly when state >= STATE_KV_DONE(instead
of > in previous code). We already persist the data in STATE_KV_DONE so
it's safe for us to do this.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/NewStore: avoid dup the data of the overlays in the WAL

When writing all the overlays, there is no need to dup the data in WAL.
Instead, we can reference the overlays in the WAL, and remove these
overlays after commiting them to the fs. When replaying, we can get
these data from the referenced overlays. Doing this way, we can save a
write and a deletion for each of the overlay data in the db.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

os/newstore: fix multiple aio case

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: more conservative default for aio queue depth

There appears to be a kernel aio bug when the queue depth is small.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore:close fd after writting with O_DIRECT

fix bug in 2b4c60e0a521ad10b94bbc82865b49f2d28c2ac9

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/NewStore: need to increase the wal op length when combining overlays

Need to add the length of the combining overlays to the length of the
wal op.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

os/Newstore:Fix collection_list_range

We need to rule out hobject_t::max before calling get_object_key
(in which will call get_filestore_key_u32 and get an assert failure)

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: fix race in _txc_aio_submit

We cannot rely on the iterator pointers being valid after we submit the
aio because we are racing with the completion.  Make our loop decision
before submitting and avoid dereferencing txc after that point.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore : Do not need to call fdatasync if using direct.

skip ::fdatasync if in direct mode.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

osd/NewStore: fix for skipping the overlay in _do_overlay_trim

When the offset of the write starts at the end of the overlay, that is,
p->first + p->second.length == offset, the overlay could be skipped as
well.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

os/NewStore: combine contiguous overlays when writing all the overlays

Combine contiguous overlay writes to reduce the numbers of WAL writes
and fs writes.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

os/Nestore: batch cleanup

batch cleanup wal.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: fix _txc_aio_submit

The aios may complete before _txc_aio_submit completes.  In fact, the aio
may complete, commit to the kv store, and then queue more wal aio's before
we finish the loop.  Move aios to a separate list to ensure we only submit
them once and do not right another CPU adjusting the list.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: throttle over entire write lifecycle

Take a global throttle when we submit ops and release when they complete.
The first throttles cover the period from submit to commit, while the wal
ones also cover the async post-commit wal work.  The configs are additive
since the wal ones cover both periods; this should make them reasonably
idiot-proof.

Signed-off-by: Sage Weil <sage@redhat.com>

os/NewStore: data_map shouldn't be empty when writing all overlays

This should be an assert instead of creating new data_map.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

os/NewStore: clear the shared_overlays after writing all the overlays

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

os/NewStore: don't clear overlay in the create/append case of write

Shouldn't clear the overlay in the create/append case of write.
Otherwise, this removes the overlay data and leads to data loss.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

os/newstore: debug io_submit EAGAIN

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: release wal throttle when wal completes, not when queued

If we take the aio path, the io is queued immediately and the resources
are released back to the pool.  Instead release them when wal completes.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: todo

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: move toward state-machine

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use aio for wal writes, too

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: a few comments about wal

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: combined O_DSYNC with O_DIRECT

This avoids the need for an explicit fdatasync when doing O_DIRECT.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: basic aio support

Signed-off-by: Sage Weil <sage@redhat.com>

os/Newstore: add newstore_db_path option

The load of Keyvalue DB is heavy, allow user to put
DB to a seperate(fast) device.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: throttle wal work

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: show # o_direct buffers in debug output

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use a threadpool for applying wal events

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: rebuild buffers to be page-aligned for O_DIRECT

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix omap test cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use fdatasync instead of fsync

On XFS at least, fdatasync is sufficient to make data readable.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: update todo

Signed-off-by: Sage Weil <sage@redhat.com>

os/Newstore: Check onode.omap_head in valid() and next()

The db iter will be set to KeyValueDB::Iterator() if onode.omap_head
not present. In that case if we touch the db iter we will get a segmentation
fault.

Prevent to touch the db iter when onode.omap_head is invalid(equals to 0).

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Use .str() to output a stringstream.

a nit.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/Newstore: Allow gap in _do_write append mode

We can allow some gap so we only need to ensure
onode.size <= offset.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Implement get_omap_iterator

implemented get_omap_iterator

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/KeyValueDB: Add raw_key() interface for IteratorImpl

raw_key() is useful to split out the prefix.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

test/store_test Add get_omap_iterator test cases

omap iterator test cases include:
  iter aganist omap
  lower_bound
  upper_bound

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: drop sync()

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: drop sync()

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: drop flush

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: drop sync_and_flush

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use FS::zero()

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use O_DIRECT is write is page-aligned

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: pass flags to _{open,create}_fid

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: drop unused FragmentHandle

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: do not call completions from kv thread

Reads may call wait_wal() holding user locks, and so we cannot block
progress on WAL completion/flushing by calling callbacks that may take
user locks.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: let wal cleanup kv txn get batched

No need to trigger another sync kv commit here; just let the next KV
commit catch it.

We could possibly do a bit better here by not waking up the kv thread at
all...

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: fix off-by-one on overlay_max_length

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use lower_bound for finding overlay extents in map

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use overlay even if it is a new object or append

This avoids the fsync for small writes.

Signed-off-by: Sage Weil <sage@redhat.com>

os/Newstore:Change assert in get_onode

db->get will return negtive when key is not found.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: open by handle

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: use fs abstaction layer

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: cap fid_max below newstore_max_dir_size

Prevent fid_max over the max_dir_size when preallocation.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: keep smallish overlay extents in kv db

If we have a small overwrite, keep the extent in the key/value database.
Only write it back to the file/fragment later, and when we do, write them
all at once.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: assigned unique nid to each new object

Use this as the key for omap (omap_head), but keep the omap_head field
so that we can tell when no omap data is present.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: consolite collection_list to a single implementation

Signed-off-by: Sage Weil <sage@redhat.com>

Clear removed_collections after reap

Previous code forgot to clear the removed_collections queues
after reaped the collections in _reap_collection.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: ref count OpSequencer

Our OpSequencer may live longer than the ObjectStore::Sequencer interface
object does.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: send complete overwrite to a new fid

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: clone omap

Signed-off-by: Sage Weil <sage@redhat.com>

newstore: initial version

This includes a bunch of new ceph_test_objectstore tests, and a ton of fixes
to existing tests so that objects actually live inside the collections they
are written to.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: debug newstore

Revert "os/Makefile.am: add os/fs/XFS.cc"

This reverts commit 32331ede41ef5b1dc4eb85304d2e86d7c027c75c.

Doh, this is in a conditional below.

Merge pull request #5173 from ceph/wip-12000-12200

Fast read for erasure coding pool and erasure code error handling

Error handling
Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Fast Read
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

os/Makefile.am: add os/fs/XFS.cc

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5695 from tchaikov/wip-12012

osd: translate sparse_read to read for ecpool

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_aio: add a test for aio_sparse_read

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph_test_rados_api_io: add tests for sparse_read

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph_test_rados: also send sparse_read in ReadOp

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: should use ec_pool() when checking for an ecpool

we were using pool.info.require_rollback() in do_osd_ops() when
handling OP_SPARSE_READ to tell if a pool is an ecpool. should
use pool.info.ec_pool() instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: translate sparse_read to read for ecpool

Fixes: #12012
Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: Fix the diagnostic logging mostly to dout(20)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5652 from dachary/wip-mailmap

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.spec: package cls_numops

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5684 from liewegas/wip-hammer-corpus

add hammer objects to ceph-object-corpus

Merge pull request #5451 from XinzeChi/wip-recovery-attr

bug fix: osd: do not cache unused buffer in attrs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5357 from ceph/wip-12325

mon: LogMonitor: handle boolean options consistently

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #5622 from ceph/wip-memstore

MemStore locking enhancements and bufferlist alternative

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5705 from dachary/wip-12786-centos-systemd

ceph-disk: {CentOS,RHEL} >= 7 && Fedora >= 22 are systemd

Reviewed-by: Travis Rhoden <trhoden@redhat.com>

Merge pull request #5699 from dachary/wip-11881-multipath

  ceph-disk: add multipath support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5655 from duduribeiro/patch-1

doc: update ruby doc with the aws-sdk gem usage

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: update ruby doc with the aws-sdk gem usage

Since I'm using ceph with the `aws-sdk` gem (https://github.com/aws/aws-sdk-ruby) instead `aws-s3` (https://github.com/marcel/aws-s3) because the aws-s3 have a trouble with the new active support (https://github.com/marcel/aws-s3/issues/98) (and the downgrade active-support wasn't a option), I proposed change the doc to receive the usage instructions with the aws-sdk gem.

I used ceph with aws-sdk gem with this commands.

Thanks so much

Signed-off-by: Carlos E Ribeiro <mail@carlosribeiro.me>

Merge pull request #4869 from joaquimrocha/cls_numops

CLS numops

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #5724 from ceph/wip-i386

perf_serialize: fix i386 build

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5713 from yuyuyu101/wip-12798

Fix keyvaluestore fails ceph_test_objectstore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5603 from wonzhq/doc-read-recency

doc: add the description for min_read_recency_for_promote

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5460 from ceph/port/new

OSX and BSD derivatives port

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4896 from xinxinsh/wip-monitor-dump-transaction

MonitorDBStore : make monitor transaction more readable

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

make: do not compile XFS.cc if --without-libxfs

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/fs: include <sys/mount.h> on osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

test_c_headers: don't use -Werror option for clang

When using clang to compile ceph, there are lots of warnings:
clang: warning: argument unused during compilation: '-Wp,-D_FORTIFY_SOURCE=2'

Signed-off-by: Yan, Zheng <zyan@redhat.com>

ceph: use 'sed -ie' to edit file in-place

On OSX, 'sed -i script' does not work, because it considers 'script'
as suffix of backup file. 'sed -ie script' works on both OSX and Linux.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: fix compile error on OSX

On OSX, type of the 3rd parameters of getgrouplist is 'int *'

Signed-off-by: Yan, Zheng <zyan@redhat.com>

common/blkdev: fix complie error on OSX/FreeBSD

Signed-off-by: Yan, Zheng <zyan@redhat.com>

common/admin_socket: fix compile error on OSX

Invalid operands to binary expression ('__bind<int &, sockaddr *,
unsigned long>' and 'int')

Without :: clang confuses C bind function and std::bind().

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tests: only use posix_fadvise on linux

Signed-off-by: Yan, Zheng <zyan@redhat.com>