tools:remove the local file when get map failed.

Signed-off-by: Bo Cai <cai.bo@h3c.com>

debian: package radosgw-object-expirer in radosgw deb

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: package new rgw files

   /usr/bin/radosgw-object-expirer
   /usr/lib64/rados-classes/libcls_timeindex.so

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5799 from liewegas/wip-cmake

cmake: fix build (newstore issues)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

CMakeLists.txt: add newstore files

Signed-off-by: Sage Weil <sage@redhat.com>

.gitignore: ignore build (usually used by cmake)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/master' into infernalis

rgw/Makefile.am: ship rgw_object_expirer_core.h

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5744 from ceph/wip-12909

cmake: update FUSE_INCLUDE_DIRS to match autoconf

Merge pull request #5610 from ceph/wip-cmake

cmake: make check

ceph.spec: build requires cmake

Signed-off-by: Sage Weil <sage@redhat.com>

debian/control: build requires cmake

Signed-off-by: Sage Weil <sage@redhat.com>

debian/control: build-requires libboost-regex-dev

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5692 from ceph/wip-rgw-swift-expiration

Wip rgw swift expiration

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

cmake: install crushtool to destdir/bin

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cmake: add blkid as dependency to libcommon

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: Changed name of crc32 target to crc32c

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Added shell script tests

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Fixed HAVE_BETTER_YASM_ELF64 variable

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Removed trailing spaces from isa .s files

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Uncommented erasure-code/shec conditional

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Removed traces of CDS, minor cmake fixes

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Fixed rbd_replay build issue

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Removed scripts, check_PROGRAMS included

Removed the unittest scripts for the time being.
Built unittests included in check_PROGRAMS target.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Cleaned up syntax for make check targets

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: check_TESTPROGRAMS tests running

Make check working, accept rocksdb tests. Clean up coming.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

README.md: Add basic CMake instructions

README.md: Fixed spacing, trimmed cmake section

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #5792 from ceph/wip-vstart-rgw

vstart: add -c argument to radosgw-admin commands

Reviewed-by: Sage Weil <sage@redhat.com>

vstart: add -c argument to radosgw-admin commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #5590 from majianpeng/mds

Mds: add osdmap epoch for setxattr.

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: don't copy delete_at attr, unless it's intra region copy

We don't want to keep the expiration value of a copied object, unless
we're doing a copy within the same zone group.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: objexp shards index by key

Not by time. This should provide better concurrency.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: delete-at and delete-after also on obj put / copy

And potentially later we could use also the S3 api, so it
could work with multipart upload, and POST obj.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5775 from dachary/wip-do-autogen

tools: fix do_autogen.sh -R

Merge pull request #5712 from yuyuyu101/wip-12801

Mon: Make ceph osd metadata support dump all osds

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

tools: fix do_autogen.sh -R

The R letter was missing from the getopts flags. Also sort the flags
alphabetically to make it easier to spot that kind of lossage.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5752 from wonzhq/doc-write-recency

doc: add the doc for min_write_recency_for_promote

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5739 from ceph/wip-12776

mds: fix shutdown while in standby

doc: add the doc for min_write_recency_for_promote

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #5736 from tianshan/wip-12864

rados: make 'rados bench' support json format output

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5748 from liewegas/wip-warnings

fix newstore warning

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-osd-prestart.sh: fix osd data dir ownership check

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5749 from liewegas/wip-async-ms

msg/async: log tx/rx at level 1

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

vstart.sh: enable all experimental features for vstart

Signed-off-by: Sage Weil <sage@redhat.com>

ms/async: log message tx/rx at level 1

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5722 from cxwshawn/vs-fix

vstart.sh: add --mon_num --osd_num --mds_num --rgw_port option

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5693 from tchaikov/wip-12730

common/SubProcess: silence compiler warnings

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5643 from dreamhost/wip-make-check-makeopt

make-check: support MAKEOPTS overrides.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5299 from hjwsm1989/pgmonitor-const

  mon: added const to dump_* functions in PGMonitor

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5156 from rubenk/fix-indentation

Fix indentation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5275 from tchaikov/wip-12287

pybind/ceph_argparse: do not choke on non-ascii prefix

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5702 from Sandy4999/wip-doc-sandy

doc:radosgw: correct typos of the command removing a subuser

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #5747 from ceph/wip-user

fix ceph-disk

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5742 from dachary/wip-user

tests: ceph-disk: dmcrypt simplification

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5746 from ceph/wip-fix-doc-build

doc: fix the code-block in ruby.rst

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: fix the code-block in ruby.rst

* and add the link to library homepage in the section titles

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: update FUSE_INCLUDE_DIRS to match autoconf

client/fuse_ll.cc is now including <fuse.h> and <fuse_lowlevel.h>
instead of <fuse/fuse.h> and <fuse/fuse_lowlevel.h>, so we need to add
the fuse directory to the FUSE_INCLUDE_DIRS variable

using find_path() with just fuse.h was finding a /usr/include/fuse.h
instead of the one in /usr/include/fuse/. looking for fuse_common.h and
fuse_lowlevel.h first causes it to generate the correct
FUSE_INCLUDE_DIRS=/usr/include/fuse

Fixes: #12909
Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/newstore: fix swarning

os/newstore/NewStore.cc: In member function 'int NewStore::_zero(NewStore::TransContext*, NewStore::CollectionRef&, const ghobject_t&, uint64_t, size_t)':
os/newstore/NewStore.cc:3693:32: warning: ignoring return value of 'int ftruncate(int, __off64_t)', declared with attribute warn_unused_result [-Wunused-result]
       ::ftruncate(fd, f.length);
                                ^

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5578 from ceph/wip-newstore

osd: newstore (experimental)

ceph_test_keyvaluedb: add simple commit latency benchmark

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: update todo

Signed-off-by: Sage Weil <sage@redhat.com>

do_autogen.sh: build static rocksdb by default

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: update alt dist rule

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: make OMapIterator test work with FileStore

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: enable newstore tests

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: update to 3.11.2

Signed-off-by: Sage Weil <sage@redhat.com>

os/RocksDBStore: make other rmkey match

No need for Slice() here; it can take a string.

Signed-off-by: Sage Weil <sage@redhat.com>

os/RocksDBStore: fix rmkey()

This took way too long to debug!

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_keyvaluedb: some simple KeyValueDB unit tests

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: fix end bound on collection_list

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: flush object before doing omap reads

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: add 'newstore backend options' to pass options to e.g. rocksdb

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: change escaping chars

# is lowest besides space and !, except for " (which would be too
confusing).

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: trim overlay when zeroing extent

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: tolerate null pnext to collection_list()

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: fix collection range for temp objects

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: Implement fiemap

For simplicity we ignore holes inside an fragment now.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: make sync/async submit_transaction optional

It seems doing this synchronously may be better for SSDs?

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: renamed TransContext::fds -> sync_items

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: queue kv transactions in kv_sync_thread

It appears that db->submit_transaction() will block if there is a sync
commit that is in progress instead of simply queueing the new txn for
later.  To work around this, submit these to the backend in the
kv_sync_thread prior to the synchronous submit_transaction_sync().

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: process multiple aio completions at a time

This isn't affecting things for a slow disk, but it will matter for faster
backends.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: clean up kv commit debug output

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: only ftruncate if i_size is incorrect

Even a no-op ftruncate can block in the kernel.  Prior to this change I
could frequently see ftruncate wait for an aio completion on the same
file.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "os/newstore: avoid sync append for small ios"

This reverts commit 69baab2f7eaca7688ce1d45802a82fc3539cd906.

This is slower.  :(

os/newstore: avoid sync append for small ios

An append is expensive in terms of latency (write, fdatasync, kv commit),
while a wal write is just the kv commit and the write and fdatasync are
async.  For small IOs doing the wal may improve performance.

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: fallocate_with_keep_size = false

This improves my 4k random writes on hdd by about 25%.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "os/NewStore: data_map shouldn't be empty when writing all overlays"

This reverts commit 0d9cce462fec61f754ddcd17cf9a3cf69581d7c5.

We may want to write an overlay if hte object is new and the write is small to defer the cost
of the fsync.

os/NewStore: delay the read of all the overlays until wal applying

The read of all the overlays can be delayed until applying the wal. If
we are doing async wal apply, this can reduce write op latency by
eliminating unnecessary reads in the write code path.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

os/newstore: fix deadlock when newstore_sync_transaction=true

There is a deadlock issue in Newstore when newstore_sync_transaction = true.
With sync_transaction to true, the txc state machine will go all the way down
from STATE_IO_DONE to STATE_FINISHING in the same thread, while holding the osr->qlock().
The deadlock is caused in _txc_finish and _osr_reap_done, when trying to
lock osr->qlock again.

Since the _txc_finish can be called with(in sync transaction mode) or without
(in async transaction mode) holding the qlock, so fix this by setting the qlock
to PTHREAD_MUTEX_RECURSIVE, thus we can recursive acquire the qlock.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/NewStore: fix the append of the later overlays when doing combination

The data of the later contiguous overlays should be claim_append to
'op->data', instead of 'bl'.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

os/Newstore: flush_commit return true on STATE_KV_DONE

There is a racing condition here, if the flush_commit() call
happened after _txc_finish_kv and before next state, the context
was pushed to on_commits but no one will handle the context since
we already pass _txc_finish_kv. This bug can be easily reproduce
by putting a sleep(5) after _txc_finish_kv, and trigger the bug by
ceph-osd -i 0 --mkfs.

Fix this bug by return true directly when state >= STATE_KV_DONE(instead
of > in previous code). We already persist the data in STATE_KV_DONE so
it's safe for us to do this.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/NewStore: avoid dup the data of the overlays in the WAL

When writing all the overlays, there is no need to dup the data in WAL.
Instead, we can reference the overlays in the WAL, and remove these
overlays after commiting them to the fs. When replaying, we can get
these data from the referenced overlays. Doing this way, we can save a
write and a deletion for each of the overlay data in the db.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

os/newstore: fix multiple aio case

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: more conservative default for aio queue depth

There appears to be a kernel aio bug when the queue depth is small.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore:close fd after writting with O_DIRECT

fix bug in 2b4c60e0a521ad10b94bbc82865b49f2d28c2ac9

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/NewStore: need to increase the wal op length when combining overlays

Need to add the length of the combining overlays to the length of the
wal op.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

os/Newstore:Fix collection_list_range

We need to rule out hobject_t::max before calling get_object_key
(in which will call get_filestore_key_u32 and get an assert failure)

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: fix race in _txc_aio_submit

We cannot rely on the iterator pointers being valid after we submit the
aio because we are racing with the completion.  Make our loop decision
before submitting and avoid dereferencing txc after that point.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore : Do not need to call fdatasync if using direct.

skip ::fdatasync if in direct mode.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

osd/NewStore: fix for skipping the overlay in _do_overlay_trim

When the offset of the write starts at the end of the overlay, that is,
p->first + p->second.length == offset, the overlay could be skipped as
well.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>