os/filestore: Fix erroneous WARNING: max attr too small

When we read the conf values from the conf file we call handle_conf_change which
calls set_xattr_limits_via_conf but this is before m_fs_type is initialised so
we fall through to the default comaprison with
filestore_max_xattr_value_size_other (1024) and erroneously issue the warning.
This happens at every startup but subsequent calls once the filestore is mounted
proceed as expected. Don't call set_xattr_limits_via_conf unless the filestore
is mounted.

Fixes: http://tracker.ceph.com/issues/17420
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11196 from tchaikov/wip-lttng-the-right-way

lttng: build the tracepoint provider lib from .c files in repo

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11090 from dillaman/wip-16974

rbd-mirror: force-promoted image will remain R/O until rbd-mirror daemon restarted

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11197 from yuyuyu101/wip-bluestore-misc

os/bluestore: misc cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11215 from stiopaa1/mon_paxos_moveContextClassesToCC

mon/Paxos: move classes to .cc file

Reviewed-by: Sage Weil <sage@redhat.com>

mon/Paxos: move classes to .cc file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11170 from ukernel/wip-17253

client: fix segment fault in Client::_invalidate_kernel_dcache().

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11142 from hzhang-wx/master

cls_rbd: clean up status from rbd-mirror if image removed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10939 from oritwas/wip-rgw-acl-underscore

rgw: fix regression with handling double underscore

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11159 from linuxbox2/rgw-setattr-920

rgw_file: fix set_attrs operation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11172 from batrick/ceph-conf-metavar

doc: add $pid metavar conf doc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11200 from yuyuyu101/wip-upper-thread

msg/async: avoid set out of range ms_async_op_threads option

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #9170 from Yan-waller/yj-wip-fixrgwnew

rgw/admin: fix some return values and indents
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11210 from xiexingguo/xxg-wip-bluestore-0923

os/bluestore: more cleanups and fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10463 from gongchuang/gc-wip-test

Client: fixup param type and return value

Reviewed-by: John Spray <john.spray@redhat.com>

journal: send update notification when tag allocated

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new journal listener event for force promotion

Fixes: http://tracker.ceph.com/issues/16974
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper class for quiescing in-flight async ops

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: unify journal event callbacks into single interface

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: expose ability to retrieve partial tag list

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: fix remove mirrored image race conditions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: new mirroring forced promotion test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11153 from dillaman/wip-17317

test/rbd: fix possible mock journal race conditions

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11054 from rzarzynski/wip-rgw-postobj-nodynalloc

rgw: eradicate dynamic memory allocation in RGWPostObj.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore: fix potential access violation during shard-key scan

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: kill unused variables "ekey"

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11208 from tchaikov/wip-17380

osd: should not look up an empty pg

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10949 from avnerbh/avnerbh-preforker-fix

preforker: prevent call to 'write' on an fd that was already closed

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: re-enable test_pg_scrub() test in ceph-helper.sh

this reverts d053705. i disabled this test in hope to bisect the
offending tests that fail the mysterious jenkins failure, which was
fixed by 6f3ce3a.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: should not look up an empty pg

this fixes the scrub timeouts we spotted in jenkins run and qa run.
it's a regression introduced by c2b7cdb

Fixes: http://tracker.ceph.com/issues/17380
Signed-off-by: Loic Dachary <loic@dachary.org>
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: drop csum_enabled parameter from blob::get_chunk_size()

The only two callers from bluestore now always set csum_enabled to true,
so there is no need to keep this parameter any more.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify BlueStore::_setup_block_symlink_or_file()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11205 from liewegas/wip-async-debug

msg/async: less verbose debug messages at debug_ms=1

Reviewed-by: Haomai Wang <haomai@xsky.com>

rgw_file: fix set_attrs operation

The effective part of this change is to always set a value
for RGW_ATTR_UNIX_KEY1 (because it is expected later).

Secondarily, do not pass the address of the to-set attributes
buffer::list as remove attrs--this is confusing.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #11181 from linuxbox2/rgw-legacy-time

rgw_file:  pre-assign times

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11048 from jimifm/wip-FDCache-master

os/filestore/FDCache: fix bug when filestore_fd_cache_shards = 0

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async: no need for ..."started." suffix

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: less noisy about EventCenter owner

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: make rx line look like normal debug

The DispatchQueue prints the traditional ==== line.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: make send debug line look like SimpleMessenger's

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/AsyncConnection: less noisy about connection top

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/EventCenter: less noisy debug on events

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11190 from tchaikov/wip-memstore-clone-fixture

tests: use a fixture for memstore clone testing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

msg/async: avoid set out of range ms_async_op_threads option

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10587 from b-ranto/wip-logrotate-root

logrotate: Run as root/ceph

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7338 from ceph/cls-lua

Lua object class support

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Ken Dreyer (Red Hat) <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10135 from david-z/wip-enhance-ceph-disk-bluestore

ceph-disk: support creating block.db and block.wal with customized size for bluestore

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10371 from Yan-waller/yj-wip-msgsimple-0720

msg/simple: close the failed socket fds

https://github.com/ceph/ceph/pull/11052

Merge pull request #11052 from yuyuyu101/wip-shutdown-deadlock

OSD: avoid FileStore finisher deadlock in osd_lock when shutdown OSD

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11082 from aiicore/nearfull_flag_fix

mon: OSDMonitor: Missing nearfull flag set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11089 from dachary/wip-16477-ceph-helper

cli: retry when the mon is not configured

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11199 from yuyuyu101/wip-17378

msg/async: remove fd output in log prefix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11195 from xiexingguo/xxg-wip-csum-improve

os/bluestore: a few fixes about  the global csum setting

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async: remove fd output in log prefix

it's tricky to prevent fd racing in logs, otherwise we need to provide with
lock to totally avoid this. But I don't think it deserve lock

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11182 from ceph/wip-docs-rm-ink

remove old references to inktank premium support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11194 from wjwithagen/wip-wjw-rocksdb-include

src/kv/CMakeLists.txt: force rocksdb/include to first include directory

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix race condition of global csum setting

The global csum_type option can be changed dynamically and
can be fixed by upgrading into std::atomic.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop the global bluestore_csum option

We don't need this option as it can be properly covered
by the bluestore_csum_type option.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: use blob internal settings for csum checking

This is because:
1. It is more accurate, as the global csum settings may change dynamically.
2. bluestore_blob_t::verify_csum() can handle both csum-on and csum-off correctly.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11151 from ceph/wip-bluestore-error-injection

os/bluestore: add error injection

Merge pull request #11184 from liewegas/wip-down-out

mon: wait 10m (not 5m) before marking down OSDs out

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #11186 from majianpeng/bluestore-allocator

os/bluestore: remove unused head file.

Reviewed-by: Sage Weil <sage@redhat.com>

rgw_file:  pre-assign times

Set unix timestamps based on RGW values for creation, modification
time for buckets, objects in stat requests.  Regard any saved
value of these in unix attributes as an overlay.

Fixes: http://tracker.ceph.com/issues/17367
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #11192 from xiexingguo/xxg-wip-bluestore-0922

os/bluestore: a few cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11152 from dillaman/wip-17198

librbd: potential deadlock closing image with in-flight readahead

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10506 from wjwithagen/wip-wjw-clock-realtime

FreeBSD: Define CLOCK_REALTIME_COARSE in compat.h

lttng: build the tracepoint provider lib from .c files in repo

we should not have `TRACEPOINT_DEFINE` deinfed in both the application
we are tracing and the tracing provider library.

* tracing/*.c: reference the header file with path
* README.md: add instructions to add .c file
* CMakeLists.c:
  - compile the .c file in repo, not the one generated by
    lttng-gen-tp
  - no need to mark the generated header file as GENEREATED,
    they are marked so by default (they are listed in OUTPUT)
  - do not generate .c file anymore, they are not used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

src/kv/CMakeLists.txt: force rocksdb/include to first include directory

 - Otherwise on FreeBSD older packages could be used for include
   this then can result to missing declared functions because
   includes and libs are out of sync

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc/rados/troubleshooting: remove references to premium support from inktank.com

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados: remove references to premium support from inktank.com

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #11147 from ukernel/wip-17286

client: fix readdir vs fragmentation race

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10393 from ukernel/wip-mds-log-flush

reduce unnecessary mds log flush

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11155 from dillaman/wip-17330

librbd: backward/forward compatibility for update_features

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11148 from dillaman/wip-17310

librbd: block name prefix might overflow fixed size C-string

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: remove unused head file.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #11176 from zealoussnow/wip-0922

ceph_fuse: use sizeof get the buf length

Reviewed-by: Kefu Chai <kchai@redhat.com>

test_memstore_clone: use StoreTestFixture to do the setup/teardown

rename test case name from `MemStore` to `MemStoreClone`, because
if we use `TEST_F` to group the tests into a fixture, `MemStore`
will be used as the class name of the fixture, and hence conflicts
with the existing `MemStore` class under testing.

but the object store is created and destroyed for every test. before
this change, we do this for every test case. if it's a concern
we need to use SetUpTestCase() and TearDownTestCase() instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph_test_objectstore: fail the test if SetUp() fails

so it will issues a fatal error if anything goes wrong, instead of continuing
with the test cases.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph_test_objectstore: extract StoreTestFixture into store_test_fixture.cc

so it can be reused by other tests like test_memstore_clone.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: clean up finishers properly

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: move lookup() of SharedBlobSet back into .h

Because the lookup() method is simple and easy,
and requires no dout for debugging.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon: wait 10m (not 5m) before marking down OSDs out

5m isn't quite enough time for many boxes to reboot.  10m will do
a better job of masking transient crashes without having much
effect on time to repair.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add error injection

This is used by the teuthology repair_test.py.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11122 from branch-predictor/bp-zlib-improvements

compressor/zlib: switch to raw deflate

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11108 from liewegas/wip-bluefs-dirty-fnodes

os/bluestore/BlueFS: log dirty files at sync time

ceph_fuse: use sizeof get the buf length

Signed-off-by: LeoZhang <nguzcf@gmail.com>

Merge pull request #11173 from tchaikov/wip-17306

test/cli/crushtool: fix the test of compile-decompile-recompile.t

Reviewed-by: Samuel Just <sjust@redhat.com>

test/cli/crushtool: fix the test of compile-decompile-recompile.t

should read the map from $TESTDIR,
it's a regression introduced by b2c0a07

Fixes: http://tracker.ceph.com/issues/17306
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add $pid metavar conf doc

$pid introduced in 2012 via cf2a0454.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11166 from xiexingguo/xxg-wip-bluestore-0921

os/bluestore: simplify calculation of collection key range

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11131 from ceph/rgw-lifecycle-testing

Rgw lifecycle testing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

client: fix segment fault in Client::_invalidate_kernel_dcache().

when umounting, root can be NULL

Fixes: http://tracker.ceph.com/issues/17253
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Lifecycle testing

Add new config option - rgw_lc_debug_interval - that modifies the
lifecycle algorithm.  It treats days as being the configured number of
seconds.  This allows lifecycle to be tested in minutes, rather than
days.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Lifecycle - Fix config locking

The lifecycle config lock was not setting a cookie.  This means that all
instances of that lock conflict, causing errors to be returned to the
client.

Set a unique cookie on each lock instance.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #11031 from yangdongsheng/du

rbd: return error if we specified a wrong image name for rbd du

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11163 from jcsp/wip-cmake-vstart

doc: Update developer docs for cmake paths

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

doc: update vstart guide for cmake-ish paths

Signed-off-by: John Spray <john.spray@redhat.com>

doc: update build instructions for cmake

Signed-off-by: John Spray <john.spray@redhat.com>

kv/RocksDBStore: no reason to make write_batch using new/delete

discards new/delete for bat

Signed-off-by: Haomai Wang <haomai@xsky.com>

os/bluestore: simplify calculation of collection key range

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>