Merge pull request #11844 from liupan1111/master

os/bluestore: speedup the performance of multi-replication flow by switc…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11799 from KerLiu/fixdupassignment

osd/OSDMap.cc: fix duplicated assignment for new_blacklist_entries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11378 from ryneli/watch_notify_add_timeout

librados: add timeout to watch/notify

Reviewed-by: Kefu Chai <kchai@redhat.com>

librados: add timeout parameter to rados_watch

Signed-off-by: Ryne Li <lizhenqiangsnake@gmail.com>

Merge pull request #10221 from Yan-waller/yj-wip-testrados-0709

test/rados: remove unused bufferlist variable

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11941 from liewegas/wip-envlibrados

qa/workunits/rados/test_envlibrados_for_rocksdb: force librados-dev install

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9357 from Aran85/wip-assert-build-push

osd/PGBackend: build_push_op segment fault

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11800 from bassamtabbara/wip-make-leveldb-optional

mon: support for building without leveldb + mon mkfs bug fix

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11664 from sysnote/wip-17719

mon: fix "OSDs marked OUT wrongly after monitor failover"

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10196 from xiaoxichen/fix_16609

log/LogClient: fill seq & who for syslog and graylog

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

BlueStore: speedup the performance of multi-replication flow by
switching the callback order in bluestore.

In Bluestore ack callback and commit callback are queued one by one in
the function "BlueStore;:_txc_finish_kv". Therefore, just only one callback
is needed to call, in order to improve performance. We do this by switching
the callback order in bluestore, and the callback work can be done in
sub_op_modify_commit and doesn't need to do it again in sub_op_modify_applied.

Signed-off-by: Pan Liu <pan.liu@istuary.com>

Merge pull request #11805 from tchaikov/wip-deprecate-readdir-r

deprecate readdir_r() with readdir()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11824 from xiexingguo/xxg-wip-kill-pg-splitting

osd: kill PG_STATE_SPLITTING

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11874 from wido/osd-no-scrub-recovery

osd: Add config option to disable new scrubs during recovery

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11880 from cbodley/wip-ms-type-random

msg: seed random engine used for ms_type="random"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11898 from liewegas/wip-filestore-hashindex

os/filestore/HashIndex: fix list_by_hash_* termination on reaching end

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/workunits/rados/test_envlibrados_for_rocksdb: force librados-dev install

On trusty we see

 WARNING: The following packages cannot be authenticated!
   librados-dev
 E: There are problems and -y was used without --force-yes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11907 from badone/wip-pipe-socket-typos

msg: Fix typos in socket creation error message

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>

Merge pull request #11392 from Oloremo/keystone_docs

Update keystone doc about v3 options

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #11925 from ktdreyer/doc-infernalis-eol

doc: add infernalis EOL date

doc: add infernalis EOL date

Remove the estimated Infernalis EOL date, and add the actual EOL date.

As discussed on the ceph-devel mailing list, we decided to say that this
infernalis EOL date is the same date that we made the first Jewel
release available.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #11924 from liewegas/wip-balloc

unittest_bit_alloc: fix type

Merging before make check because it clearly breaks the build and the build part is done
Reviewed-by: Dan Mick <dmick@redhat.com>

unittest_bitalloc: fix type

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11922 from liewegas/wip-mempool-bluestore

mempool: changes for bitmap allocator

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

Merge pull request #11920 from ifed01/wip-bluestore-tmp

os/bluestore: remove build warning in a better way

Reviewed-by: Sage Weil <sage@redhat.com>

mempool changes for bitmap allocator

[with bstore_balloc -> bluestore_alloc rename by sage]

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #11839 from liewegas/wip-mempool-buffer

buffer: put buffers in buffer_{data,meta} mempools

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

Merge pull request #11834 from liewegas/wip-bluestore-mempool

os/bluestore: move bluefs into its own mempool

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

buffer: put data and metadata in a mempool

Note that for raw_combined we leak some metadata into the data pool.

Also, we do not account for non-raw metadata or anything else in buffer.h,
as that would pollute the ABI and public interface.

Drop the namespace ceph in buffer.cc (which serves no real purpose) as it
confuses the MEMPOOL_DEFINE_* macros (they cannot be used inside a
namespace).

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: add [de]allocate_aligned

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move pg logs into osd mempool

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move peering evt into pool

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move ReplicatedPG into mempool

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: create an osd mempool

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: attach mempool asok commands to all ccts

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: move to libcommon

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: align shard_t to cacheline

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: adjust factory name

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: drop unused factory type

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: fix allocate() debug_mode check

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7784 from ceph/wip-da-SCA-20160219

Coverity and SCA fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11830 from adamemerson/wip-crush-unlocked

crush: remove the crush_lock

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11910 from xiexingguo/xxg-wip-bluestore-1111

os/bluestore: some more cleanups

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: remove build warning in a better way

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11915 from ifed01/wip-bluestore-fix-test

os/test/store_test: fix legacy bluestore cache settings application

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11905 from xiexingguo/xxg-wip-kill-boost-warn

os/bluestore: fix compiler warnings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11896 from ifed01/wip-bluestore-cleanup

os/bluestore: cleanup around Blob::ref_map

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11710 from ceph/wip-jlayton-fsync

Fix long stalls when calling ceph_fsync()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11293 from liewegas/wip-mon-markout

mon: preserve osd weight when marking osd out, then in

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/test/store_test: fix legacy bluestore cache settings application

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: make Blob::ref_map private

Signed-off-by: Igor Fedotov <ifedotov@minratis.com>

os/bluestore: start using Blob::get_ref over the code

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

client: rename flush_caps() with no arguments to flush_caps_sync()

Per Greg's recommendation, change the name of this function to better
indicate what it does now that we always request a journal flush on
the last cap flush.

Also, add a comment above the function to better explain why we do this.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: request journal flush after flushing caps in syncfs

Ensure that we ask the MDS to flush the journal on the last cap flush
from sync_fs and umount codepaths.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mds: only update change_attr and btime when client sets appropriate feature flags

The kernel client lags the userland code a bit, and feature support for
addr2 is not quite ready. Still, we want to allow the client to set the
new flags field in a cap request before then so it can get better fsync
performance.

When we go to update the cap fields, grab the features from the peer,
and verify that the appropriate flags are set before we apply updates
to the btime and change_attr.

Also, just have the function return early if dirty is 0, since it's
a no-op in that case, and turn the comment above the function into
an assertion.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: wire up the CHECK_CAPS_SYNCHRONOUS flag

Ensure that the client will request an immediate journal flush from the
MDS when we'll end up waiting on the flush response. This patch should
fix the fsync codepath, but we may need something similar for syncfs.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: change no_delay flag to a flags field

In a later patch, we'll want to have the client set the sync flag in
the cap flush, to hint to the MDS that it should process it immediately.

We could add a second bool, but let's instead do what the kernel client
does which is to have a flags field. With that, the existing no_delay
bool becomes CHECK_CAPS_NODELAY.

We'll add other flags in subsequent patches.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mds: do mds log flush if CLIENT_CAPS_SYNC is set

If the client has set the sync flag in a cap update, then it
is indicating that it's waiting on the reply. Ensure that we flush
the journal in that case.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #11790 from jcsp/wip-mclientcaps

client: remove unneeded layout on MClientCaps

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #11789 from jcsp/wip-17798

mds: require MAY_SET_POOL to set pool_ns

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11783 from jcsp/wip-17796

mds: enable rmxattr on pool_namespace attrs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11739 from SUSE/wip-17747

mds: remove "--journal-check" help text

Merge pull request #11692 from stiopaa1/mds_flock_addConstToMemFun

mds/flock: add const to member functions

Merge pull request #11545 from majianpeng/osdc-objectcache-fix-bug

osdc: After write try merge bh.

Reviewed-by: John Spray <john.spray@redhat.com>

osd: Add config option to disable new scrubs during recovery

The osd_scrub_during_recovery config option allows for configuring
if the OSD will schedule a new scrub while recovery is active.

When set to false no new scrubs will be initiated by the OSD while
there are recovery threads active on that OSD.

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #10853 from wido/docs-mon-pool-settings

doc: Add MON docs about pool flags and pool removal config settings

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Add MON docs about pool flags and pool removal config settings

Signed-off-by: Wido den Hollander <wido@42on.com>

msg: Fix typos in socket creation error message

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

os/bluestore: remove dead assert

This assert will never fire, hence it can be safely removed.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make Extent deleter as independent disposer

Duplicated definition of lambas of same function is not good.
Also switching ExtentMap::rm() to use the new disposer to
keep pace with others.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix compiler warnings

As follows:

/home/jenkins-build/build/workspace/ceph-pull-requests/build/boost/include/boost/intrusive/pointer_plus_bits.hpp: In member function ‘bool BlueStore::ExtentMap::encode_some(uint32_t, uint32_t, ceph::bufferlist&, unsigned int*)’:
/home/jenkins-build/build/workspace/ceph-pull-requests/build/boost/include/boost/intrusive/pointer_plus_bits.hpp:76:7: warning: ‘dummy’ is used uninitialized in this function [-Wuninitialized]
       n = pointer(uintptr_t(p) | (uintptr_t(n) & Mask));
       ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueStore.cc:1779:10: note: ‘dummy’ was declared here
   Extent dummy(offset);

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11902 from ceph/wip-krbd-mons-fix

krbd: kernel client expects ip[:port], not an entity_addr_t

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11903 from dmick/master

README: hint at where to look to diagnose test failures

README: hint at where to look to diagnose test failures

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #11895 from xiexingguo/xxg-wip-bluestore-blob-range

os/bluestore: readability improvements and doxygen fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11900 from linuxbox2/fix-civetweb-link

cmake: replace civetweb symlink w/file copy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

krbd: kernel client expects ip[:port], not an entity_addr_t

Commit 2ee1b9a4084f ("krbd.cc: don't rely on MonMap internal members")
inadvertently dropped .get_sockaddr() call, breaking rbd map.  Fix it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #11897 from liewegas/wip-pool-delete

tests: fix tests vs pool deletion

The default changed to disallow pool delete as of #11665; the tests assume it's allowed.
Reviewed-by: Dan Mick <dmick@redhat.com>

cmake: replace civetweb symlink w/file copy

Intended to remove an apparent race.  The two effects are
1. replace top-level command callouts w/file builtins
2. do them in the src/rgw sub-cmake

This is cleaner, and ideally avoids the race.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

os/filestore/HashIndex: fix list_by_hash_* termination on reaching end

If we set *next to max, then the caller (a few lines up) doesn't terminate
the loop and will keep trying to list objects in every following hash
dir until it reaches the end of the collection.  In fact, if we have an
end bound we will never to an efficient listing unless we hit the max
first.

For one user, this was causing OSD suicides when scrub ran because it
wasn't able to list all objects before the timeout.  In general, this would
cause scrub to stall a PG for a long time and slow down requests.

Broken by refactor in 921c4586f165ce39c17ef8b579c548dc8f6f4500.

Fixes: http://tracker.ceph.com/issues/17859
Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: allow pool deletion

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/ceph-helpers.sh: allow pool deletes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11894 from xiexingguo/xxg-wip-bluestore-1110

os/bluestore: speed up omap-key generation for same onode(the read paths)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11893 from xiexingguo/xxg-wip-bluestore-fix-clear

os/bluestore: fix potential memory leak

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11892 from tchaikov/wip-move-context

Context: std::move the callback param in FunctionContext's ctor

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: doxygen fixes

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11889 from trociny/wip-vstart-bashism

vstart.sh: fix bashism in the script

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11601 from ceph/wip-ms-async-keepalive2

msg/async/AsyncConnection: dispatch write handler on keepalive2

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11633 from ddiss/tcc17684_rados_segv_without_pool

rados: fix segfaults when run without --pool

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11665 from wido/mon-prevent-pool-delete

mon: Do not allow pools to be deleted by default

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11795 from runsisi/wip-fix-osd-noin

mon/OSDMonitor: newly created osd should not be wrongly marked in

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10049 from sponce/wip-striper-async-rm-master

librados: improvements async IO in librados and libradosstriper

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Context: std::move the callback param in FunctionContext's ctor

for better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11883 from jcsp/wip-mgr-functioncontext

mgr: remove unnecessary C_StdFunction

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: improve the readability of Extent::blob_escapes_range()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix potential memory leak

Call to extent_map.clear_and_dispose() will be noop after extent_map.clear()
hence causing memory leak.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

vstart.sh: fix bashism in the script

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: speed up omap-key generation for same onode(the read paths)

In https://github.com/ceph/ceph/pull/11807 we only modify the write paths,
this patch fixes the read parts too, which are more time-sensitive.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11886 from stiopaa1/mds_mantle_includeCorrectHeaders

mds/Mantle.h: include correct header files

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11683 from wjwithagen/wip-wjw-freebsd-vstart

src/vstart.sh: Only execute btrfs if it is available

Reviewed-by: Kefu Chai <kchai@redhat.com>