Merge pull request #12468 from bassam/pr-simplify-jemalloc

cmake: simplify find_package jemalloc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12519 from hjwsm1989/pgbackfill

osd/PG: simplify the logic of backfill_targets checking

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12585 from liewegas/wip-notify-ack

osd/osd_types: print notify-ack op properly

Merge pull request #12563 from xiexingguo/xxg-wip-osd-1217

common: make attempts of auth rotating configurable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11065 from LiumxNL/wip-fix-osdmap-update

mon/OSDMonitor: should propose osdmap update when cluster addr changed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12634 from liewegas/wip-bluestore-shard-keys

os/bluestore: do not cache shard keys

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

Merge pull request #12635 from liewegas/wip-bluestore-cache

os/bluestore: set cache meta ratio to .9

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12644 from xiexingguo/xxg-wip-bluestore-keyreserve

os/bluestore: preallocate object[extent_shard] key to avoid reallocate

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: do not precalculate/cache shard keys

Note that we are generating it on demand now.  We can
probably do better, especially when in a loop.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12149 from adamemerson/wip-g_ceph_context-exterminate

global: start removing g_ceph_context

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/kraken'

Merge pull request #12630 from liewegas/wip-workunit-retry

qa/tasks/workunit: clear clone dir before retrying checkout

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12614 from liewegas/wip-task-peer

qa/tasks/peer: update task based on current peering behavior

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12559 from Adirl/verify_transport_type

msg/async: assert if compiled code doesn't support the configured ms_…

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: assert if compiled code doesn't support the configured ms_async_transport_type

for example, currently there is a crash when using the async stack (access of null pointer)
in case the user configured rdma for a code that was not compiled with RDMA

issue: 913397

Change-Id: I4e3d88006312a713b8d3aaefbdcd0596039270e1
Signed-off-by: Avner BenHanoch <avnerb@mellanox.com>

Merge pull request #12643 from xiexingguo/xxg-wip-bluestore-1123

os/bluestore: remove intermediate key var to avoid string copy

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: whu liuchang <liuchang0812@gmail.com>

os/bluestore: preallocate object[extent_shard] key to avoid reallocate

which is less efficient.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove intermediate key var to avoid string copy

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12636 from badone/wip-doc-pg-states

doc: Remove "splitting" state

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12543 from hjwsm1989/cleanup2

  osd/ECBackend: remove unused variable "ReadCB"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Remove "splitting" state

There is no splitting state so remove it from pg-states.rst

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12632 from liewegas/wip-centos-73

qa/suites: switch from centos 7.2 to 7.x

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge remote-tracking branch 'gh/kraken'

# Conflicts:
# src/os/bluestore/BlueStore.cc

os/bluestore: set cache meta ratio to .9

More metadata, less data.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/distros: centos_7.yaml -> centos.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites: centos_7.2.yaml -> centos_7.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

qa/distros: add centos 7.3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/distros: add centos 7 yaml; use that instead

No need to be picky about the point release here.

Signed-off-by: Sage Weil <sage@redhat.com>

msg: Remove mention of g_ceph_context

Since it won't exist any more there's no point in saying it can't be
relied on, now is there?

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

objclass: Utterly destroy g_ceph_context

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os: Complete removal of g_ceph_context and g_conf

And so it continues.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os: Continuing the removal of g_ceph_context and g_conf

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os/filestore: Destroy g_ceph_context and g_conf

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os/bluestore: Remove g_ceph_context and g_conf

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os: Add CephContext to ObjectStore and ObjectMap bases

Preparatory to removing g_ceph_context and g_conf uses

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd: Remove g_conf and g_ceph_context

So begins the ruthless annihilation of g_ceph_context.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

dout: Use dout_context

As a transition for g_ceph_context removal, make all dout macros depends
on a local macro.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Remove now unused clock_offset option

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Unskew clock

In preparation to deglobalizing CephContext, remove the CephContext*
parameter to ceph_clock_now() and ceph::real_clock::now() that carries
a configurable offset.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #12629 from liewegas/wip-bluestore-shard-extents

os/bluestore: remove 'extents' from shard_info

Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #12616 from atheism/rgw-ldap-fix-simple-bind

rgw: ldap: simple_bind() should set ldap version option on tldap

qa/tasks/workunit: clear clone dir before retrying checkout

If we checkout ceph-ci.git, and don't find a branch,
we'll try again from ceph.git. But the checkout will
already exist and the clone will fail, so we'll still
fail to find the branch.

The same can happen if a previous workunit task already
checked out the repo.

Fix by removing the repo before checkout (the first and
second times).  Note that this may break if there are
multiple workunit tasks running in parallel on the same
role.  That is already racy, so if it's happening, we'll
want to switch to using a truly unique clonedir for each
instantiation.

Fixes: http://tracker.ceph.com/issues/18336
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove 'extents' from shard_info

We don't use this field; no need to consume the space
on disk.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12394 from ifed01/wip-bluestore-fix-reshard

os/bluestore: refactor ExtentMap::update to avoid preceeding db updat…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12464 from ifed01/wip-bluestore-turnoff-block-write

os/bluestore: add a debug option to bypass block device writes for bl…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12526 from hjwsm1989/ec-cleanup

osd/ECBackend: only need check missing_loc when doing recovery

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12527 from hjwsm1989/cleanup

osd/PrimaryLogPG: unify the access to primary pg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12564 from kylinstorage/wip-cleanup-pg-stat-t

osd/ECBackend: cleanup for unnecessary copy with pg_stat_t

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12576 from kylinstorage/wip-cleanup-bluestore-coll-ref

bluestore: avoid unnecessary copy with coll_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12579 from hjwsm1989/smallfix

osd/PG: check the connection first in fulfill_log

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12583 from liupan1111/wip-add-throttle-bluestore

os/bluestore: add perf variable for throttle info in bluestore

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

OSDMonitor: should propose osdmap update when OSD cluster_addr was changed

Signed-off-by: Mingxin Liu <mingxin@xsky.com>

os/bluestore/bluefs: make bluefs_preextend_wal_files default is false.

This make rocksdb log corruption. Before fix, disable this function.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge remote-tracking branch 'gh/kraken'

mon: MonMap: clear 'mon_info' during decode

It so happens that it's not safe to assume the monmap will be in an
empty state upon decoding.

Turns out the MonClient will reuse the MonMap instance when decoding
the just received map from the monitors. Should the monitors be on an
older version that do not support 'mon_info', this field will not be
decoded (after all, there's no field to decode from); but by this time,
the MonClient would already have a built monmap, which could have
populated 'mon_info' with temporary mon names from 'mon initial
members'.

Given the existing entries in 'mon_info', and the conflicting entries in
'mon_addr', we would end up asserting in 'sanitize_mons()'. This becomes
a non-issue if 'mon_info' is empty, as was unfortunately presumed.

Fixes: http://tracker.ceph.com/issues/18265
Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #12604 from ommoreno/wip-16966

os/bluestore: allow multiple SPDK BlueStore OSD instances

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12618 from Liuchang0812/fix-blue-store-unused-var

os/bluestore: remove a never read value

Merge branch 'kraken' into wip-18314

qa/tasks/peer: update task based on current peering behavior

This changed in 0be3f5f72e169fad08dcb0240de45287b567bd49.

Fixes: http://tracker.ceph.com/issues/18330
Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: output recovery_state records for Down and Incomplete states

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove a never read value

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

rgw: ldap: simple_bind() should set ldap option on tldap

    ldap_set_option() should set option for "tldap" other than "ldap".
    The current code will lead to "Protocol error" as the ldap version
    of tldap is not set.

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

Merge pull request #12556 from Adirl/support_roceV2

msg/async/rdma: Support for RoCE v2 and SL

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge remote-tracking branch 'gh/kraken'

Merge pull request #12603 from liewegas/wip-fix-rwordered

osd: fix CEPH_OSD_FLAG_RWORDERED

Reviewed-by: Samuel Just <sjust@redhat.com>

osd: fix CEPH_OSD_FLAG_RWORDERED

Broke this in 3a2ed3b136de68192f661b6992cbb96ce6aafd20.

Signed-off-by: Sage Weil <sage@redhat.com>

Added "bluestore" prefix to new config options

Signed-off-by: Orlando Moreno <orlando.moreno@intel.com>

msg/async/rdma: Support for RoCE v2 and SL

Adding GID support to ceph.conf
Search configured GID in GIDs table with specific RoCE version.
Use it instead of index 0 (default)
Add service level support to supprt PFC

Change-Id: I5369bdeb6797e1c440110bdb4ad8e54ee6db9857
Signed-off-by: Oren Duer <oren@mellanox.com> Adir Lev <adirl@mellanox.com>

Merge pull request #12598 from ly798/typo-fix-in-docs

doc: typo fix in s3_compliance

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12601 from rishabhkumar296/ceph12684

msg/simple: Remove dead code in pipe.cc

Reviewed-by: Sage Weil <sage@redhat.com>

Remove dead code in pipe.cc

Fixes: http://tracker.ceph.com/issues/12684
Signed-off-by: Rishabh Kumar <kris.kr296@gmail.com>

osd/PrimaryLogPG: enlarge omap request bytes limitation

For cephfs, 4M buffer can only encode about 5k dentries. It's too
small for directory.

Fixes: http://tracker.ceph.com/issues/18314
Signed-off-by: Yan, Zheng <zyan@redhat.com>

doc: typo fix in s3_compliance

Signed-off-by: LiuYang <yippeetry@gmail.com>

os/bluestore: add perf variable for throttle info in bluestore

Signed-off-by: Pan Liu <pan.liu@istuary.com>

Merge pull request #12587 from yuriw/wip-fix-subset-script-master

qa: fixed script to schedule rados and other suites with --subset option

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

changed script name
added kraken
parameterized suite name to be able to use for rados, rbd etc.

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #12590 from athanatos/wip-ping-write-ordered

osd: force watch PING to be write ordered

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12593 from wjwithagen/wip-wjw-freebssd-SO_reuseaddr

msg/async/net_handler.cc: Do not apply SO_REUSEADDR for FreeBSD

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12584 from mkoderer/openstack_nova_doc

doc: mention corresponding libvirt section in nova.conf

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/async/net_handler.cc: Do not apply SO_REUSEADDR for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

osd/: treat PINGs as RWORDERED

89fd030bf9436dc4e37cc3a0f935ec077455d9d5 switched them to show up
as reads to avoid logging them, but we still pipeline them with
reconnects.  Thus, also force them to be rwordered.

Fixes: http://tracker.ceph.com/issues/18310
Signed-off-by: Samuel Just <sjust@redhat.com>

osd/osd_types: print notify-ack op properly

Not like "[notify-ack 0~0]"!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12540 from dmick/wip-18297

ceph.in: allow 'flags' to not be present in cmddescs

Merge pull request #12577 from liewegas/wip-osd-backfill

qa/tasks/osd_backfill.py: wait for osd.[12] to start

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12574 from liewegas/wip-bluestore-split

os/bluestore: preserve source collection cache during split

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

DOC: Mention corresponding section in nova.conf

rbd_user and rbd_secret_uuid are part of the libvirt configuration.

Signed-off-by: Marc Koderer <marc@koderer.com>

Merge pull request #12472 from dillaman/wip-18244

librbd: remove image header lock assertions

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

  osd/PG: check the connection first in fulfill_log
          No need to prepare reply message if the connection not exist

Signed-off-by: huangjun <hjwsm1989@gmail.com>

Merge pull request #12532 from ifed01/wip-bluestore-large-onode-test

os/bluestore: add UT for an estimation of Onode in-memory size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12505 from ceph/wip-jlayton-submount

client: set metadata["root"] from mount method when it's called with …

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12570 from wjwithagen/patch-9

common/str_list.h: fix clang warning about std::move

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12541 from liewegas/wip-bluestore-omap-flush

os/bluestore: include modified objects in flush list even if onode unchanged

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/tasks/osd_backfill.py: wait for osd.[12] to start

...before sending a tell command.  Otherwise osd.2 might
start without 1, the io unblocks, and the tell fails
because osd.1 is still down.

Fixes: http://tracker.ceph.com/issues/18303
Signed-off-by: Sage Weil <sage@redhat.com>

bluestore: avoid unnecessary copy with coll_t

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

osd/OSDMap: drop redundant statements

calc_pg_rank() will do the same check inside.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

common: make attempts of auth rotating configurable

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: allow multiple SPDK BlueStore OSD instances

Fixes: http://tracker.ceph.com/issues/16966
Signed-off-by: Orlando Moreno <orlando.moreno@intel.com>

os/bluestore: preserve source collection cache during split

OSD split transactions look something like

 mkcoll new
 split old
 ...
 omap_rmkey_range old
 omap_setkeys old
 omap_setkeys new

The last part splits the log into two pieces.  The
problem is that the rmkey_range needs to wait on old
omap transactions to flush, and those are linked to the
old onode, and split clears the cache.  The result is
that we don't wait, rmkeyrange leaves some recent pg log
keys behind, and on OSD restart we get an error because
the object doesn't belong to the (old) collection.

Fix this by preserving objects in the old collection and
only clear out objects that are moving to the newly
split collections.  This will include the pgmeta object
that we care about.

(Note that we are one step closer to preserving the
cache contents across the split, but not quite there
yet: at this point we don't have all of the destination
collections.  A change in the ObjectStore interface is
probably needed to make that not be extremely awkward.)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12571 from athanatos/wip-15943

osd: Fix map gaps again (bug 15943)

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12572 from liewegas/wip-python-rados-test

qa/workunits: include extension for nose tests