osd/ReplicatedPG: do not send ACK separate from COMMIT

This never comes before COMMIT, making it useless to clients.  And it's
just unnecessary overhead.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove 'rlat' read latency metrics

These are measuring how long of a delay we see before ops are readable.
They are calculated in the on_applied path, which we'd like to remove,
and weren't reliably calculated anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: kill hit_set_flushing

This was added before we started taking the ondisk lock in order to allow
reading a just-written hitset.  Since this is a rare path anyway, just
take the ondisk lock.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12522 from hjwsm1989/ecwrite

osd/ECBackend: Send write message to peers first, then do local write

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12524 from yuyuyu101/wip-bluestore-flush

os/bluestore: prevent lock for almost "flush" calls

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12528 from kylinstorage/wip-JournalingObjectStore-cleanup

os/filestore/JournalingObjectStore cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12670 from liewegas/wip-nits

mon: fix a few nits

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12684 from xiexingguo/xxg-wip-fix-onodesizetracking

os/bluestore: fix OnodeSizeTracking testing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12707 from tchaikov/wip-dpdk-v16.07

builds with dpdk v16.07

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: bump dpdk to v16.07

as some distros include it already. so sync it up to avoid building
failures on them with dpdk-dev package(s) shipped by distro.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: be compatible with dpdk v16.07

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12698 from yuyuyu101/wip-dpdk-fix

msg/async/dpdk: fix compile errors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12699 from wjwithagen/patch-15

qa/test_rados_tool.sh: POSIX dd only accepts 'k' as multiplier

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12703 from kylinstorage/wip-remove-unused-variable

os/bluestore/BlueStore.cc: remove unused variable

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12594 from wjwithagen/wip-wjw-freebsd-rebind-timing

common/config_opts.h: FreeBSD timing changed due to no SO_REUSEADDR

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore.cc: remove unused variable

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

qa/test_rados_tool.sh: POSIX dd only accepts 'k' as multiplier

FreeBSD does not want KB as 1k multiplier.
see also: http://www.unix.com/man-page/posix/1posix/dd/

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

test_userspace_event: fix missing std

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/dpdk/TCP: remove unneeded CephContext argument

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12688 from Adirl/clean_line_end

msg/async/rdma: clean line endings

Reviewed-by: Haomai Wang <haomai@xsky.com>

os/bluestore: fix OnodeSizeTracking testing

[  FAILED  ] 1 test, listed below:
[  FAILED  ] ObjectStore/StoreTest.OnodeSizeTracking/2, where GetParam() = "bluestore"

 1 FAILED TEST

The above test failure happens as the bluestore mount() process
will try to load all collections and put them into the coll_map,
which will be also tracked as mempool::bluestore_meta_other.
So total_bytes from mempool won't be equal to zero.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12336 from kylinstorage/wip-cleanup-use-string-ref

osd: cleanup: use string & to avoid unnecessary copy

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12569 from liewegas/wip-bluestore-raw-unused

os/bluestore/bluestore_types: drop std::bitset for blob unused

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #12612 from fangyuxiangGL/master

osd/ReplicatedBackend: do not set omap header if it is empty

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12668 from hjwsm1989/pg-unfound

osd/PG: add new have_unfound() function in MissingLoc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12674 from Liuchang0812/tools-rados-offset-support

tools/rados: add a parameter "--offset" to rados put command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12675 from Liuchang0812/fix-test-case-in-rados

test: replace hard-code binary names with varibles

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: ceph osd pool get/set crush_rule by name

This means users don't have to manually translate a rule
they just created to a ruleset ID in order to map a pool
to it.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: remove CATEGORY column from 'df detail'

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: fix alignment of 'cache io' in ceph -s output

Currently it looks like

  client io 3508 B/s wr, 0 op/s rd, 5 op/s wr
  cache io 0 op/s promote

but should be

  client io 3508 B/s wr, 0 op/s rd, 5 op/s wr
   cache io 0 op/s promote

Signed-off-by: Sage Weil <sage@redhat.com>

mon: 'fs set-default' not 'fs set_default'

The convention is to use - for command names and
_ for variable names.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12655 from ceph/wip-fix-fio

test/fio_ceph_objectstore: fix fio plugin build failure caused by rec…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12682 from renhwztetecs/renhw-wip-mon-osdmonitor-sortbitwise

mon/OSDMonitor: fixup sortbitwise flag warning

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12654 from yuyuyu101/wip-fix-async-subop

messages/MOSDSubOp: Make encode_payload can be reentrant

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12662 from Liuchang0812/doc-fix

doc: fix document about rados mon

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12198 from tchaikov/wip-pybind-freebsd

pybind: conditional compile the linux specific constant

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

mon/OSDMonitor: fixup sortbitwise flag warning

"ceph -s" does not report warning when using
command "ceph osd unset sortbitwise" to drop
sortbitwise flag.
we should use "osdmap.get_up_osd_features() &
CEPH_FEATURE_OSD_BITWISE_HOBJ_SORT"
instead of "(osdmap.get_features(CEPH_ENTITY_TYPE_OSD, NULL) &
CEPH_FEATURE_OSD_BITWISE_HOBJ_SORT)",
because osdmap.get_features only get local "features"

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

ReplicatedBackend: do not set omap header if it is empty

In replicatedpg backfill, objects have no omap header or keys previously, but an empty omap header will be inserted when they are pushed to replica, which leads leveldb performance degrade seriously when cluster has huge number of objects.
so my opinion is that we does't insert omap header if is size is zero when handle PushOp in recovery.

Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>

tools/rados: add parameter offset for put cmd and test case for it

add a parameter "--offset" for put command
add a test case for rados put command

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

doc: fix document about rados mon

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12663 from badone/wip-ceph_test_rados_api_c_write_operations-valgrind-fixes

c_write_operations.cc: Fix trivial memory leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: replace hard-code binary names with varibles

This test case couldn't run under customize env CEPH_BIN.

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12672 from liupan1111/wip-fix-nvme-driver-compile-issue

bluestore: fixed compilation error when enable spdk

Reviewed-by: Haomai Wang <haomai@xsky.com>

bluestore: fixed compilation error when enable spdk

Signed-off-by: Pan Liu <pan.liu@istuary.com>

msg/async/rdma: clean line endings

Change-Id: I59359ae585b66018fe54d5e841a6506c077c606f
Signed-off-by: Adir Lev <adirl@mellanox.com>

Merge pull request #12659 from kylinstorage/wip-remove-obsolete-configuration-option

common/config_opts.h: remove obsolete configuration option

Reviewed-by: Sage Weil <sage@redhat.com>

  osd/PG: add new have_unfound() function in MissingLoc
          No need to lookup all objects in needs_recovery_map if only want to
          know whether if there are unfound objects.

Signed-off-by: huangjun <hjwsm1989@gmail.com>

Merge pull request #12648 from Adirl/fix_RoCEv2_deafult_value

msg/async/rdma: fix RoCE v2 deafult value

Reviewed-by: Haomai Wang <haomai@xsky.com>

c_write_operations.cc: Fix trivial memory leak

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12657 from badone/wip-c_read_operations-valgrind-fixes

test/librados/c_read_operations.cc: Fix valgrind errors

Reviewed-by: liuchang0812 <liuchang0812@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12660 from liupan1111/wip-fix-dpdk-compile

async: Fixed compilation error when enable -DWITH_DPDK

Reviewed-by: Haomai Wang <haomai@xsky.com>

async: Fixed compilation error when enable -DWITH_DPDK

Signed-off-by: Pan Liu <pan.liu@istuary.com>

test/librados/c_read_operations.cc: Fix valgrind errors

Fixes: http://tracker.ceph.com/issues/18354
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

common/config_opts.h: remove obsolete configuration option

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

Merge pull request #12474 from cbodley/wip-rgw-test-versioned

test/rgw: add test for versioned object sync
Reviewed-by: Orit wasserman <owasserm@redhat.com>

Merge pull request #12597 from badone/wip-osdmap-nocopy

osd/OSDMap: Uncomment code to enable private default constructors

Reviewed-by: Sage Weil <sage@redhat.com>

test/fio_ceph_objectstore: fix fio plugin build failure caused by recent g_ceph_context removal

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: prevent lock for almost "flush" calls

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12642 from theanalyst/doc/release-notes/11.1.1

doc: release notes for v11.1.1

Merge pull request #12582 from Liuchang0812/fix-store-version-bug

os/filestore: version will be uninitialized varible if store_version doesn't exist

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12581 from Liuchang0812/fix-clang-warn-filestore

os/filestore: fix clang static check warn "use-after-free“

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12517 from hjwsm1989/wip-18266

messages/MOSDPGTrim: add the missed HEAD_VERSION AND COMPAT_VERSION

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12518 from kylinstorage/wip-safe-read-cmd

common/admin-socket: fix potential buffer overflow

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12653 from wjwithagen/wjw-wip-denc-raw

src/test/test_denc.cc: Fix errors in buffer overflow

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12469 from bassam/pr-jemalloc-mallinfo

cmake: disable mallinfo for jemalloc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12468 from bassam/pr-simplify-jemalloc

cmake: simplify find_package jemalloc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12519 from hjwsm1989/pgbackfill

osd/PG: simplify the logic of backfill_targets checking

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12585 from liewegas/wip-notify-ack

osd/osd_types: print notify-ack op properly

Merge pull request #12563 from xiexingguo/xxg-wip-osd-1217

common: make attempts of auth rotating configurable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11065 from LiumxNL/wip-fix-osdmap-update

mon/OSDMonitor: should propose osdmap update when cluster addr changed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12634 from liewegas/wip-bluestore-shard-keys

os/bluestore: do not cache shard keys

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

Merge pull request #12635 from liewegas/wip-bluestore-cache

os/bluestore: set cache meta ratio to .9

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12644 from xiexingguo/xxg-wip-bluestore-keyreserve

os/bluestore: preallocate object[extent_shard] key to avoid reallocate

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

messages/MOSDSubOp: Make encode_payload can be reentrant

Otherwise, AsyncConnection will use uninitialized feature to encode
message. The caller side will set a old version header but encode with newest
feature since passing feature is ok. It will let receiver side got a old
header version but newest payload decode result.

Signed-off-by: Haomai Wang <haomai@xsky.com>

src/test/test_denc.cc: Fix errors in buffer overflow

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

os/bluestore: do not precalculate/cache shard keys

Note that we are generating it on demand now.  We can
probably do better, especially when in a loop.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12149 from adamemerson/wip-g_ceph_context-exterminate

global: start removing g_ceph_context

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/kraken'

Merge pull request #12630 from liewegas/wip-workunit-retry

qa/tasks/workunit: clear clone dir before retrying checkout

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12614 from liewegas/wip-task-peer

qa/tasks/peer: update task based on current peering behavior

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12559 from Adirl/verify_transport_type

msg/async: assert if compiled code doesn't support the configured ms_…

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: assert if compiled code doesn't support the configured ms_async_transport_type

for example, currently there is a crash when using the async stack (access of null pointer)
in case the user configured rdma for a code that was not compiled with RDMA

issue: 913397

Change-Id: I4e3d88006312a713b8d3aaefbdcd0596039270e1
Signed-off-by: Avner BenHanoch <avnerb@mellanox.com>

Merge pull request #12643 from xiexingguo/xxg-wip-bluestore-1123

os/bluestore: remove intermediate key var to avoid string copy

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: whu liuchang <liuchang0812@gmail.com>

msg/async/rdma: fix RoCE v2 deafult value

For backward compatibility, if no GID given or given GID is malformed,
use GID index 0.

Change-Id: I8c8ecbcfc0261e58c5a226ef6bc91297ec756182
Signed-off-by: Oren Duer <oren@mellanox.com> Adir Lev <adirl@mellanox.com>
Signed-off-by: Adir Lev <adirl@mellanox.com>

os/bluestore: preallocate object[extent_shard] key to avoid reallocate

which is less efficient.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove intermediate key var to avoid string copy

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: release notes for v11.1.1

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #12636 from badone/wip-doc-pg-states

doc: Remove "splitting" state

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12543 from hjwsm1989/cleanup2

  osd/ECBackend: remove unused variable "ReadCB"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Remove "splitting" state

There is no splitting state so remove it from pg-states.rst

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12632 from liewegas/wip-centos-73

qa/suites: switch from centos 7.2 to 7.x

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge remote-tracking branch 'gh/kraken'

# Conflicts:
# src/os/bluestore/BlueStore.cc

os/bluestore: set cache meta ratio to .9

More metadata, less data.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/distros: centos_7.yaml -> centos.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites: centos_7.2.yaml -> centos_7.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

qa/distros: add centos 7.3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/distros: add centos 7 yaml; use that instead

No need to be picky about the point release here.

Signed-off-by: Sage Weil <sage@redhat.com>

msg: Remove mention of g_ceph_context

Since it won't exist any more there's no point in saying it can't be
relied on, now is there?

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

objclass: Utterly destroy g_ceph_context

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os: Complete removal of g_ceph_context and g_conf

And so it continues.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>