Merge pull request #11510 from vshankar/wip-17356

librbd: batch object map updates during trim

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11766 from trociny/wip-rbd-info-fix-upgrade-client

librbd: snap_get_limit compatibility check

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10907 from jecluis/wip-mon-features

mon: have mon-specific features & rework internal monmap structures

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11772 from liewegas/wip-sortbitwise

osd,mon: require sortbitwise flag to upgrade beyond jewel

Merge pull request #11791 from liewegas/wip-remove-move-ranges

os: remove move_ranges_destroy_src

Merge pull request #10447 from stiopaa1/msg_pipe_moveDelayedDelivery2

msg/Pipe: move DelayedDelivery class to cc file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11780 from xiexingguo/xxg-wip-bluestore-1104

os/bluestore: a few more cleanups

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11498 from liewegas/wip-stop-at-jewel

mon: prevent post-jewel OSDs from booting if require_jewel_osds is not set

os/bluestore: fix output message when passing the end of list range

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix typo in comments

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check against collection existence

To keep pace with other methods.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: narrow boundary check for fiemap

By including off-by-one.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11527 from wjwithagen/wip-wjw-readable.sh

test/encoding/readable.sh: fix shell script warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11775 from jdurgin/wip-jewel-osds-warning

mon/OSDMonitor: remove duplicate jewel/kraken flag warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9345 from SUSE/wip_remove_mon_startup_magic

mon: remove ceph-create-keys from mon startup

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

src/init-ceph.in:Remove ceph-create-keys for sysv

ceph-create-keys should not be started on boot of mons with sysv init.

Signed-off-by: Owen Synge <osynge@suse.com>

systemd/CMakeLists.txt:Remove ceph-create-keys cmake

ceph-create-keys should not be started on boot of mons with systemd so should
not exist as 'After' or 'Wants' for the ceph-mon.service

Signed-off-by: Owen Synge <osynge@suse.com>

systemd/ceph-mon@.service:Remove ceph-create-keys for mon in systemd

ceph-create-keys should not be started on boot of mons with systemd so should
not exist as 'After' or 'Wants' for the ceph-mon.service

Signed-off-by: Owen Synge <osynge@suse.com>

systemd/ceph-create-keys@.service:Remove ceph-create-keys for systemd

ceph-create-keys should not be started on boot of mons with systemd so should
not exist in the systemd files

Signed-off-by: Owen Synge <osynge@suse.com>

src/upstart/ceph-create-keys.conf:Remove ceph-create-keys for upstart

ceph-create-keys should not be started on boot of mons with upstart so should
not exist in the upstart files

Signed-off-by: Owen Synge <osynge@suse.com>

debian/rules:Remove ceph-create-keys for deb packaging

ceph-create-keys should not be started on boot of mons with upstart so should
not exist in the deb.

Signed-off-by: Owen Synge <osynge@suse.com>

ceph.spec.in:Remove ceph-create-keys for systemd

ceph-create-keys should not be started on boot of mons with systemd so should
not exist in the rpm.

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #11749 from ceph/wip-17753

ceph-create-keys should not try forever to do things

Merge pull request #11788 from liewegas/wip-bluestore-blackhole

os: make filestore_blackhole -> objectstore_blackhole

os: remove move_ranges_destroy_src

We are going to take a different path (rollback support in ObjectStore
instead of roll-forward via splicing data from a temp object).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11747 from jecluis/wip-17725

mon: MonmapMonitor: return success when monitor will be removed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9768 from cxwshawn/wip-tc1-fix

log: optimize header file dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9288 from yuyuyu101/wip-add-shutdown-log

osd: print log when osd want to kill self

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11770 from liewegas/wip-kill-tbl

os/ObjectStore: remove legacy tbl support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11785 from Liuchang0812/patch-1

doc: Update crush-map.rst, fix a typo mistake

os/bluestore: implement blackhole

Signed-off-by: Sage Weil <sage@redhat.com>

os: add objectstore_blackhole option; deprecate filestore_blackhole

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11531 from yuyuyu101/wip-rdma

msg/async: ibverbs/rdma support

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mellanox <adirl@mellanox.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #11637 from liupan1111/master

common: make l_finisher_complete_lat more accurate

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Update crush-map.rst

fix typo

Merge pull request #11771 from liewegas/wip-bluestore-collection-end

os/bluestore: fix collection_list end bound off-by-one

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11784 from jcsp/wip-jcsp-docs

Documentation syntax cleanup

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/cephfs: separate front page section for dev stuff

The "additional details" section was getting longer
and longer, and I don't want developer-only stuff
in there anyway.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/mgr: fix rst syntax in plugin guide

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: fix rst syntax in mantle.rst

Signed-off-by: John Spray <john.spray@redhat.com>

cephfs/capabilities: fix rst syntax

Although we don't link this into built docs,
it generates warnings during build.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: s/CephFS Quotas/Quotas/

This looks odd in the TOC (we're already
in the cephfs section)

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: fix header depths

Some files had multiple top-level headers resulting in
subsections being listed in the TOC alongside the
main page header.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: link in mantle doc

...and add a warning at the very top of the document
that it is not for ordinary user use.

Considered just not linking this in at all, but it
seems a shame to have it here but not be able to
e.g. google it and link to it.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11741 from trociny/wip-17219

rbd-nbd: disallow mapping images >2TB in size

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11744 from trociny/wip-16962

rbd-mirror: snap protect of non-layered image results in split-brain

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11755 from trociny/wip-17791

librbd: proper check for get_data_pool compatibility

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10852 from wido/doc-mon-dns

doc: Add docs about looking up Monitors through DNS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11774 from wjwithagen/wip-wjw-test_bloom_filter

test_bloom_filter.cc: Fix a mismatch for the random_seed parameter

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11731 from kylinstorage/wip-avoid-unnecessary-verify

FileStore: Only verify split when it has been really done and done correctly

Reviewed-by: Kefu Chai <kchai@redhat.com>

OSDMonitor: remove duplicate jewel/kraken flag warning

Since 6244755a7051abe2e89d9f1ff75a10c4b76ee3b5 and
12e508313dbd5d1d38c76859cb7de2ce22404e12, the original check added in
ae0d6eb1c01aa74a2c44c017c0c5b06697348d2d (perhaps by a bad
rebase/merge?)  is duplicated and doubles the warning messages.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

test/common/test_bloom_filter.cc: Fix a mismatch for the random_seed parameter

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mon/OSDMonitor: require 'sortbitwise' before 'require_jewel_osds'

This forces users upgrading past jewel to first set
'sortbitwise', which in turn means that post-jewel OSDs
do not need to support nibblewise sort.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do not boot unless SORTBITWISE and REQUIRE_JEWEL flags are set

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11554 from mslovy/wip-fix-fiemap

os/FileStore: fix fiemap issue in xfs when #extents > 1364

Reviewed-by: Sage Weil <sage@redhat.com>

test: add test for fiemap xfs issue when #extents > 1364

Signed-off-by: Ning Yao <yaoning@unitedstack.com>

FileStore:: fix fiemap issue in xfs when #extents > 1364

Signed-off-by: Ning Yao <yaoning@unitedstack.com>

os/kstore: fix collection_list end bound off-by-one

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix collection_list end bound off-by-one

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test collection_list end bound

Ensure that it is not included in the result!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11617 from umullan/brokenlinks

doc: Fixing the broken hyperlinks by pointing to correct documentation.

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: snap_get_limit compatibility check

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10707 from SUSE/wip-fix-suse-usermod

rpm: Remove trailing whitespace in usermod command (SUSE)

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11494 from aclamk/wip-rgw-compression

[RGW] Wip rgw compression
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd/ReplicatedBackend: let MOSDRepOP record the correctly largest_data_off_in_data_bl.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/ObjectStore: fix bug about calc largest_data_off_in_data_bl.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/ObjectStore: remove the legacy tbl support completely.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore: fix build error

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11768 from linuxbox2/xio-build-2

xio: fix build

Merge pull request #11769 from liewegas/wip-option-name

kv/RocksDBStore: rename option

Reviewed-by: Mark Nelson <mnelson@redhat.com>

kv/RocksDBStore: rename option

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11767 from ceph/wip-cmake

cmake: pass CMAKE_BUILD_TYPE down to rocksdb

Reviewed-by: Bassam Tabbara <bassam.tabbara@quantum.com>

xio: build fixes (some unittest targets need XIO_LIBRARY)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #11748 from liewegas/wip-bluestore-split

os/bluestore: fix _split_collections race with osr_reap

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

xio: build fixes (XioMessenger lacks features argument)

Applies the constructor signature change to the uses in
xio_client and xio_server test programs.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

xio:  compile fix removing undeclared 'features'

This fix was proposed on ceph-devel and acked by Sage Weil.
The corresponding change in in the later print is obvious
from context.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

os/bluestore: fix _split_collections race with osr_reap

The SharedBlobSet may not yet be empty at split time because previous
transactions may not have been reaped, leaving some Blobs still alive
even after the cache refs are cleared.  Drop them explicitly here (they
will go away shortly).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9444 from Adirl/dout_prefix

xio: provide dout_prefix for XioConnection

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: pass CMAKE_BUILD_TYPE down to rocksdb

so rocksdb share the same ${CMAKE_BUILD_TYPE} with ceph

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11704 from dillaman/wip-17752

librbd: do not create empty object map object on image creation

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11696 from tanghaodong25/rocksdb-breakdown-wip

kv/rocksdb: enable rocksdb write path breakdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11752 from stiopaa1/mds_mdsrank_addConstToMemFunc

mds/MDSRank: add const to member functions

Merge pull request #11761 from xiexingguo/xxg-wip-fix-envmirror-cleanup

os/bluestore: fixes and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11762 from atheism/doc-rgw-s3-javaapi

doc: rgw: fix a typo in S3 java api example

librbd: proper check for get_data_pool compatibility

Fixes: http://tracker.ceph.com/issues/17791
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-nbd: disallow mapping images >2TB in size

Fixes: http://tracker.ceph.com/issues/17219
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

doc: rgw: fix a typo in S3 java api example

    ObjectSummary.getSize() should be objectSummary.getSize()

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

Merge pull request #11753 from yehudasa/wip-17790

rgw: support for x-robots-tag header
Reviewed-by: Orit wasserman <owasserm@redhat.com>

doc: Fixing the broken hyperlinks by pointing to correct documentation.
Signed-off-by: umullangi@walmartlabs.com

os/bluestore: fix improper call of shutdown() to mempool_thread

Because below here we haven't called mempool_thread.init() yet,
and call to mempool_thread.shutdown() will therefore raise an assert instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop safe check against g_conf

We are using g_conf unconditionly everywhere...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop redundant boundary check

As we have done the sanity check ahead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix "rm -rf" cmd

The original cmd won't do the tidy-up work completedly.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

librbd: batch ObjectMap updations upon trim

Shrinking a clone which has snapshots and does not share
majority of objects with its parent (i.e., there are less
objects to be copied up) involves huge number of object
map updates -- two (pre, post) per object. This results
in lots of requests to be send to OSDs especially when
trimming a gigantus image. This situation can be optimized
by sending batch ObjectMap updates for an object range
thereby significantly cutting down OSD traffic resulting
in faster trim times.

Fixes: http://tracker.ceph.com/issues/17356
Signed-off-by: Venky Shankar <vshankar@redhat.com>

tools: monmaptool: support monmap features

Adds three new options:

  --feature-list     will list monmap features, as well as available
                     features.
  --feature-set      sets a feature on the monmap
  --feature-unset    unsets a feature on the monmap

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

test/mon: few more cases for mon_feature_t

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: Monitor: mon debug cli via admin socket

Purpose: allow a dev to test certain portions of the monitors on-the-fly
that would otherwise be nearly impossible without requiring shutting
them off.

Description:

  Currently we are adding support to debug monitors on-the-fly via the
  ceph tool cli, via the admin socket.

  An initial attempt was made to make this interface available via
  over-the-network ceph cli, but that would require a quorum to exist,
  or significant changes to be made to the handling of commands to allow
  this interface to be an exception to the rule.

  Therefore, this interface will live on the admin sockets for the time
  being; potentially forever, because the jury is still out on whether
  there's added value making this available via any other medium that's
  not the admin socket.

  The work means to test the newly added monmap features. This new
  interface will allow one to set, unset and list supported features.

  This will make it easier to implement functional tests for the
  features themselves, for quorum features that are expected to be set
  on-quorum, as well as to test upgrade paths from version A to version
  B.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonMap: move functions from header to source file

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonMap: add a new test instance

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: MonMap: mon addr and all now held by mon_info_t

Instead of keeping several maps mapping back and forth from string to
entity_addr_t for each monitor, have the monitor's address and name in
purpose-specific class, mon_info_t, and everything references it
instead.

Although the benefits may not be obvious right now, it will allow us to
add other ip addresses for the monitors a bit more easily, instead of
having to kludge around the existing structures.

We also keep compatibility with older versions by maintaining the
traditional 'mon_addr' map. This is a logical change inside the monitor,
so we can easily accomodate older monitors without having to rely on
quorum features; just needs a bit of care and foo.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>