librbd: include pool id in data block prefix if data pool enabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: initialize the data pool IO context during image open state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: store data pool id with image header during creation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11355 from dillaman/wip-17424

rbd: add support for separate image data pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10731 from yehudasa/wip-rgw-sync-plugins

rgw: sync modules, metadata search

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11363 from xiexingguo/xxg-wip-fix-rbd-warn

cls_rbd: silence compiler warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11371 from stiopaa1/mds_cinode_removeCount

mds/CInode.h: remove unneeded use of count

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11383 from stiopaa1/mds_fsmap_passSharedPtrByConstRef

mds/FSMap: pass shared_ptr by const ref

rbd: include data pool name within info command

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: support overriding image data pool when creating images

Fixes: http://tracker.ceph.com/issues/17424
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11290 from trociny/wip-17017

rbd-mirror HA: move librbd::image_watcher::Notifier to librbd::object_watcher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11260 from runsisi/wip-fix-mirror-image-disable

librbd: fix rollback if failed to disable mirroring for image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11301 from liewegas/wip-bluestore-buffered-write

os/bluestore: allow default to buffered write

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #11370 from ceph/wip-krbd-unmap-options

rbd: expose rbd unmap options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11255 from dzafman/wip-scrub-boundary

osd: fix scrub boundary to not include a SnapSet

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11377 from xiexingguo/xxg-wip-bluestore-1008

os/bluestore: drop inline_dirty from struct ExtentMap

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: narrow scope of 'pos'

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11281 from jcsp/wip-17466

mds: handle bad standby_for_fscids in fsmap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: get rid of multiple call to dirty_blob()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10722 from stiopaa1/mon_moncap_addMoveToStrings

mon/MonCap.h: add std::move for std::string

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10916 from jordan41177/fix-17169

mon: OSDMonitor: fix the check error of pg creating

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10720 from wjwithagen/wip-wjw-freebsd-socket-shutdown

msg/simple/Accepter.cc: replace shutdown() with selfpipe event in poll() (FreeBSD)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10601 from stiopaa1/mon_monclient_subwantRemoveCount

mon/MonClient.h: remove repeated searching of map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11145 from atheism/msg-async-fix-specifier

msg: fix format specifier for unsigned value id

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/FSMap: pass shared_ptr by const ref

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

os/bluestore: fix raw blob encoded size

The maximum length of fixed fields of bluestore_blob_t shall be:

uint64_t sbid                       ///< 8 bytes
uint32_t compressed_length_orig     ///< 4 bytes
uint32_t compressed_length          ///< 4 bytes
uint32_t flags                      ///< 4 bytes
uint8_t csum_type                   ///< 1 byte
uint8_t csum_chunk_order            ///< 1 byte
unused_t unused;                    ///< 2 bytes

8 + 4 + 4 + 4 + 1 + 1 + 2 = 24

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add counter to trace number of garbage collection reads

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: kill dead lines to keep code clean

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11379 from zealoussnow/fix-dead-link

doc: fix dead link "Hardware Recommendations"

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix dead link "Hardware Recommendations"

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

os/bluestore: drop inline_dirty from struct ExtentMap

We don't need this member as we can simply clear inline_bl
to mark it as dirty.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rbd/cls_rbd: silence compiler warnings

Signed-off-by: xiexingguo <xie.xingguo@zte.com.cn>

mds/CInode.h: remove unneeded use of count

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10057 from Yan-waller/yj-wip-rgwradosccremover-0701

rgw: minor cleanup

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10674 from ceph/wip-rgw-no-empty-amzrqid

rgw/rest: don't print empty x-amz-request-id

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10849 from zhangsw/master

rgw: remove a redundant judgement when listng objects.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: allow default to buffered write

Add config option to buffered writes (in the absense of
a client WILLNEED hint).  Set the config option to false,
though, so that the default behavior is unchanged.

2Q should do a half-decent job of preventing this from
pushing actually-hot data from the cache when it's
enabled, but we'll need to do more testing first.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11353 from dillaman/wip-17423

librbd: API changes to support separate data pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rgw: index metadata in elasticsearch using realm name for path

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: sync status command shows if not syncing from zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: setting sync-from zone by name not by id

Using the zone name is easier and clearer.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: update usage

add refrence to --sync-from*

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: config options to set sync_from and sync_from_all

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: support partial mesh for zone sync

zone configuration now includes two new fields: sync_from_all
which is boolean, and sync_from, which is a least of zones to
sync from. By default sync_from_all is set to true. Sync will
only happen from all the zones, or from the specified zones if
sync_from all is false. We also check to see whether zone can
export data (depending on tier_type).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es module, remove entry on delete

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add cr to send DELETE to remove endpoint

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rest conn functions cleanup, only append zonegroup if not empty

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, store custom metadata

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, keep object mtime

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: utility function to dump iso8601

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, store acl information

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, store object attrs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, send object info to elasticsearch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cmake: fix linkage of ceph_test_librgw_file_nfsns

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial implementation of elasticsearch sync module

sync module that will handle rgw metadata indexing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a new cr to send http PUT requests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: allow null store in RGWRESTConn

We're not necessarily going to connect to rgw/s3 endpoints,
we only need store param to handle s3 signing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: REST client, don't sign requests if empty key

If key is not passed in, don't try to sign the request.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move the rgw sync code module around

No real code change

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: some abstraction around log sync module

Moving code that fetches remote object meta to its own classes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: log sync module gets source object's meta

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate attrs, mtime, size of remote object

Use new rgwx-stat http param that allows getting only object's
meta. Use that when calling stat_remote_object().

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add cr to stat remote obj

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: helper to stat remote obj

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add a simple logging sync module

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: non-rgw tier is not writeable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define sync_module on RGWRados

Instead of having it as part of the data sync module. Since we only have a
single sync_module, having it there will make it easier to get its properties.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: can set/modify zone tier's config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add tier config for zone params

Needed for sync module instance configuration

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define zone tier type, sync from appropriate tiers only

Can only sync from tiers that can export data.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define sync modules manager, instance

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rbd: expose rbd unmap options

Reuse rbd map -o infrastructure to expose rbd unmap options in
a similar fashion.  Currently it's just one bool option, but we may
need more in the future.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: fix up terminology in help texts

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #11361 from batrick/fs-rm-formatting

mon: add missing space in warning message

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11364 from badone/wip-_lsb_release_-dead-declarations

osd: Remove unused '_lsb_release_' declarations

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11349 from liewegas/wip-compressor

compressor: fixes and tests; disable zlib isal (it's broken)

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11366 from xiexingguo/xxg-wip-bluestore-1007

os/bluestore: simplify blob status checking for small writes

Reviewed-by: Sage Weil <sage@redhat.com>

unittest_compression: naming nit

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compression: restore zlib_isal setting

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11236 from renhwztetecs/renhw-wip-cephfs-journal-tool

tool/cephfs: displaying "list" in journal event mode

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10021 from rzarzynski/wip-rgw-swift-auth-errors

rgw: handle Swift auth errors in a way compatible with new Tempests.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore: make randomizing value process of kvpair faster

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fsck() on idempotent mkfs() request

E.g., if the fsck() found something during the first mkfs() try,
it can still be caught by the following retry.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix comment

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: Remove unused '_lsb_release_' declarations

_lsb_release_set and _lsb_release_parse declarations still exist despite their
definitions being removed some time ago.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

os/bluestore: sanity check iterator of BufferSpace::_rm_buffer()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify blob status checking for small writes

If a blob is marked as compressed and still mutable, something
is wrong.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11302 from asheplyakov/rados-doc-fixup

doc: fix librados example programs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: use data sync module callbacks

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial data plugin definition and default implementation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mon: add missing space in warning message

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

unittest_compression: subsume plugin test

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compression: subsume _zlib and _snappy test cases

This includes zlib and isal interop.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compressor: smaller big test

Just so it runs a bit faster!

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/zlib: disable ISA-L

This fails the unit tests.  Do not reenable until it is fixed!

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/zlib: add compressor_zlib_isal config option to disable ISA-L

We dynamically enable this if the necessary processor features are present.
Allow this probing to be disabled explicitly.

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/snappy: fix decompress buffer sizing for large buffers

4 bytes is enough for small buffers, but fails for larger buffers because
snappy encodes the length as a varint.  8 is enough.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compressor: unified compression tests, parameterized by plugin

This obsoletes the per-plugin plugin tests, which are tedious anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: utilize new API methods for image id and block name prefix

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new API methods to retrieve image id and block name prefix

Signed-off-by: Jason Dillaman <dillaman@redhat.com>