rgw: avoid corruption when running old radosgw-admin on a newer rgw

Fixes:http://tracker.ceph.com/issues/17371
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Fix missing master zone for default region conversion

Fixes:http://tracker.ceph.com/issues/17371
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: mark configuration as converted to avoid corruption when running older admins

Fixes: http://tracker.ceph.com/issues/17371
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10731 from yehudasa/wip-rgw-sync-plugins

rgw: sync modules, metadata search

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11363 from xiexingguo/xxg-wip-fix-rbd-warn

cls_rbd: silence compiler warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11371 from stiopaa1/mds_cinode_removeCount

mds/CInode.h: remove unneeded use of count

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11383 from stiopaa1/mds_fsmap_passSharedPtrByConstRef

mds/FSMap: pass shared_ptr by const ref

Merge pull request #11290 from trociny/wip-17017

rbd-mirror HA: move librbd::image_watcher::Notifier to librbd::object_watcher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11260 from runsisi/wip-fix-mirror-image-disable

librbd: fix rollback if failed to disable mirroring for image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11301 from liewegas/wip-bluestore-buffered-write

os/bluestore: allow default to buffered write

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #11370 from ceph/wip-krbd-unmap-options

rbd: expose rbd unmap options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11255 from dzafman/wip-scrub-boundary

osd: fix scrub boundary to not include a SnapSet

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11377 from xiexingguo/xxg-wip-bluestore-1008

os/bluestore: drop inline_dirty from struct ExtentMap

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: narrow scope of 'pos'

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11281 from jcsp/wip-17466

mds: handle bad standby_for_fscids in fsmap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: get rid of multiple call to dirty_blob()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10722 from stiopaa1/mon_moncap_addMoveToStrings

mon/MonCap.h: add std::move for std::string

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10916 from jordan41177/fix-17169

mon: OSDMonitor: fix the check error of pg creating

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10720 from wjwithagen/wip-wjw-freebsd-socket-shutdown

msg/simple/Accepter.cc: replace shutdown() with selfpipe event in poll() (FreeBSD)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10601 from stiopaa1/mon_monclient_subwantRemoveCount

mon/MonClient.h: remove repeated searching of map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11145 from atheism/msg-async-fix-specifier

msg: fix format specifier for unsigned value id

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/FSMap: pass shared_ptr by const ref

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

os/bluestore: fix raw blob encoded size

The maximum length of fixed fields of bluestore_blob_t shall be:

uint64_t sbid                       ///< 8 bytes
uint32_t compressed_length_orig     ///< 4 bytes
uint32_t compressed_length          ///< 4 bytes
uint32_t flags                      ///< 4 bytes
uint8_t csum_type                   ///< 1 byte
uint8_t csum_chunk_order            ///< 1 byte
unused_t unused;                    ///< 2 bytes

8 + 4 + 4 + 4 + 1 + 1 + 2 = 24

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add counter to trace number of garbage collection reads

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: kill dead lines to keep code clean

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11379 from zealoussnow/fix-dead-link

doc: fix dead link "Hardware Recommendations"

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix dead link "Hardware Recommendations"

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

os/bluestore: drop inline_dirty from struct ExtentMap

We don't need this member as we can simply clear inline_bl
to mark it as dirty.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rbd/cls_rbd: silence compiler warnings

Signed-off-by: xiexingguo <xie.xingguo@zte.com.cn>

mds/CInode.h: remove unneeded use of count

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10057 from Yan-waller/yj-wip-rgwradosccremover-0701

rgw: minor cleanup

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10674 from ceph/wip-rgw-no-empty-amzrqid

rgw/rest: don't print empty x-amz-request-id

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10849 from zhangsw/master

rgw: remove a redundant judgement when listng objects.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: allow default to buffered write

Add config option to buffered writes (in the absense of
a client WILLNEED hint).  Set the config option to false,
though, so that the default behavior is unchanged.

2Q should do a half-decent job of preventing this from
pushing actually-hot data from the cache when it's
enabled, but we'll need to do more testing first.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11353 from dillaman/wip-17423

librbd: API changes to support separate data pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rgw: index metadata in elasticsearch using realm name for path

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: sync status command shows if not syncing from zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: setting sync-from zone by name not by id

Using the zone name is easier and clearer.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: update usage

add refrence to --sync-from*

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: config options to set sync_from and sync_from_all

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: support partial mesh for zone sync

zone configuration now includes two new fields: sync_from_all
which is boolean, and sync_from, which is a least of zones to
sync from. By default sync_from_all is set to true. Sync will
only happen from all the zones, or from the specified zones if
sync_from all is false. We also check to see whether zone can
export data (depending on tier_type).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es module, remove entry on delete

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add cr to send DELETE to remove endpoint

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rest conn functions cleanup, only append zonegroup if not empty

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, store custom metadata

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, keep object mtime

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: utility function to dump iso8601

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, store acl information

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, store object attrs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: es sync module, send object info to elasticsearch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cmake: fix linkage of ceph_test_librgw_file_nfsns

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial implementation of elasticsearch sync module

sync module that will handle rgw metadata indexing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a new cr to send http PUT requests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: allow null store in RGWRESTConn

We're not necessarily going to connect to rgw/s3 endpoints,
we only need store param to handle s3 signing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: REST client, don't sign requests if empty key

If key is not passed in, don't try to sign the request.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move the rgw sync code module around

No real code change

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: some abstraction around log sync module

Moving code that fetches remote object meta to its own classes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: log sync module gets source object's meta

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate attrs, mtime, size of remote object

Use new rgwx-stat http param that allows getting only object's
meta. Use that when calling stat_remote_object().

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add cr to stat remote obj

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: helper to stat remote obj

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add a simple logging sync module

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: non-rgw tier is not writeable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define sync_module on RGWRados

Instead of having it as part of the data sync module. Since we only have a
single sync_module, having it there will make it easier to get its properties.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: can set/modify zone tier's config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add tier config for zone params

Needed for sync module instance configuration

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define zone tier type, sync from appropriate tiers only

Can only sync from tiers that can export data.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define sync modules manager, instance

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rbd: expose rbd unmap options

Reuse rbd map -o infrastructure to expose rbd unmap options in
a similar fashion.  Currently it's just one bool option, but we may
need more in the future.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: fix up terminology in help texts

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #11361 from batrick/fs-rm-formatting

mon: add missing space in warning message

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11364 from badone/wip-_lsb_release_-dead-declarations

osd: Remove unused '_lsb_release_' declarations

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11349 from liewegas/wip-compressor

compressor: fixes and tests; disable zlib isal (it's broken)

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11366 from xiexingguo/xxg-wip-bluestore-1007

os/bluestore: simplify blob status checking for small writes

Reviewed-by: Sage Weil <sage@redhat.com>

unittest_compression: naming nit

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compression: restore zlib_isal setting

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11236 from renhwztetecs/renhw-wip-cephfs-journal-tool

tool/cephfs: displaying "list" in journal event mode

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10021 from rzarzynski/wip-rgw-swift-auth-errors

rgw: handle Swift auth errors in a way compatible with new Tempests.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore: make randomizing value process of kvpair faster

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fsck() on idempotent mkfs() request

E.g., if the fsck() found something during the first mkfs() try,
it can still be caught by the following retry.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix comment

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: Remove unused '_lsb_release_' declarations

_lsb_release_set and _lsb_release_parse declarations still exist despite their
definitions being removed some time ago.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

os/bluestore: sanity check iterator of BufferSpace::_rm_buffer()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify blob status checking for small writes

If a blob is marked as compressed and still mutable, something
is wrong.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11302 from asheplyakov/rados-doc-fixup

doc: fix librados example programs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: use data sync module callbacks

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial data plugin definition and default implementation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mon: add missing space in warning message

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

unittest_compression: subsume plugin test

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compression: subsume _zlib and _snappy test cases

This includes zlib and isal interop.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compressor: smaller big test

Just so it runs a bit faster!

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/zlib: disable ISA-L

This fails the unit tests.  Do not reenable until it is fixed!

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/zlib: add compressor_zlib_isal config option to disable ISA-L

We dynamically enable this if the necessary processor features are present.
Allow this probing to be disabled explicitly.

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/snappy: fix decompress buffer sizing for large buffers

4 bytes is enough for small buffers, but fails for larger buffers because
snappy encodes the length as a varint.  8 is enough.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compressor: unified compression tests, parameterized by plugin

This obsoletes the per-plugin plugin tests, which are tedious anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: utilize new API methods for image id and block name prefix

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new API methods to retrieve image id and block name prefix

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: relocate image option processing to create state machine

Replace the individual image option parameters with the ImageOptions
wrapper and extract the options within the state machine itself.
This will simplify adding support for the data pool and other options
in the future.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add data pool option to API and config

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: fix compiler warning due to feature bit macro type change

Signed-off-by: Jason Dillaman <dillaman@redhat.com>