librbd: track maximum object count during mirror sync

If the sync is interrupted, it is possible the image the source
image is shrunk.  Therefore, the sync process needs to track
the maximum known object extent of the image so objects can
be properly freed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: test cases for rbd-mirror image sync point state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: new image sync point create/prune state machines

These are responsible for creating/removing snapshots in the
remote image and updating the client registration to account
for the snapshots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added equality operator to MirrorPeer journal types

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: async version of snap create and snap remove

This will be utilized by the RBD mirror daemon when creating
sync points.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: prefix local maintenance ops methods

This will help to differentiate between the methods that send an RPC
request if the lock isn't owned vs execute locally if it is owned.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: maintenance ops now send asynchronous requests

The RBD mirror daemon will need to request the creation/removal
of snapshots from a remote image.  These operations need to be
performed asynchronously as opposed to the current synchronous
style of the librbd API.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track starting point snapshot for mirror sync point

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: move get_mock_io_ctx helper method to common location

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: unit test cases for rbd::mirror::image_sync::SnapshotCopyRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: sync snapshots between remote-local images during bootstrap

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track local to peer snapshot id mapping within journal

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: unit test cases for rbd::mirror::image_sync::ObjectCopyRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: updated mock expectation for librados remove op

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: added missing forward declarations for librbd mocks

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd_mirror: deep-copy of object and its associated snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper method for wrappign rados safe callbacks

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: updated API for calc_snap_set_diff

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: calc_snap_set_diff now provides last known object size

This allows truncate requests to be detected between snapshots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: support for truncating/removing with old snap contexts

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8066 from yehudasa/wip-rgw-swift-versioning-disabled

Wip rgw swift versioning disabled

Reviewed-by: Josh Durgin <joshd@redhat.com>

Merge pull request #7985 from linuxbox2/rgw-token-7

rgw ldap (Matt Benjamin)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge branch 'wip-15086' of git://github.com/SUSE/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8068 from SUSE/wip-15104

qa/workunits/rbd: use POSIX function definition

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7663 from javacruft/wip-pybind-flags

pybind: Ensure correct python flags are passed

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7856 from jdurgin/wip-cython-cleanup

configure.ac: update help strings for cython

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge remote-tracking branch 'origin/wip-josh-testing'

Merge pull request #8067 from athanatos/wip-fix-filestore-warning

FileStore: fix initialization order for m_disable_wbthrottle

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/workunits/rbd: use POSIX function definition

In scripts with /bin/sh shebang, we must use POSIX function definitions to
avoid syntax errors on platforms that link /bin/sh to something other than
/bin/bash.

See https://wiki.ubuntu.com/DashAsBinSh#function for more information.

http://tracker.ceph.com/issues/15104 Fixes: #15104

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: disable swift versioning by default

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: improve naming of Swift versioned object's copies.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: dump X-Versions-Location HTTP header of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: copy swift versioned object to destination if overwritten

Only allow if target bucket owner is the source bucket owner.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: swift object versioning data structure changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8034 from SUSE/wip-15047

packaging: make infernalis -> jewel upgrade work

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

debian: make infernalis -> jewel upgrade work

Attempts to install jewel ceph-common, ceph-mon, ceph-osd, and ceph-base
package over infernalis ceph package fail due to files existing in both.

See comment #4 in the tracker issue for a deeper analysis.

http://tracker.ceph.com/issues/15047 Fixes: #15047

Signed-off-by: Nathan Cutler <ncutler@suse.com>

FileStore: fix initialization order for m_disable_wbthrottle

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph-detect-init: return systemd on Debian Jessie

http://tracker.ceph.com/issues/15086 Fixes: #15086

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Revert "test/time: no need to abs(uint64_t) for comparing"

This reverts commit da0f6608dd6cc3356e727e739575f496c2b6a590.

unittest_time fails:

[==========] Running 4 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from RealClock
[ RUN      ] RealClock.Sanity
[       OK ] RealClock.Sanity (0 ms)
[ RUN      ] RealClock.Conversions
test/common/test_time.cc:125: Failure
Expected: ((Clock::from_double(bd) - brt).count()) < (30U), actual: 18446744073709551595 vs 30
[  FAILED  ] RealClock.Conversions (0 ms)
[----------] 2 tests from RealClock (0 ms total)

[----------] 2 tests from CoarseRealClock
[ RUN      ] CoarseRealClock.Sanity
[       OK ] CoarseRealClock.Sanity (0 ms)
[ RUN      ] CoarseRealClock.Conversions
test/common/test_time.cc:125: Failure
Expected: ((Clock::from_double(bd) - brt).count()) < (30U), actual: 18446744073709551595 vs 30
[  FAILED  ] CoarseRealClock.Conversions (0 ms)
[----------] 2 tests from CoarseRealClock (0 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 2 test cases ran. (0 ms total)
[  PASSED  ] 2 tests.
[  FAILED  ] 2 tests, listed below:
[  FAILED  ] RealClock.Conversions
[  FAILED  ] CoarseRealClock.Conversions

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8063 from gregsfortytwo/wip-fix-fsmap-decode

Wip fix fsmap decode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7752 from somnathr/wip-odsync-write

FileStore: Added O_DSYNC write scheme

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

FSMap: actually include the legacy MDSMap in Filesystems when upgrading

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

FSMap: decode old MDSMaps properly

We accidentally left a few members out that the MDSMap had!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8058 from yehudasa/wip-rgw-clean-log

cleanup

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rgw_ldap:  merge cleanups

* match LDAPHelper op return codes to RGW (-POSIX)
* make temporary LDAP handle local (MT-safe)
* add required --encode and --ttype to usage
* use versioned encoding macros

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: silence compilation warnings

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cleanup of debug messages

use ldout instead of derr and lderr and increase logging level

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7944 from trociny/wip-rbd-mirror-image-replayer-async

rbd-mirror: ImageReplayer async start/stop

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8028 from dillaman/wip-librbd-journal-replay

librbd: replaying a journal op post-refresh requires locking

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7483 from liewegas/wip-osd-boot

osd: resolve boot vs NOUP set + clear race

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8030 from yehudasa/wip-rgw-sync-status

adds the radosgw-admin sync status command that gives a human readable status of the sync process at a specific zone

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7615 from ceph/wip-14678

rgw: store system object meta in cache when creating it

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #8037 from yehudasa/wip-bucket-link

rgw:bucket link now set the bucket.instance acl (bug fix)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8013 from yehudasa/wip-rgw-multisite-fixes

rgw multisite fixes

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #6953 from jcsp/wip-fsmap

Multi-filesystem support

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7465 from liewegas/wip-promote-throttle

osd: probabilistic cache tier promotion throttling

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge tag 'v10.0.5'

v10.0.5

Conflicts:
debian/changelog

v10.0.5

Signed-off-by: Sage Weil <sage@redhat.com>

debian/changelog: Remove stray 'v' in version

Signed-off-by: Dan Mick <dan.mick@redhat.com>
(cherry picked from commit 0f7730c8f444657d4aaffa28c12bbcd602a2e3b8)

Merge branch 'master' of github.com:ceph/ceph

Merge branch 'debian-stretch-dep-fix' of git://github.com/rldleblanc/ceph

Reviewed-by: Sage Weil <sage@redhat.com>
# Conflicts:
# debian/control

Merge pull request #8045 from tchaikov/wip-scrubstore-cleanup

osd/ScrubStore: remove unused function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7707 from xiexingguo/xxg-wip-14815

osd: filestore: fix wrong scope of result code for error cases during mount

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7726 from tchaikov/wip-fix-ftbfs-gcc6

test/time: no need to abs(uint64_t) for comparing

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #7955 from ErwanAliasr1/evelu-check-bufferlist

test/bufferlist: Avoid false-positive tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7980 from liewegas/wip-14952

mon: initialize last_* timestamps on new pgs to creation time

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7982 from liewegas/wip-14606

qa/workunits/rest/test.py: add confirmation to 'mds setmap'

Merge pull request #7983 from liewegas/wip-14616

log: do not repeat errors to stderr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8004 from liewegas/wip-rados-test-stderr

qa/workunits/rados/test.sh: capture stderr too

Merge pull request #8006 from liewegas/wip-big-pages

os/bluestore/KernelDevice: force block size

Merge pull request #8019 from kmroz/wip-upstream-civetweb-low-ports-6

rgw: drop permissions of rgw/civetweb after startup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8007 from wido/se-mirror-mail

mirrors: Change contact e-mail address for se.ceph.com

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7991 from xiexingguo/xxg-wip-remove-upthrupending

osd: remove up_thru_pending field, which is never used

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8023 from roidayan/xio-remove_unused_variable

xio: remove unused variable

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test: add rbd-mirror test script

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: rbd_mirror_image_replay improve commit position parsing

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: ImageReplayer on_stop/on_start error tests

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: ImageReplayer async start/stop

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: protect against duplicate journal events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: protect against journal replay shutdown race condition

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: replaying a journal op post-refresh requires locking

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: periodic failure in TestJournalReplay.SnapRename

Force the refresh of the image to pick up the new snapshot
name before testing that it successfully was renamed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7923 from RustShen/master

doc: add cinder backend section to rbd-openstack.rst

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/ScrubStore: remove unused function

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7999 from liewegas/wip-armhf

a few armhf (32-bit build) fixes

Reviewed-by: Dan Mick <dmick@redhat.com>

rgw: store system object meta in cache when creating it

Fixes: #14678
When creating the system object we can easily also get the object's meta, so
keep it in the cache.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw:bucket link now set the bucket.instance acl

Fixes: #11076
Signed-off-by: Zengran Zhang <zhangzengran@h3c.com>

Merge pull request #7891 from rzarzynski/wip-rgw-content-disp

rgw: multiple Swift API compliance improvements for TempURL (Radoslaw Zarzynsk)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7990 from x11562/rgw_unused_vector

rgw: remove unused vector

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8031 from ceph/wip-rgw-usage

rgw: admin api for retrieving usage info (Ji Chen)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #8036 from yehudasa/wip-rgw-support-json-format-v2

rgw: support json format for admin policy API (Dunrong Huang)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #7874 from badone/wip-coverity-rgw_lib-uninit

[rgw] RGWLib::env is not used so remove it

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: policy dump --xml backward compatibility

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

osd: make promote_probability_millis atomic_t

This is almost always read, so it should be cheap.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: remove unused header file

Signed-off-by: Dunrong Huang <riegamaths@gmail.com>

rgw: support json format output for rados-admin policy command

when use rados-admin to get policy with parameter --format=json:
$ radosgw-admin policy --bucket=m8x --format=json

policy information with json format should be output.

Signed-off-by: Dunrong Huang <riegamaths@gmail.com>

rgw: support json format for admin policy API

Fixes: #14090, #14091
- before this patch:
$ curl -s -X GET http://radosrgw:9090/admin/bucket?format=json&policy&bucket=m8x | xmllint --format -

got XML response with error format

- after this patch:
got correct response with json format

Signed-off-by: Dunrong Huang <riegamaths@gmail.com>

Merge pull request #8027 from liewegas/wip-reweight-fix

test/pybind/test_ceph_argparse: fix reweight-by-utilization tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6656 from jmunhoz/wip-bucket-request-payment-usage-log

rgw: add bucket request payment feature usage statistics integration

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>