rgw: propagate low time precision for user operations

When user set a conditional timestamp to compare (e.g. if-modified-since),
it uses a low precision time resolution. Need to adjust checks for that.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use higher precision time when encoding / decoding external apis

When dealing with external entities (e.g., remote rgw), use higher precision time
format for object mtime.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

utime: add gmtime_nsec()

higher precision than gmtime()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: extend date/time parsing

So that we can parse correctly time encoded using utime_t::gmtime()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: metadata notifications go to all zones

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch remote obj uses high def clock for if-[un]modified-since

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: timespan fix

use make_timespan() to generate a correct timespan struct

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

time: For tests, subtract in the right order

This was a bug in the test rather than a bug in the code the test was testing.

Use the ternary operator to replace labs.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: adapt to narrow stat2() and mtime2()

The librados stat2() and mtime2() ops no longer provide an override
for the internal ceph_realtime type.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: high resolution time

replace time_t, and some utime_t with ceph::real_clock. Use new
librados interfaces to stat() and set mtime.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

utime: add conversion methods to/from real_time

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

common/ceph_time: add real_clock::is_zero()

to check whether time is not zero

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #8079 from dillaman/wip-14413

rbd-mirror: integrate with image sync state machine

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: remove broken negative test cases

These should be reimplemented using google mock instead of
virtual functions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: integrate new bootstrap state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: simple image bootstrap state machine

This state machine will handle the simple cases of creating
a local image from a remote image, registering to the remote
journal, and sync'ing the image contents.

There are several TODOs for future remaining work.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: helper state machine for opening local image

This state machine will open a local image and request the
exclusive lock.  If the open or lock fails, it will automatically
close the image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: helper state machine for closing librbd images

librbd ImageCtx's cannot be deleted from within the librbd thread.  This
state machine will perform the deletion from the rbd-mirror work queue
thread.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7979 from dillaman/wip-14413

rbd-mirror: remote to local cluster image sync

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8102 from dillaman/wip-librados-aio-notify-redux

librados: cancel aio notification linger op upon completion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8099 from dillaman/wip-15121

cls_rbd: protect against excessively large object maps

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8100 from linuxbox2/ldap-cmake

cmake: avoid false-positive LDAP header detect

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8104 from dillaman/wip-rbd-read-flags

qa/workunits/rbd: do not use object map during read flag testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/rbd: do not use object map during read flag testing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: cancel aio notification linger op upon completion

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cmake: avoid false-positive LDAP header detect

Pre-setting a value in OPENLDAP_INCLUDE_DIR broke the subsequent
test.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cls_rbd: protect against excessively large object maps

Fixes: #15121
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8094 from dillaman/wip-librbd-image-watch-unregister

librbd: handle unregistering the image watcher when disconnected

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: handle unregistering the image watcher when disconnected

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7913 from liewegas/wip-list-encode

include/encoding: do not try to be clever with list encoding

Merge pull request #7915 from ceph/wip-librados-stat2

librados: stat2 with higher time precision

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7964 from liewegas/wip-crush-jewel

crush: update tunable docs.  change default profile to jewel

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8035 from liewegas/wip-bluestore

osd: bluestore updates, scrub fixes

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8050 from liewegas/wip-15077

os/filestore: fix punch hole usage in _zero

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8064 from liewegas/wip-zlib-test

unittest_compression_zlib: do not assume buffer will be null terminated

Merge pull request #7870 from stiopaa1/blueStoreRemoveIncludes

osd: bluestore: remove unneeded includes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7871 from xiexingguo/xxg-wip-cleanasync

msg: async: small cleanups

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7170 from trociny/wip-freebsd-misc

FreeBSD related fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7947 from saru95/patch---2

ceph.in: Minor python3 specific changes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7974 from tchaikov/wip-buffer-eq-neq

buffer: add symmetry operator==() and operator!=()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8074 from tchaikov/wip-ceph-detect-init-test

ceph-detect-init: add debian/jessie test

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-detect-init: return None at seeing unknown debian distro

* return None at seeing unknown debian distros
* add test for this accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8046 from xiexingguo/xxg-wip-fixpc

msg: async: fix perf counter description and simplify _send_keepalive_or_ack

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7949 from wjin/fix

Event: fix clock skew problem

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8076 from ceph/wip-rgw-ldap

Wip rgw ldap

This follows a revert of the previous PR. This branch has some build related fixes.

Merge pull request #8075 from ceph/revert-7985-rgw-token-7

Revert "rgw ldap"

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Revert "rgw ldap"

test: verify proper update of object map during rbd-mirror image sync

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: update object map while performing image sync

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: temporarily disable support for cloned images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: unit test cases for rbd mirror image sync

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: image sync state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: unit tests for rbd::mirror::image_sync::ImageCopyRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: state machine to iterate and copy all image objects

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track maximum object count during mirror sync

If the sync is interrupted, it is possible the image the source
image is shrunk.  Therefore, the sync process needs to track
the maximum known object extent of the image so objects can
be properly freed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: test cases for rbd-mirror image sync point state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: new image sync point create/prune state machines

These are responsible for creating/removing snapshots in the
remote image and updating the client registration to account
for the snapshots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added equality operator to MirrorPeer journal types

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: async version of snap create and snap remove

This will be utilized by the RBD mirror daemon when creating
sync points.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: prefix local maintenance ops methods

This will help to differentiate between the methods that send an RPC
request if the lock isn't owned vs execute locally if it is owned.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: maintenance ops now send asynchronous requests

The RBD mirror daemon will need to request the creation/removal
of snapshots from a remote image.  These operations need to be
performed asynchronously as opposed to the current synchronous
style of the librbd API.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track starting point snapshot for mirror sync point

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: move get_mock_io_ctx helper method to common location

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: unit test cases for rbd::mirror::image_sync::SnapshotCopyRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: sync snapshots between remote-local images during bootstrap

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track local to peer snapshot id mapping within journal

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: unit test cases for rbd::mirror::image_sync::ObjectCopyRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: updated mock expectation for librados remove op

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: added missing forward declarations for librbd mocks

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd_mirror: deep-copy of object and its associated snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper method for wrappign rados safe callbacks

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: updated API for calc_snap_set_diff

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: calc_snap_set_diff now provides last known object size

This allows truncate requests to be detected between snapshots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: support for truncating/removing with old snap contexts

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8066 from yehudasa/wip-rgw-swift-versioning-disabled

Wip rgw swift versioning disabled

Reviewed-by: Josh Durgin <joshd@redhat.com>

Merge pull request #7985 from linuxbox2/rgw-token-7

rgw ldap (Matt Benjamin)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'origin/master' into wip-rgw-ldap

test: disable test_rgw_ldap

for now

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/Makefile.am: add a missing .h file

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_ldap:  merge cleanups

* match LDAPHelper op return codes to RGW (-POSIX)
* make temporary LDAP handle local (MT-safe)
* add required --encode and --ttype to usage
* use versioned encoding macros

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: require openldap library client headers

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: free components on shutdown

Also, use ldout in rgw_file.h in 3 places.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: LDAP pass-through authentication

Implement a new external authenticator based on LDAP and the
new external token format.

External LDAP auth now works, at least with openldap/X.500 style
naming and ldaps:// (SSL).

The latter is AD-friendly, but since AD uses dnattr=cn (IIRC)
everywhere, AD will need testing.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge branch 'wip-15086' of git://github.com/SUSE/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8068 from SUSE/wip-15104

qa/workunits/rbd: use POSIX function definition

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7663 from javacruft/wip-pybind-flags

pybind: Ensure correct python flags are passed

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7856 from jdurgin/wip-cython-cleanup

configure.ac: update help strings for cython

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge remote-tracking branch 'origin/wip-josh-testing'

Merge pull request #8067 from athanatos/wip-fix-filestore-warning

FileStore: fix initialization order for m_disable_wbthrottle

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/workunits/rbd: use POSIX function definition

In scripts with /bin/sh shebang, we must use POSIX function definitions to
avoid syntax errors on platforms that link /bin/sh to something other than
/bin/bash.

See https://wiki.ubuntu.com/DashAsBinSh#function for more information.

http://tracker.ceph.com/issues/15104 Fixes: #15104

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: disable swift versioning by default

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: improve naming of Swift versioned object's copies.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: dump X-Versions-Location HTTP header of Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: copy swift versioned object to destination if overwritten

Only allow if target bucket owner is the source bucket owner.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: swift object versioning data structure changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8034 from SUSE/wip-15047

packaging: make infernalis -> jewel upgrade work

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

debian: make infernalis -> jewel upgrade work

Attempts to install jewel ceph-common, ceph-mon, ceph-osd, and ceph-base
package over infernalis ceph package fail due to files existing in both.

See comment #4 in the tracker issue for a deeper analysis.

http://tracker.ceph.com/issues/15047 Fixes: #15047

Signed-off-by: Nathan Cutler <ncutler@suse.com>

FileStore: fix initialization order for m_disable_wbthrottle

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph-detect-init: return systemd on Debian Jessie

http://tracker.ceph.com/issues/15086 Fixes: #15086

Signed-off-by: Nathan Cutler <ncutler@suse.com>