Merge pull request #8165 from tchaikov/wip-fix-rgw-encoding

rgw/rgw_common.h: fix the RGWBucketInfo decoding

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #8158 from dillaman/wip-rbd-mirror-fixes

rbd-mirror: fix image replay test failures

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8133 from ceph/wip-15145-cephfs-tools

packaging: move cephfs repair tools to ceph-common

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

rgw/rgw_common.h: fix the RGWBucketInfo decoding

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8154 from tchaikov/wip-fix-rgw-decoding

rgw/rgw_common.h: fix the RGWBucketInfo decoding

Reviewed-by: Loic Dachary <ldachary@redhat.com>

qa/workunits/rbd: rbd_mirror was extracting the incorrect image id

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: use unique logs for each rbd-mirror daemon

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: refetch active object before defaulting to new tag

If a live replay is in progress, it's possible that object offset 0
was pulled and a new tag is discovered before the current object is
(re-)pulled to determine that the old tag still has entries remaining.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7894 from SUSE/wip-14828

 ceph.spec.in: do not install Ceph RA on systemd platforms

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw/rgw_common.h: fix the RGWBucketInfo decoding

introduced by 416234b

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: Make ceph-common require libcephfs1

We recently moved fsck/repair tools to ceph-common package. We should
also make the version/release requirements tight.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #8107 from liewegas/wip-crush

crush: fix cli tests for new crush tunables

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8073 from ceph/wip-fsmap-upgrade-jewel

mds: fix FSMap upgrade with daemons in the map

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8142 from tchaikov/wip-rgw-ftbfs

cls/rgw: fix FTBFS

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: fix FTBFS

BIVerObjEntry::mtime() is a method not a member variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8002 from rzarzynski/wip-rgw-putobjmeta

rgw: add support for metadata upload during PUT on Swift container.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8138 from dillaman/wip-15141

rbd: add support for mirror image promotion/demotion/resync

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8139 from yehudasa/wip-rgw-implicit-tenant

rgw: Allow an implicit tenant in case of Keystone

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: implement mirror image resync request API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track if a mirror image resync has been requested

A secondary image will store its resync request to the primary
image's journal. When the local rbd mirror process detects the
flag being set, it can initiate a resync from the remote.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: add placeholder for marking clients are disconnected

When a client is disconnected from the journal, entries can be
pruned regardless of the affected client's commit position.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: initial implementation of mirror image promote/demote

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper methods for manipulating journal tags from API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: dump mirroring primary status on info command

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: expose image mirroring primary/secondary status

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: update rbd cli tests to incorporate new mirror image commands

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add mirror image promote/demote/resync commands

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: stub API methods for mirror image promotion/demotion/resync

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: Allow an implicit tenant in case of Keystone

This, unfortunately, introduces possible double lookups, but
they should be cached. Also, the logic appears somewhat convoluted,
although the intent is quite simple: if you're an OpenStack user
with a Keystone authentication, we allow an implicit tenant of
the same name as the user.

Signed-off-by: Pete Zaitcev <zaitcev@redhat.com>
Conflicts:
src/rgw/rgw_swift.cc

Merge pull request #8095 from yehudasa/wip-rgw-sync-fixes

rgw sync fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: data shard sync doesn't exit on error

We don't handle the error at the top level, so no
need to exit. Just continue with the backoff mechanism.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8108 from yehudasa/wip-rgw-highres-3

Wip rgw highres 3

rgw: propagate low time precision for user operations

When user set a conditional timestamp to compare (e.g. if-modified-since),
it uses a low precision time resolution. Need to adjust checks for that.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use higher precision time when encoding / decoding external apis

When dealing with external entities (e.g., remote rgw), use higher precision time
format for object mtime.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

utime: add gmtime_nsec()

higher precision than gmtime()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: extend date/time parsing

So that we can parse correctly time encoded using utime_t::gmtime()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7631 from liewegas/wip-agent-flush-low

osd: consider high/low mode when putting agent to sleep

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8040 from ukernel/jewel-15050

mds: fix stray purging in 'stripe_count > 1' case

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7713 from ukernel/wip-filelock-deadlock

Wip filelock deadlock

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8136 from athanatos/wip-split-lfn

osd: filestore: fix race condition with split vs collection_move_rename and long object names

Reviewed-by: Sage Weil <sage@redhat.com>

packaging: move cephfs repair tools to ceph-common

The CephFS repair tools are generally useful to run on any node, not
just the MDS nodes themselves.

Move the utilities from the ceph-mds package to the ceph-common package.

Fixes: #15145
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #8132 from dillaman/wip-librbd-memcheck

librbd: correct issues discovered via valgrind memcheck

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

journal: reschedule watch if no entries available during live replay

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: correct valgrind memcheck errors

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8120 from SUSE/wip-15138

RPM: fix openldap and openssl build dependencies for SUSE

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Jun Huang <hjwsm1989@gmail.com>

Merge pull request #8114 from liewegas/wip-tmap-migrate

qa/workunits/rados/test.sh: test tmap_migrate

Merge pull request #8119 from dillaman/wip-qa-rbd-copy

rbd: clone operation should default to image format 2

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8090 from trociny/fixup-rbd-mirror-test

test: fixup and improvements for rbd-mirror test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8128 from jcsp/wip-15135

tools/cephfs: fix tmap_upgrade

Merge pull request #8129 from dillaman/wip-qa-rbd-permissions

test: fix issues discovered via the rbd permissions test case

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8124 from dillaman/wip-qa-rbd-lock-fence

test: handle exception thrown from close during rbd lock test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/rbd: disable deep-flatten during permissions testing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tools/cephfs: fix tmap_upgrade

Move it up to handle the command before we try
and initialize DataScan::driver, because drivers
require a target filesystem, and that doesn't
apply to this command.

Fixes: #15135
Signed-off-by: John Spray <john.spray@redhat.com>

rbd: snap list should open image as read-only

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #7243 from tchaikov/wip-py3-jewel

ceph-detect-init: fix py3 test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7975 from JiYou/fix-config-opts

config: fix osd_crush_initial_weight

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8125 from dillaman/wip-librbd-mock

test: add missing shut_down mock method

test: add missing shut_down mock method

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: handle exception thrown from close during lock test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

RPM: fix openldap and openssl build dependencies for SUSE

http://tracker.ceph.com/issues/15138 Fixes: #15138

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd: clone operation should default to image format 2

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8116 from SUSE/wip-docdev-nuke

doc/dev: add section on interrupting a running suite

Reviewed-by: Loic Dachary <ldachary@redhat.com>

MDSMonitor: check for duplicated fs creation *before* multiple-fs creation

If we check for creating an FS with one already existing first, we will
reject (with EINVAL) duplicate attempts to create the same FS with
the same params. That makes it non-idempotent, which is bad!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

doc/dev: add section on interrupting a running suite

This commit adds a new section, entitled "Interrupt a running suite", to
the "Testing in the cloud" chapter.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #8105 from SUSE/wip-ceph-detect-init-tox

ceph-detect-init: add missing test case

Merge pull request #8088 from liewegas/wip-typo

osd/OSDMap: fix typo in summarize_mapping_stats

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8098 from liewegas/wip-fix-crush

unittest_osdmap: default crush tunables now firefly

Merge pull request #8089 from liewegas/wip-mark-pg-stale

mon/PGMonitor: reliably mark PGs state

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7943 from llua/init-ceph

init-ceph.in: allow case-insensitive true in `osd crush update on start'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #8055 from liewegas/wip-hobject

ghobject_t: use # instead of ! as a separator

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #8056 from rjfd/wip-13296

rbd: support for enabling/disabling mirroring on specific images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rados/test.sh: test tmap_migrate

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7971 from ceph/wip-selinux-update

selinux: Update policy to grant additional access

Merge pull request #8033 from liewegas/wip-pg-create

osd: handle dup pg_create that races with pg deletion

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8054 from ErwanAliasr1/evelu-check

common: SubProcess: Avoid buffer corruption when calling err()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8057 from liewegas/wip-pglog-warning

osd/PGLog: fix warning

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8059 from ceph/wip-include-scope

rados: add ceph:: namespace to bufferlist type

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8071 from xiexingguo/xxg-wip-fixptr

osd: initialize last_recalibrate field at construction

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8081 from renhwsky/renhw-wip-osdmonitor-poolset

osd: min_write_recency_for_promote & min_read_recency_for_promote are tiering only

Reviewed-by: Sage Weil <sage@redhat.com>

test: fixup and improvements for rbd-mirror test

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #7879 from xiexingguo/xxg-wip-doc-rados-api

doc/rados/api/librados-intro.rst: fix typo

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

doc: fix typo

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: metadata notifications go to all zones

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch remote obj uses high def clock for if-[un]modified-since

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: timespan fix

use make_timespan() to generate a correct timespan struct

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

time: For tests, subtract in the right order

This was a bug in the test rather than a bug in the code the test was testing.

Use the ternary operator to replace labs.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: adapt to narrow stat2() and mtime2()

The librados stat2() and mtime2() ops no longer provide an override
for the internal ceph_realtime type.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: high resolution time

replace time_t, and some utime_t with ceph::real_clock. Use new
librados interfaces to stat() and set mtime.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

utime: add conversion methods to/from real_time

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

common/ceph_time: add real_clock::is_zero()

to check whether time is not zero

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #8079 from dillaman/wip-14413

rbd-mirror: integrate with image sync state machine

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: remove broken negative test cases

These should be reimplemented using google mock instead of
virtual functions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: integrate new bootstrap state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: simple image bootstrap state machine

This state machine will handle the simple cases of creating
a local image from a remote image, registering to the remote
journal, and sync'ing the image contents.

There are several TODOs for future remaining work.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: helper state machine for opening local image

This state machine will open a local image and request the
exclusive lock.  If the open or lock fails, it will automatically
close the image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: helper state machine for closing librbd images

librbd ImageCtx's cannot be deleted from within the librbd thread.  This
state machine will perform the deletion from the rbd-mirror work queue
thread.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7979 from dillaman/wip-14413

rbd-mirror: remote to local cluster image sync

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8102 from dillaman/wip-librados-aio-notify-redux

librados: cancel aio notification linger op upon completion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>