rgw: send appropriate op to cancel bucket index pending operation

Fixes: #10770
Backport: firefly, giant

Reported-by: baijiaruo <baijiaruo@126.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/hammer'

Merge remote-tracking branch 'gh/wip-assert-version'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3532 from dachary/wip-run-make-check

run-make-check: use nproc and shorter git expression

Merge pull request #3611 from kylinstorage/wip-rbd-recover-tool

Wip rbd recover tool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3544 from vuhuong/xio-spread-portals

Xio spread portals

Reviewed-by: Matt Benjamin <matt@cohortfs.com>

Merge pull request #3628 from SUSE/wip-initradosgw-fixup

Unconditionally chown rados log file.

Reviewed-by: Sage Weil <sage@redhat.com>

Unconditionally chown rados log file.

This fixes bnc#905047 (in a somewhat ad-hoc way). Sadly the log
file gets created from several places, so its existence does not
mean init-radosgw had actually run.

Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

Merge pull request #3623 from ceph/wip-opath

client: fix O_PATH on older Linux systems

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3622 from athanatos/wip-10340

Objecter::_op_submit_with_budget: add timeout before call

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3621 from athanatos/wip-10717

ReplicatedPG::do_op: ignore snapset_obc if !exists

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3616 from ceph/wip-export-dir

mds: implement export dir asok

rbd_recover_tool: move rbd_recover_tool directory to src/tools subdirectory

Signed-off-by: Min Chen <minchen@ubuntukylin.com>

client: fix O_PATH on older Linux systems

O_PATH was introduced in 2.6.39, but many Linux systems in use
today still have older kernels than that. Check for them.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3607 from ceph/wip-librbd-invalid-map

librbd: verify image's object map validity

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3592 from ceph/wip-librbd-watch-errors

librbd: better handling for image watch errors

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

include/encoding: fix an compile warning

./include/encoding.h:317:7: warning: 't' may be used uninitialized in
this function [-Wmaybe-uninitialized]

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3619 from ceph/wip-8903

librbd: removed lingering TODO message from ImageWatcher

Reviewed-by: Sage Weil <sage@redhat.com>

msg: fixup for 2ffacbe (crc configuration in messenger)

* Update the remaining Message::encode() calls, which now expect crc
  flags to be passed as a flags parameter instead of a bool (this e.g.
  fixes the issue with routing messages forwarded from an older client).

* In Message::encode() data crc is calculated when MSG_CRC_DATA is
  set, but in decode_message(), Pipe::read/write_message() data crc is
  calculated when MSG_CRC_HEADER is set. Fix this.

Note, 2ffacbe changed the behavior of Pipe::read_message/write_message():
previously the methods always calculated crc, now they calculate it
only if crc is enabled in the config. This means crc can not be disabled
if there are monitors of older version in the cluster.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3613 from rzarzynski/wip-4920

client: add support for O_NOFOLLOW and O_PATH

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: implement export dir asok

This was available as a legacy mds tell command
previously: make it availably via the cleaner
asok interface.  asok rather than new `tell` because
it's a low level/debug thing.

Signed-off-by: John Spray <john.spray@redhat.com>

client: implement support for O_PATH on Linux.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

client: add support for O_NOFOLLOW in Client::open().

Fixes: #4920
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rbd-recover-tool: add usefull information of this tool
include README, FAQ, TODO

Signed-off-by: Min Chen <minchen@ubuntukylin.com>

rbd-recover-tool: add a test case for raw image & snapshot

Signed-off-by: Min Chen <minchen@ubuntukylin.com>

rbd-recover-tool: implement the function framework
add control files & config files

rbd-recover-tool is an offline tool for dead ceph cluster
to recever rbd image in replicated pool from all osds.
it is a simple but usefull policy to keep data secruity on ceph

Sigbed-off-by: Min Chen <minchen@ubuntukylin.com>

rbd-recover-tool: add the basic function files

Signed-off-by: Min Chen <minchen@ubuntukylin.com>

Merge pull request #3603 from dachary/wip-10729-debian-install-deps

install-deps.sh: get lsb_release if needed

Reviewed-by: Yann Dupont <yann@objoo.org>

Merge pull request #3565 from rzarzynski/wip-7264

pybind/rados.py: fix error hiding and inconsistency on librados load.

Reviewed-by: Dan Mick <dan.mick@redhat.com>

doc: Removed references to Inktank suppport.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

doc: Removed Inktank reference.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3598 from ceph/wip-10720

mds: fix leaked MDCache::logger

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3262 from dachary/wip-jerasure

erasure-code: update links to jerasure upstream

Reviewed-by: John Wilkins <jowilkin@redhat.com>

install-deps.sh: babeltrace is not on wheezy

The babeltrace package is not found on wheezy, not just squeeze.
Fix the regular expression that incorrectly replaced
libbabeltrace-ctf-dev with -ctf-dev leading to a strange error

E: Opening configuration file tf-dev - ifstream::ifstream (2: No such file or directory)

when running apt-get install.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ReplicatedPG::do_op: ignore snapset_obc if !exists

Fixes: 10717
Signed-off-by: Samuel Just <sjust@redhat.com>

install-deps.sh: get lsb_release if needed

Debian does not have lsb_release installed by default.

http://tracker.ceph.com/issues/10729 Fixes: #10729

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3601 from dachary/wip-10729-fedora-install-deps

install-deps.sh: get lsb_release if needed

Reviewed-by: Haïkel Guémar <hguemar@redhat.com>

doc/release-notes: v0.92

Signed-off-by: Sage Weil <sage@redhat.com>

install-deps.sh: get lsb_release if needed

Fedora does not have lsb_release installed by default.

http://tracker.ceph.com/issues/10729 Fixes: #10729

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge commit '4ada7cc8e908a26b4640497b64b2ed9986889a92' into next

librbd: removed lingering TODO message from ImageWatcher

The TODO was a reminder to integrate with the watch/notify
error handler.  This integration was already completed under
previous commits.

Fixes: #8903
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mds: fix leaked MDCache::logger

Fixes: #10720
Signed-off-by: John Spray <john.spray@redhat.com>

librbd: fixed object map issues discovered via fsx

The object map wasn't being properly refreshed after setting
the snapshot context on the parent image. Additionally fixed
a potential deadlock that could have occurred if no object
map update was required when trimming an image.

Fixes: #10706
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librados_test_stub: added Rados::watch_flush

RBD now uses this method, so it was added to the test stub.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: better handling for image watch errors

When the librados watcher fails, librbd will now continuously
attempt to re-register the watch until it succeeds or the image
is closed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

vstart.sh: fix .ceph_port typo

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: prevent copyup during image shrink operations

If a cloned image was shrunk, it was possible for a copyup
to be performed on objects outside of the new image range.
Resize now pre-shrinks the parent overlap to disable copyup
when removing objects.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3597 from karelstriegel/master

rdb: update init-rbdmap to fix duplicate mount point

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rdb: update init-rbdmap to fix duplicate mount point

According to the manual 'mount -f' adds an entry in /etc/mtab.
Therefore the 'mount -v' command needs the flag -n to avoid
duplicate entries in /etc/mtab.

Signed-off-by: Karel Striegel <karel@striegel.be>

cls_rbd: fixed object_map_resize boundary issue

The object_map_resize sanity check ensures that a resize will
not erase state for an in-use object.  The check was incorrectly
including a block within the new object map range.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge remote-tracking branch 'origin/hammer'

librbd: removing objects can lead to infinite loop

The AioRemove state machine is not properly advanced from
_PRE to _FLAT.  This will result in an infinite state machine
loop.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: pending AIO operations are now flushed asynchronously

If exclusive locking was enabled, the librbd aio_flush command
would block waiting for queued AIO operations to proceed once
the exclusive lock was obtained.  Now librbd will no longer
block when aio_flush is invoked and AIO operations are waiting
on the exclusive lock.

Fixes: #10714
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3594 from ceph/wip-10707

test/cls_rgw: fix test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/hammer'

Merge remote-tracking branch 'gh/next'

Merge pull request #3590 from ceph/wip-10722

rgw: move perf cleanup before context cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

Fix memory leak in python rados bindings

A circular reference was inadvertently created when using the
CFUNCTYPE binding for callbacks for the asynchronous i/o callbacks.
This commit refactors the usage of the callbacks such that the
Ioctx object does not have a class reference to the callbacks.

Fixes: #10723
Backport: giant, firefly, dumpling
Signed-off-by: Billy Olsen <billy.olsen@gmail.com>
Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: RBD tests now check object map validity

If the RBD object map is invalidated during the run of the test,
the tests will now return a failed status to alert to an issue
with the object map.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add image flags to 'rbd info'

The 'rbd info' CLI tool now includes the image's flags.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added get_flags API methods

Exposed the RBD image flags bitmask through the API via
new methods in the C/C++/Python RBD APIs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: get_flags now reads all snapshot flags

As a convenience, get_flags now retrieves the current image
flags as well as all historical snapshop flags with a single
librados operation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/cls_rgw: fix test

Test needed to be updated, now using meta.accounted_size for object
size.

Fixes: #10707
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3593 from ceph/wip-autogen

Fix do_autogen.sh so that -L is allowed

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fix do_autogen.sh so that -L is allowed

Signed-off-by: David Zafman <dzafman@redhat.com>

rgw: move perf cleanup before context cleanup

Fixes: #10722
Fixes: #10572
Backport: giant, firefly

This fixes a regression introduced in commit
de2e5fa048639de6c9ee004a93ab295625fa3b94.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

pybind: fix error hiding and inconsistency on librados load.

When loading of librados failed, the real cause was hidden in some
circumstances due to exception rewriting. Additionally, the error
reporting was inconsistent -- two paths with different exceptions
and messages existed.

Fixes: #7264
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge remote-tracking branch 'upstream/next'

Objecter::_op_submit_with_budget: add timeout before call

Objecter::_send_op depends on the ontimeout field being filled in
to avoid 10340 and 9582.

Fixes: 10340
Signed-off-by: Samuel Just <sjust@redhat.com>

0.92

Merge pull request #3588 from athanatos/wip-10693

Wip 10693

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3587 from athanatos/wip-10633

ReplicatedPG::C_ProxyRead: fix dropped lock

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3538 from athanatos/wip-7862

osd: allow recovery below min size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3539 from athanatos/wip-7861

osd: allow writes on degraded objects

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'origin/next'

Conflicts:
src/rgw/rgw_op.cc
src/rgw/rgw_rados.cc
src/rgw/rgw_rados.h

Merge pull request #3578 from ceph/wip-10709

rbd: watch command should unwatch before exiting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3583 from jdurgin/wip-qemu-iotests

qa: use correct binary path on rpm-based systems

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3584 from ceph/wip-optional-encoding

encoding: ignore uninitialized instantiation in boost::optional decode

Reviewed-by: Sage Weil <sage@redhat.com>

qa: use correct binary path on rpm-based systems

Fixes: #10715
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rbd: watch command should unwatch before exiting

Switched the rbd CLI tool to use the new watch/unwatch API
and properly unwatch the image before exiting. Additionally,
the watch command opened the image twice -- once R/W and another
R/O.  This resulted in two watches being created for the image.
Cleaned up old/new format detection by using the existing API
methods.

Fixes: #10709
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

encoding: ignore uninitialized instantiation in boost::optional decode

POD types won't be initialized until the decode writes to them.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3562 from liewegas/wip-lttng

do_autogen.sh: default to --with-lttng, -L to build without

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3485 from jdurgin/wip-merge-diff-testing

Improve merge-diff test

do_autogen.sh: default to --with-lttng, -L to build without

Get developers building with LTTNG by default.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3549 from Ved-vampir/master

Ceph: Fix

Reviewed-by: Sage Weil <sage@redhat.com>

librados: add missing tracepoints

Signed-off-by: Sage Weil <sage@redhat.com>

osd:  change pg_stat plain to display CRUSH_ITEM_NONE in pgmap output section.

Fixes: #10519
Signed-off-by: Kiseleva Alyona <akiselyova@mirantis.com>

Merge pull request #3501 from yuyuyu101/wip-fix-async

AsyncMessenger: Fix mix auth assert failed

Merge branch 'wip-rbd-listwatchers'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

man: add rbd status to doc/man/8/rbd.rst

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

Merge pull request #3533 from ceph/wip-disaster-doc

doc: add cephfs disaster recovery guidance

Reviewed-by: Greg Farnum <gfarnum@redhat.com

doc: add cephfs disaster recovery guidance

This is a place to put some useful notes about
the new offline recovery tooling.

Signed-off-by: John Spray <john.spray@redhat.com>

test: add rbd status to src/test/cli/rbd/help.t

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

rbd: add status command to list the watchers of an image

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

Merge branch 'librbd-cache-iohint' of git://github.com/majianpeng/ceph

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librbd/ImageCtx.cc
src/librbd/ImageCtx.h
src/osdc/ObjectCacher.cc
src/osdc/ObjectCacher.h

Merge pull request #3556 from trociny/image_watcher-fixup

librbd: RemoteAsyncRequest: fix comparison operator

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3537 from ceph/wip-10688-boost-157

#10688: support Boost 1.57.0

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librbd: RemoteAsyncRequest: fix comparison operator

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: use IO hints for read/write operations

Import, export, and benchmark operations now use the librados
IO hints to indicate the read/write pattern and whether caching
is required.

Fixes: #10462
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>