ReplicatedPG::on_change: requeue in_progress_async_reads close to last

Fixes: 10768
Signed-off-by: Samuel Just <sjust@redhat.com>

include/encoding: fix an compile warning

./include/encoding.h:317:7: warning: 't' may be used uninitialized in
this function [-Wmaybe-uninitialized]

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3619 from ceph/wip-8903

librbd: removed lingering TODO message from ImageWatcher

Reviewed-by: Sage Weil <sage@redhat.com>

msg: fixup for 2ffacbe (crc configuration in messenger)

* Update the remaining Message::encode() calls, which now expect crc
  flags to be passed as a flags parameter instead of a bool (this e.g.
  fixes the issue with routing messages forwarded from an older client).

* In Message::encode() data crc is calculated when MSG_CRC_DATA is
  set, but in decode_message(), Pipe::read/write_message() data crc is
  calculated when MSG_CRC_HEADER is set. Fix this.

Note, 2ffacbe changed the behavior of Pipe::read_message/write_message():
previously the methods always calculated crc, now they calculate it
only if crc is enabled in the config. This means crc can not be disabled
if there are monitors of older version in the cluster.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3613 from rzarzynski/wip-4920

client: add support for O_NOFOLLOW and O_PATH

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: implement support for O_PATH on Linux.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

client: add support for O_NOFOLLOW in Client::open().

Fixes: #4920
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #3603 from dachary/wip-10729-debian-install-deps

install-deps.sh: get lsb_release if needed

Reviewed-by: Yann Dupont <yann@objoo.org>

Merge pull request #3565 from rzarzynski/wip-7264

pybind/rados.py: fix error hiding and inconsistency on librados load.

Reviewed-by: Dan Mick <dan.mick@redhat.com>

doc: Removed references to Inktank suppport.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

doc: Removed Inktank reference.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3598 from ceph/wip-10720

mds: fix leaked MDCache::logger

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3262 from dachary/wip-jerasure

erasure-code: update links to jerasure upstream

Reviewed-by: John Wilkins <jowilkin@redhat.com>

install-deps.sh: babeltrace is not on wheezy

The babeltrace package is not found on wheezy, not just squeeze.
Fix the regular expression that incorrectly replaced
libbabeltrace-ctf-dev with -ctf-dev leading to a strange error

E: Opening configuration file tf-dev - ifstream::ifstream (2: No such file or directory)

when running apt-get install.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

install-deps.sh: get lsb_release if needed

Debian does not have lsb_release installed by default.

http://tracker.ceph.com/issues/10729 Fixes: #10729

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3601 from dachary/wip-10729-fedora-install-deps

install-deps.sh: get lsb_release if needed

Reviewed-by: Haïkel Guémar <hguemar@redhat.com>

doc/release-notes: v0.92

Signed-off-by: Sage Weil <sage@redhat.com>

install-deps.sh: get lsb_release if needed

Fedora does not have lsb_release installed by default.

http://tracker.ceph.com/issues/10729 Fixes: #10729

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge commit '4ada7cc8e908a26b4640497b64b2ed9986889a92' into next

librbd: removed lingering TODO message from ImageWatcher

The TODO was a reminder to integrate with the watch/notify
error handler.  This integration was already completed under
previous commits.

Fixes: #8903
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mds: fix leaked MDCache::logger

Fixes: #10720
Signed-off-by: John Spray <john.spray@redhat.com>

librbd: fixed object map issues discovered via fsx

The object map wasn't being properly refreshed after setting
the snapshot context on the parent image. Additionally fixed
a potential deadlock that could have occurred if no object
map update was required when trimming an image.

Fixes: #10706
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

vstart.sh: fix .ceph_port typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3597 from karelstriegel/master

rdb: update init-rbdmap to fix duplicate mount point

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rdb: update init-rbdmap to fix duplicate mount point

According to the manual 'mount -f' adds an entry in /etc/mtab.
Therefore the 'mount -v' command needs the flag -n to avoid
duplicate entries in /etc/mtab.

Signed-off-by: Karel Striegel <karel@striegel.be>

Merge remote-tracking branch 'origin/hammer'

librbd: removing objects can lead to infinite loop

The AioRemove state machine is not properly advanced from
_PRE to _FLAT.  This will result in an infinite state machine
loop.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: pending AIO operations are now flushed asynchronously

If exclusive locking was enabled, the librbd aio_flush command
would block waiting for queued AIO operations to proceed once
the exclusive lock was obtained.  Now librbd will no longer
block when aio_flush is invoked and AIO operations are waiting
on the exclusive lock.

Fixes: #10714
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3594 from ceph/wip-10707

test/cls_rgw: fix test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/hammer'

Merge remote-tracking branch 'gh/next'

Merge pull request #3590 from ceph/wip-10722

rgw: move perf cleanup before context cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

Fix memory leak in python rados bindings

A circular reference was inadvertently created when using the
CFUNCTYPE binding for callbacks for the asynchronous i/o callbacks.
This commit refactors the usage of the callbacks such that the
Ioctx object does not have a class reference to the callbacks.

Fixes: #10723
Backport: giant, firefly, dumpling
Signed-off-by: Billy Olsen <billy.olsen@gmail.com>
Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/cls_rgw: fix test

Test needed to be updated, now using meta.accounted_size for object
size.

Fixes: #10707
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3593 from ceph/wip-autogen

Fix do_autogen.sh so that -L is allowed

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fix do_autogen.sh so that -L is allowed

Signed-off-by: David Zafman <dzafman@redhat.com>

rgw: move perf cleanup before context cleanup

Fixes: #10722
Fixes: #10572
Backport: giant, firefly

This fixes a regression introduced in commit
de2e5fa048639de6c9ee004a93ab295625fa3b94.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

pybind: fix error hiding and inconsistency on librados load.

When loading of librados failed, the real cause was hidden in some
circumstances due to exception rewriting. Additionally, the error
reporting was inconsistent -- two paths with different exceptions
and messages existed.

Fixes: #7264
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge remote-tracking branch 'upstream/next'

0.92

Merge pull request #3588 from athanatos/wip-10693

Wip 10693

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3587 from athanatos/wip-10633

ReplicatedPG::C_ProxyRead: fix dropped lock

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3538 from athanatos/wip-7862

osd: allow recovery below min size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3539 from athanatos/wip-7861

osd: allow writes on degraded objects

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'origin/next'

Conflicts:
src/rgw/rgw_op.cc
src/rgw/rgw_rados.cc
src/rgw/rgw_rados.h

Merge pull request #3578 from ceph/wip-10709

rbd: watch command should unwatch before exiting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3583 from jdurgin/wip-qemu-iotests

qa: use correct binary path on rpm-based systems

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3584 from ceph/wip-optional-encoding

encoding: ignore uninitialized instantiation in boost::optional decode

Reviewed-by: Sage Weil <sage@redhat.com>

qa: use correct binary path on rpm-based systems

Fixes: #10715
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rbd: watch command should unwatch before exiting

Switched the rbd CLI tool to use the new watch/unwatch API
and properly unwatch the image before exiting. Additionally,
the watch command opened the image twice -- once R/W and another
R/O.  This resulted in two watches being created for the image.
Cleaned up old/new format detection by using the existing API
methods.

Fixes: #10709
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

encoding: ignore uninitialized instantiation in boost::optional decode

POD types won't be initialized until the decode writes to them.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3562 from liewegas/wip-lttng

do_autogen.sh: default to --with-lttng, -L to build without

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3485 from jdurgin/wip-merge-diff-testing

Improve merge-diff test

do_autogen.sh: default to --with-lttng, -L to build without

Get developers building with LTTNG by default.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3549 from Ved-vampir/master

Ceph: Fix

Reviewed-by: Sage Weil <sage@redhat.com>

osd:  change pg_stat plain to display CRUSH_ITEM_NONE in pgmap output section.

Fixes: #10519
Signed-off-by: Kiseleva Alyona <akiselyova@mirantis.com>

Merge pull request #3501 from yuyuyu101/wip-fix-async

AsyncMessenger: Fix mix auth assert failed

Merge branch 'wip-rbd-listwatchers'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

man: add rbd status to doc/man/8/rbd.rst

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

Merge pull request #3533 from ceph/wip-disaster-doc

doc: add cephfs disaster recovery guidance

Reviewed-by: Greg Farnum <gfarnum@redhat.com

doc: add cephfs disaster recovery guidance

This is a place to put some useful notes about
the new offline recovery tooling.

Signed-off-by: John Spray <john.spray@redhat.com>

test: add rbd status to src/test/cli/rbd/help.t

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

rbd: add status command to list the watchers of an image

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

Merge branch 'librbd-cache-iohint' of git://github.com/majianpeng/ceph

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librbd/ImageCtx.cc
src/librbd/ImageCtx.h
src/osdc/ObjectCacher.cc
src/osdc/ObjectCacher.h

Merge pull request #3556 from trociny/image_watcher-fixup

librbd: RemoteAsyncRequest: fix comparison operator

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3537 from ceph/wip-10688-boost-157

#10688: support Boost 1.57.0

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librbd: RemoteAsyncRequest: fix comparison operator

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: use IO hints for read/write operations

Import, export, and benchmark operations now use the librados
IO hints to indicate the read/write pattern and whether caching
is required.

Fixes: #10462
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/wip-4087'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librbd/LibrbdWriteback.cc

Merge pull request #3545 from thesues/fix-cmp-xattr

Fix ReplicatedPG do_xattr_cmp_u64

Backport: giant, firefly
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3555 from ceph/wip-10106

rgw: flush xml header on get acl request

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3527 from wonzhq/op-stat

osd: don't log op stats for proxy read in the cache tier

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3503 from ceph/wip-pg-reqids

osd: index PGLog with optional extra_reqids in each log entry

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Wang, Zhiqiang <zhiqiang.wang@intel.com>

Merge pull request #3554 from ceph/wip-10572

rgw: finalize perfcounters after shutting down storage

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3323 from ceph/wip-10445

rgw: use gc for multipart abort

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph_test_rados: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3452 from majianpeng/iohint

Iohint in librbd cache.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: flush xml header on get acl request

Fixes: #10106
Backport: firefly, giant

dump_start() updates the formatter with the appropriate prefix, however,
we never flushed the formatter.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: finalize perfcounters after shutting down storage

Fixes: #10572
Backport: giant, firefly
First disable the storage subsystem, then disable perfcounters.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: Don't do readahead for random read.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3516 from ceph/wip-mdsmon-args

mon/MDSMonitor: fix gid/rank/state parsing

Reviewed-by: Huang Jun <hjwsm1989@gmail.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #3551 from ceph/wip-10698

rgw: fail s3 POST auth if keystone not configured

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ReplicatedPG::C_ProxyRead: fix dropped lock

Fixes: 10633
Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: only populate_obc_watchers if active

Signed-off-by: Samuel Just <sjust@redhat.com>

OSD: allow recovery in peered

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: go "peered" instead of "active" when < min_size

In the case of a replicated pool, the pg will transition to "peered"
rather than "active", where it can perform backfill bringing itself up
to min_size peers.  To that end, most operations now require
is_peered rather than is_active (OSDOps being the primary exception).
Also, rather than using the query_epoch on the activation message as the
activation epoch (for last_epoch_started) on the replica, we instead
use the last_epoch_started in the info sent by the primary.  This
allows the primary to not advance last_epoch_started past the last
known actual activation.  This will prevent later peering epochs from
requiring the last_update from a peered epoch to go active (might be
divergent).

Fixes: #7862
Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: move most of all_activated_and_commited into the AllReplicasActivated handler

It's confusing to have that logic split up so much.

Signed-off-by: Samuel Just <sjust@redhat.com>

PG::proc_master_log: assume les and history.les from log source

Otherwise, we might later go peered (not active) and not distribute
a non-0 last_epoch_started.  This should be safe as the log recipient
will have a last_update reflecting that interval as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd_types: add last_peered, last_became_peered

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: break waiting_for_peered out of waiting_for_active

waiting_for_peered now holds ops until peering completes (activation,
not necessarily state active).  waiting_for_active now holds
specifically MOSDOp blocked on:
- scrub
- replay
- state active

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: remove very obsolete comment about backfill_pos

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: add state PG_STATE_PEERED

We need a state to represent the pg state where we are below
min_size, but trying to recover.

Signed-off-by: Samuel Just <sam.just@inktank.com>

doc: add documentation for undersized

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #3553 from ceph/wip-10701

rgw: use strict_strtoll() for content length

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ReplicatedPG: block writes on degraded objects for ec pools

Signed-off-by: Samuel Just <sjust@redhat.com>

append_log: use remove_snap_mapped_object

Backport: giant, firefly
Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::agent_maybe_evict: take write lock, skip on failure

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: if we got ENOENT and an obc, we need to check the locks

Otherwise, we might return ENOENT out of order.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::get_rw_locks: use excl lock for read & write_ordered

Signed-off-by: Samuel Just <sjust@redhat.com>