osd: apply affinity if any affinity is not default

osd,mon: add norebalance osdmap flag

Fixes: #10559
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3616 from ceph/wip-export-dir

mds: implement export dir asok

Merge pull request #3607 from ceph/wip-librbd-invalid-map

librbd: verify image's object map validity

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3592 from ceph/wip-librbd-watch-errors

librbd: better handling for image watch errors

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

include/encoding: fix an compile warning

./include/encoding.h:317:7: warning: 't' may be used uninitialized in
this function [-Wmaybe-uninitialized]

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3619 from ceph/wip-8903

librbd: removed lingering TODO message from ImageWatcher

Reviewed-by: Sage Weil <sage@redhat.com>

msg: fixup for 2ffacbe (crc configuration in messenger)

* Update the remaining Message::encode() calls, which now expect crc
  flags to be passed as a flags parameter instead of a bool (this e.g.
  fixes the issue with routing messages forwarded from an older client).

* In Message::encode() data crc is calculated when MSG_CRC_DATA is
  set, but in decode_message(), Pipe::read/write_message() data crc is
  calculated when MSG_CRC_HEADER is set. Fix this.

Note, 2ffacbe changed the behavior of Pipe::read_message/write_message():
previously the methods always calculated crc, now they calculate it
only if crc is enabled in the config. This means crc can not be disabled
if there are monitors of older version in the cluster.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3613 from rzarzynski/wip-4920

client: add support for O_NOFOLLOW and O_PATH

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: implement export dir asok

This was available as a legacy mds tell command
previously: make it availably via the cleaner
asok interface.  asok rather than new `tell` because
it's a low level/debug thing.

Signed-off-by: John Spray <john.spray@redhat.com>

client: implement support for O_PATH on Linux.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

client: add support for O_NOFOLLOW in Client::open().

Fixes: #4920
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #3603 from dachary/wip-10729-debian-install-deps

install-deps.sh: get lsb_release if needed

Reviewed-by: Yann Dupont <yann@objoo.org>

Merge pull request #3565 from rzarzynski/wip-7264

pybind/rados.py: fix error hiding and inconsistency on librados load.

Reviewed-by: Dan Mick <dan.mick@redhat.com>

doc: Removed references to Inktank suppport.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

doc: Removed Inktank reference.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3598 from ceph/wip-10720

mds: fix leaked MDCache::logger

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3262 from dachary/wip-jerasure

erasure-code: update links to jerasure upstream

Reviewed-by: John Wilkins <jowilkin@redhat.com>

install-deps.sh: babeltrace is not on wheezy

The babeltrace package is not found on wheezy, not just squeeze.
Fix the regular expression that incorrectly replaced
libbabeltrace-ctf-dev with -ctf-dev leading to a strange error

E: Opening configuration file tf-dev - ifstream::ifstream (2: No such file or directory)

when running apt-get install.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

install-deps.sh: get lsb_release if needed

Debian does not have lsb_release installed by default.

http://tracker.ceph.com/issues/10729 Fixes: #10729

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3601 from dachary/wip-10729-fedora-install-deps

install-deps.sh: get lsb_release if needed

Reviewed-by: Haïkel Guémar <hguemar@redhat.com>

doc/release-notes: v0.92

Signed-off-by: Sage Weil <sage@redhat.com>

install-deps.sh: get lsb_release if needed

Fedora does not have lsb_release installed by default.

http://tracker.ceph.com/issues/10729 Fixes: #10729

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge commit '4ada7cc8e908a26b4640497b64b2ed9986889a92' into next

librbd: removed lingering TODO message from ImageWatcher

The TODO was a reminder to integrate with the watch/notify
error handler.  This integration was already completed under
previous commits.

Fixes: #8903
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mds: fix leaked MDCache::logger

Fixes: #10720
Signed-off-by: John Spray <john.spray@redhat.com>

librbd: fixed object map issues discovered via fsx

The object map wasn't being properly refreshed after setting
the snapshot context on the parent image. Additionally fixed
a potential deadlock that could have occurred if no object
map update was required when trimming an image.

Fixes: #10706
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librados_test_stub: added Rados::watch_flush

RBD now uses this method, so it was added to the test stub.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: better handling for image watch errors

When the librados watcher fails, librbd will now continuously
attempt to re-register the watch until it succeeds or the image
is closed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

vstart.sh: fix .ceph_port typo

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: prevent copyup during image shrink operations

If a cloned image was shrunk, it was possible for a copyup
to be performed on objects outside of the new image range.
Resize now pre-shrinks the parent overlap to disable copyup
when removing objects.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3597 from karelstriegel/master

rdb: update init-rbdmap to fix duplicate mount point

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rdb: update init-rbdmap to fix duplicate mount point

According to the manual 'mount -f' adds an entry in /etc/mtab.
Therefore the 'mount -v' command needs the flag -n to avoid
duplicate entries in /etc/mtab.

Signed-off-by: Karel Striegel <karel@striegel.be>

cls_rbd: fixed object_map_resize boundary issue

The object_map_resize sanity check ensures that a resize will
not erase state for an in-use object.  The check was incorrectly
including a block within the new object map range.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge remote-tracking branch 'origin/hammer'

librbd: removing objects can lead to infinite loop

The AioRemove state machine is not properly advanced from
_PRE to _FLAT.  This will result in an infinite state machine
loop.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: pending AIO operations are now flushed asynchronously

If exclusive locking was enabled, the librbd aio_flush command
would block waiting for queued AIO operations to proceed once
the exclusive lock was obtained.  Now librbd will no longer
block when aio_flush is invoked and AIO operations are waiting
on the exclusive lock.

Fixes: #10714
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3594 from ceph/wip-10707

test/cls_rgw: fix test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/hammer'

Merge remote-tracking branch 'gh/next'

Merge pull request #3590 from ceph/wip-10722

rgw: move perf cleanup before context cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

Fix memory leak in python rados bindings

A circular reference was inadvertently created when using the
CFUNCTYPE binding for callbacks for the asynchronous i/o callbacks.
This commit refactors the usage of the callbacks such that the
Ioctx object does not have a class reference to the callbacks.

Fixes: #10723
Backport: giant, firefly, dumpling
Signed-off-by: Billy Olsen <billy.olsen@gmail.com>
Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: RBD tests now check object map validity

If the RBD object map is invalidated during the run of the test,
the tests will now return a failed status to alert to an issue
with the object map.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add image flags to 'rbd info'

The 'rbd info' CLI tool now includes the image's flags.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added get_flags API methods

Exposed the RBD image flags bitmask through the API via
new methods in the C/C++/Python RBD APIs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: get_flags now reads all snapshot flags

As a convenience, get_flags now retrieves the current image
flags as well as all historical snapshop flags with a single
librados operation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/cls_rgw: fix test

Test needed to be updated, now using meta.accounted_size for object
size.

Fixes: #10707
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3593 from ceph/wip-autogen

Fix do_autogen.sh so that -L is allowed

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fix do_autogen.sh so that -L is allowed

Signed-off-by: David Zafman <dzafman@redhat.com>

rgw: move perf cleanup before context cleanup

Fixes: #10722
Fixes: #10572
Backport: giant, firefly

This fixes a regression introduced in commit
de2e5fa048639de6c9ee004a93ab295625fa3b94.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

pybind: fix error hiding and inconsistency on librados load.

When loading of librados failed, the real cause was hidden in some
circumstances due to exception rewriting. Additionally, the error
reporting was inconsistent -- two paths with different exceptions
and messages existed.

Fixes: #7264
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge remote-tracking branch 'upstream/next'

0.92

Merge pull request #3588 from athanatos/wip-10693

Wip 10693

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3587 from athanatos/wip-10633

ReplicatedPG::C_ProxyRead: fix dropped lock

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3538 from athanatos/wip-7862

osd: allow recovery below min size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3539 from athanatos/wip-7861

osd: allow writes on degraded objects

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'origin/next'

Conflicts:
src/rgw/rgw_op.cc
src/rgw/rgw_rados.cc
src/rgw/rgw_rados.h

Merge pull request #3578 from ceph/wip-10709

rbd: watch command should unwatch before exiting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3583 from jdurgin/wip-qemu-iotests

qa: use correct binary path on rpm-based systems

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3584 from ceph/wip-optional-encoding

encoding: ignore uninitialized instantiation in boost::optional decode

Reviewed-by: Sage Weil <sage@redhat.com>

qa: use correct binary path on rpm-based systems

Fixes: #10715
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rbd: watch command should unwatch before exiting

Switched the rbd CLI tool to use the new watch/unwatch API
and properly unwatch the image before exiting. Additionally,
the watch command opened the image twice -- once R/W and another
R/O.  This resulted in two watches being created for the image.
Cleaned up old/new format detection by using the existing API
methods.

Fixes: #10709
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

encoding: ignore uninitialized instantiation in boost::optional decode

POD types won't be initialized until the decode writes to them.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3562 from liewegas/wip-lttng

do_autogen.sh: default to --with-lttng, -L to build without

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3485 from jdurgin/wip-merge-diff-testing

Improve merge-diff test

do_autogen.sh: default to --with-lttng, -L to build without

Get developers building with LTTNG by default.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3549 from Ved-vampir/master

Ceph: Fix

Reviewed-by: Sage Weil <sage@redhat.com>

osd:  change pg_stat plain to display CRUSH_ITEM_NONE in pgmap output section.

Fixes: #10519
Signed-off-by: Kiseleva Alyona <akiselyova@mirantis.com>

Merge pull request #3501 from yuyuyu101/wip-fix-async

AsyncMessenger: Fix mix auth assert failed

Merge branch 'wip-rbd-listwatchers'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

man: add rbd status to doc/man/8/rbd.rst

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

Merge pull request #3533 from ceph/wip-disaster-doc

doc: add cephfs disaster recovery guidance

Reviewed-by: Greg Farnum <gfarnum@redhat.com

doc: add cephfs disaster recovery guidance

This is a place to put some useful notes about
the new offline recovery tooling.

Signed-off-by: John Spray <john.spray@redhat.com>

test: add rbd status to src/test/cli/rbd/help.t

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

rbd: add status command to list the watchers of an image

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

Merge branch 'librbd-cache-iohint' of git://github.com/majianpeng/ceph

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librbd/ImageCtx.cc
src/librbd/ImageCtx.h
src/osdc/ObjectCacher.cc
src/osdc/ObjectCacher.h

Merge pull request #3556 from trociny/image_watcher-fixup

librbd: RemoteAsyncRequest: fix comparison operator

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3537 from ceph/wip-10688-boost-157

#10688: support Boost 1.57.0

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librbd: RemoteAsyncRequest: fix comparison operator

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: use IO hints for read/write operations

Import, export, and benchmark operations now use the librados
IO hints to indicate the read/write pattern and whether caching
is required.

Fixes: #10462
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/wip-4087'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librbd/LibrbdWriteback.cc

Merge pull request #3545 from thesues/fix-cmp-xattr

Fix ReplicatedPG do_xattr_cmp_u64

Backport: giant, firefly
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3555 from ceph/wip-10106

rgw: flush xml header on get acl request

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3527 from wonzhq/op-stat

osd: don't log op stats for proxy read in the cache tier

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3503 from ceph/wip-pg-reqids

osd: index PGLog with optional extra_reqids in each log entry

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Wang, Zhiqiang <zhiqiang.wang@intel.com>

Merge pull request #3554 from ceph/wip-10572

rgw: finalize perfcounters after shutting down storage

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3323 from ceph/wip-10445

rgw: use gc for multipart abort

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph_test_rados: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3452 from majianpeng/iohint

Iohint in librbd cache.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: flush xml header on get acl request

Fixes: #10106
Backport: firefly, giant

dump_start() updates the formatter with the appropriate prefix, however,
we never flushed the formatter.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: finalize perfcounters after shutting down storage

Fixes: #10572
Backport: giant, firefly
First disable the storage subsystem, then disable perfcounters.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: Don't do readahead for random read.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3516 from ceph/wip-mdsmon-args

mon/MDSMonitor: fix gid/rank/state parsing

Reviewed-by: Huang Jun <hjwsm1989@gmail.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #3551 from ceph/wip-10698

rgw: fail s3 POST auth if keystone not configured

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ReplicatedPG::C_ProxyRead: fix dropped lock

Fixes: 10633
Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: only populate_obc_watchers if active

Signed-off-by: Samuel Just <sjust@redhat.com>

OSD: allow recovery in peered

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: go "peered" instead of "active" when < min_size

In the case of a replicated pool, the pg will transition to "peered"
rather than "active", where it can perform backfill bringing itself up
to min_size peers.  To that end, most operations now require
is_peered rather than is_active (OSDOps being the primary exception).
Also, rather than using the query_epoch on the activation message as the
activation epoch (for last_epoch_started) on the replica, we instead
use the last_epoch_started in the info sent by the primary.  This
allows the primary to not advance last_epoch_started past the last
known actual activation.  This will prevent later peering epochs from
requiring the last_update from a peered epoch to go active (might be
divergent).

Fixes: #7862
Signed-off-by: Samuel Just <sam.just@inktank.com>