librbd: improved ImageWatcher duplicate message detection

Added a unique client id to announcement messages so that duplicate
lock release / acquired / requested messages can be detected and
ignored by the client.  Also fixed an issue processing the result
code for async operations.

Fixes: #10898
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add test instances for watch/notify messages

Ensure that the librbd watch/notify messages are tested
for backwards compatibility.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: minor cleanup of ImageWatcher messages

Moved all RPC messages to their own classes to facilitate cleaner
version control and backward compatibility.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3756 from ceph/wip-10899

librbd: fixed ImageWatcher recursive locking issues

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: fixed ImageWatcher recursive locking issues

It was possible for ImageWatcher to attempt to re-acquire held locks
via context callbacks.  This issue affected resizing/flattening when
no work was required and rescheduling a watch upon two successive
failures.

Fixes: #10899
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3749 from ceph/wip-10900

test/librbd/fsx.c: disable RBD object map for krbd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3747 from ceph/wip-10892

osd: clear obc cache on_shutdown

Reviewed-by: Samuel Just <sjust@redhat.com>

test/librbd/fsx.c: disable RBD object map for krbd

RBD object map requires exclusive lock support, which is not
yet available in the kernel client.

Fixes: #10900
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: clear obc cache on_shutdown

Fixes: #10892
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3746 from nwl/rgw-docs

doc: RGW is "built on top of librgw" changed to "librados"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: RGW is "built on top of librgw" changed to "librados"

Signed-off-by: Neil Levine <levine@yoyo.org>

Merge branch 'hammer'

Merge branch 'wip-librbd-snaps-object-map' into hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: enforce write ordering with snapshot

The md_lock is now held for reading when scheduling write/discards.
Since snap_create now holds the lock for writing and flushes all
pending IO, write/discard operations will now be consistent for a
given request across objects.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use separate files for snapshot object maps

Instead of relying on the built-in object snapshot support,
create a separate object map object for each image snapshot.
This will allow a future repair utility to rebuild the object
map for an image's snapshots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: added IoCtx::write_full implementation

Mapped IoCtx::write_full to existing test method used by the
ObjectWriteOperation::write_full API method. Also added missing
cls_log implementation for debugging.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'wip-librbd-object-map-crc' into hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: skip invalidating an invalid object map

If the object map is already flagged as invalid, avoid
attempting to set the invalid object map flag again.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: added CRC validation to object map

Added a footer to the object map which stores a header CRC and
and data CRCs for each 4KB chunk.  Updates to the object map only
require recomputing the CRC to the affected 4KB chunk.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3739 from athanatos/wip-10840

PG,ReplicatedPG: make missing_digests local to scrub_compare_maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3738 from athanatos/wip-10830

osd/: include version_t in extra_reqids with promote

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3737 from athanatos/wip-10768

ReplicatedPG::on_change: requeue in_progress_async_reads close to last

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3694 from ceph/wip-10784

Fix 10784: librbd: image has watchers - not removing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3735 from ceph/wip-da-fix-xio-configure

fix configure to reflect the missing libs for xio

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3716 from t-miyamae/wip-mshec-fix10839

erasure-code: remove tests for when init() is not called in shec

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: remove tests for when init() is not called in shec (#10839)

init2_1, init2_2, init2_3 are equivalent to init_1 and also removed.
encode_6, decode_6, create_ruleset_3 are null argument tests,
but the arguments are references of C++, so also removed.

Signed-off-by: Takeshi Miyamae <miyamae.takeshi@jp.fujitsu.com>

Merge pull request #3732 from Abioy/patch-1

rm some useless codes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rm some useless codes

tls is not used here

Signed-off-by: Yongyue Sun <abioy.sun@gmail.com>

Merge pull request #3720 from dachary/wip-10800-shec-experimental

erasure-code: mark the shec plugin as experimental

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: mark the shec plugin as experimental

Reject the creation of an erasure code profile including plugin=shec
unless --enable-experimental-unrecoverable-data-corrupting-features=shec
is set.

http://tracker.ceph.com/issues/10800 Fixes: #10800

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: fix unused variable warning

Signed-off-by: Loic Dachary <ldachary@redhat.com>

common: capture check_experimental_feature_enabled message

Implement check_experimental_feature_enabled so that it returns the
message instead of unconditionally displaying it via derr. It allows the
caller to display it in another context.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3727 from POPSuL/patch-2

README.md: fix the indent of a command line usage

Reviewed-by: Kefu Chai <kchai@redhat.com>

README.md: fix the indent of a command line usage

Spaces replaced with tab

Signed-off-by: Viktor Suprun <popsul1993@gmail.com>

Merge pull request #3725 from XinzeChi/wip-kvstore-op

cleanup: delete useless member variable

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

cleanup: delete useless member variable

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #3717 from ceph/wip-decoding-oldest-snap

mds: fix decoding of InodeStore::oldest_snap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

PG,ReplicatedPG: make missing_digests local to scrub_compare_maps

We don't want/need to carry this state between scrub stages, so lets
remove the tempation by simply making it a local to scrub_compare_maps.

This caused a bug where we would attempt to write out the
missing_digests from the previous chunk during all future chunks, which
caused trouble.

Fixes: 10840
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge remote-tracking branch 'origin/wip-10785' into hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librbd/ImageWatcher.cc

Merge pull request #3723 from concubidated/master

Docs: OSD name, not id, needs to be given to remove an item from crush map

Reviewed-by: Sage Weil <sage@redhat.com>

Docs: OSD name, not id, needs to be given to remove an item from the CRUSH map.
Include command for deleteing bucket from the CRUSH map.

Signed-off-by: Tyler Brekke <tbrekke@redhat.com>

mds: fix decoding of InodeStore::oldest_snap

There is no ENCODE_START/FINISH block when encoding inode that
embedded in dentry. So we can't use encoding version to check
if the buffer contains InodeStore::oldest_snap. Instead, we check
if the buffer iterator reaches end of buffer.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3698 from dachary/wip-10836-erasure-code-corpus

tests: clone ceph-erasure-code-corpus from ceph

Reviewed-by: Takeshi Miyamae <miyamae.takeshi@jp.fujitsu.com>

Merge pull request #3696 from dachary/wip-10755-centos-docker

tests: add support for centos 6

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3714 from tchaikov/fix-osdcap-parser-10757

osd: fix OSDCap parser on old boost/spirit

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: Dockerfile COPY with two arguments

Avoid using COPY with multiple files because it will not behave as
intended with older (1.2) versions of docker.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: update docker helper documentation

The tags for the centos repository changed from centos6, centos7 to 6
and 7 which is consistent with the other distribution
repositories. Update the documentation accordingly.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: add Dockerfile for centos-6

http://tracker.ceph.com/issues/10755 Fixes: #10755

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: one Dockerfile per repository:tag

There cannot be a common Dockerfile for all repository:tag combination
of a given operating system. The only way to customize a Dockerfile is
via variable substitution and it cannot conveniently address all
differences between versions.

Create one Dockerfile per operating system version instead. I.e. one
dockerfile for centos:7, one for centos:6 etc.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

osd: fix OSDCap parser on old boost/spirit

* on boost 1.41, the ascii::space skipper fails to skip the spaces at the
  beginning of the parsed string, so as a workaround we replace the `lit(' ')`
  in grammar spec with `ascii::blank`. this also simplifies the grammar
  a little bit.

Fixes: #10757
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3707 from dachary/wip-erasure-code-uninitialized

erasure-code: initialize all data members

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #3713 from tchaikov/pass-string-by-ref

Pass string by ref

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd/OpRequest: pass string by const&

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3711 from ceph/hadoop

qa: hadoop plays nice with new teuthology task

Merge remote-tracking branch 'gh/master' into hammer

Merge pull request #3522 from ceph/wip-8600

mon: OSDMonitor: tolerate catastrophically bad crush maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3703 from vuhuong/wip-xio-hammer

Wip xio hammer

Reviewed-by: Matt Benjamin <matt@cohortfs.com>

osd/: include version_t in extra_reqids with promote

Otherwise, we can't return the correct user version on a dup request.

Note: This patch does not handle compatilibity with the variant which
does not include the version_t field.  Since it's been less than 2 weeks
and we haven't had a release, I think that's ok since handling
compatilibity would require some overhead in the encode/decode
methods.

Fixes: 10830
Introduced (merge): b79c349067ab82b1ccb5d6cdac702f651b27a435
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3664 from liewegas/wip-10695

mon: fix 'nan' from osd df output

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3704 from ceph/wip-10761

librbd: blacklist clients before breaking locks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3513 from rzarzynski/wip-10425

librados: Wip 10425

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3515 from rzarzynski/wip-10458

librados: Wip 10458 - rectify the pool name caching in IoCtxImpl.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3674 from jdurgin/wip-python-leak

rados.py: keep reference to python callbacks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3676 from majianpeng/misc

Misc

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3678 from tchaikov/fix-pg-leak

osd: fix PG leak in SnapTrimWQ._clear()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge remote-tracking branch 'ceph/hammer'

qa: hadoop plays nice with new teuthology task

This brings the hadoop wordcount up-to-date with the new teuthology
hadoop task.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #3683 from dachary/wip-10845-python-flask

tests: no need for python-flask at build time

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

tests: no need for python-flask at build time

http://tracker.ceph.com/issues/10845 Refs: #10845

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3699 from ceph/wip-early-adopters

doc: Early adopter notes for CephFS

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3617 from ceph/wip-10737

mds: disallow layout changes for files with data

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #3683 from ceph/wip-10791-stopping

MDSMonitor: do not allow MDS to transition from STATE_STOPPING

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #3599 from ceph/wip-traceless-reply

Wip traceless reply

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3700 from ceph/wip-librbd-maint-tests

tests: add simple tests for proxied maintenance operations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3706 from dachary/wip-10839-shec-uninitialized-hammer

erasure-code: fix uninitialized data members on SHEC

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: initialize all data members

As a safeguard against unitialized memory in isa and jerasure
plugins. Since the init method is supposed to be called before the
plugin is used, it is mostly not relevant. However it can lead to
problems that are non trivial to figure out.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

erasure-code: fix uninitialized data members on SHEC

Also fix the tests that verify the result when init is not called and
factorize the two tests that were almost identical for simplicity.

http://tracker.ceph.com/issues/10839 Fixes: #10839

Backport: hammer
Signed-off-by: Loic Dachary <ldachary@redhat.com>

ReplicatedPG::on_change: requeue in_progress_async_reads close to last

Fixes: 10768
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3692 from athanatos/wip-8753

ReplicatedPG::on_change: clean up callbacks_for_degraded_object

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG::on_change: clean up callbacks_for_degraded_object

Backport: dumpling, firefly, giant
Fixes: 8753
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3643 from athanatos/wip-10734

ObjectStore: fix Transaction encoding version number

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/master' into hammer

Merge pull request #3092 from abartlet/ceph-dmcrypt-plain

Ceph dmcrypt with LUKS (split into 4 patches)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge branch 'hammer' of github.com:ceph/ceph into hammer

mon bits
Reviewed-by: Sage Weil <sage@redhat.com>
cli bits
Reviewed-by: Dan Mick <dmick@redhat.com>

mon/PGMonitor: drop pg ls debug line

Signed-off-by: Sage Weil <sage@redhat.com>

fix configure to reflect the missing libs for xio

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

doc: add a page for CephFS early adopters

This is where we should point people who are
trying out CephFS in evaluation environments.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: Ceph FS -> CephFS

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephfs: clarify the 'not production ready' warning

Continue to run into people who see this and run away (occasionally to
less robust solutions) without understanding why.

Signed-off-by: Sage Weil <sage@redhat.com>

build-doc: package checks for fedora

Signed-off-by: John Spray <john.spray@redhat.com>

build-doc: don't ignore errors

It's hard to see what went wrong when e.g. a pip
install fails, if it just steams right past.

Signed-off-by: John Spray <john.spray@redhat.com>

tests: clone ceph-erasure-code-corpus from ceph

Instead of the http://github.com/dachary namespace. It is not an issue
in the common case because it should be cloned because it is in the
.gitmodules file with the proper namespace.

http://tracker.ceph.com/issues/10836 Fixes: #10836

Signed-off-by: Loic Dachary <ldachary@redhat.com>

xio: Fix out-of-order responses

Without this clients received out-of-order responses from xio messenger

Signed-off-by: Raju.Kurunkad@sandisk.com

cmake: Fix cmakelist to compile librbd

Signed-off-by: Vu Pham <vu@melanox.com>

osd: Update object state after removing watch from object info

Fixes: #10784
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Simplify handle_watch_timeout() using existing support functions

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #3691 from athanatos/wip-6003

FileJournal: fix journalq population in do_read_entry()

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3690 from athanatos/wip-10817

Wip 10817

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3689 from athanatos/wip-10771

ReplicatedPG::eval_repop: check waiting_for_* even if !m

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-7701' of git://github.com/xinxinsh/ceph into hammer

Conflicts:
src/osd/osd_types.h