qa: update old replication tests

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

hadoop: add terasort workunit

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge remote-tracking branch 'gh/hammer'

Merge pull request #3741 from ceph/wip-cmake-vstart

cmake fixes and enable vstart with cmake build

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3729 from guangyy/wip-4254-hammer

osd: number of degraded objects in EC pool is wrong when there is OSD down(in).

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3722 from ceph/wip-10787

mon: fix osd_epoch cache bug 10787

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge branch 'osd-fix' of git://github.com/wonzhq/ceph into hammer

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge branch 'wip-5639' of git://github.com/rzarzynski/ceph into hammer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3760 from ceph/wip-10883

osd: Fix FileJournal wrap to get header out first

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3767 from athanatos/wip-10881

Wip 10881

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3768 from athanatos/wip-10780

Wip 10780

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3769 from athanatos/wip-10908

Wip 10908

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: radosgw, radosgw-admin related fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

vstart.sh: can use binaries outside of ceph/src

If setting CEPH_BUILD_ROOT, will use that path, otherwise runs
everything from current directory as before.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3612 from zhouyuan/acct_get_resp_headers

rgw: Adding stats in the header of GET response on account

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3648 from theanalyst/doc/swift-temp-url

doc: swift tempurl functionality

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3766 from tchaikov/chain_xattr-wrong-size-for-snprintf

os/chain_xattr: fix wrong `size` for snprintf()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3744 from ceph/wip-10884-hammer-rpm-devel-split

ceph.spec: split ceph-devel to appropriate *-devel packages

Reviewed-by: Sandon Van Ness <sandon@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

os/chain_xattr: fix wrong `size` for snprintf()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3764 from ceph/wip-10919

cls_rbd: invalidate bufferlist CRC when updating object map

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3754 from ceph/wip-10907

rgw: pass civetweb configurables to civetweb

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3762 from fghaas/master

doc: explain that ceph tell osd.N bench is non-destructive

Merge pull request #3697 from yuyuyu101/wip-async-fix-2

AsyncMessenger: Fix fd leak and lossless_peer_reuse policy problem

cls_rbd: invalidate bufferlist CRC when updating object map

The bit vector was not invalidating the bufferlist's CRC, resulting
in peer OSDs rejecting write op due to a mismatched CRC on the
message.

Fixes: #10919
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: explain that tell bench is non-destructive

Mention that "ceph tell osd.N bench" will not overwrite data in a live
OSD, but might temporarily affect OSD performance.

Signed-off-by: Florian Haas <florian@hastexo.com>

Merge remote-tracking branch 'gh/hammer'

Merge pull request #3759 from ceph/wip-10914

osdc: pass fadvise op flags to WritebackHandler read requests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd: Fix FileJournal wrap to get header out first

Correct and restore assert that was removed

Cause by f46b1b473fce0322a672b16c7739e569a45054b6
Fixes: #10883
Backport: dumpling, firefly, giant

Signed-off-by: David Zafman <dzafman@redhat.com>

osdc: pass fadvise op flags to WritebackHandler read requests

librbd was previously attempting to cast the provided Context to
retrieve the fadvise flags.  To eliminate the unsafe cast, now
the fadvise flags are directly passed to the WritebackHandler::read
callback.

Fixes: #10914
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd/OSDMap: include pg_temp count in summary

It is useful to know how big the pg_temp map is.  Strictly speaking
this is part of the OSDMap so I'm including it here.  It looks like
this:

     osdmap e25: 3 osds: 3 up, 3 in; 1 remapped pgs

It might be more user-friendly to put it in a line with the pgmap
somewhere (where other pg counts are included), but it doesn't quite
fit there either.  So sticking with where it lives in the data
structure!

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit db06582a067439a57e0d7f0da2193fc479736200)

Merge pull request #3629 from liewegas/wip-pg-temp

osd/OSDMap: include pg_temp count in summary

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3663 from ceph/wip-10765

librados: close watch/notify race

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3693 from ceph/wip-objecter-linger-locking

objecter: clean up linger op locking

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3710 from ceph/wip-10844

mon: MonCap: take EntityName instead when expanding profiles

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3748 from ceph/wip-pg-refs

osd: a few things to fix pg ref tracking

Reviewed-by: Kefu Chai <kchai@redhat.com>

PG: compensate for bug 10780 on older peers

Previously, there was a harmless bug where we didn't fill in the
last_epoch_started field for a peer which we are resetting the
last_backfill line for.  It's no longer harmless since we use that
as the activation epoch, so if the peer is missing the MIN_SIZE
feature bit, we fill in the last_epoch_started it meant to fill in.

Signed-off-by: Samuel Just <sjust@redhat.com>

PG: add a config option to enable (by default) recovery below min_size

This may be useful at some point.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3756 from ceph/wip-10899

librbd: fixed ImageWatcher recursive locking issues

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3755 from t-miyamae/wip-mshec-documentation

erasure code: add shec's documentation / change default layout

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librbd: fixed ImageWatcher recursive locking issues

It was possible for ImageWatcher to attempt to re-acquire held locks
via context callbacks.  This issue affected resizing/flattening when
no work was required and rescheduling a watch upon two successive
failures.

Fixes: #10899
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

erasure code: add shec's documentation / change default layout

Signed-off-by: Takeshi Miyamae <miyamae.takeshi@jp.fujitsu.com>

Merge pull request #3651 from swamireddy/wip-v092-organizationmap

mailmap: v092 organizationmap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: add Matt Richards to organizationmap

Signed-off-by: Swami Reddy M R <swami.reddy@ril.com>

mailmap: add Xie Rui to organizationmap

Signed-off-by: Swami Reddy M R <swami.reddy@ril.com>

ReplicatedPG: only allow a degraded write if we have at least min_size copies

Also, add a config option to disable the feature.

Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend: use tbl for empty transaction as well if necessary

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend: use tbl for empty transaction to backfill peer also

Fixes: 10908
Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: s/is_degraded_object/is_degraded_or_backfilling_object

Currently, this method also returns true if the object is backfilling.

Signed-off-by: Samuel Just <sjust@redhat.com>

rgw: pass civetweb configurables to civetweb

Fixes: #10907
Backport: firefly

Pass any configurables in the rgw frontends config line to civetweb.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3749 from ceph/wip-10900

test/librbd/fsx.c: disable RBD object map for krbd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3747 from ceph/wip-10892

osd: clear obc cache on_shutdown

Reviewed-by: Samuel Just <sjust@redhat.com>

test/librbd/fsx.c: disable RBD object map for krbd

RBD object map requires exclusive lock support, which is not
yet available in the kernel client.

Fixes: #10900
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: dump pg ref ids on shutdown

(if ref debugging is enabled)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make lock() and unlock() const

Signed-off-by: Sage Weil <sage@redhat.com>

common/tracked_int_ptr: const variants

Signed-off-by: Sage Weil <sage@redhat.com>

osd: clear obc cache on_shutdown

Fixes: #10892
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3746 from nwl/rgw-docs

doc: RGW is "built on top of librgw" changed to "librados"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.spec: split ceph-devel to appropriate *-devel packages

ceph-devel contains various header files/bindings for several
libraries, this patch creates *-devel packages for all the libraries
separately and provides the compatibility layer for the split.

http://tracker.ceph.com/issues/10884 Refs: #10884

Signed-off-by: Boris Ranto <branto@redhat.com>
Amended by Ken Dreyer <kdreyer@redhat.com> to add version numbers to the
Obsoletes, add Obsoletes to the libradosstriper1-devel and
libcephfs_jni1-devel subpackages, adjust the librados documentation, and
add the Redmine issue number to this commit log.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: RGW is "built on top of librgw" changed to "librados"

Signed-off-by: Neil Levine <levine@yoyo.org>

Merge branch 'hammer'

Merge branch 'wip-librbd-snaps-object-map' into hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: enforce write ordering with snapshot

The md_lock is now held for reading when scheduling write/discards.
Since snap_create now holds the lock for writing and flushes all
pending IO, write/discard operations will now be consistent for a
given request across objects.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use separate files for snapshot object maps

Instead of relying on the built-in object snapshot support,
create a separate object map object for each image snapshot.
This will allow a future repair utility to rebuild the object
map for an image's snapshots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: added IoCtx::write_full implementation

Mapped IoCtx::write_full to existing test method used by the
ObjectWriteOperation::write_full API method. Also added missing
cls_log implementation for debugging.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'wip-librbd-object-map-crc' into hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: skip invalidating an invalid object map

If the object map is already flagged as invalid, avoid
attempting to set the invalid object map flag again.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: added CRC validation to object map

Added a footer to the object map which stores a header CRC and
and data CRCs for each 4KB chunk.  Updates to the object map only
require recomputing the CRC to the affected 4KB chunk.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3739 from athanatos/wip-10840

PG,ReplicatedPG: make missing_digests local to scrub_compare_maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3738 from athanatos/wip-10830

osd/: include version_t in extra_reqids with promote

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3737 from athanatos/wip-10768

ReplicatedPG::on_change: requeue in_progress_async_reads close to last

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3694 from ceph/wip-10784

Fix 10784: librbd: image has watchers - not removing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3735 from ceph/wip-da-fix-xio-configure

fix configure to reflect the missing libs for xio

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3716 from t-miyamae/wip-mshec-fix10839

erasure-code: remove tests for when init() is not called in shec

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: remove tests for when init() is not called in shec (#10839)

init2_1, init2_2, init2_3 are equivalent to init_1 and also removed.
encode_6, decode_6, create_ruleset_3 are null argument tests,
but the arguments are references of C++, so also removed.

Signed-off-by: Takeshi Miyamae <miyamae.takeshi@jp.fujitsu.com>

Merge pull request #3732 from Abioy/patch-1

rm some useless codes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rm some useless codes

tls is not used here

Signed-off-by: Yongyue Sun <abioy.sun@gmail.com>

osd: number of degraded objects in EC pool is wrong when there is OSD down(in)

With EC pool (crush rule choose indep), when there is an OSD down, the size of the 'acting' list does not change (CRUSH_ITEM_NONE is used to replace the down OSD), in this case, 'actingset' should be used to calculate the degraded objects.

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

cmake: add librbd/test_fixture.cc

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cmake: add os/XfsFileStoreBackend.cc to rules

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3720 from dachary/wip-10800-shec-experimental

erasure-code: mark the shec plugin as experimental

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: mark the shec plugin as experimental

Reject the creation of an erasure code profile including plugin=shec
unless --enable-experimental-unrecoverable-data-corrupting-features=shec
is set.

http://tracker.ceph.com/issues/10800 Fixes: #10800

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: fix unused variable warning

Signed-off-by: Loic Dachary <ldachary@redhat.com>

common: capture check_experimental_feature_enabled message

Implement check_experimental_feature_enabled so that it returns the
message instead of unconditionally displaying it via derr. It allows the
caller to display it in another context.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mon/OSDMonitor: note osd map epoch in one other place

Also add a note about why we don't do it in another.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: move osd epoch cache update into helper, add fixme

There is no semantic change here, just a helper to make things more
clear.

Add warning about a potential bug so we don't lose track.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3727 from POPSuL/patch-2

README.md: fix the indent of a command line usage

Reviewed-by: Kefu Chai <kchai@redhat.com>

README.md: fix the indent of a command line usage

Spaces replaced with tab

Signed-off-by: Viktor Suprun <popsul1993@gmail.com>

AsyncConnection: release message when connection is closed

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3725 from XinzeChi/wip-kvstore-op

cleanup: delete useless member variable

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

cleanup: delete useless member variable

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #3717 from ceph/wip-decoding-oldest-snap

mds: fix decoding of InodeStore::oldest_snap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

PG,ReplicatedPG: make missing_digests local to scrub_compare_maps

We don't want/need to carry this state between scrub stages, so lets
remove the tempation by simply making it a local to scrub_compare_maps.

This caused a bug where we would attempt to write out the
missing_digests from the previous chunk during all future chunks, which
caused trouble.

Fixes: 10840
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge remote-tracking branch 'origin/wip-10785' into hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librbd/ImageWatcher.cc

Merge pull request #3723 from concubidated/master

Docs: OSD name, not id, needs to be given to remove an item from crush map

Reviewed-by: Sage Weil <sage@redhat.com>

Docs: OSD name, not id, needs to be given to remove an item from the CRUSH map.
Include command for deleteing bucket from the CRUSH map.

Signed-off-by: Tyler Brekke <tbrekke@redhat.com>

osd/OSDMap: include pg_temp count in summary

It is useful to know how big the pg_temp map is.  Strictly speaking
this is part of the OSDMap so I'm including it here.  It looks like
this:

     osdmap e25: 3 osds: 3 up, 3 in; 1 remapped pgs

It might be more user-friendly to put it in a line with the pgmap
somewhere (where other pg counts are included), but it doesn't quite
fit there either.  So sticking with where it lives in the data
structure!

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: do not trust small values in osd epoch cache

If the epoch cache says the osd has epoch 100 and the osd is asking for
epoch 200+, do not send it 100+.

Fixes: #10787
Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>

mon: MonCap: take EntityName instead when expanding profiles

entity_name_t is tightly coupled to the messenger, while EntityName is
tied to auth.  When expanding profiles we want to tie the profile
expansion to the entity that was authenticated.  Otherwise we may incur
in weird behavior such as having caps validation failing because a given
client messenger inst does not match the auth entity it used.

e.g., running

ceph --name osd.0 config-key exists foo daemon-private/osd.X/foo

has entity_name_t 'client.12345' and EntityName 'osd.0'.  Using
entity_name_t during profile expansion would not allow the client access
to daemon-private/osd.X/foo (client.12345 != osd.X).

Fixes: #10844
Backport: firefly,giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

mds: fix decoding of InodeStore::oldest_snap

There is no ENCODE_START/FINISH block when encoding inode that
embedded in dentry. So we can't use encoding version to check
if the buffer contains InodeStore::oldest_snap. Instead, we check
if the buffer iterator reaches end of buffer.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3698 from dachary/wip-10836-erasure-code-corpus

tests: clone ceph-erasure-code-corpus from ceph

Reviewed-by: Takeshi Miyamae <miyamae.takeshi@jp.fujitsu.com>