common: add perf counters description

Signed-off-by: Kiseleva Alyona <akiselyova@mirantis.com>

Merge pull request #3782 from ceph/hadoop-workunits

Hadoop workunits

Merge pull request #3753 from ceph/wip-10743

mds: wait for mydir replicas to go away in stopping

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3775 from ceph/wip-10862

mon: do not try and "deactivate" the last MDS

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa: update old replication tests

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

hadoop: add terasort workunit

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #3686 from ceph/wip-crushtool

crushtool: process commands in a sensible order

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3778 from dachary/wip-crushtool

crush: re-organize the help to clarify ordering

crush: re-organize the help to clarify ordering

The options are grouped according to the stage they relate to

   - input/build
   - tunables adjustments
   - modifications
   - display/test
   - output

Signed-off-by: Loic Dachary <ldachary@redhat.com>

crushtool: add test for crushtool ordering

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: send --tree to stdout

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: name osds with --build function

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: do not dump tree on build

This pollutes the output.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v0.87.1

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3776 from dachary/wip-development-workflows

doc: development workflows

Reviewed-by: Yuri Weinstein <yuriw@redhat.com>

doc: development workflows

Document the high level view of the development workflows which are
otherwise implicit and non trivial to figure out.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

doc: page title suitable for :doc:

When using :doc:`/dev/sepia` the page title is inlined. The modified
title is easier to include as part of a sensible sentence.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

doc: spellcheck quick development guide

Signed-off-by: Loic Dachary <ldachary@redhat.com>

doc: do not doxygen src/tracing

because it does not publish an API that needs to be documented and
because it triggers https://github.com/ceph/asphyxiate/issues/1 in some
contexts.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mon: do not try and "deactivate" the last MDS

Fixes: #10862
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3763 from Ved-vampir/master

common: add description in PerfCounter class

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/hammer'

Merge pull request #3741 from ceph/wip-cmake-vstart

cmake fixes and enable vstart with cmake build

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3729 from guangyy/wip-4254-hammer

osd: number of degraded objects in EC pool is wrong when there is OSD down(in).

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3722 from ceph/wip-10787

mon: fix osd_epoch cache bug 10787

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge branch 'osd-fix' of git://github.com/wonzhq/ceph into hammer

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge branch 'wip-5639' of git://github.com/rzarzynski/ceph into hammer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3760 from ceph/wip-10883

osd: Fix FileJournal wrap to get header out first

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3767 from athanatos/wip-10881

Wip 10881

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3768 from athanatos/wip-10780

Wip 10780

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3769 from athanatos/wip-10908

Wip 10908

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: radosgw, radosgw-admin related fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

vstart.sh: can use binaries outside of ceph/src

If setting CEPH_BUILD_ROOT, will use that path, otherwise runs
everything from current directory as before.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3612 from zhouyuan/acct_get_resp_headers

rgw: Adding stats in the header of GET response on account

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3648 from theanalyst/doc/swift-temp-url

doc: swift tempurl functionality

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3766 from tchaikov/chain_xattr-wrong-size-for-snprintf

os/chain_xattr: fix wrong `size` for snprintf()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3744 from ceph/wip-10884-hammer-rpm-devel-split

ceph.spec: split ceph-devel to appropriate *-devel packages

Reviewed-by: Sandon Van Ness <sandon@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

os/chain_xattr: fix wrong `size` for snprintf()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3764 from ceph/wip-10919

cls_rbd: invalidate bufferlist CRC when updating object map

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3754 from ceph/wip-10907

rgw: pass civetweb configurables to civetweb

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3762 from fghaas/master

doc: explain that ceph tell osd.N bench is non-destructive

mon: fix bug with counter name (win counter was inc-ed in lose func)

Signed-off-by: Kiseleva Alyona akiselyova@mirantis.com

Merge pull request #3697 from yuyuyu101/wip-async-fix-2

AsyncMessenger: Fix fd leak and lossless_peer_reuse policy problem

common: add description in PerfCounter class

Signed-off-by: Kiseleva Alyona <akiselyova@mirantis.com>

cls_rbd: invalidate bufferlist CRC when updating object map

The bit vector was not invalidating the bufferlist's CRC, resulting
in peer OSDs rejecting write op due to a mismatched CRC on the
message.

Fixes: #10919
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: explain that tell bench is non-destructive

Mention that "ceph tell osd.N bench" will not overwrite data in a live
OSD, but might temporarily affect OSD performance.

Signed-off-by: Florian Haas <florian@hastexo.com>

Merge remote-tracking branch 'gh/hammer'

Merge pull request #3759 from ceph/wip-10914

osdc: pass fadvise op flags to WritebackHandler read requests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd: Fix FileJournal wrap to get header out first

Correct and restore assert that was removed

Cause by f46b1b473fce0322a672b16c7739e569a45054b6
Fixes: #10883
Backport: dumpling, firefly, giant

Signed-off-by: David Zafman <dzafman@redhat.com>

osdc: pass fadvise op flags to WritebackHandler read requests

librbd was previously attempting to cast the provided Context to
retrieve the fadvise flags.  To eliminate the unsafe cast, now
the fadvise flags are directly passed to the WritebackHandler::read
callback.

Fixes: #10914
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd/OSDMap: include pg_temp count in summary

It is useful to know how big the pg_temp map is.  Strictly speaking
this is part of the OSDMap so I'm including it here.  It looks like
this:

     osdmap e25: 3 osds: 3 up, 3 in; 1 remapped pgs

It might be more user-friendly to put it in a line with the pgmap
somewhere (where other pg counts are included), but it doesn't quite
fit there either.  So sticking with where it lives in the data
structure!

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit db06582a067439a57e0d7f0da2193fc479736200)

Merge pull request #3629 from liewegas/wip-pg-temp

osd/OSDMap: include pg_temp count in summary

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3663 from ceph/wip-10765

librados: close watch/notify race

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3693 from ceph/wip-objecter-linger-locking

objecter: clean up linger op locking

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3710 from ceph/wip-10844

mon: MonCap: take EntityName instead when expanding profiles

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3748 from ceph/wip-pg-refs

osd: a few things to fix pg ref tracking

Reviewed-by: Kefu Chai <kchai@redhat.com>

PG: compensate for bug 10780 on older peers

Previously, there was a harmless bug where we didn't fill in the
last_epoch_started field for a peer which we are resetting the
last_backfill line for.  It's no longer harmless since we use that
as the activation epoch, so if the peer is missing the MIN_SIZE
feature bit, we fill in the last_epoch_started it meant to fill in.

Signed-off-by: Samuel Just <sjust@redhat.com>

PG: add a config option to enable (by default) recovery below min_size

This may be useful at some point.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3756 from ceph/wip-10899

librbd: fixed ImageWatcher recursive locking issues

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3755 from t-miyamae/wip-mshec-documentation

erasure code: add shec's documentation / change default layout

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librbd: fixed ImageWatcher recursive locking issues

It was possible for ImageWatcher to attempt to re-acquire held locks
via context callbacks.  This issue affected resizing/flattening when
no work was required and rescheduling a watch upon two successive
failures.

Fixes: #10899
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

erasure code: add shec's documentation / change default layout

Signed-off-by: Takeshi Miyamae <miyamae.takeshi@jp.fujitsu.com>

Merge pull request #3651 from swamireddy/wip-v092-organizationmap

mailmap: v092 organizationmap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: add Matt Richards to organizationmap

Signed-off-by: Swami Reddy M R <swami.reddy@ril.com>

mailmap: add Xie Rui to organizationmap

Signed-off-by: Swami Reddy M R <swami.reddy@ril.com>

ReplicatedPG: only allow a degraded write if we have at least min_size copies

Also, add a config option to disable the feature.

Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend: use tbl for empty transaction as well if necessary

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend: use tbl for empty transaction to backfill peer also

Fixes: 10908
Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: s/is_degraded_object/is_degraded_or_backfilling_object

Currently, this method also returns true if the object is backfilling.

Signed-off-by: Samuel Just <sjust@redhat.com>

rgw: pass civetweb configurables to civetweb

Fixes: #10907
Backport: firefly

Pass any configurables in the rgw frontends config line to civetweb.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mds: wait for mydir replicas to go away in stopping

Previously, while we were waiting for locks related
to mydir to be acknowledged by other ranks, we could
advance to the point of capping the log.  When the
locks were acked, we would try to write to the log
and assert out.  Avoid this by waiting for mydir
to not be replicated before capping the log.

Fixes: #10743
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3749 from ceph/wip-10900

test/librbd/fsx.c: disable RBD object map for krbd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3747 from ceph/wip-10892

osd: clear obc cache on_shutdown

Reviewed-by: Samuel Just <sjust@redhat.com>

test/librbd/fsx.c: disable RBD object map for krbd

RBD object map requires exclusive lock support, which is not
yet available in the kernel client.

Fixes: #10900
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: dump pg ref ids on shutdown

(if ref debugging is enabled)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make lock() and unlock() const

Signed-off-by: Sage Weil <sage@redhat.com>

common/tracked_int_ptr: const variants

Signed-off-by: Sage Weil <sage@redhat.com>

osd: clear obc cache on_shutdown

Fixes: #10892
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3746 from nwl/rgw-docs

doc: RGW is "built on top of librgw" changed to "librados"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.spec: split ceph-devel to appropriate *-devel packages

ceph-devel contains various header files/bindings for several
libraries, this patch creates *-devel packages for all the libraries
separately and provides the compatibility layer for the split.

http://tracker.ceph.com/issues/10884 Refs: #10884

Signed-off-by: Boris Ranto <branto@redhat.com>
Amended by Ken Dreyer <kdreyer@redhat.com> to add version numbers to the
Obsoletes, add Obsoletes to the libradosstriper1-devel and
libcephfs_jni1-devel subpackages, adjust the librados documentation, and
add the Redmine issue number to this commit log.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: RGW is "built on top of librgw" changed to "librados"

Signed-off-by: Neil Levine <levine@yoyo.org>

Merge branch 'hammer'

Merge branch 'wip-librbd-snaps-object-map' into hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: enforce write ordering with snapshot

The md_lock is now held for reading when scheduling write/discards.
Since snap_create now holds the lock for writing and flushes all
pending IO, write/discard operations will now be consistent for a
given request across objects.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use separate files for snapshot object maps

Instead of relying on the built-in object snapshot support,
create a separate object map object for each image snapshot.
This will allow a future repair utility to rebuild the object
map for an image's snapshots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: added IoCtx::write_full implementation

Mapped IoCtx::write_full to existing test method used by the
ObjectWriteOperation::write_full API method. Also added missing
cls_log implementation for debugging.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'wip-librbd-object-map-crc' into hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: skip invalidating an invalid object map

If the object map is already flagged as invalid, avoid
attempting to set the invalid object map flag again.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: added CRC validation to object map

Added a footer to the object map which stores a header CRC and
and data CRCs for each 4KB chunk.  Updates to the object map only
require recomputing the CRC to the affected 4KB chunk.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3739 from athanatos/wip-10840

PG,ReplicatedPG: make missing_digests local to scrub_compare_maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3738 from athanatos/wip-10830

osd/: include version_t in extra_reqids with promote

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3737 from athanatos/wip-10768

ReplicatedPG::on_change: requeue in_progress_async_reads close to last

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3694 from ceph/wip-10784

Fix 10784: librbd: image has watchers - not removing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3735 from ceph/wip-da-fix-xio-configure

fix configure to reflect the missing libs for xio

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3716 from t-miyamae/wip-mshec-fix10839

erasure-code: remove tests for when init() is not called in shec

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: remove tests for when init() is not called in shec (#10839)

init2_1, init2_2, init2_3 are equivalent to init_1 and also removed.
encode_6, decode_6, create_ruleset_3 are null argument tests,
but the arguments are references of C++, so also removed.

Signed-off-by: Takeshi Miyamae <miyamae.takeshi@jp.fujitsu.com>

Merge pull request #3732 from Abioy/patch-1

rm some useless codes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rm some useless codes

tls is not used here

Signed-off-by: Yongyue Sun <abioy.sun@gmail.com>

osd: number of degraded objects in EC pool is wrong when there is OSD down(in)

With EC pool (crush rule choose indep), when there is an OSD down, the size of the 'acting' list does not change (CRUSH_ITEM_NONE is used to replace the down OSD), in this case, 'actingset' should be used to calculate the degraded objects.

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

cmake: add librbd/test_fixture.cc

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>