Merge pull request #3423 from yuyuyu101/rename-test-msgr

test_msgr: Rename unittest_msgr to ceph_test_msgr

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test_msgr: Rename unittest_msgr to ceph_test_msgr

Because ceph_test_msgr will run a lot of times than unittest expected,
move it into qa suites.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3192 from gravitystorm/patch-1

Replace placeholder with 'gateway'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Replace placeholder with 'gateway'

This section is where all the placeholders are replaced by example text, this one had been missed out.

Signed-off-by: Andy Allan <github@gravitystorm.co.uk>

Merge pull request #3414 from vuhuong/wip-cmake-fix

cmake: Fix CMakelist.txt to remove depleted files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3419 from Nairolf21/wip-documentation

doc: Introduce the term MON at http://ceph.com/docs/master/rados/

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Introduce the term MON at http://ceph.com/docs/master/rados/

http://tracker.ceph.com/issues/10203 Fixes:#10203

Signed-off-by: Florian Coste <fcoste21@gmail.com>

Merge pull request #3421 from hazem92/wip-documentation

doc: mon should be listed before osd

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: mon should be listed before osd

When deploying a Ceph cluster, the mon must be run first.
In the list shown at http://ceph.com/docs/master/rados/configuration/
it would therefore be better to have mon listed before osd.

http://tracker.ceph.com/issues/10204 Fixes: #10204

Signed-off-by: Hazem <hazem.amara@telecom-bretagne.eu>

Merge pull request #3420 from Anols/wip-documentation

doc: missing the reference to "ceph-deploy-new"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3418 from ThomasCantin/wip-documentation

doc: Network Configuration Reference duplicate string #3418

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Doc : Documentation#10205

Missing the reference of "ceph-deploy-new".

https://tracker.ceph.com/issues/10205

Signed-off-by: AYARI <ayari_anis@live.fr>

doc bug 10206

Merge pull request #3164 from ceph/wip-librbd-rados-stub

librados test driver and librbd unit test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3411 from ceph/wip-10576

ceph_test_rados_api_misc: do not assert rbd feature match

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph_test_rados_api_misc: do not assert rbd feature match

This test fails on upgrades when we (or the server) have new
features.  Make it less fragile.

Fixes: #10576
Signed-off-by: Sage Weil <sage@redhat.com>

cmake: Fix CMakelist.txt to remove depleted files

- remove depleted tools/ceph_mon_store_converter_srcs
- replace librbd source file (WatchCtx.cc to ImageWatcher.cc)

Signed-off-by: Vu Pham <vu@mellanox.com>

Merge pull request #3334 from dachary/wip-10216-jerasure-sync

erasure-code: update jerasure/gf-complete submodules

Merge pull request #3375 from XinzeChi/wip-journal-seq

osd: fix journal header.committed_up_to

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3320 from wonzhq/lfn_open

FileStore: return error if get_index fails in lfn_open

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3211 from yuyuyu101/wip-10172

AsyncMessenger: Bind thread to core, use buffer read and fix some bugs

Merge pull request #3221 from ceph/wip-9440

mon: log health changes to clog

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3382 from xinxinsh/wip-fix

fix command 'ceph pg dump_stuck degraded'

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix keyvaluestore name

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3338 from ceph/wip-recover-dentries

#9883 tools/cephfs: add recover_dentries to journaltool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3400 from kylinstorage/fix-rbd-watch

fix rbd watch command for v2 image

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/wip-bi-sharding-3'

Merge pull request #3396 from leseb/doc-openstack-fix-glance

doc: Fix OpenStack Glance configuration

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3401 from FrankYu/master

Doc: rbd-snapshot: Fix the typo

Merge pull request #3374 from dachary/wip-mailmap

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tools: output per-event errors from recover dentries

tools: handle hardlinks in recover_dentries

Signed-off-by: Johh Spray <john.spray@redhat.com>

tools: recover_dentries efficiency

Avoid a redundant stat, and gather updates to a frag
into a single OMAP get/set.

Still could be heaps more efficient in the case of
many updates to the same dirs by adding in a little
cache and batching the updates.

Signed-off-by: John Spray <john.spray@redhat.com>

tweak comment wording in recover_dentries

fixup some oversized lines

fix handling of io.read retval

(it returns length read, which was falling through as
a spurious nonzero "error")

tools: remove duplicated InoTable encoding

...and add a method to InoTable so that we can
artifically acquire inodes.

Signed-off-by: John Spray <john.spray@redhat.com>

JournalTool: handle corrupt fnodes

tools/cephfs: add recover_dentries to journaltool

This is intended as a comparatively safe recovery
operation, where we compare the versions
of journalled dentries with backing store dentries,
and write into the backing store only when the
existing contents are older than the journal
or invalid.

Fixes: #9883
Signed-off-by: John Spray <john.spray@redhat.com>

Doc: rbd-snapshot: Fix the typo

Signed-off-by: Frank Yu <flyxiaoyu@gmail.com>

rbd: fix bug about rbd watch command

the header oid should be prefix+image_id, rather than prefix+image_name

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>

Merge pull request #3397 from liewegas/wip-prealloc

mon: fix globalid when prealloc value is larger than max

mon: handle case where mon_globalid_prealloc > max_global_id

This triggers with the new larger mon_globalid_prealloc value.  It didn't
trigger on the existing cluster I tested on because it already had a very
large max.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: Fix OpenStack Glance configuration

Glance has not completely moved to 'store' yet so we need to configure
the store in the DEFAULT section as well.

Fixes: #10478
Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

Merge pull request #3361 from wonzhq/watch-notify

osd/ReplicatedPG: force promotion for watch/notify ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3395 from liewegas/wip-cache-none

osd: skip all of maybe_handle_cache if cachemode is none

Merge pull request #3315 from majianpeng/fix6

bug fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3385 from majianpeng/misc

Misc

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3194 from dachary/wip-10350-erasure-code-choose-tries

resolve and document most common erasure coded pool pain points

Documentation-Reviewed-by: Italo Santos <okdokk@gmail.com>

mon: change mon_globalid_prealloc to 10000 (from 100)

100 ids (session 100 authentications) can be consumed quite quickly if
the monitor is being queried by the CLI via scripts or on a large cluster,
especially if the propose interval is long (many seconds).  These live in
a 64-bit value and are only "lost" if we have a mon election before they
are consumed, so there's no real risk here.

Backport: giant, firefly
Reviewed-by: Joao Eduardo Luis <joao@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

mon: silently ignore mark_down, mark_disposable on AnonConnection

This mirrors 0a49db8e6fa141a36ca964e68017d02b81ae7a3c but was not captured
by 9fff0c53bdc7bb332df1a710da3de71e3c41bec7.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/wip-xio'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3391 from liewegas/wip-pool-delete

mon: global option to prevent pool deletion

Reviewed-by: John Spray <john.spray@redhat.com>

osd/ReplicatedPG: skip all of maybe_handle_cache if caching is off

Return quickly and avoid all of the checks.

Signed-off-by: Sage Weil <sage@redhat.com>

AsyncConnection: Fix memory leak for AsyncConnection

*_handler will store a reference to AsyncConnection, it need to explicit reset
it.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge remote-tracking branch 'origin/next'

Merge remote-tracking branch 'origin/wip-10271' into next

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3292 from kylinstorage/rbd-merge-diff-v2

rbd: merge diff files

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3318 from XinzeChi/wip-scrub

osd: support schedule scrub between some time defined by users

Reviewed-by David Zafman <dzafman@redhat.com>

Merge pull request #3090 from ceph/wip-mon-fixes

mon: fix issues with mixed-version monitors features

Reviewed-by: Sage Weil <sage@redhat.com>
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

rgw: bilog marker related fixes

Fix the way we parse the marker. Instead of specifying whether it's a
sharded or not sharded bucket, we pass a shard_id. If string itself
points to a singe shard, we'll use the passed shard_id, otherwise we'll
parse the string and determine the shard id by that. In this way when
referencing a single shard we can get the marker with either shard id
specified or not. This works with the non-shard case too.
Adjust the bilog listing function, set it to work with the new
interface. It was broken before, and there are multiple fixes to it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3390 from ceph/wip-librbd-coverity

librbd: fix coverity false-positives for tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon: Monitor: return 'required_features' on get_required_features()

We were returning 'quorum_features' instead.  This would lead to funny
and weird behavior.  I hate funny.

Backport: emperor,firefly,giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

mon: Elector: output features in handle_propose()

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

mon: Elector: put dangling message reference

Backport: emperor,firefly,giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

mon: mkfs compatset may be different from runtime compatset

When we create a monitor we set a given number of compat features on
disk to clearly state the features a given monitor supports -- mostly to
break backward compatibility when such compatibility cannot be
guaranteed.

However, we may wish to toggle some features during runtime; e.g., wait
for all the monitors in the quorum to support a given feature before
flipping a switch and state that all monitors now require feature X.

We are already flipping those switches during runtime, but we weren't
allowing the monitor to set a subset of those features during mkfs.
While the initial approach worked fine with clusters being upgraded and
fresh clusters, it could become weird in a mixed-version environment.

Backport: emperor,firefly,giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

mon/OSDMonitor: require mon_allow_pool_delete = true to remove pools

This is a simple safety check.  Since we default to true it is currently
opt-in.

Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3384 from liewegas/wip-crush-tests

crush: minor reorg of crush unit tests

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: fix journal header.committed_up_to

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

test: add test for osd scrub

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

osd: support schedule scrub between some time defined by users

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

test: Using different filename for different test case.

Some test case use tmp file to test.But they used same file and create
in the same directory. If we do in parallel, it will cause error.
So different test case use own their tmp file.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Striep: s/OSDExtent/ObjectExtent

OSDExtent already removed.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librados: clean up code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: clean up code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3335 from ceph/wip-cephfs-tabletool

Create cephfs-table-tool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3383 from ceph/wip-10552

client: fix getting zero-length xattr

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3358 from ceph/wip-mon-propose

mon: improve paxos proposals

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3342 from ceph/wip-10311

rgw: only keep track for cleanup of rados objects that were written

Reviewed-by: Ray Lv <xiangyulv@gmail.com>

Merge pull request #3362 from FrankYu/master

Doc: Fix the indentation in doc/rbd/rbd-snapshot.rst

Merge pull request #3346 from timfreund/update-radosgw-python-swift-example

doc: Replace cloudfiles with swiftclient in Python Swift example

Merge pull request #3359 from ceph/wip-mon-converter

drop ceph_mon_store_converter

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3373 from jdurgin/wip-rados-ls-dups

qa: ignore duplicates in rados ls

Reviewed-by: Sage Weil <sage@redhat.com>

crush: move two crush tests over

CrushWrapper handles map manipulation, crush.cc tests the placement.

Signed-off-by: Sage Weil <sage@redhat.com>

crush: rename unit tests

Signed-off-by: Sage Weil <sage@redhat.com>

client: fix getting zero-length xattr

Fixes: #10552
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3378 from xinxinsh/wip-cleanup

cleanup unused varibles

Reviewed-by: Sage Weil <sage@redhat.com>

tools: create cephfs-table-tool

It was unnatural to shoehorn resetting tables
into the journaltool.  This new tool initially
can simply dump or reset the session/snap/ino
tables, and would also be a place for any
more complex operations in future.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: give MDSTables a `rank` attribute

...so that they (like the new SessionMapStore)
can be used outside of a live MDS in tool code.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: abstract SessionMapStore from SessionMap

This is similar to what I did for InodeStore a while back:
introduce a logical separation between the persisted attributers
(and their encoding) and the live/runtime behavioural code.  This
results in a handy SessionMapStore class that can be used for
encode/decode from tools.

Also give it a reset_state method so that it matches the
prototype of the MDSTable subclasses for the benefit of
cephfs-table-tool.

Signed-off-by: John Spray <john.spray@redhat.com>

erasure-code: tests use different pool/profile names

Use different erasure coded pool names and profiles to avoid deletion /
creation races. The more expensive alternative is to run a different
cluster for each test.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

documentation: add troubleshooting erasure coded PGs section

Add a new section to the PG troubleshooting section that covers the most
common problems reported when an erasure coded pool fails to properly
map PGs to enough OSDs.

http://tracker.ceph.com/issues/10350 Fixes: #10350

Signed-off-by: Loic Dachary <ldachary@redhat.com>

erasure-code: set max_size to chunk_count() instead of 20

The ruleset created for an erasure coded pool has max_size set to a
fixed value of 20, which may be incorrect when more than 20 chunks are
needed and lead to obscure errors. Set it to the number of chunks,
i.e. k+m most of the time.

In a cluster with few OSDs (9 for instance), setting max_size to 20
causes performance problems when injecting a new crushmap. The monitor
will call CrushTester::test which tries 1024 mappins for all sizes
ranging from min_size to max_size. Each attempt to map more OSDs than
available will exhaust all retries (50 by default) and it takes a
significant amount of time. In a cluster with 9 OSDs, testing one such
ruleset can take up to 5 seconds.

Since the test blocks the monitor leader, a few erasure coded rulesets
will block the monitor long enough to exceed the timeouts and trigger an
election.

http://tracker.ceph.com/issues/10363 Fixes: #10363

Signed-off-by: Loic Dachary <ldachary@redhat.com>

crush: set_choose_tries = 100 for erasure code rulesets

It is common for people to try to map 9 OSDs out of a 9 OSDs total ceph
cluster. The default tries (50) will frequently lead to bad mappings for
this use case. Changing it to 100 makes no significant CPU performance
difference, as tested manually by running crushtool on one million
mappings.

http://tracker.ceph.com/issues/10353 Fixes: #10353

Signed-off-by: Loic Dachary <ldachary@redhat.com>

crush: update tries statistics for indep rules

http://tracker.ceph.com/issues/10349 Fixes: #10349

Signed-off-by: Loic Dachary <ldachary@redhat.com>

erasure-code: update jerasure/gf-complete submodules

jerasure:

git log --no-merges --pretty=%s \
  8fe20c6608385d6a1f38db89aec5cba85ccf04ac..02731df4c1eae1819c4453c9d3ab6d408cadd085
use assert(0) instead of exit(1)

gf-complete:

git log --no-merges --pretty=%s \
  39768c55bb00917691364f6f9f7bf688235aedf8..d384952c68a64d93ac7af6341d5519ea5d2958b9
gitignore: add src/.dirstamp
use assert(0) instead of exit(1)

http://tracker.ceph.com/issues/10216 Fixes: #10216

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3380 from trhoden/doc_cephextras

doc: add cases where ceph-extras is not needed

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: add cases where ceph-extras is not needed

The Ceph Extras repo is not needed on EL7 distributions or
Fedora

http://tracker.ceph.com/issues/9793 Refs: #9793

Signed-off-by: Travis Rhoden <trhoden@redhat.com>

Merge pull request #3379 from ceph/wip-mon-drop-conversion

mon: drop store conversion code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3377 from ceph/wip-fail-idempotent

mon/MDSMonitor: make 'mds fail' idempotent for IDs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>