ceph-debugpack: fix bashism of {1..10}

* replaces {1..10} with $(seq 10)
* fixes 772220@bugs.debian.org

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Revert "Revert "Merge remote-tracking branch 'origin/wip-bi-sharding-3' into next""

Following a merge of next to master, the feature got reverted (because
it was reverted on next). Undoing.

This reverts commit 6613358ddc5339c8e33c409387fd6044db0b6f26.

Merge pull request #3246 from ceph/wip-9780-9781

ceph-objectstore-tool import/export and related fixes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge branch 'next'

Merge pull request #3251 from theanalyst/fix/rgw/s3-10062

wip: rgw: check keystone auth also for s3 post requests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3381 from liewegas/wip-pg-stat-updates

osd: avoid spamming monitor with unchanged pg stats

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3393 from ceph/wip-osd-df

osd: ceph osd df [tree] command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3409 from trociny/wip-osd_pool_flags

osd: new pool safeguard flags: nodelete, nopgchange, nosizechange

Reviewed-by: Sage Weil <sage@redhat.com>

rbd: ensure aio_write buffer isn't invalidated during image import

The buffer provided to aio_write shouldn't be invalidated until
after aio_write has indicated that the operation has completed.

Fixes: #10590
Backport: giant
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3426 from jdurgin/wip-10592

qa: disable automatic locking for manual locking test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3259 from trociny/pg_dump_plain

mon: improvements for 'ceph pg dump' in plain format

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3431 from XinzeChi/wip-compile-warning

osd: fix some compile warning

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3436 from ceph/wip-10579

qa: move fs quota to its own dir

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa: move fs quota to its own dir

This is because running fs/misc on a kclient
should not include quota (quota not supported
on kernel client).

Fixes: #10579
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3402 from ceph/wip-10388

#10388 PerfCounters for MDCache: strays and recovery

osd: fix some compile warning

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #3433 from FrankYu/master

Doc: Fix the typo in doc/rbd/rados-rbd-cmds.rst

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Doc: Fix the typo in doc/rbd/rados-rbd-cmds.rst

Fix the typo in doc/rbd/rados-rbd-cmds.rst, also add explanation/example
when creating an image without specifying pool.

Signed-off-by: Frank Yu <flyxiaoyu@gmail.com>

Merge pull request #3427 from jdurgin/wip-cram

test: fix rbd cli tests for new feature bit

Reviewed-by: Sage Weil <sage@redhat.com>

test: fix rbd cli tests for new feature bit

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: disable automatic locking for manual locking test

Automatic locking hides the ESHUTDOWN from the caller, which is how
this test detects that blacklisting works.

Fixes: #10592
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3408 from ceph/wip-10257

mon: PGMonitor: several stats output error fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3423 from yuyuyu101/rename-test-msgr

test_msgr: Rename unittest_msgr to ceph_test_msgr

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test_msgr: Rename unittest_msgr to ceph_test_msgr

Because ceph_test_msgr will run a lot of times than unittest expected,
move it into qa suites.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

mon: PGMonitor: skip zeroed osd stats on get_rule_avail()

Fixes: #10257
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

mon: PGMonitor: available size 0 if no osds on pool's ruleset

get_rule_avail() may return < 0, which we were using blindly assuming it
would always return an unsigned value.  We would end up with weird
values if the ruleset had no osds.

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3192 from gravitystorm/patch-1

Replace placeholder with 'gateway'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Replace placeholder with 'gateway'

This section is where all the placeholders are replaced by example text, this one had been missed out.

Signed-off-by: Andy Allan <github@gravitystorm.co.uk>

Merge pull request #3414 from vuhuong/wip-cmake-fix

cmake: Fix CMakelist.txt to remove depleted files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3419 from Nairolf21/wip-documentation

doc: Introduce the term MON at http://ceph.com/docs/master/rados/

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Introduce the term MON at http://ceph.com/docs/master/rados/

http://tracker.ceph.com/issues/10203 Fixes:#10203

Signed-off-by: Florian Coste <fcoste21@gmail.com>

Merge pull request #3421 from hazem92/wip-documentation

doc: mon should be listed before osd

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: mon should be listed before osd

When deploying a Ceph cluster, the mon must be run first.
In the list shown at http://ceph.com/docs/master/rados/configuration/
it would therefore be better to have mon listed before osd.

http://tracker.ceph.com/issues/10204 Fixes: #10204

Signed-off-by: Hazem <hazem.amara@telecom-bretagne.eu>

Merge pull request #3420 from Anols/wip-documentation

doc: missing the reference to "ceph-deploy-new"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3418 from ThomasCantin/wip-documentation

doc: Network Configuration Reference duplicate string #3418

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Doc : Documentation#10205

Missing the reference of "ceph-deploy-new".

https://tracker.ceph.com/issues/10205

Signed-off-by: AYARI <ayari_anis@live.fr>

doc bug 10206

Merge pull request #3164 from ceph/wip-librbd-rados-stub

librados test driver and librbd unit test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3411 from ceph/wip-10576

ceph_test_rados_api_misc: do not assert rbd feature match

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph_test_rados_api_misc: do not assert rbd feature match

This test fails on upgrades when we (or the server) have new
features.  Make it less fragile.

Fixes: #10576
Signed-off-by: Sage Weil <sage@redhat.com>

cmake: Fix CMakelist.txt to remove depleted files

- remove depleted tools/ceph_mon_store_converter_srcs
- replace librbd source file (WatchCtx.cc to ImageWatcher.cc)

Signed-off-by: Vu Pham <vu@mellanox.com>

Merge pull request #3334 from dachary/wip-10216-jerasure-sync

erasure-code: update jerasure/gf-complete submodules

Merge pull request #3375 from XinzeChi/wip-journal-seq

osd: fix journal header.committed_up_to

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3320 from wonzhq/lfn_open

FileStore: return error if get_index fails in lfn_open

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3211 from yuyuyu101/wip-10172

AsyncMessenger: Bind thread to core, use buffer read and fix some bugs

Merge pull request #3221 from ceph/wip-9440

mon: log health changes to clog

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3382 from xinxinsh/wip-fix

fix command 'ceph pg dump_stuck degraded'

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix keyvaluestore name

Signed-off-by: Sage Weil <sage@redhat.com>

osd: new pool safeguard flags: nodelete, nopgchange, nosizechange

* nodelete - pool can't be deleted
* nopgchange - pool's pg and pgp num can't be changed
* nosizechange - pool's size and min size can't be changed

This is intended to help some poor admin to avoid a very bad day.

Fixes: #9792 (but in a different way than it was proposed there)
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #3338 from ceph/wip-recover-dentries

#9883 tools/cephfs: add recover_dentries to journaltool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3400 from kylinstorage/fix-rbd-watch

fix rbd watch command for v2 image

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/wip-bi-sharding-3'

Revert "Merge remote-tracking branch 'origin/wip-bi-sharding-3' into next"

This reverts commit f79d8f24e9c0bf0d0b37270eba2745a878f2caed, reversing
changes made to 896c8899ac28eb0403bfaa20454f3756f3705c51.

Merge remote-tracking branch 'origin/wip-bi-sharding-3' into next

Merge pull request #3396 from leseb/doc-openstack-fix-glance

doc: Fix OpenStack Glance configuration

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3401 from FrankYu/master

Doc: rbd-snapshot: Fix the typo

Merge pull request #3374 from dachary/wip-mailmap

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tools: output per-event errors from recover dentries

tools: handle hardlinks in recover_dentries

Signed-off-by: Johh Spray <john.spray@redhat.com>

tools: recover_dentries efficiency

Avoid a redundant stat, and gather updates to a frag
into a single OMAP get/set.

Still could be heaps more efficient in the case of
many updates to the same dirs by adding in a little
cache and batching the updates.

Signed-off-by: John Spray <john.spray@redhat.com>

tweak comment wording in recover_dentries

fixup some oversized lines

fix handling of io.read retval

(it returns length read, which was falling through as
a spurious nonzero "error")

tools: remove duplicated InoTable encoding

...and add a method to InoTable so that we can
artifically acquire inodes.

Signed-off-by: John Spray <john.spray@redhat.com>

JournalTool: handle corrupt fnodes

tools/cephfs: add recover_dentries to journaltool

This is intended as a comparatively safe recovery
operation, where we compare the versions
of journalled dentries with backing store dentries,
and write into the backing store only when the
existing contents are older than the journal
or invalid.

Fixes: #9883
Signed-off-by: John Spray <john.spray@redhat.com>

mds: perfcounters for RecoveryQueue

Define these at MDCache scope, as RecoveryQueue is
not an official subsystem to be made visible to
external parties.

Signed-off-by: John Spray <john.spray@redhat.com>

mds/MDCache: stray/purge perfcounters

Fixes: #10388
Signed-off-by: John Spray <john.spray@redhat.com>

Doc: rbd-snapshot: Fix the typo

Signed-off-by: Frank Yu <flyxiaoyu@gmail.com>

rbd: fix bug about rbd watch command

the header oid should be prefix+image_id, rather than prefix+image_name

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>

Merge pull request #3397 from liewegas/wip-prealloc

mon: fix globalid when prealloc value is larger than max

mon: handle case where mon_globalid_prealloc > max_global_id

This triggers with the new larger mon_globalid_prealloc value.  It didn't
trigger on the existing cluster I tested on because it already had a very
large max.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: Fix OpenStack Glance configuration

Glance has not completely moved to 'store' yet so we need to configure
the store in the DEFAULT section as well.

Fixes: #10478
Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

Merge pull request #3361 from wonzhq/watch-notify

osd/ReplicatedPG: force promotion for watch/notify ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3395 from liewegas/wip-cache-none

osd: skip all of maybe_handle_cache if cachemode is none

Merge pull request #3315 from majianpeng/fix6

bug fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3385 from majianpeng/misc

Misc

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3194 from dachary/wip-10350-erasure-code-choose-tries

resolve and document most common erasure coded pool pain points

Documentation-Reviewed-by: Italo Santos <okdokk@gmail.com>

mon: change mon_globalid_prealloc to 10000 (from 100)

100 ids (session 100 authentications) can be consumed quite quickly if
the monitor is being queried by the CLI via scripts or on a large cluster,
especially if the propose interval is long (many seconds).  These live in
a 64-bit value and are only "lost" if we have a mon election before they
are consumed, so there's no real risk here.

Backport: giant, firefly
Reviewed-by: Joao Eduardo Luis <joao@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

mon: silently ignore mark_down, mark_disposable on AnonConnection

This mirrors 0a49db8e6fa141a36ca964e68017d02b81ae7a3c but was not captured
by 9fff0c53bdc7bb332df1a710da3de71e3c41bec7.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/wip-xio'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3391 from liewegas/wip-pool-delete

mon: global option to prevent pool deletion

Reviewed-by: John Spray <john.spray@redhat.com>

mon/OSDMonitor: pull totals into the table format

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: don't -nan for variance when no osds

Signed-off-by: Sage Weil <sage@redhat.com>

common/TextTable: do padding during final step

We don't know the full width until we get all table entries.  Do not do
any padding before that.  In particular, setw may left-pad on a column
that is left-justified.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: drop unnecessary stringify

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: fix cli tests for new tree output

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: skip all of maybe_handle_cache if caching is off

Return quickly and avoid all of the checks.

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: add --tree option

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: drop weights from crush dump_tree

Signed-off-by: Sage Weil <sage@redhat.com>

osd, crush: caps for headers

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushTreeDumper: add missing close_section

Signed-off-by: Sage Weil <sage@redhat.com>

crush, osdmap: use weightf_t helper for all weights

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use TextTable for osd tree command output

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osd: use weightf_t type when outputing weights by osd tree command

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

mon: ceph osd df -- a new command to show OSD utilization

Fixes: #10452
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Add a type (weightf_t) to output float weights.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osd: use CrushTreeDumper for printing osd tree.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

crush: use CrushTreeDumper for dumping crush tree.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

crush: introduce a helper class for dumping crush tree.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>