add tests for ceph pg ls-by-osd

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add CLI ceph pg ls-by-osd

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add tests for ceph pg ls-by-primary

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

fix error : ceph pg ls 0

no valid command found; 10 closest matches:
pg ls-by-primary <osdname (id|osd.id)> {<int>} {active|clean|down|replay|splitting|scrubbing|scrubq|degraded|inconsistent|peering|repair|recovery|backfill_wait|incomplete|stale|remapped|deep_scrub|backfill|backfill_toofull|recovery_wait|undersized [active|clean|down|replay|splitting|scrubbing|scrubq|degraded|inconsistent|peering|repair|recovery|backfill_wait|incomplete|stale|remapped|deep_scrub|backfill|backfill_toofull|recovery_wait|undersized...]}
Error EINVAL: invalid command

this is due to partial compare of prefix

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add CLI ceph pg ls-by-primary

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add tests for ceph pg ls

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add CLI ceph pg ls [pool] [state]

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

get pg status with specific pool, osd, state

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add is_acting_osd function to check if pg is on that osd

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add pg_string_state function

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #3469 from tchaikov/fix-bashism-in-ceph-debugpack

ceph-debugpack: fix bashism of {1..10}

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3449 from wonzhq/rocksdb-del

RocksDBStore: filterpolicy is not deleted when RocksDBStore destructs

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

ceph-debugpack: fix bashism of {1..10}

* replaces {1..10} with $(seq 10)
* fixes 772220@bugs.debian.org

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #3467 from ktdreyer/wip-packaging-description-formatting

ceph.spec.in: standardize capitalization and punctuation

ceph.spec.in: capitalization and punctuation

Ensure that each sub-package's %description starts with a capital letter
and ends with a dot.

rpmlint has a rule that checks the Summary field, but not %description
(yet).

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #3424 from dachary/wip-reuse-dmcrypt

ceph-disk: do not reuse partition if encryption required

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3464 from ceph/wip-10614

test/librbd/fsx.c: disable RBD exclusive locking for krbd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3285 from ceph/wip-10439

pybind: ceph_argparse: validate incorrectly formed targets

Reviewed-by: Dan Mick <dan.mick@redhat.com>

Merge remote-tracking branch 'origin/next'

rbd image_read.sh: disable exclusive locking

Until the kernel supports RBD exclusive locking, this test
has been updated to create shared images (exclusive locking
disabled).

Fixes: #10613
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3458 from liewegas/wip-vstart

vstart.sh: pull default CEPH_PORT from .ceph_port

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3317 from ceph/wip-10474

FileJournal: Fix hang in FileJournal::close()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3460 from athanatos/wip-10524

Wip 10524

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3461 from athanatos/wip-10535

ceph_test_objectstore: enable keyvaluestore experimental option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3462 from athanatos/wip-10536

ReplicatedPG::_scrub: don't record digests for snapdirs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3459 from athanatos/wip-10603

ObjectStore::_update_op: treat CLONERANGE2 like CLONE

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3350 from liewegas/wip-10477

osd: populate blocked_by with peers we are trying to activate

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3457 from athanatos/wip-txn-append

Wip txn append

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3351 from liewegas/wip-10512

osd: fix blocked vs blocking rqeuest requeue order on interval change

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3363 from liewegas/wip-cache-fix

osd: fix ReplicatedPG::cancel_proxy_read_ops()

Reviewed-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #3365 from liewegas/wip-peering

osd: new activating pg state between peering and active

Reviewed-by: Samuel Just <sjust@redhat.com>

vstart.sh: pull default CEPH_PORT from .ceph_port

This lets you put a unique port in .ceph_port in your working dir and
vstart.sh instances will avoid each other without having to pass
CEPH_PORT=... to each one each time.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3425 from vuhuong/wip-xio-fork

xio: Enable xio option to call fork init

Reviewed-by: Matt Benjamin <matt@cohortfs.com>

Merge pull request #3453 from FrankYu/master

Doc: Fix the extra blank space in doc/start/quick-rbd.rst

Merge pull request #3451 from xiaoxichen/fix_memstore_statfs

Fix memstore free space caculation

Reviewed-by: Sage Weil <sage@redhat.com>

FileJournal: Fix hang in FileJournal::close()

On FileStore::umount() when stopping and in ENOSPC condition,
don't let write_thread_entry() get stuck in commit_cond.Wait(write_lock).

Fixes: #10474
Signed-off-by: David Zafman <dzafman@redhat.com>

Doc: Fix the extra blank space in doc/start/quick-rbd.rst

Signed-off-by: Frank Yu <flyxiaoyu@gmail.com>

Merge pull request #3450 from zhouyuan/rgw_doc_typo

doc: Fix a typo in radosgw-admin doc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fix memstore free space caculation

Originally if memstore run out of space, it will report a very
large positive number as free space. For example:

root@bigmem:~# rados df
pool name                 KB      objects       clones     degraded      unfound           rd        rd KB           wr        wr KB
rbd                 12366704        92240            0            0           0            0            0        92240     12366704
  total used        12375877        92240
  total avail   36028797009199167
  total space        2611076

st->f_bavail = st->f_blocks - used_bytes / st->f_bsize

This is due to used_bytes is an unsigned value, so compiler make the whole statement unsigned.

Fix it by adding explicit type cast,

st->f_bavail = long(st->f_blocks) - long(used_bytes / st->f_bsize)

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

RocksDBStore: filterpolicy is not deleted when RocksDBStore destructs

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

test/librbd/fsx.c: disable RBD exclusive locking for krbd

Until the kernel supports RBD exclusive locking, this test
has been updated to disable the new feature.

Fixes: #10614
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: Fix a typo in radosgw-admin doc

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Revert "Revert "Merge remote-tracking branch 'origin/wip-bi-sharding-3' into next""

Following a merge of next to master, the feature got reverted (because
it was reverted on next). Undoing.

This reverts commit 6613358ddc5339c8e33c409387fd6044db0b6f26.

Merge pull request #3246 from ceph/wip-9780-9781

ceph-objectstore-tool import/export and related fixes

Reviewed-by: Samuel Just <sjust@redhat.com>

PGBackend: fix and clarify be_select_auth_object

Previously, auth would end up containing every object without a
self-evident defect -- even if they did not match each other.  Instead
of filtering out the non-matching items there, be_compare_scrubmaps now
returns one valid object and be_compare_scrubmaps gathers the other
which match it.

We can be smarter by doing this in be_select_auth_object and selecting
the largest matching set, but for now this is simpler.

Fixes: 10524
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge branch 'next'

ObjectStore::_update_op: treat CLONERANGE2 like CLONE

Fixes: #10603
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3251 from theanalyst/fix/rgw/s3-10062

wip: rgw: check keystone auth also for s3 post requests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3381 from liewegas/wip-pg-stat-updates

osd: avoid spamming monitor with unchanged pg stats

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3393 from ceph/wip-osd-df

osd: ceph osd df [tree] command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3409 from trociny/wip-osd_pool_flags

osd: new pool safeguard flags: nodelete, nopgchange, nosizechange

Reviewed-by: Sage Weil <sage@redhat.com>

rbd: ensure aio_write buffer isn't invalidated during image import

The buffer provided to aio_write shouldn't be invalidated until
after aio_write has indicated that the operation has completed.

Fixes: #10590
Backport: giant
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3426 from jdurgin/wip-10592

qa: disable automatic locking for manual locking test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3259 from trociny/pg_dump_plain

mon: improvements for 'ceph pg dump' in plain format

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3431 from XinzeChi/wip-compile-warning

osd: fix some compile warning

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3436 from ceph/wip-10579

qa: move fs quota to its own dir

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa: move fs quota to its own dir

This is because running fs/misc on a kclient
should not include quota (quota not supported
on kernel client).

Fixes: #10579
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3402 from ceph/wip-10388

#10388 PerfCounters for MDCache: strays and recovery

osd: fix some compile warning

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #3433 from FrankYu/master

Doc: Fix the typo in doc/rbd/rados-rbd-cmds.rst

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Doc: Fix the typo in doc/rbd/rados-rbd-cmds.rst

Fix the typo in doc/rbd/rados-rbd-cmds.rst, also add explanation/example
when creating an image without specifying pool.

Signed-off-by: Frank Yu <flyxiaoyu@gmail.com>

Merge pull request #3427 from jdurgin/wip-cram

test: fix rbd cli tests for new feature bit

Reviewed-by: Sage Weil <sage@redhat.com>

test: fix rbd cli tests for new feature bit

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: disable automatic locking for manual locking test

Automatic locking hides the ESHUTDOWN from the caller, which is how
this test detects that blacklisting works.

Fixes: #10592
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

xio: Enable xio option to call fork init

Ceph fork when running as daemon. This fix call xio option to enforce
Accelio to explicitly call ibv_fork_init() to avoid cq overrun issue.

Signed-off-by: Vu Pham <vu@mellanox.com>

osd: revert d427ca35404a30e1f428859c3274e030f2f83ef6

d427ca35404a30e1f428859c3274e030f2f83ef6 reversed the order of localt
and op_t, while some logic depends on the order of localt and op_t.

This commit revert d427ca35404a30e1f428859c3274e030f2f83ef6.

Fix http://tracker.ceph.com/issues/10534.

Change-Id: I5adba561d10274e3d12f89418a755c39e5e5f086
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: Transction::append SHOULD NOT modify other.op_bl

the other.op_bl SHOULD NOT be changes during append operation,
we use additional bufferlist to avoid this problem

Change-Id: I5d4a8c9ed3d0b926ca4bbdcdadf72cb4d5fde53c
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

Merge pull request #3408 from ceph/wip-10257

mon: PGMonitor: several stats output error fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3423 from yuyuyu101/rename-test-msgr

test_msgr: Rename unittest_msgr to ceph_test_msgr

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: do not reuse partition if encryption required

If encryption is required, an existing journal partition must not be
reused. If an existing partition that was not prepared with ceph-disk is
found and reused, the caller will assume it is encrypted although it is
not.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

test_msgr: Rename unittest_msgr to ceph_test_msgr

Because ceph_test_msgr will run a lot of times than unittest expected,
move it into qa suites.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

mon: PGMonitor: skip zeroed osd stats on get_rule_avail()

Fixes: #10257
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

mon: PGMonitor: available size 0 if no osds on pool's ruleset

get_rule_avail() may return < 0, which we were using blindly assuming it
would always return an unsigned value.  We would end up with weird
values if the ruleset had no osds.

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3192 from gravitystorm/patch-1

Replace placeholder with 'gateway'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Replace placeholder with 'gateway'

This section is where all the placeholders are replaced by example text, this one had been missed out.

Signed-off-by: Andy Allan <github@gravitystorm.co.uk>

Merge pull request #3414 from vuhuong/wip-cmake-fix

cmake: Fix CMakelist.txt to remove depleted files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3419 from Nairolf21/wip-documentation

doc: Introduce the term MON at http://ceph.com/docs/master/rados/

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Introduce the term MON at http://ceph.com/docs/master/rados/

http://tracker.ceph.com/issues/10203 Fixes:#10203

Signed-off-by: Florian Coste <fcoste21@gmail.com>

Merge pull request #3421 from hazem92/wip-documentation

doc: mon should be listed before osd

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: mon should be listed before osd

When deploying a Ceph cluster, the mon must be run first.
In the list shown at http://ceph.com/docs/master/rados/configuration/
it would therefore be better to have mon listed before osd.

http://tracker.ceph.com/issues/10204 Fixes: #10204

Signed-off-by: Hazem <hazem.amara@telecom-bretagne.eu>

Merge pull request #3420 from Anols/wip-documentation

doc: missing the reference to "ceph-deploy-new"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3418 from ThomasCantin/wip-documentation

doc: Network Configuration Reference duplicate string #3418

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Doc : Documentation#10205

Missing the reference of "ceph-deploy-new".

https://tracker.ceph.com/issues/10205

Signed-off-by: AYARI <ayari_anis@live.fr>

doc bug 10206

Merge pull request #3164 from ceph/wip-librbd-rados-stub

librados test driver and librbd unit test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3411 from ceph/wip-10576

ceph_test_rados_api_misc: do not assert rbd feature match

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph_test_rados_api_misc: do not assert rbd feature match

This test fails on upgrades when we (or the server) have new
features.  Make it less fragile.

Fixes: #10576
Signed-off-by: Sage Weil <sage@redhat.com>

cmake: Fix CMakelist.txt to remove depleted files

- remove depleted tools/ceph_mon_store_converter_srcs
- replace librbd source file (WatchCtx.cc to ImageWatcher.cc)

Signed-off-by: Vu Pham <vu@mellanox.com>

Merge pull request #3334 from dachary/wip-10216-jerasure-sync

erasure-code: update jerasure/gf-complete submodules

Merge pull request #3375 from XinzeChi/wip-journal-seq

osd: fix journal header.committed_up_to

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3320 from wonzhq/lfn_open

FileStore: return error if get_index fails in lfn_open

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3211 from yuyuyu101/wip-10172

AsyncMessenger: Bind thread to core, use buffer read and fix some bugs

Merge pull request #3221 from ceph/wip-9440

mon: log health changes to clog

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3382 from xinxinsh/wip-fix

fix command 'ceph pg dump_stuck degraded'

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix keyvaluestore name

Signed-off-by: Sage Weil <sage@redhat.com>

PGBackend: do not update oi digest on inconsistent object

Otherwise, repair can race with the oi update.  Also, it doesn't really
make sense to update the oi attr if the object is inconsistent.

Fixes: #10524
Signed-off-by: Samuel Just <sjust@redhat.com>

osd: new pool safeguard flags: nodelete, nopgchange, nosizechange

* nodelete - pool can't be deleted
* nopgchange - pool's pg and pgp num can't be changed
* nosizechange - pool's size and min size can't be changed

This is intended to help some poor admin to avoid a very bad day.

Fixes: #9792 (but in a different way than it was proposed there)
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #3338 from ceph/wip-recover-dentries

#9883 tools/cephfs: add recover_dentries to journaltool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3400 from kylinstorage/fix-rbd-watch

fix rbd watch command for v2 image

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/wip-bi-sharding-3'