Merge pull request #14055 from hrchu/radosgw-admin-api-user-caps

doc: correct and improve add user capability section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13956 from hrchu/rgw-admin-doc

Indicate how to add multiple admin capbabilies

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14372 from hrchu/rgwadmin-admin-doc-usage

doc: rgw: correct get usage parameter default value

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14547 from smithfarm/wip-create-pool

rgw: cleanup: fix variable name in RGWRados::create_pool() declaration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14697 from ceph/wip-rgw-upgrade-s3tests-encryption

qa/rgw: don't scan radosgw logs for encryption keys on jewel upgrade test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14424 from dachary/wip-jerasure-sync

erasure-code: sync jerasure/gf-complete submodules

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14558 from liewegas/wip-18293

osd: fix occasional MOSDMap leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14633 from SUSE/wip-yasm-wrapper-strip-E-flag

yasm-wrapper: strip -E (stops ccache trashing source files)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14640 from ovh/bp-fiemap-burden

osd, os: reduce fiemap burden

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14644 from liewegas/wip-rados-df-json

rados: out json 'df' values as numbers, not strings

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #14645 from liewegas/wip-mgr-mon

mon/MgrMonitor: only propose if we updated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14648 from gregsfortytwo/wip-17916-master

osd: pglog: with config, don't assert in the presence of stale diverg…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13978 from trociny/wip-18787

rbd-mirror A/A: proxy InstanceReplayer APIs via InstanceWatcher RPC

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14647 from dzafman/wip-19698

Fix full testing in cephtool/test.sh when used by rados suite
Replace using sleep with new wait_for_health() bash function

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cephtool.yaml: Add "failsafe" osd error messages to whitelist

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/workunits/ceph-helpers.sh: Improve test_mon_pg() by starting healthy

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa/workunits/ceph-helpers.sh: introduce (and use) wait_for_health

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Fix incorrect health output and test code to match

Signed-off-by: David Zafman <dzafman@redhat.com>

/workunits/cephtool/test.sh: Improve testing of health-output.

Instead of 9ff401e65b1a66a320953aeef8ab65b6f1fb6f10 use regex to handle:
       HEALTH_ERR 2 pgs stuck unclean; Full ratio(s) out of order

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix test_admin_heap_profiler() and enable it

Signed-off-by: David Zafman <dzafman@redhat.com>

cephtool/test.sh: Get socket for whichever environment this test is running

Need sudo for qa suite runs

Fixes: http://tracker.ceph.com/issues/19698
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #14689 from liewegas/wip-jewel-x

qa/suites/upgrade/jewel-x: add mgr.x role

rados: out json 'df' values as numbers, not strings

Fixes: http://tracker.ceph.com/issues/15546
Signed-off-by: Sage Weil <sage@redhat.com

Merge pull request #14423 from tchaikov/wip-cmake-crc32-cleanup

cmake: should not compile crc32c_ppc.c on intel arch.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14434 from ifed01/wip-bluestore-params

os/bluestore: make bluestore_max_blob_size parameter hdd/ssd case dependant

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14448 from majianpeng/bluestore-kv-queued-lat

os/bluestore: Record l_bluestore_state_kv_queued_lat for sync_submit_…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14549 from tchaikov/wip-kill-clang-warnings

misc: kill clang warnings

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14557 from liewegas/wip-bluefs-balance-interval

os/bluestore: do not balance bluefs on every kv_sync_thread iteration

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

qa/rgw: don't scan radosgw logs for encryption keys on jewel upgrade test

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/suites/upgrade/jewel-x/point-to-point: no xenial

Avoid conflict with upstream repo having a newer version
than what we're installing.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/jewel-x/stress-split: not pg import/export tests during upgrade

These aren't meant to be run on a mixed-version cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11796 from linuxbox2/wip-bucket-size-ck

rgw: add bucket size limit check to radosgw-admin

Merge pull request #14690 from ceph/luminous

build/ops: add 12.0.1 release tag in master

qa/suites/upgrade/jewel-x: add mgr.x role

Signed-off-by: Sage Weil <sage@redhat.com>

ReplicatedPG: use new fiemap interface

Reduce memory allocator load during sparse reads and recovery by not
encoding sparse file extents from map into bufferlist, then back again
into map.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

rgw: add bucket size limit check to radosgw-admin

The change adds a new list of all buckets x all users, with
fields for bucket name, tenant name, current num_objects,
current num_shards, current objects per shard, and the
corresponding fill_status--the latter consisting of 'OK',
'WARN <n>%', or 'OVER <n>%.'

The warning check is relative to two new tunables.  The threshold
max objects per shard is set as rgw_bucket_safe_max_objects_per_shard,
which defaults to 100K.  The value rgw_bucket_warning_threshold is
a percent of the current safe max at which to warn (defaults to
90% of full).

From review:

* fix indentation (rgw_admin)
* if user a user_id is provided, check only buckets for that user
* update shard warn pct to be pct-of-fill (not 100 - pct-of-fill)
* print only buckets near or over per-shard limit, if --warnings-only
* s/bucket limitcheck/bucket limit check */
* sanity shard limit should be 90, not 10 (because that changed)
* fixes for memleaks and other points found by cbodley

Fixes: http://tracker.ceph.com/issues/17925
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

ObjectStore, *Store: extend fiemap interface

This extends fiemap interface in objectstore and all data stores so
it's possible to pass in the map used later by caller, instead of
encoding internal map into bufferlist, then parsing it back into the
same type of data container, wasting memory and CPU time in the process.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

common/interval_set.h: add move_into() and new ctor

move_into() will be used to move data from internal (private)
std::map into other, external std::map, without exposing internal one
to public. New ctor improves moving data from external map to internal
one.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #14447 from ukernel/wip-19450

mds: drop partial entry and adjust write_pos when opening PurgeQueue

Reviewed-by: John Spray <john.spray@redhat.com>

rbd-mirror A/A: proxy InstanceReplayer APIs via InstanceWatcher RPC

Fixes: http://tracker.ceph.com/issues/18787
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test/librados_test_stub: mock wrapper for aio_notify

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: possible lockdep false positive

(when InstanceWatcher::remove_instance is called)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: race between notify context completion and Notifier destroy

The assert "m_pending_aio_notifies == 0" could fail in the destructor.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: don't use deprecated syntax

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: remove recently added public peer image types

They were supposed to be used to pass a list of peer images.
It has turned out unnecessary.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14638 from dillaman/wip-19692

test: rbd master/slave notify test should test active features

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #14197 from jcsp/wip-19395-counter

mds: use same inode count in health check as in trim

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14649 from hikycool/master

doc: update packages mentioned by build-doc and related doc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14650 from optimistyzy/420_1

NVMEDevice: remove unnessary dpdk header file

Reviewed-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: remove unnessary dpdk header file

Signed-off-by: optimistyzy <optimistyzy@gmail.com>

doc: correct the package name in ceph/admin/build-doc

In document, it is called 'libxml2-devel', one of the dependent
package of CentOS/RHEL7. But the 'build-doc' check 'libxml-devel'.
And it id also called 'libxml2-devel' in software of yum repo .
So correct the checked package name in ceph/admin/build-doc.

Signed-off-by: Yu Shengzuo <yu.shengzuo@99cloud.net>

doc: update dependent packages links and remove a 'important' tip

Some links in document has been deprecated, such as:
http://rpmfind.net/linux/centos/7.0.1406/, So update it;
The 'fc21, fc24, fc26(newest)' rpm 'ditaa' can be used in
CentOS/RHEL7 to build doc. So remove the '...important' tip.

Signed-off-by: Yu Shengzuo <yu.shengzuo@99cloud.net>

Merge pull request #14608 from tchaikov/wip-19594

qa/tasks: assert on pg status with a timeout

Reviewed-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: use inc.modified as the ctime for MOSDPGCreate

so the last_scrub_time of the newly created pg is consistent

Fixes: http://tracker.ceph.com/issues/19594
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks: update tests with helper to wait for pg-stats

and remove unused helpers

Fixes: http://tracker.ceph.com/issues/19594
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: add a "wait_for_pg_stats()" decorator

and accompany it with two helpers to access the pg stats in a more
natural way

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: pglog: with config, don't assert in the presence of stale divergent_priors

Fixes: http://tracker.ceph.com/issues/17916
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #14496 from jdurgin/wip-ec-overwrites-cleanup

osd: make ec overwrites ready to use

Reviewed-by: Sage Weil <sage@redhat.com>

qa: use bluestore for (k)rbd ec overwrites tests

Only bluestore supports ec overwrites

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: use bluestore for cephfs cephtool tests

This lets the tests enable ec overwrites

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: use 4k stripe_width again for test_rados_tool.sh

The stripe_unit change altered the default stripe_width to depend on k.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: move ec overwrites tests to a bluestore-only subsuite

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc: update ec docs for overwrite support

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

PendingReleaseNotes: note about ec overwrites

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd,OSDMonitor: try to protect against ec overwrites with filestore

This isn't perfect, but it's better than nothing. Prevent enabling the
allow_ec_overwrites flag if any of a sample of pgs in the pool map to
osds using filestore. This mainly protects filestore-only clusters
from enabling it.

If a filestore osd is started later, warn in the cluster log when it
gets a pg with ec overwrites enabled.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

test/osd/osd-scrub-repair.sh: add ec overwrites test cases

Move pool and profile creation into a single function, and
add a 'allow_overwrites' parameter for it so each ec test
can be paramterized by it.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/rados: test sparse reads with ec overwrites

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd, OSDMonitor, qa: mark ec overwrites non-experimental

Keep the pool flag around so we can distinguish between a pool that
should maintain hashes for each chunk, and a missing one is a bug, vs
an overwrites pool where we rely on bluestore checksums for detecting
corruption.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

OSDMonitor: get stripe_width via stripe_unit in ec profile

With bluestore, making the smallest write match min_alloc_size avoids
write amplification. With EC pools this is the stripe unit, or
stripe_width / num_data_chunks. Rather than requiring people to divide
by k to get the smallest ec write, allow it to be specified directly
via stripe_unit. Store it in the ec profile so changing a monitor
config option isn't necessary to set it.

This is particularly important for ec overwrites since they allow random i/o
which should match bluestore's checksum granularity (aka min_alloc_size).

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14256 from dillaman/wip-19297

osd: new op for calculating an extent checksum

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14414 from Adirl/is_ready

msg/async: Postpone bind if network stack is not ready

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14486 from dachary/wip-multiweight

crush: weight_set and id remapping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14556 from liewegas/wip-pgupmap

osd: pg-remap -> pg-upmap

Reviewed-by: David Zafman <dzafman@redhat.com>

mon/MgrMonitor: only propose if we updated

Otherwise we generate a new map on every beacon, even
though there is no change.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14554 from ukernel/wip-19630

mds: fix stray creation/removal notification

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14433 from cbodley/wip-rgw-multi-py

test/rgw: refactor test_multi.py for use in qa suite

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14623 from liewegas/wip-bluestore-compression

qa/objectstore: test bluestore with aggressive compression

Merge pull request #14642 from linuxbox2/wip-rgw-poolcreate-erange

rgw: error more verbosely in RGWRados::create_pool

rgw: error more verbosely in RGWRados::create_pool

Adds detail when pool creation fails with ERANGE, which can be
caused by pool settings.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #14637 from ovh/bp-test-osd-dup-warn-on-low-limit

test/osd/osd-dup.sh: warn on low open file limit

Reviewed-by: Sage Weil <sage@redhat.com>

test: rbd master/slave notify test should test active features

Fixes: http://tracker.ceph.com/issues/19692
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites: client-upgrade/jewel-client-x should test multiple features

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14631 from idryomov/wip-krbd-data-pool-fix

qa: krbd_data_pool.sh: account for rbd_info metadata object

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14385 from smithfarm/wip-release-script-strip-periods

tools: ceph-release-notes: strip trailing punctuation

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

test/osd/osd-dup.sh: warn on low open file limit

This test fails badly when open file limit is low. Increasing to around
1536 seems to be doing the trick, so warn the user with appropriate
message and try to proceed anyway.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #14599 from smithfarm/wip-crushtool-mappings

doc: mention --show-mappings in crushtool manpage

Reviewed-by: Loic Dachary <ldachary@redhat.com>

yasm-wrapper: strip -E (stops ccache trashing source files)

When building with ccache (`./do_cmake.sh -DWITH_CCACHE=ON`), it seems
ccache invokes `yast-wrapper` more than once per source file, the first
time with '-E' immediately before the source file in the argument list.
'-E' is an option for gcc which tells it to stop after the preprocessing
stage.  With yasm however, -E means "redirect error messages to file",
which makes yasm thinks there's no input files, and writes "yasm: No input
files specified" *over* the source file.  Then, the second run of
`yast-wrapper` (attempting to actually compile) fails with:

~/src/common/crc32c_intel_fast_asm.S:1: error: instruction expected after label

Curiously, this doesn't seem to affect all asm files; for me it
trashes crc32c_intel_fast_asm.S, but various other asm files build fine
without being clobbered.  I'm uncertain whether this is an issue
peculiar to openSUSE Tumbleweed, to ccache 3.3.3, or something else,
but in any case, having `yast-wrapper` strip '-E' fixes it.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #14483 from wjwithagen/wip-wjw-freebsd-ceph-disk-list

src/ceph-disk/ceph_disk/main.py: Make 'ceph-disk list' work on FreeBSD

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/rgw: rewrite test_multi.py in terms of rgw_multi

logging changes inspired by Abhishek Lekshmanan <abhishek@suse.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add tests in rgw_multi/tests.py based on multisite.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add rgw_multi/multisite.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14614 from wjwithagen/wip-wjw-cephtool-test-mon.sh

qa/workunits/cephtool/test.sh: Be more liberal in testing health-output.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14106 from theanalyst/release/12.0.1

doc: add 12.0.1 release notes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/objecstore/bluestore-comp: test bluestore with compression on

Enable aggressive compression.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/objectstore/bluestore: drop old option

This test code has been removed.

Signed-off-by: Sage Weil <sage@redhat.com>

qa: krbd_data_pool.sh: account for rbd_info metadata object

... added in commit a16beba78435 ("librbd: image create validates that
pool supports overwrites").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/rgw: add skeleton rgw_multi module

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: update and fixes for test-rgw-multisite.sh

the script was incomplete and unused, but it seems useful in itself
to bring up a simple multisite cluster without having to go through
test_multi.py. it's also a good test for functions in the other
test-rgw-*.sh scripts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: test_multi.py uses cluster_id for calls

this makes the underlying scripts more flexible, because they don't
depend on having a cluster name in the zgX-cY format

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14619 from cbodley/wip-rgw-encryption-build-fix

rgw: remove extra RGWMPObj in rgw_multi.h

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>