Merge pull request #14647 from dzafman/wip-19698

Fix full testing in cephtool/test.sh when used by rados suite
Replace using sleep with new wait_for_health() bash function

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cephtool.yaml: Add "failsafe" osd error messages to whitelist

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/workunits/ceph-helpers.sh: Improve test_mon_pg() by starting healthy

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa/workunits/ceph-helpers.sh: introduce (and use) wait_for_health

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Fix incorrect health output and test code to match

Signed-off-by: David Zafman <dzafman@redhat.com>

/workunits/cephtool/test.sh: Improve testing of health-output.

Instead of 9ff401e65b1a66a320953aeef8ab65b6f1fb6f10 use regex to handle:
       HEALTH_ERR 2 pgs stuck unclean; Full ratio(s) out of order

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix test_admin_heap_profiler() and enable it

Signed-off-by: David Zafman <dzafman@redhat.com>

cephtool/test.sh: Get socket for whichever environment this test is running

Need sudo for qa suite runs

Fixes: http://tracker.ceph.com/issues/19698
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #14689 from liewegas/wip-jewel-x

qa/suites/upgrade/jewel-x: add mgr.x role

Merge pull request #14423 from tchaikov/wip-cmake-crc32-cleanup

cmake: should not compile crc32c_ppc.c on intel arch.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14434 from ifed01/wip-bluestore-params

os/bluestore: make bluestore_max_blob_size parameter hdd/ssd case dependant

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14448 from majianpeng/bluestore-kv-queued-lat

os/bluestore: Record l_bluestore_state_kv_queued_lat for sync_submit_…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14549 from tchaikov/wip-kill-clang-warnings

misc: kill clang warnings

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14557 from liewegas/wip-bluefs-balance-interval

os/bluestore: do not balance bluefs on every kv_sync_thread iteration

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

qa/suites/upgrade/jewel-x/point-to-point: no xenial

Avoid conflict with upstream repo having a newer version
than what we're installing.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/jewel-x/stress-split: not pg import/export tests during upgrade

These aren't meant to be run on a mixed-version cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11796 from linuxbox2/wip-bucket-size-ck

rgw: add bucket size limit check to radosgw-admin

Merge pull request #14690 from ceph/luminous

build/ops: add 12.0.1 release tag in master

qa/suites/upgrade/jewel-x: add mgr.x role

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: add bucket size limit check to radosgw-admin

The change adds a new list of all buckets x all users, with
fields for bucket name, tenant name, current num_objects,
current num_shards, current objects per shard, and the
corresponding fill_status--the latter consisting of 'OK',
'WARN <n>%', or 'OVER <n>%.'

The warning check is relative to two new tunables.  The threshold
max objects per shard is set as rgw_bucket_safe_max_objects_per_shard,
which defaults to 100K.  The value rgw_bucket_warning_threshold is
a percent of the current safe max at which to warn (defaults to
90% of full).

From review:

* fix indentation (rgw_admin)
* if user a user_id is provided, check only buckets for that user
* update shard warn pct to be pct-of-fill (not 100 - pct-of-fill)
* print only buckets near or over per-shard limit, if --warnings-only
* s/bucket limitcheck/bucket limit check */
* sanity shard limit should be 90, not 10 (because that changed)
* fixes for memleaks and other points found by cbodley

Fixes: http://tracker.ceph.com/issues/17925
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #14447 from ukernel/wip-19450

mds: drop partial entry and adjust write_pos when opening PurgeQueue

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14638 from dillaman/wip-19692

test: rbd master/slave notify test should test active features

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #14197 from jcsp/wip-19395-counter

mds: use same inode count in health check as in trim

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14649 from hikycool/master

doc: update packages mentioned by build-doc and related doc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14650 from optimistyzy/420_1

NVMEDevice: remove unnessary dpdk header file

Reviewed-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: remove unnessary dpdk header file

Signed-off-by: optimistyzy <optimistyzy@gmail.com>

doc: correct the package name in ceph/admin/build-doc

In document, it is called 'libxml2-devel', one of the dependent
package of CentOS/RHEL7. But the 'build-doc' check 'libxml-devel'.
And it id also called 'libxml2-devel' in software of yum repo .
So correct the checked package name in ceph/admin/build-doc.

Signed-off-by: Yu Shengzuo <yu.shengzuo@99cloud.net>

doc: update dependent packages links and remove a 'important' tip

Some links in document has been deprecated, such as:
http://rpmfind.net/linux/centos/7.0.1406/, So update it;
The 'fc21, fc24, fc26(newest)' rpm 'ditaa' can be used in
CentOS/RHEL7 to build doc. So remove the '...important' tip.

Signed-off-by: Yu Shengzuo <yu.shengzuo@99cloud.net>

Merge pull request #14608 from tchaikov/wip-19594

qa/tasks: assert on pg status with a timeout

Reviewed-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: use inc.modified as the ctime for MOSDPGCreate

so the last_scrub_time of the newly created pg is consistent

Fixes: http://tracker.ceph.com/issues/19594
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks: update tests with helper to wait for pg-stats

and remove unused helpers

Fixes: http://tracker.ceph.com/issues/19594
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: add a "wait_for_pg_stats()" decorator

and accompany it with two helpers to access the pg stats in a more
natural way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14496 from jdurgin/wip-ec-overwrites-cleanup

osd: make ec overwrites ready to use

Reviewed-by: Sage Weil <sage@redhat.com>

qa: use bluestore for (k)rbd ec overwrites tests

Only bluestore supports ec overwrites

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: use bluestore for cephfs cephtool tests

This lets the tests enable ec overwrites

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: use 4k stripe_width again for test_rados_tool.sh

The stripe_unit change altered the default stripe_width to depend on k.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: move ec overwrites tests to a bluestore-only subsuite

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc: update ec docs for overwrite support

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

PendingReleaseNotes: note about ec overwrites

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd,OSDMonitor: try to protect against ec overwrites with filestore

This isn't perfect, but it's better than nothing. Prevent enabling the
allow_ec_overwrites flag if any of a sample of pgs in the pool map to
osds using filestore. This mainly protects filestore-only clusters
from enabling it.

If a filestore osd is started later, warn in the cluster log when it
gets a pg with ec overwrites enabled.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

test/osd/osd-scrub-repair.sh: add ec overwrites test cases

Move pool and profile creation into a single function, and
add a 'allow_overwrites' parameter for it so each ec test
can be paramterized by it.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/rados: test sparse reads with ec overwrites

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd, OSDMonitor, qa: mark ec overwrites non-experimental

Keep the pool flag around so we can distinguish between a pool that
should maintain hashes for each chunk, and a missing one is a bug, vs
an overwrites pool where we rely on bluestore checksums for detecting
corruption.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

OSDMonitor: get stripe_width via stripe_unit in ec profile

With bluestore, making the smallest write match min_alloc_size avoids
write amplification. With EC pools this is the stripe unit, or
stripe_width / num_data_chunks. Rather than requiring people to divide
by k to get the smallest ec write, allow it to be specified directly
via stripe_unit. Store it in the ec profile so changing a monitor
config option isn't necessary to set it.

This is particularly important for ec overwrites since they allow random i/o
which should match bluestore's checksum granularity (aka min_alloc_size).

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14256 from dillaman/wip-19297

osd: new op for calculating an extent checksum

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14414 from Adirl/is_ready

msg/async: Postpone bind if network stack is not ready

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14486 from dachary/wip-multiweight

crush: weight_set and id remapping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14556 from liewegas/wip-pgupmap

osd: pg-remap -> pg-upmap

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #14554 from ukernel/wip-19630

mds: fix stray creation/removal notification

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14433 from cbodley/wip-rgw-multi-py

test/rgw: refactor test_multi.py for use in qa suite

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14623 from liewegas/wip-bluestore-compression

qa/objectstore: test bluestore with aggressive compression

Merge pull request #14642 from linuxbox2/wip-rgw-poolcreate-erange

rgw: error more verbosely in RGWRados::create_pool

rgw: error more verbosely in RGWRados::create_pool

Adds detail when pool creation fails with ERANGE, which can be
caused by pool settings.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #14637 from ovh/bp-test-osd-dup-warn-on-low-limit

test/osd/osd-dup.sh: warn on low open file limit

Reviewed-by: Sage Weil <sage@redhat.com>

test: rbd master/slave notify test should test active features

Fixes: http://tracker.ceph.com/issues/19692
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites: client-upgrade/jewel-client-x should test multiple features

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14631 from idryomov/wip-krbd-data-pool-fix

qa: krbd_data_pool.sh: account for rbd_info metadata object

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14385 from smithfarm/wip-release-script-strip-periods

tools: ceph-release-notes: strip trailing punctuation

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

test/osd/osd-dup.sh: warn on low open file limit

This test fails badly when open file limit is low. Increasing to around
1536 seems to be doing the trick, so warn the user with appropriate
message and try to proceed anyway.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #14599 from smithfarm/wip-crushtool-mappings

doc: mention --show-mappings in crushtool manpage

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #14483 from wjwithagen/wip-wjw-freebsd-ceph-disk-list

src/ceph-disk/ceph_disk/main.py: Make 'ceph-disk list' work on FreeBSD

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/rgw: rewrite test_multi.py in terms of rgw_multi

logging changes inspired by Abhishek Lekshmanan <abhishek@suse.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add tests in rgw_multi/tests.py based on multisite.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add rgw_multi/multisite.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14614 from wjwithagen/wip-wjw-cephtool-test-mon.sh

qa/workunits/cephtool/test.sh: Be more liberal in testing health-output.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14106 from theanalyst/release/12.0.1

doc: add 12.0.1 release notes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/objecstore/bluestore-comp: test bluestore with compression on

Enable aggressive compression.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/objectstore/bluestore: drop old option

This test code has been removed.

Signed-off-by: Sage Weil <sage@redhat.com>

qa: krbd_data_pool.sh: account for rbd_info metadata object

... added in commit a16beba78435 ("librbd: image create validates that
pool supports overwrites").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/rgw: add skeleton rgw_multi module

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: update and fixes for test-rgw-multisite.sh

the script was incomplete and unused, but it seems useful in itself
to bring up a simple multisite cluster without having to go through
test_multi.py. it's also a good test for functions in the other
test-rgw-*.sh scripts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: test_multi.py uses cluster_id for calls

this makes the underlying scripts more flexible, because they don't
depend on having a cluster name in the zgX-cY format

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14619 from cbodley/wip-rgw-encryption-build-fix

rgw: remove extra RGWMPObj in rgw_multi.h

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

rgw: remove extra RGWMPObj in rgw_multi.h

two separate PRs had done refactoring around RGWMPObj, and it ended up
in two different places. remove the one in rgw_multi.h, because
rgw_rados.h now depends on its definition

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14418 from songbaisen/m20

mon: fix synchronise pgmap with others

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14436 from dachary/wip-19513-ceph-disk-device-class

ceph-disk,osd: add support for crush device classes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14464 from ceph/wip-systemd

qa/tasks: use sudo to check ceph health for systemd test

Merge pull request #14615 from dillaman/wip-19636

librbd: corrected resize RPC message backwards compatibility

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14381 from joscollin/wip-cleanup-osdmap-rangefor

osd: Cleanup-Updated OSDMap.cc with C++11 style range-for loops

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14431 from liewegas/wip-mgrclient-leak

mgr/MgrClient: fix reconnect event leak

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14507 from tchaikov/wip-19590

mgr: always free allocated MgrPyModule

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

osd: pg-remap -> pg-upmap

'remap' is to non-specific a name.  In particular, it
sounds like it is related to the 'remapped' PG state
but in reality it is not related.

'upmap' or 'pg-upmap' is more specific: it maps a pgid
to the 'up' set value (or item)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13597 from cbodley/wip-s3tests-crypto

qa/rgw: add configuration for server-side encryption tests

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #13483 from ceph/wip-rgw-encryption-doc

doc: rgw server-side encryption and barbican

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #11049 from aclamk/wip-rgw-s3-crypto-sse-kms-plugin

rgw: s3 server-side encryption (SSE-C, SSE-KMS)

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14415 from smithfarm/wip-19556

tests: Thrasher: handle "OSD has the store locked" gracefully

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14456 from tchaikov/wip-rocksdb-freebsd-build

rocksdb: sync with upstream

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #14609 from linuxbox2/wip-rgw-file-wattrs

rgw_file:  pre-compute unix attrs in write_finish()

Merge pull request #14488 from dzafman/wip-15789

Give requested scrub work a higher priority

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: corrected resize RPC message backwards compatibility

Commit d1f2c557 incorrectly changed the order of variables within
the payload. This resulted in breaking the resize RPC message
with older versions of Ceph.

Fixes: http://tracker.ceph.com/issues/19636
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14600 from tchaikov/wip-19390

debian: package ceph.logroate properly

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

rgw_file:  pre-compute unix attrs in write_finish()

New serialized Unix attrs need to reflect the change being made,
and should be reverted if the change fails.

Fixes: http://tracker.ceph.com/issues/19653
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #14396 from jcsp/wip-19550

qa: re-enable ENOSPC tests for kclient

Merge pull request #14589 from jcsp/wip-19640

client: refine fsync/close writeback error handling

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #14272 from jcsp/wip-vstart-fixup

qa: fix test_standby_for_invalid_fscid with vstart_runner

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14196 from jcsp/wip-cephfs-relnotes

PendingReleaseNotes: recent cephfs changes

Merge pull request #14105 from jcsp/wip-pretty-tell

mds: pretty json from `tell` commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14104 from jcsp/wip-18509

mds: include advisory `path` field in damage

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #14164 from jcsp/wip-16842-mitigation

Mitigation for #16842, validate sessions after load

Reviewed-by: Yan, Zheng <zyan@redhat.com>

client: refine fsync/close writeback error handling

Previously, errors stuck indelibly to the inode, which
meant that a close call would see an error even if the
user already dutifully fsync()'d and handled it.

We should emit each error only once per file handle.

Signed-off-by: John Spray <john.spray@redhat.com>