CMakeLists.txt: don't do crypto/isa-l if not Intel

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #14720 from liewegas/wip-19591

mgr: fix session leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add --valgrind_args

To do leak checking, you can do

 vstart.sh ... \
--valgrind_mgr memcheck \
--valgrind_args '--leak-check=full --show-reachable=yes'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrSession: allow ref tracing, and fix ref leak

The first RefCountedObj ctor arg is a cct, not a ref count.

Fixes: http://tracker.ceph.com/issues/19591
Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: add --valgrind_mgr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12272 from kylinstorage/wip-writesame

osd: use append(bufferlist &) to avoid unnecessary copy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11684 from runsisi/wip-fix-asok-mode

common/admin_socket: add config for admin socket permission bits

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13019 from wonzhq/wip-long-io

osd: support dumping long ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14358 from majianpeng/osd-remove-modsubop

osd/ReplicatedBackend: remove MOSDSubOp cruft from repop_applied

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14268 from hrchu/swift/java

doc: rgw: Rewrite Java swift examples

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10384 from wjwithagen/wip-wjw-reduce-noise

common/ceph_context.cc: Use CEPH_DEV to reduce logfile noise

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14693 from mogeb/init-ceph-libs

init-ceph: add ceph libraries path to environment

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12898 from XinzeChi/wip-interval-return

common/interval_set: return int64_t for size()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14232 from jcsp/wip-19412

mgr: fix python module teardown & add tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14584 from tchaikov/wip-19631

qa/suites: Revert "qa/suites: add mon-reweight-min-pgs-per-osd = 4"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14504 from tchaikov/wip-another-mgr-command

mon,mgr: trim osdmap without the help of pgmap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14055 from hrchu/radosgw-admin-api-user-caps

doc: correct and improve add user capability section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13956 from hrchu/rgw-admin-doc

Indicate how to add multiple admin capbabilies

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14372 from hrchu/rgwadmin-admin-doc-usage

doc: rgw: correct get usage parameter default value

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14547 from smithfarm/wip-create-pool

rgw: cleanup: fix variable name in RGWRados::create_pool() declaration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14697 from ceph/wip-rgw-upgrade-s3tests-encryption

qa/rgw: don't scan radosgw logs for encryption keys on jewel upgrade test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14424 from dachary/wip-jerasure-sync

erasure-code: sync jerasure/gf-complete submodules

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14558 from liewegas/wip-18293

osd: fix occasional MOSDMap leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14633 from SUSE/wip-yasm-wrapper-strip-E-flag

yasm-wrapper: strip -E (stops ccache trashing source files)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14640 from ovh/bp-fiemap-burden

osd, os: reduce fiemap burden

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14644 from liewegas/wip-rados-df-json

rados: out json 'df' values as numbers, not strings

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #14645 from liewegas/wip-mgr-mon

mon/MgrMonitor: only propose if we updated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14648 from gregsfortytwo/wip-17916-master

osd: pglog: with config, don't assert in the presence of stale diverg…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13978 from trociny/wip-18787

rbd-mirror A/A: proxy InstanceReplayer APIs via InstanceWatcher RPC

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14647 from dzafman/wip-19698

Fix full testing in cephtool/test.sh when used by rados suite
Replace using sleep with new wait_for_health() bash function

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cephtool.yaml: Add "failsafe" osd error messages to whitelist

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/workunits/ceph-helpers.sh: Improve test_mon_pg() by starting healthy

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa/workunits/ceph-helpers.sh: introduce (and use) wait_for_health

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Fix incorrect health output and test code to match

Signed-off-by: David Zafman <dzafman@redhat.com>

/workunits/cephtool/test.sh: Improve testing of health-output.

Instead of 9ff401e65b1a66a320953aeef8ab65b6f1fb6f10 use regex to handle:
       HEALTH_ERR 2 pgs stuck unclean; Full ratio(s) out of order

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix test_admin_heap_profiler() and enable it

Signed-off-by: David Zafman <dzafman@redhat.com>

cephtool/test.sh: Get socket for whichever environment this test is running

Need sudo for qa suite runs

Fixes: http://tracker.ceph.com/issues/19698
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #14689 from liewegas/wip-jewel-x

qa/suites/upgrade/jewel-x: add mgr.x role

rados: out json 'df' values as numbers, not strings

Fixes: http://tracker.ceph.com/issues/15546
Signed-off-by: Sage Weil <sage@redhat.com

Merge pull request #14423 from tchaikov/wip-cmake-crc32-cleanup

cmake: should not compile crc32c_ppc.c on intel arch.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14434 from ifed01/wip-bluestore-params

os/bluestore: make bluestore_max_blob_size parameter hdd/ssd case dependant

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14448 from majianpeng/bluestore-kv-queued-lat

os/bluestore: Record l_bluestore_state_kv_queued_lat for sync_submit_…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14549 from tchaikov/wip-kill-clang-warnings

misc: kill clang warnings

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14557 from liewegas/wip-bluefs-balance-interval

os/bluestore: do not balance bluefs on every kv_sync_thread iteration

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

init-ceph: add ceph libraries path to environment

These libraries are set in vstart.sh, but not in init-ceph. When
init-ceph is not invoked through vstart.sh, library paths are missing.

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

qa/rgw: don't scan radosgw logs for encryption keys on jewel upgrade test

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/suites/upgrade/jewel-x/point-to-point: no xenial

Avoid conflict with upstream repo having a newer version
than what we're installing.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/jewel-x/stress-split: not pg import/export tests during upgrade

These aren't meant to be run on a mixed-version cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11796 from linuxbox2/wip-bucket-size-ck

rgw: add bucket size limit check to radosgw-admin

Merge pull request #14690 from ceph/luminous

build/ops: add 12.0.1 release tag in master

qa/suites/upgrade/jewel-x: add mgr.x role

Signed-off-by: Sage Weil <sage@redhat.com>

ReplicatedPG: use new fiemap interface

Reduce memory allocator load during sparse reads and recovery by not
encoding sparse file extents from map into bufferlist, then back again
into map.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

rgw: add bucket size limit check to radosgw-admin

The change adds a new list of all buckets x all users, with
fields for bucket name, tenant name, current num_objects,
current num_shards, current objects per shard, and the
corresponding fill_status--the latter consisting of 'OK',
'WARN <n>%', or 'OVER <n>%.'

The warning check is relative to two new tunables.  The threshold
max objects per shard is set as rgw_bucket_safe_max_objects_per_shard,
which defaults to 100K.  The value rgw_bucket_warning_threshold is
a percent of the current safe max at which to warn (defaults to
90% of full).

From review:

* fix indentation (rgw_admin)
* if user a user_id is provided, check only buckets for that user
* update shard warn pct to be pct-of-fill (not 100 - pct-of-fill)
* print only buckets near or over per-shard limit, if --warnings-only
* s/bucket limitcheck/bucket limit check */
* sanity shard limit should be 90, not 10 (because that changed)
* fixes for memleaks and other points found by cbodley

Fixes: http://tracker.ceph.com/issues/17925
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

ObjectStore, *Store: extend fiemap interface

This extends fiemap interface in objectstore and all data stores so
it's possible to pass in the map used later by caller, instead of
encoding internal map into bufferlist, then parsing it back into the
same type of data container, wasting memory and CPU time in the process.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

common/interval_set.h: add move_into() and new ctor

move_into() will be used to move data from internal (private)
std::map into other, external std::map, without exposing internal one
to public. New ctor improves moving data from external map to internal
one.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #14447 from ukernel/wip-19450

mds: drop partial entry and adjust write_pos when opening PurgeQueue

Reviewed-by: John Spray <john.spray@redhat.com>

qa: s/REQUIRE_MGRS/MGRS_REQUIRED/ for consistency

Signed-off-by: John Spray <john.spray@redhat.com>

qa: additions to mgr.test_failover

Reproducers for recent fixes:
http://tracker.ceph.com/issues/19407
http://tracker.ceph.com/issues/19258

Signed-off-by: John Spray <john.spray@redhat.com>

qa: fix vstart_runner --create for mgr tests

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix waiting for module shutdown() before destroying

Fixes: http://tracker.ceph.com/issues/19412
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix MgrPyModule destruction

Fixes: http://tracker.ceph.com/issues/19412
Signed-off-by: John Spray <john.spray@redhat.com>

rbd-mirror A/A: proxy InstanceReplayer APIs via InstanceWatcher RPC

Fixes: http://tracker.ceph.com/issues/18787
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test/librados_test_stub: mock wrapper for aio_notify

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: possible lockdep false positive

(when InstanceWatcher::remove_instance is called)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: race between notify context completion and Notifier destroy

The assert "m_pending_aio_notifies == 0" could fail in the destructor.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: don't use deprecated syntax

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: remove recently added public peer image types

They were supposed to be used to pass a list of peer images.
It has turned out unnecessary.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14638 from dillaman/wip-19692

test: rbd master/slave notify test should test active features

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #14197 from jcsp/wip-19395-counter

mds: use same inode count in health check as in trim

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14649 from hikycool/master

doc: update packages mentioned by build-doc and related doc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14650 from optimistyzy/420_1

NVMEDevice: remove unnessary dpdk header file

Reviewed-by: Haomai Wang <haomai@xsky.com>

qa/suites: Revert "qa/suites: add mon-reweight-min-pgs-per-osd = 4"

This reverts commit ef7583d7178061b47c737d9138dae81685be720e.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: be able to store options with service type

so we are able to change options for services other than mon while
thrashing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

NVMEDevice: remove unnessary dpdk header file

Signed-off-by: optimistyzy <optimistyzy@gmail.com>

doc: correct the package name in ceph/admin/build-doc

In document, it is called 'libxml2-devel', one of the dependent
package of CentOS/RHEL7. But the 'build-doc' check 'libxml-devel'.
And it id also called 'libxml2-devel' in software of yum repo .
So correct the checked package name in ceph/admin/build-doc.

Signed-off-by: Yu Shengzuo <yu.shengzuo@99cloud.net>

doc: update dependent packages links and remove a 'important' tip

Some links in document has been deprecated, such as:
http://rpmfind.net/linux/centos/7.0.1406/, So update it;
The 'fc21, fc24, fc26(newest)' rpm 'ditaa' can be used in
CentOS/RHEL7 to build doc. So remove the '...important' tip.

Signed-off-by: Yu Shengzuo <yu.shengzuo@99cloud.net>

Merge pull request #14608 from tchaikov/wip-19594

qa/tasks: assert on pg status with a timeout

Reviewed-by: Sage Weil <sage@redhat.com>

osd: support dumping historic slow ops

ceph daemon osd.x dump_historic_slow_ops

Signed-off-by: Zhiqiang Wang <zhiqiang@xsky.com>

common: support tracking slow op

Add slow op in op tracker history.

Signed-off-by: Zhiqiang Wang <zhiqiang@xsky.com>

mon/OSDMonitor: use inc.modified as the ctime for MOSDPGCreate

so the last_scrub_time of the newly created pg is consistent

Fixes: http://tracker.ceph.com/issues/19594
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks: update tests with helper to wait for pg-stats

and remove unused helpers

Fixes: http://tracker.ceph.com/issues/19594
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: add a "wait_for_pg_stats()" decorator

and accompany it with two helpers to access the pg stats in a more
natural way

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: pglog: with config, don't assert in the presence of stale divergent_priors

Fixes: http://tracker.ceph.com/issues/17916
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #14496 from jdurgin/wip-ec-overwrites-cleanup

osd: make ec overwrites ready to use

Reviewed-by: Sage Weil <sage@redhat.com>

qa: use bluestore for (k)rbd ec overwrites tests

Only bluestore supports ec overwrites

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: use bluestore for cephfs cephtool tests

This lets the tests enable ec overwrites

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: use 4k stripe_width again for test_rados_tool.sh

The stripe_unit change altered the default stripe_width to depend on k.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: move ec overwrites tests to a bluestore-only subsuite

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc: update ec docs for overwrite support

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

PendingReleaseNotes: note about ec overwrites

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd,OSDMonitor: try to protect against ec overwrites with filestore

This isn't perfect, but it's better than nothing. Prevent enabling the
allow_ec_overwrites flag if any of a sample of pgs in the pool map to
osds using filestore. This mainly protects filestore-only clusters
from enabling it.

If a filestore osd is started later, warn in the cluster log when it
gets a pg with ec overwrites enabled.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

test/osd/osd-scrub-repair.sh: add ec overwrites test cases

Move pool and profile creation into a single function, and
add a 'allow_overwrites' parameter for it so each ec test
can be paramterized by it.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/rados: test sparse reads with ec overwrites

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd, OSDMonitor, qa: mark ec overwrites non-experimental

Keep the pool flag around so we can distinguish between a pool that
should maintain hashes for each chunk, and a missing one is a bug, vs
an overwrites pool where we rely on bluestore checksums for detecting
corruption.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

OSDMonitor: get stripe_width via stripe_unit in ec profile

With bluestore, making the smallest write match min_alloc_size avoids
write amplification. With EC pools this is the stripe unit, or
stripe_width / num_data_chunks. Rather than requiring people to divide
by k to get the smallest ec write, allow it to be specified directly
via stripe_unit. Store it in the ec profile so changing a monitor
config option isn't necessary to set it.

This is particularly important for ec overwrites since they allow random i/o
which should match bluestore's checksum granularity (aka min_alloc_size).

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14256 from dillaman/wip-19297

osd: new op for calculating an extent checksum

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14414 from Adirl/is_ready

msg/async: Postpone bind if network stack is not ready

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14486 from dachary/wip-multiweight

crush: weight_set and id remapping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14556 from liewegas/wip-pgupmap

osd: pg-remap -> pg-upmap

Reviewed-by: David Zafman <dzafman@redhat.com>

mon/MgrMonitor: only propose if we updated

Otherwise we generate a new map on every beacon, even
though there is no change.

Signed-off-by: Sage Weil <sage@redhat.com>