ceph-object-corpus: prune old types from decode tests

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove compat cruft for object_stat_sum_t

v15 is 169cd4d962140fc0fff8cd721976a2a8430c6250 (pre-jewel), but
the decode method still said v14.  Therefose, we leave the compat
version when we encode as 14.

v14 is 0643845665c08b4eb1d0c9d49488183b7bbabb63 (pre-jewel).

Signed-off-by: Sage Weil <sage@redhat.com>

include/encoding: fix compat version error messages

These were just...wrong.  Correctly report the encoding version we
got and the oldest we can decode.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove pg_stat_t compat cruft

v22 is 0643845665c08b4eb1d0c9d49488183b7bbabb63 (pre-jewel).

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove pg_info_t compat cruft

v32 is d5ca7e9d1e75418290d439d9801e6bc15e2d917a, pre-infernalis.

This is rewritten by PG::upgrade() on major upgrades so we don't need to
be able to decode old (very) versions.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove pg_notify_t compat cruft

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove pg_query_t compat cruft

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MMDSBeacon: remove (some) compat cruft

v7 is cdf8a29f81475a000824ce8942f02d9d570ec3bb (pre-kraken).
v6 is bcf4953b455456c81b1f917705d4867cfab17b4a (pre-jewel).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MRoute: remove compat cruft

v3 was 39e06ef8f070e136e54452bdea3f6105cd79bb73 (pre-jewel).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MRecoveryReserve: remove compat cruft

v2 was 9013efd3a3cf92d1ec8e2a39639214792067d0d2 (pre-firefly).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDScrub: remove compat cruft

v2 was 9013efd3a3cf92d1ec8e2a39639214792067d0d2 (pre-bobtail).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDRepScrub: remove compat cruft

v6 was 7d73f41f0d6ae2bf971c72b0b8aca6579e1c0d62 (pre-hammer).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPing: remove compat cruft

v2 was 4e2f0d149d9bb9310dde12c201417d7747c52a85 (pre-argonaut).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPGTrim: remove compat cruft

v2 was 8b46dd1dc009981f2fcae7b0e05155e81841c4a5 (pre-firefly)

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPGScan: remove compat cruft

v2 was 8b46dd1dc009981f2fcae7b0e05155e81841c4a5 (pre-firefly)

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPGRemove: simplify encoding for luminous+

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPGRemove: remove compat cruft

v2 was 8b46dd1dc009981f2fcae7b0e05155e81841c4a5 (pre-firefly)

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPGQuery: simplify encoding for luminous+

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPGQuery: remove compat cruft

v3 was pre-firefly (8b46dd1dc009981f2fcae7b0e05155e81841c4a5).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPGPull: remove compat cruft

v2 was 8b46dd1dc009981f2fcae7b0e05155e81841c4a5 (pre-firefly).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPGPull: remove compat cruft

v2 is 8b46dd1dc009981f2fcae7b0e05155e81841c4a5 (pre-firefly).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPGCreate: remove compat cruft

v3 is 52cb44c4dde337924342bb99f8ca8d1265421077 (pre-hammer).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPGBackfill: drop compat cruft

v3 is 8b46dd1dc009981f2fcae7b0e05155e81841c4a5 (pre-firefly).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDMarkMeDown: drop compat cruft

v2 is 58ace1aac3fccd79b252d8c485c343d75e5175d3 (pre-giant).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDFailure: drop compat cruft

v3 was 4f1792d7694e065e3bcdb48d94d66e936c390daf (bobtail?).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDBoot: drop compat cruft

v6 was bee91548b07c6a28314dddc50fef8b6a2677e774 (pre-hammer).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MMonScrub: drop compat cruft

v2 was 0a19fd4a2686d07cce632d20d91cf419e41dada6 (pre-infernalis).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MMonPaxos: drop compat cruft

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MBackfilLReserve: drop compat cruft

v3 was 8b46dd1dc009981f2fcae7b0e05155e81841c4a5 (pre-firefly).

Signed-off-by: Sage Weil <sage@redhat.com>

message/MForward: drop compat cruft

v3 was 87734747dca113d6a13d1e78da52e6009a9e935a (pre-hammer).

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MMonElection: drop old compat cruft

v5 was 8025fb33add620cb89f795281067c5bddf0467b5 (pre-firefly); v6 was
34b4ad730bfb5a90760705553415582ac01a4312 (pre-kraken).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14716 from tchaikov/wip-mgr-pystring-leak

mgr: release allocated PyString

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14720 from liewegas/wip-19591

mgr: fix session leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add --valgrind_args

To do leak checking, you can do

 vstart.sh ... \
--valgrind_mgr memcheck \
--valgrind_args '--leak-check=full --show-reachable=yes'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrSession: allow ref tracing, and fix ref leak

The first RefCountedObj ctor arg is a cct, not a ref count.

Fixes: http://tracker.ceph.com/issues/19591
Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: add --valgrind_mgr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12272 from kylinstorage/wip-writesame

osd: use append(bufferlist &) to avoid unnecessary copy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11684 from runsisi/wip-fix-asok-mode

common/admin_socket: add config for admin socket permission bits

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13019 from wonzhq/wip-long-io

osd: support dumping long ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14358 from majianpeng/osd-remove-modsubop

osd/ReplicatedBackend: remove MOSDSubOp cruft from repop_applied

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14268 from hrchu/swift/java

doc: rgw: Rewrite Java swift examples

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10384 from wjwithagen/wip-wjw-reduce-noise

common/ceph_context.cc: Use CEPH_DEV to reduce logfile noise

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14693 from mogeb/init-ceph-libs

init-ceph: add ceph libraries path to environment

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12898 from XinzeChi/wip-interval-return

common/interval_set: return int64_t for size()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14232 from jcsp/wip-19412

mgr: fix python module teardown & add tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14584 from tchaikov/wip-19631

qa/suites: Revert "qa/suites: add mon-reweight-min-pgs-per-osd = 4"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14504 from tchaikov/wip-another-mgr-command

mon,mgr: trim osdmap without the help of pgmap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14055 from hrchu/radosgw-admin-api-user-caps

doc: correct and improve add user capability section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13956 from hrchu/rgw-admin-doc

Indicate how to add multiple admin capbabilies

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14372 from hrchu/rgwadmin-admin-doc-usage

doc: rgw: correct get usage parameter default value

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr: release allocated PyString

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14547 from smithfarm/wip-create-pool

rgw: cleanup: fix variable name in RGWRados::create_pool() declaration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14697 from ceph/wip-rgw-upgrade-s3tests-encryption

qa/rgw: don't scan radosgw logs for encryption keys on jewel upgrade test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14424 from dachary/wip-jerasure-sync

erasure-code: sync jerasure/gf-complete submodules

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14558 from liewegas/wip-18293

osd: fix occasional MOSDMap leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14633 from SUSE/wip-yasm-wrapper-strip-E-flag

yasm-wrapper: strip -E (stops ccache trashing source files)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14640 from ovh/bp-fiemap-burden

osd, os: reduce fiemap burden

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14644 from liewegas/wip-rados-df-json

rados: out json 'df' values as numbers, not strings

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #14645 from liewegas/wip-mgr-mon

mon/MgrMonitor: only propose if we updated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14648 from gregsfortytwo/wip-17916-master

osd: pglog: with config, don't assert in the presence of stale diverg…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13978 from trociny/wip-18787

rbd-mirror A/A: proxy InstanceReplayer APIs via InstanceWatcher RPC

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14647 from dzafman/wip-19698

Fix full testing in cephtool/test.sh when used by rados suite
Replace using sleep with new wait_for_health() bash function

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cephtool.yaml: Add "failsafe" osd error messages to whitelist

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/workunits/ceph-helpers.sh: Improve test_mon_pg() by starting healthy

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa/workunits/ceph-helpers.sh: introduce (and use) wait_for_health

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Fix incorrect health output and test code to match

Signed-off-by: David Zafman <dzafman@redhat.com>

/workunits/cephtool/test.sh: Improve testing of health-output.

Instead of 9ff401e65b1a66a320953aeef8ab65b6f1fb6f10 use regex to handle:
       HEALTH_ERR 2 pgs stuck unclean; Full ratio(s) out of order

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix test_admin_heap_profiler() and enable it

Signed-off-by: David Zafman <dzafman@redhat.com>

cephtool/test.sh: Get socket for whichever environment this test is running

Need sudo for qa suite runs

Fixes: http://tracker.ceph.com/issues/19698
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #14689 from liewegas/wip-jewel-x

qa/suites/upgrade/jewel-x: add mgr.x role

rados: out json 'df' values as numbers, not strings

Fixes: http://tracker.ceph.com/issues/15546
Signed-off-by: Sage Weil <sage@redhat.com

Merge pull request #14423 from tchaikov/wip-cmake-crc32-cleanup

cmake: should not compile crc32c_ppc.c on intel arch.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14434 from ifed01/wip-bluestore-params

os/bluestore: make bluestore_max_blob_size parameter hdd/ssd case dependant

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14448 from majianpeng/bluestore-kv-queued-lat

os/bluestore: Record l_bluestore_state_kv_queued_lat for sync_submit_…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14549 from tchaikov/wip-kill-clang-warnings

misc: kill clang warnings

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14557 from liewegas/wip-bluefs-balance-interval

os/bluestore: do not balance bluefs on every kv_sync_thread iteration

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

init-ceph: add ceph libraries path to environment

These libraries are set in vstart.sh, but not in init-ceph. When
init-ceph is not invoked through vstart.sh, library paths are missing.

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

qa/rgw: don't scan radosgw logs for encryption keys on jewel upgrade test

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/suites/upgrade/jewel-x/point-to-point: no xenial

Avoid conflict with upstream repo having a newer version
than what we're installing.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/jewel-x/stress-split: not pg import/export tests during upgrade

These aren't meant to be run on a mixed-version cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11796 from linuxbox2/wip-bucket-size-ck

rgw: add bucket size limit check to radosgw-admin

Merge pull request #14690 from ceph/luminous

build/ops: add 12.0.1 release tag in master

qa/suites/upgrade/jewel-x: add mgr.x role

Signed-off-by: Sage Weil <sage@redhat.com>

ReplicatedPG: use new fiemap interface

Reduce memory allocator load during sparse reads and recovery by not
encoding sparse file extents from map into bufferlist, then back again
into map.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

rgw: add bucket size limit check to radosgw-admin

The change adds a new list of all buckets x all users, with
fields for bucket name, tenant name, current num_objects,
current num_shards, current objects per shard, and the
corresponding fill_status--the latter consisting of 'OK',
'WARN <n>%', or 'OVER <n>%.'

The warning check is relative to two new tunables.  The threshold
max objects per shard is set as rgw_bucket_safe_max_objects_per_shard,
which defaults to 100K.  The value rgw_bucket_warning_threshold is
a percent of the current safe max at which to warn (defaults to
90% of full).

From review:

* fix indentation (rgw_admin)
* if user a user_id is provided, check only buckets for that user
* update shard warn pct to be pct-of-fill (not 100 - pct-of-fill)
* print only buckets near or over per-shard limit, if --warnings-only
* s/bucket limitcheck/bucket limit check */
* sanity shard limit should be 90, not 10 (because that changed)
* fixes for memleaks and other points found by cbodley

Fixes: http://tracker.ceph.com/issues/17925
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

ObjectStore, *Store: extend fiemap interface

This extends fiemap interface in objectstore and all data stores so
it's possible to pass in the map used later by caller, instead of
encoding internal map into bufferlist, then parsing it back into the
same type of data container, wasting memory and CPU time in the process.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

common/interval_set.h: add move_into() and new ctor

move_into() will be used to move data from internal (private)
std::map into other, external std::map, without exposing internal one
to public. New ctor improves moving data from external map to internal
one.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #14447 from ukernel/wip-19450

mds: drop partial entry and adjust write_pos when opening PurgeQueue

Reviewed-by: John Spray <john.spray@redhat.com>

qa: s/REQUIRE_MGRS/MGRS_REQUIRED/ for consistency

Signed-off-by: John Spray <john.spray@redhat.com>

qa: additions to mgr.test_failover

Reproducers for recent fixes:
http://tracker.ceph.com/issues/19407
http://tracker.ceph.com/issues/19258

Signed-off-by: John Spray <john.spray@redhat.com>

qa: fix vstart_runner --create for mgr tests

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix waiting for module shutdown() before destroying

Fixes: http://tracker.ceph.com/issues/19412
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix MgrPyModule destruction

Fixes: http://tracker.ceph.com/issues/19412
Signed-off-by: John Spray <john.spray@redhat.com>

rbd-mirror A/A: proxy InstanceReplayer APIs via InstanceWatcher RPC

Fixes: http://tracker.ceph.com/issues/18787
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test/librados_test_stub: mock wrapper for aio_notify

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: possible lockdep false positive

(when InstanceWatcher::remove_instance is called)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: race between notify context completion and Notifier destroy

The assert "m_pending_aio_notifies == 0" could fail in the destructor.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: don't use deprecated syntax

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: remove recently added public peer image types

They were supposed to be used to pass a list of peer images.
It has turned out unnecessary.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14638 from dillaman/wip-19692

test: rbd master/slave notify test should test active features

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>