test: use 7130 for crush-classes.sh

7127 is already used by mon/misc.sh

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #14707 from wjwithagen/wip-wjw-freebsd-kqueue-logging

msg: Increase loglevels on some messages

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9853 from dx9/wip-alpine-dev-support

alpine: add alpine linux dev support

Merge pull request #14189 from tchaikov/wip-mgr-py-stderr

mgr: redirect python stdout,stderr to ceph log

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14559 from liewegas/wip-pg-map

mon: move 'pg map' to OSDMonitor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14583 from joscollin/wip-cleanup-osdmap-sections

osd: renamed the new vector name in OSDMap::build_simple_crush_map_from_conf

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14721 from dmick/wip-crypto-arm

CMakeLists.txt: don't do crypto/isa-l if not Intel

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14501 from mdw-at-linuxbox/wip-rgw-rvk

rgw: swift: disable revocation thread if sleep == 0

CMakeLists.txt: don't do crypto/isa-l if not Intel

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #14716 from tchaikov/wip-mgr-pystring-leak

mgr: release allocated PyString

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14720 from liewegas/wip-19591

mgr: fix session leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add --valgrind_args

To do leak checking, you can do

 vstart.sh ... \
--valgrind_mgr memcheck \
--valgrind_args '--leak-check=full --show-reachable=yes'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrSession: allow ref tracing, and fix ref leak

The first RefCountedObj ctor arg is a cct, not a ref count.

Fixes: http://tracker.ceph.com/issues/19591
Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: add --valgrind_mgr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12272 from kylinstorage/wip-writesame

osd: use append(bufferlist &) to avoid unnecessary copy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11684 from runsisi/wip-fix-asok-mode

common/admin_socket: add config for admin socket permission bits

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13019 from wonzhq/wip-long-io

osd: support dumping long ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14358 from majianpeng/osd-remove-modsubop

osd/ReplicatedBackend: remove MOSDSubOp cruft from repop_applied

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14268 from hrchu/swift/java

doc: rgw: Rewrite Java swift examples

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10384 from wjwithagen/wip-wjw-reduce-noise

common/ceph_context.cc: Use CEPH_DEV to reduce logfile noise

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14693 from mogeb/init-ceph-libs

init-ceph: add ceph libraries path to environment

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12898 from XinzeChi/wip-interval-return

common/interval_set: return int64_t for size()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14232 from jcsp/wip-19412

mgr: fix python module teardown & add tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: cleanup: use range-based loop

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: use 'pg map' for get_pg_{primary,replica}

Pulling this out of the 'pg dump' heap is inefficient.
Also, pg dump data comes from the mgr and may be stale.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14584 from tchaikov/wip-19631

qa/suites: Revert "qa/suites: add mon-reweight-min-pgs-per-osd = 4"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14504 from tchaikov/wip-another-mgr-command

mon,mgr: trim osdmap without the help of pgmap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14055 from hrchu/radosgw-admin-api-user-caps

doc: correct and improve add user capability section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13956 from hrchu/rgw-admin-doc

Indicate how to add multiple admin capbabilies

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14372 from hrchu/rgwadmin-admin-doc-usage

doc: rgw: correct get usage parameter default value

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr: release allocated PyString

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14547 from smithfarm/wip-create-pool

rgw: cleanup: fix variable name in RGWRados::create_pool() declaration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14697 from ceph/wip-rgw-upgrade-s3tests-encryption

qa/rgw: don't scan radosgw logs for encryption keys on jewel upgrade test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14424 from dachary/wip-jerasure-sync

erasure-code: sync jerasure/gf-complete submodules

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14558 from liewegas/wip-18293

osd: fix occasional MOSDMap leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14633 from SUSE/wip-yasm-wrapper-strip-E-flag

yasm-wrapper: strip -E (stops ccache trashing source files)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14640 from ovh/bp-fiemap-burden

osd, os: reduce fiemap burden

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14644 from liewegas/wip-rados-df-json

rados: out json 'df' values as numbers, not strings

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #14645 from liewegas/wip-mgr-mon

mon/MgrMonitor: only propose if we updated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14648 from gregsfortytwo/wip-17916-master

osd: pglog: with config, don't assert in the presence of stale diverg…

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/EventKqueue.cc: set logging from Kqueue higher

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr: move handle_pyerror() from MgrPyModules to MgrPyModule

so we can reuse it in a more sane way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/MgrPyModule::handle_command: order stringstreams to match intent

ds is intended for command output (like stdout), to be shown/saved
when there's no error returned; ss is for error status (like stderr)
to further explain the error.  mgr Python tracebacks should go to
the latter.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr/PyModules handle_pyerror: don't add extra newlines to tb

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr/MgrPyModule.cc: get Python traceback in all CallMethod spots

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr: log module name before PyErr_Print()

If an exception is raised in any of the python module functions
invoked by mgr (for example, ImportError if rest_framework isn't
available when serve() in called in the rest module), print the
module name along with the failed method name.

This commit also fixes what looks to me like a missed call to
PyGILState_Release() in the failure case in MgrPyModule::serve().

Signed-off-by: Tim Serong <tserong@suse.com>

mgr: redirect python stdout,stderr to ceph log

PyErr_Print() uses a file object "stderr" for printing messages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13978 from trociny/wip-18787

rbd-mirror A/A: proxy InstanceReplayer APIs via InstanceWatcher RPC

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14647 from dzafman/wip-19698

Fix full testing in cephtool/test.sh when used by rados suite
Replace using sleep with new wait_for_health() bash function

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cephtool.yaml: Add "failsafe" osd error messages to whitelist

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/workunits/ceph-helpers.sh: Improve test_mon_pg() by starting healthy

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa/workunits/ceph-helpers.sh: introduce (and use) wait_for_health

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Fix incorrect health output and test code to match

Signed-off-by: David Zafman <dzafman@redhat.com>

/workunits/cephtool/test.sh: Improve testing of health-output.

Instead of 9ff401e65b1a66a320953aeef8ab65b6f1fb6f10 use regex to handle:
       HEALTH_ERR 2 pgs stuck unclean; Full ratio(s) out of order

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix test_admin_heap_profiler() and enable it

Signed-off-by: David Zafman <dzafman@redhat.com>

cephtool/test.sh: Get socket for whichever environment this test is running

Need sudo for qa suite runs

Fixes: http://tracker.ceph.com/issues/19698
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #14689 from liewegas/wip-jewel-x

qa/suites/upgrade/jewel-x: add mgr.x role

rados: out json 'df' values as numbers, not strings

Fixes: http://tracker.ceph.com/issues/15546
Signed-off-by: Sage Weil <sage@redhat.com

Merge pull request #14423 from tchaikov/wip-cmake-crc32-cleanup

cmake: should not compile crc32c_ppc.c on intel arch.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14434 from ifed01/wip-bluestore-params

os/bluestore: make bluestore_max_blob_size parameter hdd/ssd case dependant

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14448 from majianpeng/bluestore-kv-queued-lat

os/bluestore: Record l_bluestore_state_kv_queued_lat for sync_submit_…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14549 from tchaikov/wip-kill-clang-warnings

misc: kill clang warnings

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14557 from liewegas/wip-bluefs-balance-interval

os/bluestore: do not balance bluefs on every kv_sync_thread iteration

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

init-ceph: add ceph libraries path to environment

These libraries are set in vstart.sh, but not in init-ceph. When
init-ceph is not invoked through vstart.sh, library paths are missing.

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

qa/rgw: don't scan radosgw logs for encryption keys on jewel upgrade test

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/suites/upgrade/jewel-x/point-to-point: no xenial

Avoid conflict with upstream repo having a newer version
than what we're installing.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/jewel-x/stress-split: not pg import/export tests during upgrade

These aren't meant to be run on a mixed-version cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11796 from linuxbox2/wip-bucket-size-ck

rgw: add bucket size limit check to radosgw-admin

Merge pull request #14690 from ceph/luminous

build/ops: add 12.0.1 release tag in master

qa/suites/upgrade/jewel-x: add mgr.x role

Signed-off-by: Sage Weil <sage@redhat.com>

ReplicatedPG: use new fiemap interface

Reduce memory allocator load during sparse reads and recovery by not
encoding sparse file extents from map into bufferlist, then back again
into map.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

rgw: add bucket size limit check to radosgw-admin

The change adds a new list of all buckets x all users, with
fields for bucket name, tenant name, current num_objects,
current num_shards, current objects per shard, and the
corresponding fill_status--the latter consisting of 'OK',
'WARN <n>%', or 'OVER <n>%.'

The warning check is relative to two new tunables.  The threshold
max objects per shard is set as rgw_bucket_safe_max_objects_per_shard,
which defaults to 100K.  The value rgw_bucket_warning_threshold is
a percent of the current safe max at which to warn (defaults to
90% of full).

From review:

* fix indentation (rgw_admin)
* if user a user_id is provided, check only buckets for that user
* update shard warn pct to be pct-of-fill (not 100 - pct-of-fill)
* print only buckets near or over per-shard limit, if --warnings-only
* s/bucket limitcheck/bucket limit check */
* sanity shard limit should be 90, not 10 (because that changed)
* fixes for memleaks and other points found by cbodley

Fixes: http://tracker.ceph.com/issues/17925
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

ObjectStore, *Store: extend fiemap interface

This extends fiemap interface in objectstore and all data stores so
it's possible to pass in the map used later by caller, instead of
encoding internal map into bufferlist, then parsing it back into the
same type of data container, wasting memory and CPU time in the process.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

common/interval_set.h: add move_into() and new ctor

move_into() will be used to move data from internal (private)
std::map into other, external std::map, without exposing internal one
to public. New ctor improves moving data from external map to internal
one.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #14447 from ukernel/wip-19450

mds: drop partial entry and adjust write_pos when opening PurgeQueue

Reviewed-by: John Spray <john.spray@redhat.com>

qa: s/REQUIRE_MGRS/MGRS_REQUIRED/ for consistency

Signed-off-by: John Spray <john.spray@redhat.com>

qa: additions to mgr.test_failover

Reproducers for recent fixes:
http://tracker.ceph.com/issues/19407
http://tracker.ceph.com/issues/19258

Signed-off-by: John Spray <john.spray@redhat.com>

qa: fix vstart_runner --create for mgr tests

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix waiting for module shutdown() before destroying

Fixes: http://tracker.ceph.com/issues/19412
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix MgrPyModule destruction

Fixes: http://tracker.ceph.com/issues/19412
Signed-off-by: John Spray <john.spray@redhat.com>

rbd-mirror A/A: proxy InstanceReplayer APIs via InstanceWatcher RPC

Fixes: http://tracker.ceph.com/issues/18787
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test/librados_test_stub: mock wrapper for aio_notify

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: possible lockdep false positive

(when InstanceWatcher::remove_instance is called)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: race between notify context completion and Notifier destroy

The assert "m_pending_aio_notifies == 0" could fail in the destructor.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: don't use deprecated syntax

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: remove recently added public peer image types

They were supposed to be used to pass a list of peer images.
It has turned out unnecessary.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14638 from dillaman/wip-19692

test: rbd master/slave notify test should test active features

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #14197 from jcsp/wip-19395-counter

mds: use same inode count in health check as in trim

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14649 from hikycool/master

doc: update packages mentioned by build-doc and related doc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14650 from optimistyzy/420_1

NVMEDevice: remove unnessary dpdk header file

Reviewed-by: Haomai Wang <haomai@xsky.com>

qa/suites: Revert "qa/suites: add mon-reweight-min-pgs-per-osd = 4"

This reverts commit ef7583d7178061b47c737d9138dae81685be720e.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: be able to store options with service type

so we are able to change options for services other than mon while
thrashing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

NVMEDevice: remove unnessary dpdk header file

Signed-off-by: optimistyzy <optimistyzy@gmail.com>

doc: correct the package name in ceph/admin/build-doc

In document, it is called 'libxml2-devel', one of the dependent
package of CentOS/RHEL7. But the 'build-doc' check 'libxml-devel'.
And it id also called 'libxml2-devel' in software of yum repo .
So correct the checked package name in ceph/admin/build-doc.

Signed-off-by: Yu Shengzuo <yu.shengzuo@99cloud.net>

doc: update dependent packages links and remove a 'important' tip

Some links in document has been deprecated, such as:
http://rpmfind.net/linux/centos/7.0.1406/, So update it;
The 'fc21, fc24, fc26(newest)' rpm 'ditaa' can be used in
CentOS/RHEL7 to build doc. So remove the '...important' tip.

Signed-off-by: Yu Shengzuo <yu.shengzuo@99cloud.net>

Merge pull request #14608 from tchaikov/wip-19594

qa/tasks: assert on pg status with a timeout

Reviewed-by: Sage Weil <sage@redhat.com>

osd: support dumping historic slow ops

ceph daemon osd.x dump_historic_slow_ops

Signed-off-by: Zhiqiang Wang <zhiqiang@xsky.com>

common: support tracking slow op

Add slow op in op tracker history.

Signed-off-by: Zhiqiang Wang <zhiqiang@xsky.com>

mon/OSDMonitor: use inc.modified as the ctime for MOSDPGCreate

so the last_scrub_time of the newly created pg is consistent

Fixes: http://tracker.ceph.com/issues/19594
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks: update tests with helper to wait for pg-stats

and remove unused helpers

Fixes: http://tracker.ceph.com/issues/19594
Signed-off-by: Kefu Chai <kchai@redhat.com>