doc: mention teuthology-worker security group

teuthology-openstack now creates a teuthology-worker security group, as well as
teuthology, so we need to delete that one as well when starting over.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #14741 from trociny/wip-19405

test/rbd_mirror: race in TestMockLeaderWatcher.AcquireError

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12792 from david-z/wip-mds-use-tcmalloc

mds: remove boost::pool usage and use tcmalloc directly

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14575 from jcsp/wip-cephfs-suites

qa: Tidy up fs/ suite

Reviewed-by: Yan, Zheng <zyan@redhat.com>

qa: switch off btrfs for multimds and kcephfs

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update links for modified fs subsuite names

Signed-off-by: John Spray <john.spray@redhat.com>

qa/tasks/cephfs: clean up mount point setup

Previously were sometimes trying to maintain a mounted
client across a filesystem destroy/create.

Signed-off-by: John Spray <john.spray@redhat.com>

qa/suites: don't use btrfs for cephfs testing

This change happened a while back, but it got rolled back
when the generic objectstore/ dir had its filestore
entry split out into xfs and btrfs in 208675af.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: remove unused fs/ dir

The objectstore/ dir is now selecting the backing fs.

Signed-off-by: John Spray <john.spray@redhat.com>

qa/suites/fs: more meaningful names for sub suites

The "recovery" sub suite was originally tests for
client/mds recovery in certain failure cases, it has
since grown to include lots of unit testing of
various features using CephFSTestCase.

The "basic" suite is now specifically just running workloads
now that I've moved out the smaller functional tests.

Signed-off-by: John Spray <john.spray@redhat.com>

qa/suites/fs: put simple functional tests together

Most of what's in basic/ is "workload" type testing
(i.e. a simple cluster cluster configuration and then
running a script inside the client), which gets
permuted in various ways.  Move the simpler
functional tests out with the others like themselves.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: don't permute object store for functional tests

These are unit tests for specific CephFS functionality,
it is gratuitous to repeat them with different underlying
RADOS object stores.

We retain coverage of XFS vs. bluestore in the workload tests.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #14018 from jcsp/wip-17939

client: getattr before returning quota/layout xattrs

Reviewed-by: Yan, Zheng <zyan@redhat.com>

mds: remove boost::pool usage and use tcmalloc directly

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

Merge pull request #14729 from joscollin/wip-cleanup-osdc-comparision

osdc: silence warning from `-Wsign-compare`

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14711 from jcsp/wip-19738

mon: don't call propose_pending in prepare_update()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14740 from petermaloney/master

doc: fix link that pointed to a nonexistent file

Reviewed-by: Kefu Chai <kchai@redhat.com>

osdc: warning: comparison between signed and unsigned integer expressions

The following warning appears during make. Fixed the review comments too.
[ 28%] Building CXX object src/mds/CMakeFiles/mds.dir/__/osdc/Journaler.cc.o
./src/osdc/Journaler.cc: In member function ‘void Journaler::_do_flush(unsigned int)’:
./src/osdc/Journaler.cc:643:16: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (newlen < len) {
         ~~~~~~~^~~~~

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #14705 from joscollin/wip-googletest-warning

test: warning: comparison between signed and unsigned integer expressions

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13898 from songbaisen/m5

osd: check fsid is normal before osd mkfs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10893 from yonghengdexin735/wip-zzz-1

rados: check for negative return value of rados_create_with_context() as its comment put

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14442 from liewegas/wip-midkraken-upgrade

mon/OSDMonitor: tolerate upgrade from post-kraken dev cluster

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/rbd_mirror: race in TestMockLeaderWatcher.AcquireError

Due to handle_get_locker is executed in async context, shut_down could
start before handle_get_locker.

Fixes: http://tracker.ceph.com/issues/19405
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

doc: fix link that pointed to a nonexistent file

commit d788bae60dac374dcf5c7a30fca04426cbee660a renamed config.rst to config-fcgi.rst

Reported-by: fusl in #ceph on irc.oftc.net
Signed-off-by: Peter Maloney <peter.maloney@brockmann-consult.de>

Merge pull request #14712 from dillaman/wip-rbd-mirror-watch

rbd-mirror: remove tracking of image names from pool watcher

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14722 from cbodley/wip-19746

radosgw-admin: warn that 'realm rename' does not update other clusters
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14338 from aclamk/fix-fio-ceph-objectstore-lifetime

fio_ceph_objectstore: fixes improper write request data lifetime

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14335 from hrchu/radosgw-admin-api-subuser

doc: rgw: Clean up create subuser parameters

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14707 from wjwithagen/wip-wjw-freebsd-kqueue-logging

msg: Increase loglevels on some messages

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9853 from dx9/wip-alpine-dev-support

alpine: add alpine linux dev support

Merge pull request #14189 from tchaikov/wip-mgr-py-stderr

mgr: redirect python stdout,stderr to ceph log

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14559 from liewegas/wip-pg-map

mon: move 'pg map' to OSDMonitor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14583 from joscollin/wip-cleanup-osdmap-sections

osd: renamed the new vector name in OSDMap::build_simple_crush_map_from_conf

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14721 from dmick/wip-crypto-arm

CMakeLists.txt: don't do crypto/isa-l if not Intel

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14501 from mdw-at-linuxbox/wip-rgw-rvk

rgw: swift: disable revocation thread if sleep == 0

radosgw-admin: warn that 'realm rename' does not update other clusters

Fixes: http://tracker.ceph.com/issues/19746
Signed-off-by: Casey Bodley <cbodley@redhat.com>

CMakeLists.txt: don't do crypto/isa-l if not Intel

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #14716 from tchaikov/wip-mgr-pystring-leak

mgr: release allocated PyString

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14720 from liewegas/wip-19591

mgr: fix session leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add --valgrind_args

To do leak checking, you can do

 vstart.sh ... \
--valgrind_mgr memcheck \
--valgrind_args '--leak-check=full --show-reachable=yes'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrSession: allow ref tracing, and fix ref leak

The first RefCountedObj ctor arg is a cct, not a ref count.

Fixes: http://tracker.ceph.com/issues/19591
Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: add --valgrind_mgr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12272 from kylinstorage/wip-writesame

osd: use append(bufferlist &) to avoid unnecessary copy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11684 from runsisi/wip-fix-asok-mode

common/admin_socket: add config for admin socket permission bits

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13019 from wonzhq/wip-long-io

osd: support dumping long ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14358 from majianpeng/osd-remove-modsubop

osd/ReplicatedBackend: remove MOSDSubOp cruft from repop_applied

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14268 from hrchu/swift/java

doc: rgw: Rewrite Java swift examples

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10384 from wjwithagen/wip-wjw-reduce-noise

common/ceph_context.cc: Use CEPH_DEV to reduce logfile noise

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14693 from mogeb/init-ceph-libs

init-ceph: add ceph libraries path to environment

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12898 from XinzeChi/wip-interval-return

common/interval_set: return int64_t for size()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14232 from jcsp/wip-19412

mgr: fix python module teardown & add tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: cleanup: use range-based loop

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: use 'pg map' for get_pg_{primary,replica}

Pulling this out of the 'pg dump' heap is inefficient.
Also, pg dump data comes from the mgr and may be stale.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14584 from tchaikov/wip-19631

qa/suites: Revert "qa/suites: add mon-reweight-min-pgs-per-osd = 4"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14504 from tchaikov/wip-another-mgr-command

mon,mgr: trim osdmap without the help of pgmap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14055 from hrchu/radosgw-admin-api-user-caps

doc: correct and improve add user capability section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13956 from hrchu/rgw-admin-doc

Indicate how to add multiple admin capbabilies

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14372 from hrchu/rgwadmin-admin-doc-usage

doc: rgw: correct get usage parameter default value

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr: release allocated PyString

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14547 from smithfarm/wip-create-pool

rgw: cleanup: fix variable name in RGWRados::create_pool() declaration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14697 from ceph/wip-rgw-upgrade-s3tests-encryption

qa/rgw: don't scan radosgw logs for encryption keys on jewel upgrade test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14424 from dachary/wip-jerasure-sync

erasure-code: sync jerasure/gf-complete submodules

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14558 from liewegas/wip-18293

osd: fix occasional MOSDMap leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14633 from SUSE/wip-yasm-wrapper-strip-E-flag

yasm-wrapper: strip -E (stops ccache trashing source files)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14640 from ovh/bp-fiemap-burden

osd, os: reduce fiemap burden

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14644 from liewegas/wip-rados-df-json

rados: out json 'df' values as numbers, not strings

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #14645 from liewegas/wip-mgr-mon

mon/MgrMonitor: only propose if we updated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14648 from gregsfortytwo/wip-17916-master

osd: pglog: with config, don't assert in the presence of stale diverg…

Reviewed-by: Sage Weil <sage@redhat.com>

mon: don't call propose_pending in dispatch

This was happening indirectly because the command handling
code was calling tick() as a shortcut to "promote something"
and tick calls propose_pending because.

Fixes: http://tracker.ceph.com/issues/19738
Signed-off-by: John Spray <john.spray@redhat.com>

msg/async/EventKqueue.cc: set logging from Kqueue higher

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr: move handle_pyerror() from MgrPyModules to MgrPyModule

so we can reuse it in a more sane way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/MgrPyModule::handle_command: order stringstreams to match intent

ds is intended for command output (like stdout), to be shown/saved
when there's no error returned; ss is for error status (like stderr)
to further explain the error.  mgr Python tracebacks should go to
the latter.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr/PyModules handle_pyerror: don't add extra newlines to tb

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr/MgrPyModule.cc: get Python traceback in all CallMethod spots

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr: log module name before PyErr_Print()

If an exception is raised in any of the python module functions
invoked by mgr (for example, ImportError if rest_framework isn't
available when serve() in called in the rest module), print the
module name along with the failed method name.

This commit also fixes what looks to me like a missed call to
PyGILState_Release() in the failure case in MgrPyModule::serve().

Signed-off-by: Tim Serong <tserong@suse.com>

mgr: redirect python stdout,stderr to ceph log

PyErr_Print() uses a file object "stderr" for printing messages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: warning: comparison between signed and unsigned integer expressions

The following warning appears during make:
[ 61%] Building CXX object src/test/CMakeFiles/unittest_simple_spin.dir/simple_spin.cc.o
In file included from ./src/test/simple_spin.cc:1:0:
./src/googletest/googletest/include/gtest/gtest.h: In instantiation of ‘testing::AssertionResult testing::internal::CmpHelperEQ(const char*, const char*, const T1&, const T2&) [with T1 = unsigned int; T2 = int]’:
./src/googletest/googletest/include/gtest/gtest.h:1421:23:   required from ‘static testing::AssertionResult testing::internal::EqHelper<lhs_is_null_literal>::Compare(const char*, const char*, const T1&, const T2&) [with T1 = unsigned int; T2 = int; bool lhs_is_null_literal = false]’
./src/test/simple_spin.cc:49:3:   required from here
./src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {
       ~~~~^~~~~~

Signed-off-by: Jos Collin <jcollin@redhat.com>

rbd-mirror: remove image name retrieval from pool watcher

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: eliminate use of image name during bootstrap

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: supply image id of new images to bootstrap state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: specify image id for create and clone operations

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: removed unused pool name from cluster watcher

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: renamed Replayer to PoolReplayer

This is a stepping stone to support multiple peers within a single
pool.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13978 from trociny/wip-18787

rbd-mirror A/A: proxy InstanceReplayer APIs via InstanceWatcher RPC

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14647 from dzafman/wip-19698

Fix full testing in cephtool/test.sh when used by rados suite
Replace using sleep with new wait_for_health() bash function

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cephtool.yaml: Add "failsafe" osd error messages to whitelist

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/workunits/ceph-helpers.sh: Improve test_mon_pg() by starting healthy

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa/workunits/ceph-helpers.sh: introduce (and use) wait_for_health

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Signed-off-by: David Zafman <dzafman@redhat.com>

mon: Fix incorrect health output and test code to match

Signed-off-by: David Zafman <dzafman@redhat.com>

/workunits/cephtool/test.sh: Improve testing of health-output.

Instead of 9ff401e65b1a66a320953aeef8ab65b6f1fb6f10 use regex to handle:
       HEALTH_ERR 2 pgs stuck unclean; Full ratio(s) out of order

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix test_admin_heap_profiler() and enable it

Signed-off-by: David Zafman <dzafman@redhat.com>

cephtool/test.sh: Get socket for whichever environment this test is running

Need sudo for qa suite runs

Fixes: http://tracker.ceph.com/issues/19698
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #14689 from liewegas/wip-jewel-x

qa/suites/upgrade/jewel-x: add mgr.x role

rados: out json 'df' values as numbers, not strings

Fixes: http://tracker.ceph.com/issues/15546
Signed-off-by: Sage Weil <sage@redhat.com

Merge pull request #14423 from tchaikov/wip-cmake-crc32-cleanup

cmake: should not compile crc32c_ppc.c on intel arch.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14434 from ifed01/wip-bluestore-params

os/bluestore: make bluestore_max_blob_size parameter hdd/ssd case dependant

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14448 from majianpeng/bluestore-kv-queued-lat

os/bluestore: Record l_bluestore_state_kv_queued_lat for sync_submit_…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14549 from tchaikov/wip-kill-clang-warnings

misc: kill clang warnings

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14557 from liewegas/wip-bluefs-balance-interval

os/bluestore: do not balance bluefs on every kv_sync_thread iteration

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

init-ceph: add ceph libraries path to environment

These libraries are set in vstart.sh, but not in init-ceph. When
init-ceph is not invoked through vstart.sh, library paths are missing.

Signed-off-by: Mohamad Gebai <mgebai@suse.com>