qa/suites/upgrade/jewel-x/parallel: upgrade mons before osds

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x/parallel: expand workload matrix

These should run independently against a racing upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x/stress-split-erasure-code: box thrashosds

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x/stress-split: finish client.0 upgrade too

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x: remove kraken references

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x/parallel: fix upgrade to luminous

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suite/upgrade/jewel-x: drop x86 ec test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x: fix upgrade order; fix split grouping

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13343 from liewegas/wip-faster-dispatch

osd: faster dispatch

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #13557 from prallabh/healthcheck

rgw: Correct the return codes for the health check feature

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13685 from tchaikov/wip-19097

mgr/MgrClient: use unique_ptr for MgrClient::session

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13716 from Adirl/move_counter

msg/async/rdma: move active_queue_pairs perf counter dec to polling

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: move active_queue_pairs perf counter dec to polling

removing dead qp's is actually done at polling. if polling is busy then
dead qp will not be removed and active_queue_pair counter is not correct.

issue: 992513

Change-Id: I825e813ce0632fd01f6d29adc87e0e33a2bc13d9
Signed-off-by: DanielBar-On <danielbo@mellanox.com>

rgw: Correct the return codes for the health check feature
Fixes: http://tracker.ceph.com/issues/19025
Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

mon/MonClient: do not assert on auth in build_authorizer()

* there is chance that some connections is still trying to authorize
  itself after the MonClient is shut down. do not assert in this case,
  but it is a sign of bug, or bad shutdown sequence, so print a message to
  dout().
* do not use active_con->get_auth() as an alternative to `this->auth` if
  it is not available. because we promote the authorized conn in
  pending_cons as the active_con, and std::swap(active_conn->auth, this->auth)
  with the monc_lock. so there is no point to return active_con->get_auth(),
  as it's always null.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13703 from linuxbox2/wip-is-linked

rgw_file:  use fh_hook::is_linked() to check residence
verified by hand f23

rgw_file:  use fh_hook::is_linked() to check residence

Previously we assumed that !deleted handles were resident--there
is an observed case where a !deleted handle is !linked.  Since
we currently use safe_link mode, an is_linked() check is
available, and exhaustive.

Fixes: http://tracker.ceph.com/issues/19111
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13671 from liewegas/wip-filestore-omap

os/filestore: debug which omap keys are set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13546 from cbodley/wip-19019

rgw: RGWMetaSyncShardControlCR retries with backoff on all error codes
tested by-had, using fault injection

Merge pull request #13415 from ifed01/wip-fix-storetest

test/store_test: add deferred test case setup to support explicit min…

Reviewed-by: Varada Kari <varada.kari@sandisk.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13476 from songbaisen/s18

osd: small clear up and optimize on _recover_now and should_share_map function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13686 from wjwithagen/wip-wjw-fsx-linux-2

test: Need to exclude the fsx executable also on FreeBSD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13523 from wjwithagen/wip-wjw-clang-mempool

include/mempool.h: fix Clangs complaint about types

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13659 from majianpeng/compiler-warning

common/EventTrace: fix compiler warning.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13667 from jcsp/wip-mgr-misc

mgr: use unique_ptr for MgrStandby::active_mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13669 from jcsp/wip-mgr-get-version

mgr: add a get_version to the python interface

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13692 from Liuchang0812/wip-add-override-in-auth-headers

common/auth: add override in headers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13591 from batrick/mds-static-counters

mds: automate MDS object count tracking

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13581 from jcsp/wip-mdsmonitor-refactor

mon: refactor MDSMonitor command handling

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #13532 from ukernel/wip-18995

ceph_fuse: fix daemonization when pid file is non-empty

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13464 from jcsp/wip-sessionmap-ioerror

mds: fix IO error handling in SessionMap

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #13282 from jcsp/wip-fuse-mount-teardown

tasks/cephfs: tear down on mount() failure

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #13676 from yehudasa/wip-19096

rgw: don't init rgw_obj from rgw_obj_key when it's incorrect to do so
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #13600 from dachary/wip-17821-revert

ceph-disk: Revert "Revert "change get_dmcrypt_key test to support different cluster name""

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/auth: add override in headers

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

src/test/librbd/CMakeLists.txt: Need to exclude the fsx executable also on FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #13682 from tahia-khan/wip-doc-architecture-rados-hyperlink

doc: fixes a broken hyperlink to RADOS paper in architecture

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/MgrClient: use unique_ptr for MgrClient::session

* use unique_ptr<MgrSessionState> for MgrClient::session, so we don't leak it.
* also reset it in MgrClient::shutdown(), so the connection held by session can
  be released properly, and hence the MonClient will not be asked for
  an AuthAuthorizer after being shut down.

Fixes: http://tracker.ceph.com/issues/19097
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fixes a broken hyperlink to RADOS paper in architecture

Signed-off-by: Tahia Khan <tahia.khan@gmail.com>

Merge pull request #13138 from liewegas/wip-vstart

vstart: clean up usage a bit

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13663 from tchaikov/wip-ceph-git-for-workunit

qa/tasks/workunit: use ceph.git as an alternative of ceph-ci.git for cloning workunit

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13650 from trociny/wip-entity_addr-parse

msg: end parameter in entity_addr_t::parse is optional

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13630 from wjwithagen/wip-wjw-fsx-linux

test/librbd/CMakeLists.txt: ceph_test_librbd_fsx requires linux includes/libs

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13196 from vshankar/rbd-ut-fix

test: fix rbd unit test cases w/ striping feature

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12645 from guihecheng/wip-rbd-writesame-api

librbd: add writesame API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: RGWMetaSyncShardControlCR retries with backoff on all error codes

RGWBackoffControlCR only treats EBUSY and EAGAIN as 'temporary' error
codes, with all other errors being fatal when exit_on_error is set

to RGWMetaSyncShardControlCR, a 'fatal' error means that no further sync
is possible on that shard until the gateway restarts

this changes RGWMetaSyncShardControlCR to set exit_on_error to false, so
that it will continue to retry with backoff no matter what error code it
gets

Fixes: http://tracker.ceph.com/issues/19019
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13672 from zmc/wip-openstack-volumes

qa/suites/{ceph-ansible,rest}: OpenStack volumes

vstart.sh: set host for mgr.* conf sections

This makes init-ceph work.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: don't init rgw_obj from rgw_obj_key when it's incorrect to do so

Fixes: http://tracker.ceph.com/issues/19096
rgw_obj_key currently deals with the bucket index key, and not
representing a (name, instance, ns) tupple. Need to initialize
it in two steps.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13675 from liewegas/wip-words

doc/release-notes: avoid 'production-ready' in describing kraken

Reviewed-by: Ian Colle <icolle@redhat.com>

Merge pull request #13547 from liewegas/wip-kill-copy-get-classic

osd: remove copy-get-classic

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13574 from liewegas/wip-mon-mapping

mon/OSDMonitor: make mapping job behave if mon_osd_prime_pg_temp = false

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/EventTrace: fix compiler warning.

/mnt/ceph/src/osd/OSD.cc: In member function ‘void
OSD::dequeue_op(PGRef, OpRequestRef, ThreadPool::TPHandle&)’:
/mnt/ceph/src/osd/OSD.cc:9112:42: error: invalid conversion from ‘const
Message*’ to ‘Message*’ [-fpermissive]
   OID_EVENT_TRACE_WITH_MSG(op->get_req(), "DEQUEUE_OP_BEGIN", false);
                                          ^
In file included from /mnt/ceph/src/osd/OSD.h:57:0,
                 from /mnt/ceph/src/osd/OSD.cc:38:
/mnt/ceph/src/common/EventTrace.h:72:15: note:   initializing argument 1
of ‘static void EventTrace::trace_oid_event(Message*, const char*, const
char*, const char*, int, bool)’
   static void trace_oid_event(Message *m, const char *event, const char
*file,
               ^
/mnt/ceph/src/osd/OSD.cc:9158:42: error: invalid conversion from ‘const
Message*’ to ‘Message*’ [-fpermissive]
   OID_EVENT_TRACE_WITH_MSG(op->get_req(), "DEQUEUE_OP_END", true);
                                          ^
In file included from /mnt/ceph/src/osd/OSD.h:57:0,
                 from /mnt/ceph/src/osd/OSD.cc:38:
/mnt/ceph/src/common/EventTrace.h:72:15: note:   initializing argument 1
of ‘static void EventTrace::trace_oid_event(Message*, const char*, const
char*, const char*, int, bool)’
   static void trace_oid_event(Message *m, const char *event, const char
*file,
               ^
src/osd/CMakeFiles/osd.dir/build.make:62: recipe for target
'src/osd/CMakeFiles/osd.dir/OSD.cc.o' failed

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

doc/release-notes: choose terms carefully

Production-ready is in the eye of the beholder.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rest: Openstack volumes

Signed-off-by: Zack Cerza <zack@redhat.com>

qa/suites/ceph-ansible: Openstack volumes

Signed-off-by: Zack Cerza <zack@redhat.com>

os/filestore: debug which omap keys are set

This may help us find http://tracker.ceph.com/issues/19067

Signed-off-by: Sage Weil <sage@redhat.com>

test/CMakeLists.txt: include ceph-mgr

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: fix init-ceph stop on vstart

Do this before the old ceph.conf is deleted!

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: fix init-ceph path in cmake environment

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_common.sh: add mgr support to init-ceph

This is mainly for the benefit of vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

stop.sh: add 'mgr' arg

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: drop -r arg for radosgw, and use RGW=n instead

Simpler!  The start_* vars are all stupid.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: respect CEPH_NUM_RGW

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: start 1 ceph-mgr by default

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: default to -n ; add -N | --not-new for restart case

I never don't do -n; I'm guessing others are in the same boat.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13665 from xiexingguo/wip-fix-ec-op-out

osd/ECBackend: drop duplicated pending_commit field from << operator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13661 from dachary/wip-crush-doc

doc: crush typo in algorithm description

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: add a get_version to the python interface

We had the metadata for all the remote services,
but no quick way for the python module to
ask "what version is the process I'm running in?"

Signed-off-by: John Spray <john.spray@redhat.com>

test/store_test: add deferred test case setup to support explicit min_alloc_size specification.

Fixes: #18857
Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

mgr: use unique_ptr for MgrStandby::active_mgr

It should always have been one of these.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #13509 from yuyuyu101/wip-tx-zerocopy

msg/async/rdma: cleanup

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #13531 from LiumxNL/fix-osd-utilization

mon/OSDMonitor: fix dividing by zero in OSDUtilizationDumper

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13526 from ganeshmaharaj/tp_library_fixes

debian: Add missing tp files in deb packaging

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13553 from yonghengdexin735/wip-zzz-dbev

os/bluestore/BlueStore.cc:remove unuse code in _open_bdev()

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #13554 from songbaisen/s19

mon:  crush straw_calc_version value is 0 or 1 not 0 to 2.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13653 from rzarzynski/wip-bs-bitmap-fast-exhausted-check

os/bluestore: the exhausted check in BitMapZone can be lock-less.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13656 from tchaikov/wip-19015

mon/MonClient: discard stray messages from non-acitve conns

Reviewed-by: Sage Weil <sage@redhat.com>

osd/ECBackend: drop duplicated pending_commit field from << operator

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/tasks/workunit: use ceph.git as an alternative of ceph-ci.git for workunit repo

if we run upgrade test, where, for example, "jewel" is not in
ceph-ci.git repo, we should check ceph.git to clone the workunits.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: accelerate tx/rx buffer ownership lookup

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: cleanup

Signed-off-by: Haomai Wang <haomai@xsky.com>

doc: crush typo in algorithm description

Signed-off-by: Loic Dachary <ldachary@redhat.com>

msg/async/rdma: cleanup Infiniband, remove unused functions

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13603 from Adirl/fix_compile

msg/async/rdma: Fix broken compilation

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13652 from liupan1111/wip-cleanup-rbd-nbd

rbd-nbd: fix a typo "moudle"

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

auth/cephx: print challenges in hex

this matches the CephxServiceHandler::handle_request(), hence would be
easier for debugging.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: discard stray messages from non-acitve conns

Fixes: http://tracker.ceph.com/issues/19015
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13227 from ukernel/wip-multimds-misc

mds: misc multimds fixes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13648 from tchaikov/wip-doc-osd-agent-options

doc: document `osd-agent-{max,low}-ops` options

Reviewed-by: Zhiqiang Wang <zhiqiang@xsky.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

rbd-nbd: fix a typo "moudle"

Signed-off-by: Pan Liu <liupan1111@gmail.com>

doc: document `osd-agent-{max,low}-ops` options

Signed-off-by: Kefu Chai <kchai@redhat.com>

bluestore: the exhausted check in BitMapZone can be lock-less.

Before the patch BitMapZone::is_exhausted() required from its
callers to acquire appropriate lock. However, fulfilling this
condition is not really necessary to use the method correctly
while it can significantly hurt performance.

The change allows BitMapAreaLeaf::child_check_n_lock() to not
acquire the lock while examining zones for being exhausted.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13649 from liewegas/wip-ceph-scrub-debug

qa/tasks/ceph.py: debug which pgs aren't scrubbing

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

msg: end parameter in entity_addr_t::parse is optional

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osd: restructure op_shardedwq

This is difficult to break into pieces, so one big fat commit it is.

A few trivial bits

- include epoch in PGQueueable.
- PGQueuable operator<<
- remove op_wq ref from OSDService; use simple set of queue methods instead

The big stuff:

- Fast dispatch now passes messages directly to the queue based on an
spg_t.  The exception is MOSDOp's from legacy clients.  We add a
waiting_for_map mechanism on the front-side that is similar to but simpler
than the previous one so that we can map those legacy requests to an
accurate spg_t.
- The dequeue path now has a waiting_for_pg mechanism.  It also uses a
much simpler set of data structures that should make it much faster than
the previous incarnation.
- Shutdown works a bit differently; we drain the queue instead of trying
to remove work for individual PGs.  This lets us remove the dequeue_pg
machinery.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Dispatcher: pass const Message* to ms_can_fast_dispatch

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph.py: debug which pgs aren't scrubbing

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13478 from xiaoxichen/fix_osdc_perfcounter

osdc: fix osdc_osd_seesion perf counter.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13626 from wjwithagen/wip-wjw-freebsd-enodata-fixed

do_freebsd.sh: Remove ENODATA requirement

Reviewed-by: Kefu Chai <kchai@redhat.com>