vstart.sh: fix init-ceph stop on vstart

Do this before the old ceph.conf is deleted!

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: fix init-ceph path in cmake environment

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_common.sh: add mgr support to init-ceph

This is mainly for the benefit of vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

stop.sh: add 'mgr' arg

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: drop -r arg for radosgw, and use RGW=n instead

Simpler!  The start_* vars are all stupid.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: respect CEPH_NUM_RGW

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: start 1 ceph-mgr by default

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: default to -n ; add -N | --not-new for restart case

I never don't do -n; I'm guessing others are in the same boat.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13665 from xiexingguo/wip-fix-ec-op-out

osd/ECBackend: drop duplicated pending_commit field from << operator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13661 from dachary/wip-crush-doc

doc: crush typo in algorithm description

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13509 from yuyuyu101/wip-tx-zerocopy

msg/async/rdma: cleanup

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #13531 from LiumxNL/fix-osd-utilization

mon/OSDMonitor: fix dividing by zero in OSDUtilizationDumper

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13526 from ganeshmaharaj/tp_library_fixes

debian: Add missing tp files in deb packaging

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13553 from yonghengdexin735/wip-zzz-dbev

os/bluestore/BlueStore.cc:remove unuse code in _open_bdev()

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #13554 from songbaisen/s19

mon:  crush straw_calc_version value is 0 or 1 not 0 to 2.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13653 from rzarzynski/wip-bs-bitmap-fast-exhausted-check

os/bluestore: the exhausted check in BitMapZone can be lock-less.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13656 from tchaikov/wip-19015

mon/MonClient: discard stray messages from non-acitve conns

Reviewed-by: Sage Weil <sage@redhat.com>

osd/ECBackend: drop duplicated pending_commit field from << operator

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

msg/async/rdma: accelerate tx/rx buffer ownership lookup

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: cleanup

Signed-off-by: Haomai Wang <haomai@xsky.com>

doc: crush typo in algorithm description

Signed-off-by: Loic Dachary <ldachary@redhat.com>

msg/async/rdma: cleanup Infiniband, remove unused functions

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13603 from Adirl/fix_compile

msg/async/rdma: Fix broken compilation

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13652 from liupan1111/wip-cleanup-rbd-nbd

rbd-nbd: fix a typo "moudle"

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

auth/cephx: print challenges in hex

this matches the CephxServiceHandler::handle_request(), hence would be
easier for debugging.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: discard stray messages from non-acitve conns

Fixes: http://tracker.ceph.com/issues/19015
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13227 from ukernel/wip-multimds-misc

mds: misc multimds fixes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13648 from tchaikov/wip-doc-osd-agent-options

doc: document `osd-agent-{max,low}-ops` options

Reviewed-by: Zhiqiang Wang <zhiqiang@xsky.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

rbd-nbd: fix a typo "moudle"

Signed-off-by: Pan Liu <liupan1111@gmail.com>

doc: document `osd-agent-{max,low}-ops` options

Signed-off-by: Kefu Chai <kchai@redhat.com>

bluestore: the exhausted check in BitMapZone can be lock-less.

Before the patch BitMapZone::is_exhausted() required from its
callers to acquire appropriate lock. However, fulfilling this
condition is not really necessary to use the method correctly
while it can significantly hurt performance.

The change allows BitMapAreaLeaf::child_check_n_lock() to not
acquire the lock while examining zones for being exhausted.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13649 from liewegas/wip-ceph-scrub-debug

qa/tasks/ceph.py: debug which pgs aren't scrubbing

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/tasks/ceph.py: debug which pgs aren't scrubbing

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13478 from xiaoxichen/fix_osdc_perfcounter

osdc: fix osdc_osd_seesion perf counter.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13626 from wjwithagen/wip-wjw-freebsd-enodata-fixed

do_freebsd.sh: Remove ENODATA requirement

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13640 from zmc/wip-fs-openstack

qa/suites/fs: Add openstack volume configuration

Merge pull request #13619 from kylinstorage/cleanup_bitallocator

BitAllocator: delete useless codes

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/suites/fs: Add openstack volume configuration

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #13594 from athanatos/wip-snap-trim-sleep

osd: add snap trim reservation and re-implement osd_snap_trim_sleep

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13570 from athanatos/wip-18937

osd: don't use ORDERSNAP for flush; always request/send ondisk ack

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13637 from zmc/wip-openstack-volumes

qa/suites/{knfs,hadoop,samba}: OpenStack volume configuration

Merge pull request #13634 from zmc/wip-kcephfs-openstack

qa/suites/kcephfs: Openstack volume configuration

qa/suites/samba: Openstack volume configuration

Signed-off-by: Zack Cerza <zack@redhat.com>

qa/suites/hadoop: Openstack volume configuration

Signed-off-by: Zack Cerza <zack@redhat.com>

qa/suites/knfs: Add openstack volume configuration

Signed-off-by: Zack Cerza <zack@redhat.com>

qa/suites/kcephfs: Openstack volume configuration

Signed-off-by: Zack Cerza <zack@redhat.com>

osd,osdc: eliminate FLAG_ONDISK and helpers

The objecter actually always needs to get a response in order to
be able to not continually resend ops (even if the caller didn't
provide a callback).  Thus, it makes no sense for an MOSDOp to
ever not have FLAG_ONDISK set.  Therefore, we'll just remove the
helper and assume it's always there (it's safe to send a response
the client didn't ask for, the error paths already do that).  On
the Objecter side, we'll just unconditionally fill in ONDISK for
the benefit of pre-luminous OSDs.

Fixes: http://tracker.ceph.com/issues/18961
Signed-off-by: Samuel Just <sjust@redhat.com>

PrimaryLogPG::start_flush: don't use ORDERSNAP, eliminate the second delete

I think that whole thing was a misguided attempt to avoid deleting head
if it exists in the base tier (in reality it doesn't matter since head
would have to be logically dirty and anything we actually care about
would be preserved by sending a new enough seq to cause a clone).

Introduced in 4843fd510b33a71999cdf9c2cfa2b4c318fa80fd, but the real
logical error happened in f3df50188b54e60e28a276762c370477538bbb07.

I suggest never backporting this patch.  If you want to try, keep in
mind that the last version didn't turn up as busted for 2 years.

Fixes: f3df50188b54e60e28a276762c370477538bbb07
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #13631 from ceph/wip-krbd-openstack

qa/suites/krbd: Add openstack volume configuration

Merge pull request #13534 from liewegas/wip-objecter-fixes

osdc/Objecter: fix bugs in explicit naming of op spg_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/krbd: Add openstack volume configuration

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #13545 from liewegas/wip-18933

osd: constify OpRequest::get_req(); fix a few cases of operator<< vs mutated message races

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13625 from tchaikov/wip-suite-repo-for-workunit

qa/tasks/workunit: use the suite repo for cloning workunit

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13614 from linuxbox2/wip-valid-dirname

rgw_file: ensure valid_s3_object_name for directories, too

Merge pull request #12102 from yangdongsheng/async_remove

librbd: asynchronous image removal state machine

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

do_freebsd.sh: the EMBEDDED target is not (yet) working for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

do_freebsd.sh: Remove ENODATA requirement

 - This was there because cython did not know about ENOATTR
   But since cython 0.25.2 tus is fixed.
   And we don't need to edit /usr/include/errno.h

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #13620 from tchaikov/wip-sepia-bt

script/sepia_bt.sh: get sha1,release from t.log if it's not in core

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/tasks/workunit: use the suite repo for cloning workunit

as "workunits" reside in ceph/qa/workunits, it's more intuitive to
respect suite-repo option when cloning workunits.

Signed-off-by: Kefu Chai <kchai@redhat.com>

script/sepia_bt.sh: get sha1,release from t.log if it's not in core

* sometimes, the coredump comes from python, so we should get the sha1 and
  release in a different and more fragile way.
* also, the distro of Centos7 comes from python is "Centos Linux", so we
  should normalize its distro name and distro version.
* add "-v" option to be more chatty.
* normalize the $prog if $prog is */python*
* fix the pkg_path if the distro is centos7

Signed-off-by: Kefu Chai <kchai@redhat.com>

BitAllocator: delete useless codes

Signed-off-by: Jie Wang <jie.wang@kylin-cloud.com>

Merge pull request #13312 from trociny/wip-18783

rbd-mirror: InstanceWatcher watch/notify stub for leader/follower RPC

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13616 from liewegas/wip-primary-temp

mon/OSDMonitor: remove trivial PGMap dependency for 'osd primary-temp' command

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #13611 from zmc/wip-rgw-openstack

qa/suites/rgw: Add openstack volume configuration

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

mon/OSDMonitor: remove trivial PGMap dependency for 'osd primary-temp' command

Signed-off-by: Sage Weil <sage@redhat.com>

rgw_file: ensure valid_s3_object_name for directories, too

The logic in RGWLibFS::mkdir() validated bucket names, but not
object names (though RGWLibFS::create() did so).

The negative side effect of this was not creating illegal objects
(we won't), but in a) failing with -EIO and b) more importantly,
not removing up the proposed object from FHCache, which produced a
boost assert when recycled.

Fixes: http://tracker.ceph.com/issues/19066
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13607 from linuxbox2/wip-parentref

rgw: wip parentref

qa/suites/rgw: Add openstack volume configuration

Without this, OSDs will fail to create on instances whose root fs isn't
xfs.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #13480 from wido/srpm

make-srpm: Pass first parameter to make-dist for building SRPM

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file:  return of RGWFileHandle::FLAG_EXACT_MATCH

Allow callers of rgw_lookup() on objects attested in an
rgw_readdir() callback the ability to bypass exact match in
RGWLibFS::stat_leaf() case 2, but restore exact match enforcement
for general lookups.

This preserves required common_prefix namespace behavior, but
prevents clients from eerily permitting things like "cd sara0" via
partial name match on "sara01."

Fixes: http://tracker.ceph.com/issues/19059
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13595 from tchaikov/wip-enable-bluestore-ceph-dencoder

ceph-dencoder: enable bluestore types

Reviewed-by: Sage Weil <sage@redhat.com>

osdc/Objecter: _calc_target on all ops so that we notice splits

We need to make sure we update the mapping and get an accurate actual_pgid
value by recalcuating the mapping on every map change.  Otherwise, we may
not notice a split (and subsequent actual_pgid change) and resend the same
op with a stale spg_t.  To fix this,

- _calc_target on need_resend
- update target regardless of current con

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: refactor pool dne check to make op->session optional

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: track latest epoch in op_target_t

Signed-off-by: Sage Weil <sage@redhat.com>

osd: warn on ops directed to the wrong pg_t

Check whether the request hobj maps to the current pg_t.  If we have the
osd_debug_misdirected_ops setting enabled (as teuthology does), assert out
as well so that the error is easy to spot.  This catches bugs in the
Objecter (especially the new code that explicitly names the spg_t for the
request).

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: osd debug misdirected ops = true

Signed-off-by: Sage Weil <sage@redhat.com>

rgw_file: rgw_lookup: don't ref for "/" or ".."

These refs won't be returned by nfs-ganesha, and are sufficiently
magical that other consumers should be persuaded to understand
their specialness.

Fixes: http://tracker.ceph.com/issues/19060
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13569 from athanatos/wip-17831-18583-18809-18927

ReplicatedBackend: don't queue Context outside of ObjectStore with obc

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13470 from wangzhengyong/master

os/bluestore/BitAllocator: fix bug of checking required blocks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13516 from wjwithagen/wip-wjw-test-onjectstore

test/ceph-test-objectstore: Don't always include BlueStore code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13521 from dachary/wip-crush-uniform

crush: allow uniform buckets with no items

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13540 from liewegas/wip-debug-osd

osd: debug con in ms_handle_connect

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13565 from liewegas/wip-fix-pool-race-test

ceph_test_rados_api_misc: avoid livelock from PoolCreationRace

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd-mirror A/A: InstanceWatcher watch/notify stub for leader/follower RPC

Fixes: http://tracker.ceph.com/issues/18783
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: add ManagedLock destroy method (useful for mock tests)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls/rbd: methods to track rbd-mirror instances

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

msg/async/rdma: Fix broken compilation

Resposible commit is: eb0f624

issue:985021

Change-Id: I5b4afc537a9351dd18f5ea08bf4d72a1b03a8635
Signed-off-by: Sarit Zubakov <saritz@mellanox.com>

Merge pull request #13568 from dillaman/wip-18990

rbd-mirror: deleting a snapshot during sync can result in read errors

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph-dencoder: s/WITH_LIBAIO/HAVE_LIBAIO/

* s/WITH_LIBAIO/HAVE_LIBAIO/: as HAVE_LIBAIO is used to detect if libaio
  is installed and is exposed in the acconfig.h.
* do not test bluestore_blob_t with ceph-dencoder, as it repurposes the
  "feature" parameter for struct_v.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix bluestore_blob_t::generate_test_instances()

add extends before marking unused ranges, otherwise add_unused()
asserts if (offset + len < blob_len)
this method is supposed to be used by ceph-dencoder, but
bluestore_blob_t's codec is quite different. we are not testing its
encoding in ceph-dencoder at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

unittest: add unittest for RemoveRequest

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

unittest: fixup the mock method of open

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

unittest: add mock method of try_acquire_lock

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: cleanup: remove the unused function ObjectMap::remove()

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: add a aio_remove() in objectMap

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: use create/destroy methods in RemoveRequest

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

unittest: add a unittest for removing non-exist image.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

qa: rbd: rename the copy.sh to generic.sh

The copy.sh is not only testing the rbd copy, but also
others such as rbd ls, rbd remove. Then rename it to generic.sh

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

qa: add a test for rbd rm NOT_EXIST

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: use async image removal state machine

Signed-off-by: Venky Shankar <vshankar@redhat.com>