Merge pull request #13534 from liewegas/wip-objecter-fixes

osdc/Objecter: fix bugs in explicit naming of op spg_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13545 from liewegas/wip-18933

osd: constify OpRequest::get_req(); fix a few cases of operator<< vs mutated message races

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13625 from tchaikov/wip-suite-repo-for-workunit

qa/tasks/workunit: use the suite repo for cloning workunit

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13614 from linuxbox2/wip-valid-dirname

rgw_file: ensure valid_s3_object_name for directories, too

Merge pull request #12102 from yangdongsheng/async_remove

librbd: asynchronous image removal state machine

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13620 from tchaikov/wip-sepia-bt

script/sepia_bt.sh: get sha1,release from t.log if it's not in core

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/tasks/workunit: use the suite repo for cloning workunit

as "workunits" reside in ceph/qa/workunits, it's more intuitive to
respect suite-repo option when cloning workunits.

Signed-off-by: Kefu Chai <kchai@redhat.com>

script/sepia_bt.sh: get sha1,release from t.log if it's not in core

* sometimes, the coredump comes from python, so we should get the sha1 and
  release in a different and more fragile way.
* also, the distro of Centos7 comes from python is "Centos Linux", so we
  should normalize its distro name and distro version.
* add "-v" option to be more chatty.
* normalize the $prog if $prog is */python*
* fix the pkg_path if the distro is centos7

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13312 from trociny/wip-18783

rbd-mirror: InstanceWatcher watch/notify stub for leader/follower RPC

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13616 from liewegas/wip-primary-temp

mon/OSDMonitor: remove trivial PGMap dependency for 'osd primary-temp' command

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #13611 from zmc/wip-rgw-openstack

qa/suites/rgw: Add openstack volume configuration

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

mon/OSDMonitor: remove trivial PGMap dependency for 'osd primary-temp' command

Signed-off-by: Sage Weil <sage@redhat.com>

rgw_file: ensure valid_s3_object_name for directories, too

The logic in RGWLibFS::mkdir() validated bucket names, but not
object names (though RGWLibFS::create() did so).

The negative side effect of this was not creating illegal objects
(we won't), but in a) failing with -EIO and b) more importantly,
not removing up the proposed object from FHCache, which produced a
boost assert when recycled.

Fixes: http://tracker.ceph.com/issues/19066
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13607 from linuxbox2/wip-parentref

rgw: wip parentref

qa/suites/rgw: Add openstack volume configuration

Without this, OSDs will fail to create on instances whose root fs isn't
xfs.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #13480 from wido/srpm

make-srpm: Pass first parameter to make-dist for building SRPM

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file:  return of RGWFileHandle::FLAG_EXACT_MATCH

Allow callers of rgw_lookup() on objects attested in an
rgw_readdir() callback the ability to bypass exact match in
RGWLibFS::stat_leaf() case 2, but restore exact match enforcement
for general lookups.

This preserves required common_prefix namespace behavior, but
prevents clients from eerily permitting things like "cd sara0" via
partial name match on "sara01."

Fixes: http://tracker.ceph.com/issues/19059
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13595 from tchaikov/wip-enable-bluestore-ceph-dencoder

ceph-dencoder: enable bluestore types

Reviewed-by: Sage Weil <sage@redhat.com>

osdc/Objecter: _calc_target on all ops so that we notice splits

We need to make sure we update the mapping and get an accurate actual_pgid
value by recalcuating the mapping on every map change.  Otherwise, we may
not notice a split (and subsequent actual_pgid change) and resend the same
op with a stale spg_t.  To fix this,

- _calc_target on need_resend
- update target regardless of current con

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: refactor pool dne check to make op->session optional

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: track latest epoch in op_target_t

Signed-off-by: Sage Weil <sage@redhat.com>

osd: warn on ops directed to the wrong pg_t

Check whether the request hobj maps to the current pg_t.  If we have the
osd_debug_misdirected_ops setting enabled (as teuthology does), assert out
as well so that the error is easy to spot.  This catches bugs in the
Objecter (especially the new code that explicitly names the spg_t for the
request).

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: osd debug misdirected ops = true

Signed-off-by: Sage Weil <sage@redhat.com>

rgw_file: rgw_lookup: don't ref for "/" or ".."

These refs won't be returned by nfs-ganesha, and are sufficiently
magical that other consumers should be persuaded to understand
their specialness.

Fixes: http://tracker.ceph.com/issues/19060
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13569 from athanatos/wip-17831-18583-18809-18927

ReplicatedBackend: don't queue Context outside of ObjectStore with obc

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13470 from wangzhengyong/master

os/bluestore/BitAllocator: fix bug of checking required blocks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13516 from wjwithagen/wip-wjw-test-onjectstore

test/ceph-test-objectstore: Don't always include BlueStore code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13521 from dachary/wip-crush-uniform

crush: allow uniform buckets with no items

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13540 from liewegas/wip-debug-osd

osd: debug con in ms_handle_connect

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13565 from liewegas/wip-fix-pool-race-test

ceph_test_rados_api_misc: avoid livelock from PoolCreationRace

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd-mirror A/A: InstanceWatcher watch/notify stub for leader/follower RPC

Fixes: http://tracker.ceph.com/issues/18783
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: add ManagedLock destroy method (useful for mock tests)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls/rbd: methods to track rbd-mirror instances

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13568 from dillaman/wip-18990

rbd-mirror: deleting a snapshot during sync can result in read errors

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph-dencoder: s/WITH_LIBAIO/HAVE_LIBAIO/

* s/WITH_LIBAIO/HAVE_LIBAIO/: as HAVE_LIBAIO is used to detect if libaio
  is installed and is exposed in the acconfig.h.
* do not test bluestore_blob_t with ceph-dencoder, as it repurposes the
  "feature" parameter for struct_v.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix bluestore_blob_t::generate_test_instances()

add extends before marking unused ranges, otherwise add_unused()
asserts if (offset + len < blob_len)
this method is supposed to be used by ceph-dencoder, but
bluestore_blob_t's codec is quite different. we are not testing its
encoding in ceph-dencoder at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

unittest: add unittest for RemoveRequest

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

unittest: fixup the mock method of open

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

unittest: add mock method of try_acquire_lock

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: cleanup: remove the unused function ObjectMap::remove()

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: add a aio_remove() in objectMap

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: use create/destroy methods in RemoveRequest

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

unittest: add a unittest for removing non-exist image.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

qa: rbd: rename the copy.sh to generic.sh

The copy.sh is not only testing the rbd copy, but also
others such as rbd ls, rbd remove. Then rename it to generic.sh

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

qa: add a test for rbd rm NOT_EXIST

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: use async image removal state machine

Signed-off-by: Venky Shankar <vshankar@redhat.com>

librbd: asynchronous image removal state machine

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #13589 from dachary/wip-crush-doc

doc: crush: API documentation fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13590 from linuxbox2/wip-badbuck

rgw_file: avoid stranding invalid-name bucket handles in fhcache

Merge pull request #13324 from dongbula/make-sync-thread-name-clear

rgw: make sync thread name clear

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13361 from ddiss/rbdmap_unmap_from_file

rbdmap: consider /etc/ceph/rbdmap when unmapping images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13387 from Jing-Scott/clean-up-rgw-list-redundant-code

rgw: clean up the redundant assignment in last_entry_in_listing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13460 from theanalyst/fix/rgw-rest-user-fixme

wip: rgw: rest_admin/user avoid double checking input args

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13461 from theanalyst/doc/rgw-system-user

doc: rgw: make a note abt system users vs normal users

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13576 from jronak/doc

rgw:Fixes typo in rgw_admin.cc

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13578 from Liuchang0812/wip-fix-typo-in-rgw

rgw: fix typo in comment

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw_file: avoid stranding invalid-name bucket handles in fhcache

To avoid a string copy in the common mkdir path, handles for
proposed buckets currently are staged in the handle table, before
being rejected.  They need to be destaged, not just marked deleted
(because deleted objects are now assumed not to be linked, as of
beaeff059375b44188160dbde8a81dd4f4f8c6eb).

This triggered an unhandled Boost assert when deleting staged
handles, as current safe_link mode requires first removing from
the FHCache.

Fixes: http://tracker.ceph.com/issues/19036
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #12897 from renhwztetecs/renhw-wip-client-cleanup

client: priority to verify the correctness of the "flag"

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13359 from jcsp/wip-logrotate-sshexception

qa: handle SSHException in logrotate

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13429 from ukernel/wip-18941

test/libcephfs: avoid buffer overflow when testing ceph_getdents()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13455 from ukernel/wip-18953

mds: ignore fs full check for CEPH_MDS_OP_SETFILELOCK

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13465 from jcsp/wip-mds-voidfn

mds: replace C_VoidFn in MDSDaemon with lambdas

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13471 from batrick/fs-new-eexist

mon: detect existing fs and duplicate name earlier

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13572 from theanalyst/doc/changelog/v0.94.10

doc: add changelog for v0.94.10

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13152 from smithfarm/wip-hammer-release-notes

doc: hammer 0.94.10 release notes

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #13573 from dachary/wip-17821-revert

ceph-disk: dmcrypt activate must use the same cluster as prepare

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: Fixes typo in rgw_admin.cc

Issue: http://tracker.ceph.com/issues/19026
Signed-off-by: Ronak Jain <ronakjain@outlook.in>

rgw: fix typo in comment

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

ceph-disk: dmcrypt activate must use the same cluster as prepare

When dmcrypt is used, the fsid cannot be retrieved from the data
partition because it is encrypted. Store the fsid in the lockbox to
enable dmcrypt activation using the same logic as regular activation.

The fsid is used to retrive the cluster name that was used during
prepare, reason why activation does not and must not have a --cluster
argument.

Fixes: http://tracker.ceph.com/issues/17821
Signed-off-by: Loic Dachary <ldachary@redhat.com>

rbd-mirror: retry object copy after -ENOENT error

Fixes: http://tracker.ceph.com/issues/18990
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: object copy should always reference valid snapshots

If a remote snapshot is deleted while an image sync is in-progress,
associate the read request against the most recent, valid remote
snapshot for a given snapshot object clone.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: replace complex object op tuple with struct

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13515 from smithfarm/wip-17613

rpm: build ceph-resource-agents by default

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: add changelog for v0.94.10

Generated as
```
$ git log tags/v0.94.9..tags/v0.94.10 --after="Tue Aug 30 11:33:36 2016 +0000"
```

in order to avoid the qa suite merge commits
Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Revert "ceph-disk: change get_dmcrypt_key test to support different cluster name"

Merged by mistake, teuthology ceph-disk fails

This reverts commit 1c747473600a9a7f528ad71a889c35d85ac115b1.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Revert "ceph-disk: Adding cluster name support for dmcrypt"

Merged by mistake, teuthology ceph-disk fails

This reverts commit d98d4dbc2741295723f0a316b5bf81f07b140041.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12623 from LiumxNL/fix-boot-epoch

osd: correct epoch setting of osd boot msg

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13418 from dreamhost/wip-18928

msg/async: support IPv6 QoS.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13530 from tangwenjun3/wip-tang-nvme

os/bluestore rm unused variable in aio_read()

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph_test_rados_api_misc: avoid livelock from PoolCreationRace

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13536 from Liuchang0812/wip-add-override-in-rbd-headers

librbd: add override keyword in header files

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13558 from ceph/wip-19016-remove-mon_osd_min_down_reports

doc: remove mentions about mon_osd_min_down_reports

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12295 from wjwithagen/wip-wjw-TEMP_FAILURE

rgw/rgw_main.cc: fix parenteses and function result

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13291 from prallabh/obj_stat

rgw: Let the object stat command be shown in the usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: remove mentions about mon_osd_min_down_reports

Fixes: http://tracker.ceph.com/issues/19016
Signed-off-by: Bara Ancincova (bara@redhat.com)

librbd: introduce TypeTraits in librbd/image

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: add force option to DisableFeaturesRequest state machine

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cls_rbd: async helper for dir_get_id()

Signed-off-by: Venky Shankar <vshankar@redhat.com>

librbd: remove header when id missing

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #13550 from tchaikov/wip-18968

mon/MonClient: persist global_id across re-connecting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: add override in header files

Fixes: http://tracker.ceph.com/issues/19012
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

rgw: Let the object stat command be shown in the usage

Fixes: http://tracker.ceph.com/issues/19013
Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

mon/MonClient: persist global_id across re-connecting

MonClient should re-use previously assigned global-id after
re-connecting. and should not have its global-id changed once it is
authenticated unless monitor thinks otherwise.

Fixes: http://tracker.ceph.com/issues/18968
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13346 from batrick/mds-doc-fix

doc: update to new ceph fs commands

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13529 from linuxbox2/wip-dir-orphan

rgw: wip dir orphan

rgw_file: fix marker computation

Fixes: http://tracker.ceph.com/issues/19018
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13447 from dillaman/wip-13025

librbd: scatter/gather support for the C API

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

osd/OpRequest: make get_req() return const

Anybody who needs a mutable pointer should be calling get_nonconst_ref()
and justify themselves.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: cast const Message*'s where possible

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: annotate non-const MOSDOp casts

Signed-off-by: Sage Weil <sage@redhat.com>