ceph-disk: dmcrypt activate must use the same cluster as prepare

When dmcrypt is used, the fsid cannot be retrieved from the data
partition because it is encrypted. Store the fsid in the lockbox to
enable dmcrypt activation using the same logic as regular activation.

The fsid is used to retrive the cluster name that was used during
prepare, reason why activation does not and must not have a --cluster
argument.

Fixes: http://tracker.ceph.com/issues/17821
Signed-off-by: Loic Dachary <ldachary@redhat.com>

Revert "ceph-disk: change get_dmcrypt_key test to support different cluster name"

Merged by mistake, teuthology ceph-disk fails

This reverts commit 1c747473600a9a7f528ad71a889c35d85ac115b1.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Revert "ceph-disk: Adding cluster name support for dmcrypt"

Merged by mistake, teuthology ceph-disk fails

This reverts commit d98d4dbc2741295723f0a316b5bf81f07b140041.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13536 from Liuchang0812/wip-add-override-in-rbd-headers

librbd: add override keyword in header files

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13558 from ceph/wip-19016-remove-mon_osd_min_down_reports

doc: remove mentions about mon_osd_min_down_reports

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12295 from wjwithagen/wip-wjw-TEMP_FAILURE

rgw/rgw_main.cc: fix parenteses and function result

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13291 from prallabh/obj_stat

rgw: Let the object stat command be shown in the usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: remove mentions about mon_osd_min_down_reports

Fixes: http://tracker.ceph.com/issues/19016
Signed-off-by: Bara Ancincova (bara@redhat.com)

Merge pull request #13550 from tchaikov/wip-18968

mon/MonClient: persist global_id across re-connecting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: add override in header files

Fixes: http://tracker.ceph.com/issues/19012
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

rgw: Let the object stat command be shown in the usage

Fixes: http://tracker.ceph.com/issues/19013
Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

mon/MonClient: persist global_id across re-connecting

MonClient should re-use previously assigned global-id after
re-connecting. and should not have its global-id changed once it is
authenticated unless monitor thinks otherwise.

Fixes: http://tracker.ceph.com/issues/18968
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13346 from batrick/mds-doc-fix

doc: update to new ceph fs commands

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13529 from linuxbox2/wip-dir-orphan

rgw: wip dir orphan

rgw_file: fix marker computation

Fixes: http://tracker.ceph.com/issues/19018
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13447 from dillaman/wip-13025

librbd: scatter/gather support for the C API

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13537 from liewegas/wip-19005

messages/MOSDOp: fix pg_t decoding for version <7 decoding

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #13385 from zhangsw/feature-lifecycle-noncurrent

rgw: add support for noncurrentversion expiration in s3 lifecycle.

Merge pull request #13518 from tchaikov/wip-fix-pgp-num

test: Thrasher: do not update pools_to_fix_pgp_num if nothing happens

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12223 from cbodley/wip-rgw-meta-sync-yield

rgw: fix for broken yields in RGWMetaSyncShardCR

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13517 from liewegas/wip-kraken-x

qa/suites/upgrade/kraken-x

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #13538 from Adirl/ENOMEM

msg/async/rdma: Print error only on ENOMEM

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13539 from Adirl/destructor

msg/async/rdma: update destructor message

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: update destructor message

issue:939258

Change-Id: Ib00bb5661a0e80fb258babb63d23458daa8b7025
Signed-off-by: Sarit Zubakov <saritz@mellanox.com>

msg/async/rdma: Print error only on ENOMEM

issue: 931114

Change-Id: I1e1710245c2c056bd2b8cccae9d96676a8f49678
Signed-off-by: Sarit Zubakov <saritz@mellanox.com>

messages/MOSDOp: fix pg_t decoding for version <7 decoding

This was broken by f6e219a4df8553336839bb914733a78090561ec8

Fixes: http://tracker.ceph.com/issues/19005
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13065 from ceph/wip-fuse-client-config-edits

doc: minor changes in fuse client config reference

Reviewed-by: John Spray <john.spray@redhat.com>

librbd: new rbd_aio_readv/rbd_aio_writev API methods

The RBD C API now has scatter/gather IO support via the API. The C++
API uses bufferlists which can also support scatter/gather IO.

Fixes: http://tracker.ceph.com/issues/13025
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ReadResult helper class for reassembling partial reads

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move IO-related classes to new io namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13528 from yuyuyu101/wip-fix-rdma-fd

msg/async/rdma: don't need to delete event when tcp connection isn't …

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #13468 from vumrao/wip-vumrao-18932

auth: 'ceph auth import -i' overwrites caps, if caps are not specified

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13267 from yuyuyu101/wip-fix-lock

msg/simple: fix missing unlock when already bind

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10125 from stiopaa1/msg_asyncmess_changeReturnTypeIntToVoid

msg/AsyncMessenger: remove unused method

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13339 from wjwithagen/wip-wjw-include-ceph_features

common: include/ceph_features.h uses uint64_t, which is in sys/types.h

Reviewed-by: Sage Weil <sage@redhat.com>

rgw_file: rgw_readdir can't list multi-segment dirs

This issue has one root cause in librgw, namely that the marker
argument to these requests was incorrectly formatted (though the
marker cache was working as intended).

Secondarily, for nfs-ganesha users, there is a compounding issue
that the RGW fsal was required by "temporary" convention to
populate the entire dirent cache for a directory on a single
readdir() invocation--the cache_inode/mdcache implementations
invariantly pass (before future 2.5 changesets, currently in
progress) a null pointer for the start cookie offset, intended
to convey this.

Fixes: http://tracker.ceph.com/issues/18991
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: allow setattr on placeholder directories

When a POSIX path <bucket>/foo/ is known only as an implicit path
segment from other objects (e.g., <bucket>/foo/bar.txt), a case
that would usually arise from S3 upload of such an object, an
RGWFileHandle object representing "<bucket>/foo/" will be constructed
as needed, with no backing in RGW.

This is by design, but subsequently, if a setattr is performed on
such a handle, we must be ready to create the object inline with
storing the attributes.

Fixes: http://tracker.ceph.com/issues/18989
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: invalid use of RGWFileHandle::FLAG_EXACT_MATCH

The change which introduced this flag also caused it to be
given as the flags argument to RGWLibFS::stat_leaf() when called
from rgw_lookup().

This was incorrect:  in particular, when a directory is known only
as a common prefix of other objects, the AWS namespace mapping
convention requires lookup("foo") to match a non-materialized
instance of "foo/" (case 2 in RGWLibFS::stat_leaf's stat loop).

Fixes: http://tracker.ceph.com/issues/18992
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

msg/async/rdma: don't need to delete event when tcp connection isn't built

Signed-off-by: Haomai Wang <haomai@xsky.com>

auth: 'ceph auth import -i' overwrites caps, if caps are not specified
in given keyring file, should alert user and should not allow this import.
Because in 'ceph auth list' we keep all the keyrings with caps and importing
'client.admin' user keyring without caps locks the cluster with error[1]
because admin keyring caps are missing in 'ceph auth'.

[1] Error connecting to cluster: PermissionDeniedError

Fixes: http://tracker.ceph.com/issues/18932
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

Merge pull request #13519 from smithfarm/wip-drop-boost-devel

rpm: drop boost build dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13524 from tchaikov/wip-debian-drop-boost-builddeps

debian: drop boost build dependencies

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10487 from yangdongsheng/rbd_export

rbd: introduce v2 format for rbd export/import

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: import/export/mergediff: use STD[IN|OUT]_FILENO instead of 0 or 1

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import: use read() instead of lseek() when from_stdin is true

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: add docs for new format of rbd export.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: combine the namespaces of export_diff and export_full, import_diff and import.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import: decode file data in le64 for compatability on different platform

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import_diff/export_diff: encode length for each tag.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: refactor export/import to v1 and v2 methods.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: use the unified --export-format option for rbd export and rbd import

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

qa/workunit/rbd: import_export.sh: add image priorities exporting and importing test cases.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import/export image stripe_unit and stripe_count.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import/export features of image

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import/export image_order.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

qa/workunit/rbd: add test case for --import/export-format in import_export.sh

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: introduce --export-format to export command

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: introduce --import-format to import command

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: refactor do_import_diff to accept fd.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: refactor do_export_diff to accept fd

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: move do_import_diff out from namespace of import_diff

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: move do_export_diff out from namespace of export_diff.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: squash ImportDiff.cc into Import.cc.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: squash ExportDiff.cc into Export.cc

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: use macro for rbd diff tags

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: finish the pc in importing even if from_stdin.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: remove image in error handling of importing

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd/Import: fix the indent of labels

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

debian: drop boost build dependencies

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: drop boost build dependencies

The boost library is now a submodule.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

test: Thrasher: do not update pools_to_fix_pgp_num if nothing happens

we should not update pools_to_fix_pgp_num if the pool is not expanded or
the pg_num is not increased due to pgs being created. this prevent us
from fixing the pgp_num after done with thrashing if we actually did
nothing when fixing the pgp_num when thrashing, but we removed the pool
from pools_to_fix_pgp_num after set_pool_pgpnum() returns.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/upgrade/kraken-x: test bluestore and filestore

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph.py: fix timing of wait-for-* and osd markdown

Mark down osds, *then* wait for them to come up or for the cluster to be
healthy!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/kraken-x: Initial checkin

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #13207 from liewegas/wip-mapping

mon: restructure prime_pg_temp around a full pg mapping calculated on multiple CPUs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13420 from liewegas/wip-18929

osd/PG: restrict want_acting to up+acting on recovery completion

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #13479 from LiumxNL/fix-monc-rand

mon/MonClient: random all ranks then pick first_n

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13482 from idryomov/wip-qa-krbd-data-pool

qa: add workunit to test krbd data-pool support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon/MonClient: random all ranks then pick first_n

Signed-off-by: Mingxin Liu <mingxin@xsky.com>

Merge pull request #13493 from liewegas/wip-log-split-index

osd/PGLog: fix index for parent and child log on split

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13458 from tchaikov/wip-silence-gcc-warnings

include/denc, kv: silence gcc warnings

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13462 from tanghaodong25/reweight-by-pg

mon/OSDMonitor: some cleanup for reweight-by-pg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13473 from tchaikov/wip-coverity-fixes

msg, messages: coverity fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12977 from weixiaowilliam/crc32c_opt

crc32c: optimize aarch64 crc32c implementation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13428 from xiexingguo/wip-flush-cache

os/bluestore: add flush_store_cache cmd

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13430 from majianpeng/tracepointprovier

common/TracepointProvider: add assert if dlopen error.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13335 from ovh/wip-clang-compilation-fix

common: Fix clang compilation

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async: fix missing unlock when already bind

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13435 from yuyuyu101/wip-rdma-leak

msg/async/rdm: fix leak when existing failure in ip network

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #13511 from tangwenjun3/wip-nvme

os/bluestore: fix bug in aio_read()

Reviewed-by: Haomai Wang <haomai@xsky.com>

os/bluestore:fix bug in aio_read()

fixed compilation issue  in aio_read() when enable spdk

Signed-off-by: tangwenjun <tang.wenjun3@zte.com.cn>

Merge pull request #13484 from yuyuyu101/wip-rdma-perf-counter

msg/async/rdma: add perf counters to RDMA backend

Reviewed-by: Adir lev <adirl@mellanox.com>

msg/async/rdma: add perf counters to RDMA backend

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10156 from rzarzynski/wip-rgw-controllable-204

rgw: make sending Content-Length in 204 and 304 responses controllable

Merge pull request #13467 from liewegas/wip-init-ceph

init-ceph: fix ceph user args

Reviewed-by: Dan Mick <dmick@redhat.com>

osd/PGLog: avoid return by value on ginormous log

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGLog: reindex properly on pg log split

When pg_log_t::split_out_child() runs it builds the list, which means the
old indexes are wrong (the point to bad memory), but index() will not
rebuild them because ever since b858e869e78927dccebaa350d246bd74af7f1de9
we won't rebuild them if they are already built.

Fix that by calling unindex() before the split.

Further, the new child log also needs to be indexed.  Fix that too.

Fixes: http://tracker.ceph.com/issues/18975
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13495 from ceph/revert-13452-wip-suite-repo-for-workunit

Revert "qa/tasks/workunit: use the suite repo for cloning workunit"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/perf_histogram: initialize member vars of axis_config_d

addresses CID 1400670:    (UNINIT_CTOR)

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "qa/tasks/workunit: use the suite repo for cloning workunit"