messages/MOSDOp: fix pg_t decoding for version <7 decoding

This was broken by f6e219a4df8553336839bb914733a78090561ec8

Fixes: http://tracker.ceph.com/issues/19005
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13065 from ceph/wip-fuse-client-config-edits

doc: minor changes in fuse client config reference

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13528 from yuyuyu101/wip-fix-rdma-fd

msg/async/rdma: don't need to delete event when tcp connection isn't …

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #13468 from vumrao/wip-vumrao-18932

auth: 'ceph auth import -i' overwrites caps, if caps are not specified

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13267 from yuyuyu101/wip-fix-lock

msg/simple: fix missing unlock when already bind

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10125 from stiopaa1/msg_asyncmess_changeReturnTypeIntToVoid

msg/AsyncMessenger: remove unused method

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13339 from wjwithagen/wip-wjw-include-ceph_features

common: include/ceph_features.h uses uint64_t, which is in sys/types.h

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/rdma: don't need to delete event when tcp connection isn't built

Signed-off-by: Haomai Wang <haomai@xsky.com>

auth: 'ceph auth import -i' overwrites caps, if caps are not specified
in given keyring file, should alert user and should not allow this import.
Because in 'ceph auth list' we keep all the keyrings with caps and importing
'client.admin' user keyring without caps locks the cluster with error[1]
because admin keyring caps are missing in 'ceph auth'.

[1] Error connecting to cluster: PermissionDeniedError

Fixes: http://tracker.ceph.com/issues/18932
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

Merge pull request #13519 from smithfarm/wip-drop-boost-devel

rpm: drop boost build dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13524 from tchaikov/wip-debian-drop-boost-builddeps

debian: drop boost build dependencies

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10487 from yangdongsheng/rbd_export

rbd: introduce v2 format for rbd export/import

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: import/export/mergediff: use STD[IN|OUT]_FILENO instead of 0 or 1

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import: use read() instead of lseek() when from_stdin is true

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: add docs for new format of rbd export.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: combine the namespaces of export_diff and export_full, import_diff and import.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import: decode file data in le64 for compatability on different platform

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import_diff/export_diff: encode length for each tag.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: refactor export/import to v1 and v2 methods.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: use the unified --export-format option for rbd export and rbd import

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

qa/workunit/rbd: import_export.sh: add image priorities exporting and importing test cases.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import/export image stripe_unit and stripe_count.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import/export features of image

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: import/export image_order.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

qa/workunit/rbd: add test case for --import/export-format in import_export.sh

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: introduce --export-format to export command

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: introduce --import-format to import command

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: refactor do_import_diff to accept fd.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: refactor do_export_diff to accept fd

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: move do_import_diff out from namespace of import_diff

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: move do_export_diff out from namespace of export_diff.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: squash ImportDiff.cc into Import.cc.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: squash ExportDiff.cc into Export.cc

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: use macro for rbd diff tags

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: finish the pc in importing even if from_stdin.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: remove image in error handling of importing

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd/Import: fix the indent of labels

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

debian: drop boost build dependencies

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: drop boost build dependencies

The boost library is now a submodule.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13207 from liewegas/wip-mapping

mon: restructure prime_pg_temp around a full pg mapping calculated on multiple CPUs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13420 from liewegas/wip-18929

osd/PG: restrict want_acting to up+acting on recovery completion

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #13479 from LiumxNL/fix-monc-rand

mon/MonClient: random all ranks then pick first_n

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13482 from idryomov/wip-qa-krbd-data-pool

qa: add workunit to test krbd data-pool support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon/MonClient: random all ranks then pick first_n

Signed-off-by: Mingxin Liu <mingxin@xsky.com>

Merge pull request #13493 from liewegas/wip-log-split-index

osd/PGLog: fix index for parent and child log on split

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13458 from tchaikov/wip-silence-gcc-warnings

include/denc, kv: silence gcc warnings

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13462 from tanghaodong25/reweight-by-pg

mon/OSDMonitor: some cleanup for reweight-by-pg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13473 from tchaikov/wip-coverity-fixes

msg, messages: coverity fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12977 from weixiaowilliam/crc32c_opt

crc32c: optimize aarch64 crc32c implementation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13428 from xiexingguo/wip-flush-cache

os/bluestore: add flush_store_cache cmd

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13430 from majianpeng/tracepointprovier

common/TracepointProvider: add assert if dlopen error.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13335 from ovh/wip-clang-compilation-fix

common: Fix clang compilation

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async: fix missing unlock when already bind

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13435 from yuyuyu101/wip-rdma-leak

msg/async/rdm: fix leak when existing failure in ip network

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #13511 from tangwenjun3/wip-nvme

os/bluestore: fix bug in aio_read()

Reviewed-by: Haomai Wang <haomai@xsky.com>

os/bluestore:fix bug in aio_read()

fixed compilation issue  in aio_read() when enable spdk

Signed-off-by: tangwenjun <tang.wenjun3@zte.com.cn>

Merge pull request #13484 from yuyuyu101/wip-rdma-perf-counter

msg/async/rdma: add perf counters to RDMA backend

Reviewed-by: Adir lev <adirl@mellanox.com>

msg/async/rdma: add perf counters to RDMA backend

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10156 from rzarzynski/wip-rgw-controllable-204

rgw: make sending Content-Length in 204 and 304 responses controllable

Merge pull request #13467 from liewegas/wip-init-ceph

init-ceph: fix ceph user args

Reviewed-by: Dan Mick <dmick@redhat.com>

osd/PGLog: avoid return by value on ginormous log

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGLog: reindex properly on pg log split

When pg_log_t::split_out_child() runs it builds the list, which means the
old indexes are wrong (the point to bad memory), but index() will not
rebuild them because ever since b858e869e78927dccebaa350d246bd74af7f1de9
we won't rebuild them if they are already built.

Fix that by calling unindex() before the split.

Further, the new child log also needs to be indexed.  Fix that too.

Fixes: http://tracker.ceph.com/issues/18975
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13495 from ceph/revert-13452-wip-suite-repo-for-workunit

Revert "qa/tasks/workunit: use the suite repo for cloning workunit"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/perf_histogram: initialize member vars of axis_config_d

addresses CID 1400670:    (UNINIT_CTOR)

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "qa/tasks/workunit: use the suite repo for cloning workunit"

Merge pull request #12370 from wido/docs-cephfs-disaster

doc: Update CephFS disaster recovery documentation

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13456 from tchaikov/wip-qa-objectstore

qa/suites/rados/objectstore: enable experimental features for testing bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13437 from Liuchang0812/cleanup-rbd

librbd: use 'override' keyword instead of 'virtual'

rbd: add override in rbd subsystem

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

kv: silence the gcc warning of func foo was hidden

this silences the warnings like

ceph/ceph/src/kv/KeyValueDB.h:59:18: warning: ‘virtual void
KeyValueDB::TransactionImpl::set(const string&, const char*, size_t,
const bufferlist&)’ was hidden [-Woverloaded-\
virtual]
     virtual void set(
                  ^~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/denc: silence gcc warnings

gcc-7 complains:

ceph/ceph/src/include/denc.h:469:50: warning: enum constant in boolean
context [-Wint-in-bool-context]
 inline typename std::enable_if<traits::supported &&
                                ~~~~~~~~~~~~~~~~~~^~
           traits::featured>::type denc(
           ~~~~~~

so let's use "static constexpr bool" instead of enum.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11786 from leseb/dmcrypt-cluster-name

ceph-disk: ability to use a different cluster name with dmcrypt

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12433 from liupan1111/wip-fix-rbd-nbd-log-sock

rbd-nbd: create admin socket only for map command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

msg/async/AsyncConnection: remove no-ops

addresses CID 1400656:  Incorrect expression  (NO_EFFECT)

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/simple/Pipe: remove no-op statements

addresses CID 1400658:  Incorrect expression  (NO_EFFECT)

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MOSDBackoff: make *_VERSION static

addresses CID 1400661:  Uninitialized variables  (UNINIT)

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/objectstore/test_fuse.sh: enable experimental features

so we can test bluestore

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/objectstore/test_fuse.sh: use portable function decl

function f() is illegal in strict POSIX shells, like dash, which is
the default /bin/sh on Ubuntu

Signed-off-by: Dan Mick <dan.mick@redhat.com>

qa/workunits/objectstore/test_fuse.sh: allow failure

This script currently has a syntax error, but still exits with
success, which is hiding that failure.  Expose it by allowing
the 'sudo' exit code to be the script's exit code.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #12797 from chendave/wip_error_message

ceph.in: Fix couple of minor issues on the messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13449 from xiaoxichen/mdlog_jlat

mds/MDLog.cc Fix perf counter type for jlat

rgw: make sending Content-Length in 204 and 304 controllable

This commit introduces a new configurable "rgw print prohibited
content length" to let operator decide whether RadosGW complies
to RFC 7230 (a part of the HTTP specification) or violates it
but follows the Swift's behavior.

Fixes: http://tracker.ceph.com/issues/16602
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

init-ceph: fix ceph user args

Broken by 299b7d06ac18c5cd30b8b65c7d25df9fc00287db

Signed-off-by: Sage Weil <sage@redhat.com>

qa: add workunit to test krbd data-pool support

This is based on a script that I've been using for a while for basic
smoke testing.  The matrix has exploded with the addition of data-pool
and now it's primarily a data-pool test fixture that takes minutes to
run, so turning it into a workunit.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mon/OSDMonitor: some cleanup for reweight-by-pg

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #13350 from Jing-Scott/fix-swift-ver-error-code

rgw: fix error code of inexistence of versions location in swift api

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

mgr: mark stale PGs

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: move check_down_pgs into PGMapUpdater; use latest map

Two big changes here:

1) Move the updating into PGMapUpdater along with the other related
checks.  Cleaner!

2) We use the current OSDMap to mark things stale.  Before, we had
a weird two stage process: first we would apply the OSDMap to the
PGMap, and then the *next* time the OSDMap updated we would look
for stale PGs.  This was silly.  Instead, we mark the PGs stale
when we process the OSDMap that made them stale.

We have a be slightly careful because the PGMap stats primaries
might be a bit old and, for example, might have a smaller max_osd
than what we have now.

This change is motivated right now because the new prime_pg_temp
code is smarter about a failed OSD.  While previously failing
an OSD would install a pg_temp mapping to the old and new OSDs,
even though the old OSD is now down, the new code does not.  And
previously failing both OSDs for a PG would always result in a
subsequent OSDMap update removing those bad pg_temp values, giving
check_down_pgs() a change to mark the PGs stale, while the new code
has not subsequent OSDMap update, which would leave the PG unstale,
causing the dump_stuck.py test to fail.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMapMapping: put mappings in a mempool

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: use parallel work queue for prime_pg_temp (all)

Do the prime_pg_temp work in parallel using the mapper.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMapMapping: restructure mapper as generic PG work running

Make Job a parent class that can be specialized for any per-PG batch work.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: prime all pgs if we estimate we're approaching that anyway

If we have enough noteworthy OSDs it is likely we will touch all or most
PGs anyway, at which point it is faster and simpler to check all pgs.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMapMapper: count total pgs

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: avoid priming same pgs more than once

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMapMapping: disable up_rmap

The reverse mapping from up osds to pgs is not used; disable it for now.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: run mapping job in background; skip prime if it didn't finish

Start the background mapping job when we go active.  If it has finished
by the time we need to prime, use it.  If not, skip prime_pg_temp.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: simplify prime_pg_temp slightly

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: calculate new mapping in parallel

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add a threadpool (cpu_tp)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMapMapping: add ParallelMapper

Calculate a mapping in parallel over a workqueue + threadpool.

Signed-off-by: Sage Weil <sage@redhat.com>