os/bluestore: implement get_devices()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: return rolled-up 'devices' list as part of metadata

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: add get_devname(), get_devices()

Enumerate device(s) that we are backed by.  It may be plural because of
DM, which might mask an underlying device, or because the dm device is
combining multiple other devices.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add get_devices() method

The intent here is to return the set of hardware storage devices, as
identified by the "devname" (i.e., the directory name for the device in
/sys/block/).  For example, "sda", "nvme0n1", etc.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18639 from yaozongyou/make-readme-properly-align

doc: make the commands in README.md properly aligned

Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #18499 from batrick/ptl-tool-testing-time

ptl-tool: make branch name configurable

Reviewed-by: Jos Collin <jcollin@redhat.com>

ptl-tool: make branch name configurable

Also, include hours/minutes in branch so multiple branches in a day have
different names.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18642 from zhangsw/cleanup-rgwputobj-getdata

rgw: add logs if get_data returns error in RGWPutObj::execute.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #18645 from tchaikov/wip-cmake-Findgperftools

cmake: should use the value of GPERFTOOLS_LIBRARIES as REQUIRED_VARS

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18283 from tchaikov/wip-laggy-mgr

 qa: reduce mon-client-hunt-interval-max-multiple to 2

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

cmake: should check the availability of aligned_alloc() by running the test

this mimics the failure of rocksdb without a working
aligned_alloc()/free().

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: should use the value of GPERFTOOLS_LIBRARIES as REQUIRED_VARS

otherwise it's always true, even if the libaries are not found. also
the cmake output will be wrong without this change:

without this change, it is:

-- Found gperftools: GPERFTOOLS_TCMALLOC_LIBRARY;GPERFTOOLS_TCMALLOC_MINIMAL_LIBRARY;GPERFTOOLS_PROFILER_LIBRARY (found version "2.5.93")

with this change, it would be:

-- Found gperftools: /tmp/gperf/lib/libtcmalloc.so (found version "2.5.93")

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: add logs if get_data returns error in RGWPutObj::execute.

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

doc: make the commands in README.md properly aligned

The commands in README.md are aligned using one tab or eight spaces,
the github markdown engine not handle this properly, so using one tab
universely.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #18547 from batrick/ptl-tool-trackers

ptl-tool: print bzs/tickets cited in commit msgs/comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18495 from ceph/wip-da-SCA-20171013

osd,rgw,librbd: SCA fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18535 from yanghonggang/master

os/bluestore/bluestore_tool: add log-dump command to dump bluefs's log

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #18574 from shashalu/fix-strict-strtol

common: strict_strtol INT_MAX and INT_MIN is valid

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18577 from tchaikov/wip-kill-clang-warning

ceph-syn: silence clang analyzer warning

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

ptl-tool: print bzs/tickets cited in commit msgs

This makes it easier to update issues after a merge.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18631 from gohighsec/cleanup

os/bluestore: delete redundant header file in KernelDevice.cc

Reviewed-by: Sage Weil <sage@redhat.com>

bluestore: delete redundant header file in KernelDevice.cc

Signed-off-by: Jing Li <lijing@gohighsec.com>

Merge pull request #18603 from batrick/i21945

mon/MDSMonitor: wait for readable OSDMap before sanitizing

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #18039 into master

* refs/pull/18039/head:
mgr/dashboard: redirect away if viewed filesystem is removed
mgr/dashboard: 404 instead of 500 on missing filesystem
mds: fix FSMap copy constructor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18477 from ZVampirEM77/wip-em-cleanup

rgw: drop unused variable in copy_obj_data()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #18579 from idryomov/wip-doc-kclient-snapshots

doc/cephfs/experimental-features: kernel client snapshots limit

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #18513 from ceph/ceph-volume-zap

ceph-volume: adds the ceph-volume lvm zap subcommand

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18615 from rgdoliveira/alpine

rbd-fuse: make sure PATH_MAX is defined

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17363 from tchaikov/wip-ceph-disk

ceph-disk: unlock all partitions when activate

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge pull request #18585 from yaozongyou/fix-bluestore-tool-arg-bug

ceph-bluestore-tool: link target shouldn't ending with "\n"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17995 from scienceluo/wip-luo-fix-bluestore-metadata-branch

os/bluestore/BlueFS: Reduce unnecessary operations in collect_metadata

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

rbd-fuse: make sure PATH_MAX is defined

On systems without glibc, as Alpine Linux, you might lack definition of
PATH_MAX. This patch adds the limits.h header to solve this issue.

Signed-off-by: Roberto Oliveira <robertoguimaraes8@gmail.com>

Merge pull request #18611 from liewegas/wip-interval-set

include/interval_set: parameterize by map type and kill btree_interval_set.h

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18507 from hjwsm1989/skip-generation-object

ceph-objectstore-tool: skip object with generated version

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #18493 from liewegas/wip-unclean-degraded

mon/PGMap: 'unclean' does not imply damaged

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

os/bluestore/bluestore_tool: add bluefs-log-dump command to dump bluefs's log

./bin/ceph-bluestore-tool --command bluefs-log-dump --path dev/osd0/
...
0x1000: txn(seq 2 len 0xd7 crc 0x306e389b)
0x1000:  op_dir_create db
0x1000:  op_dir_create db.wal
0x1000:  op_dir_create db.slow
0x1000:  op_file_update  file

Signed-off-by: Yang Honggang <joseph.yang@xtaotech.com>

Merge pull request #18553 from liewegas/wip-preempt-feature

osd/PG: remote recovery preemption, and new feature bit to condition it on

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #18594 from zmc/wip-cephmetrics

suites/cephmetrics: Add Centos 7

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #18601 from dillaman/wip-21961

rbd-mirror: clean up spurious error log messages

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

osd/PG: handle spurious remote revoke or toofull

If we restart the recovery reservation process for some reason (e.g.,
local preemption), we may still get a racing REVOKE from the replicas in
the previous round.  Ignore.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: journal debug message should not be logged as error

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: avoid attempting to decode an empty client data buffer

Fixes: http://tracker.ceph.com/issues/21961
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18604 from dillaman/wip-21960

cls/journal: ensure tags are properly expired

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

unittest_interval_set: test with flat_map<> too

Signed-off-by: Sage Weil <sage@redhat.com>

include/btree_interval_set: remove it!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use interval_set<...,btree_map<...>>

This avoid the code duplication, yay!

Signed-off-by: Sage Weil <sage@redhat.com>

include/btree_map: clean up wrapper header around btree_map

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BitMapAllocator: drop useless include

Signed-off-by: Sage Weil <sage@redhat.com>

include/interval_set: cbegin/cend -> begin/end

c{begin,end} don't exist for btree_map<>.

Signed-off-by: Sage Weil <sage@redhat.com>

include/interval_set: tolerate maps that invalidate iterator on change

These changes picked out of the diff between the original
btree_interval_set.h and interval_set.h (sadly I had it rolled into the
initial commit so it was tedious to identify these).

Signed-off-by: Sage Weil <sage@redhat.com>

include/interval_set: templating map used

Default to std::map for backward compat.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: handle RELEASE race with Remote*Reserved on replica

If we get a REQUEST and RELEASE in quick succession, we may process a
local Remote*Reserved back in the RepNotRecovering state.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18602 from adamemerson/wip-optional-failure

rgw: Fix dereference of empty optional

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

osd/PG: fix Remote*Preempted handling in ReplicaActive

This was only half-implemented by c1e652a4547d89b889954ca4bdb82506ca0f1d83

Signed-off-by: Sage Weil <sage@redhat.com>

cls/journal: ensure tags are properly expired

Previously, if only the local image was using the journal or if
a disconnected peer was attached, the tag entries could not be
expired even if unreferenced.

Fixes: http://tracker.ceph.com/issues/21960
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-bluestore-tool: the link target should not ending with new line

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

ceph-bluestore-tool: the right action is prime-osd-dir not prime-osd-dev

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #18592 from dillaman/wip-21956

cls/journal: fixed possible infinite loop in expire_tags

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18607 from yaozongyou/fix-typo

doc: fix typo e.g,. => e.g.,

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: fix typo e.g,. => e.g.,

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #18519 from liewegas/wip-21712-c

ceph_test_objectstore: do not change model for 0-length zero

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: send release to remote reservations in progress

If we have a REQUEST outstanding and then cancel, send a RELEASE to the
REQUEST target too, so that they too can clean up.  This avoids leaving
the replica in the RepRecovering state when the primary is restarting
the whole reservation process.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Fix dereference of empty optional

Due to the lack of a return, there was a case where an invalid ARN
could cause a dereference of an uninitialized boost::optional.

As a bit of defensive programming, restructure a couple functions to
make that kind of error impossible by ensuring the optional is only in
scope when it is initialized and relying less in early return on
error.

Fixes: http://tracker.ceph.com/issues/21962
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

MDSMonitor: wait for readable OSDMap before sanitizing

Fixes: http://tracker.ceph.com/issues/21945
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/rbd: run cls tests for all dependencies

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/journal: fixed possible infinite loop in expire_tags

Fixes: http://tracker.ceph.com/issues/21956
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18571 from linuxbox2/wip-rgwfile-21940

rgw_file:  set s->obj_size from bytes_written

Merge pull request #18449 from dzafman/wip-zafman-misc

mark_unfound_lost fix and some other minor changes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Use a symlink for Ubuntu fragment

Signed-off-by: Zack Cerza <zack@redhat.com>

suites/cephmetrics: Add Centos 7

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph-volume: set journal_uuid and journal_device when using a partition

This correctly sets the tags when a partition is used for a filestore
journal.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

docs for ceph-volume lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add tests for ceph-volume lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: print success message if zap succeeds

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: terminal.write only takes one argument

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: the metadata field for the lv path is lv_path not path

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: update help text for ceph-volume lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: remove lvm metadata when zapping an lv

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds Volume.clear_tags

Will remove all tags for the current lv

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: zap should leave lvs and partitions intact

This will remove filesystems and wipe data from any lv or partition
given to 'ceph-volume lvm zap' but still leave it intact for further
use.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds utilites to wipe the fs and clear data when zapping

These should eventually move to a disk api.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds tests for api.lvm.remove_lv

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: api.lvm.remove_lv should return True when successful

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: zap logical volumes

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a remove_lv command to the lvm api

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: stubs out the ceph-volume lvm zap command

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18402 from cbodley/wip-21831

rgw: fix for pause in beast frontend

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18328 from PCzhangPC/cplostmeta

librbd: cannot copy all image-metas if we have more than 64 key/value pairs

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18419 from majianpeng/librbd-bug-fix

librbd misc cleanup.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18542 from trociny/wip-metadata-open

librbd: don't read metadata twice on image open

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18561 from dillaman/wip-librbd-test-notify

test/librbd: added update_features RPC message to test_notify

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #18583 from idryomov/wip-qa-xfstests-except

qa/tasks/rbd: run all xfstests runs to completion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18557 from tchaikov/wip-cmake-cleanup

cmake: check for aligned_alloc() instead of checking tcmalloc version

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18551 from liewegas/wip-object-stat-decode

osd/osd_types: fix object_stat_sum_t decode

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #18565 from liewegas/wip-prime-fixes

ceph-bluestore-tool: prime-osd-dir: update symlinks instead of bailing

Reviewed-by: Alfredo Deza <adeza@redhat.com>

osd/PG: handle slow Remote*Preempted events

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: allow preemption of remote recovery reservation too

We did backfill but not recovery.  This one is easier because we already
have a DeferRecovery event we can queue that does everything.

Condition on the RECOVERY_RESERVATION_2 feature.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: condition remote backfill preemption on RECOVERY_RESERVATION_2

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: define RECOVERY_RESERVATION_2

This will allow us to backport support for reservation preemption and
cancellations on the wire.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18518 from ceph/wip-cv-fix-default-flags

ceph-volume: ensure correct --filestore/--bluestore behavior

Reviewed-by: Andrew Schoen <aschoen@redhat.com>