test/pybind/test_rados.py: collect output in stdout for "bench" cmd

it was changed from stderr to stdout in 23583ccc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21804 from dillaman/wip-24000

osdc: self-managed snapshot helper should catch decode exception

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #21929 from majianpeng/fio-ceph-objectstore

test/fio: add option single_pool_mode in ceph-bluestore.fio

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #21905 from jcsp/wip-24022

osd: write "bench" output to stdout

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21950 from liewegas/wip-23899

osd: clean up smart probe

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21940 from dmick/wip-remove-cephd

cmake: Remove embedded 'cephd' code

Reviewed-by: Bassam Tabbara <Bassam.Tabbara@Quantum.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #21615 into master

* refs/pull/21615/head:
ceph-fuse: Delete inode's bufferhead was in Tx state would lead a assert fail

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21769 into master

* refs/pull/21769/head:
mds: properly reconnect client caps after loading inodes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21883 into master

* refs/pull/21883/head:
mds: tighten conditions of calling rejoin_gather_finish()
mds: avoid calling rejoin_gather_finish() two times successively

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21923 into master

* refs/pull/21923/head:
mds: set could_consume to false when no purge queue item actually executed

Reviewed-by: Zheng Yan <zyan@redhat.com>

osd: clean up smart probe code a bit

Signed-off-by: Sage Weil <sage@redhat.com>

Remove embedded 'cephd' code

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #21942 from tchaikov/wip-qa-ceph-disk-no-mon-config

qa/workunits/ceph-disk: --no-mon-config

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21945 from tchaikov/wip-dashboard-errno

mgr/dashboard: set errno via the parent class

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21947 from yuriw/wip-yuriw-crontab

qa/tests: added client-upgrade-luminous suit

qa/tests: added client-upgrade-luminous suit

Fixes https://tracker.ceph.com/issues/24021
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: set errno via the parent class

update the test accordingly. see also rados.Error.__str__()

Signed-off-by: Ricardo Dias <rdias@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21524 from dillaman/wip-23517

test/librados_test_stub: deterministically load cls shared libraries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21415 from ErwanAliasr1/evelu-cephdisk

ceph-disk: get_partition_dev() should fail until get_dev_path(partnam…

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/ceph-disk: --no-mon-config

port a095ec8b to the qa/workunits

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: fix osd smart data collection segment fault issue.
output.read_fd() could return zero length. it needs to
be handled gracefully.

always call invoke join() if spawn() is successfully returned.
otherwise there will be assert failure in ~SubProcess().

Fixes: http://tracker.ceph.com/issues/23899
Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

Merge pull request #21292 from zy751713126/snapshot_count

rbd: add snapshot count in rbd info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd/info: add snapshot count in rbd info

Signed-off-by: Zheng Yin <zhengyin@huayun.com>

test/fio: add option single_pool_mode in ceph-bluestore.fio

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ceph_release: nautilus

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/mimic'

ceph_release: we are an rc now

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21930 from dillaman/wip-rbd-backports-mimic

rbd: backport of mimic bug fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21893 from cbodley/wip-rgw-req-state-time

rgw: use coarse_real_clock for req_state::time

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #21931 from yuriw/wip-yuriw-fix-mimic-subset

tests/qa - fix mimic subset for nightlies

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests/qa - fix mimic subset for nightlies

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21680 from cbodley/wip-rm-replica-log

rgw: remove all traces of cls replica_log

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Revert "stop.sh: use --no-mon-config when trying to unmap rbd devices"

This reverts commit 6c722cd5f3f829ccf14280e7e36387bafd819c7a.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3c6b01574766ca128818461d6630866cc0c0405b)

rbd: don't load config overrides from monitor initially

The overrides will be loaded when we connect to the cluster via librados
and the current approach prevents running 'rbd help' without a running
cluster.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 46eafdabde57736d2a8b63f55cab94461885a957)

rbd-mirror: ensure remote demotion is replayed locally

The bootstrap process cannot immediately quit if it notices the remote
image is not primary. Instead, it needs to continue if the local image is
still chained to the remote.

Fixes: http://tracker.ceph.com/issues/24009
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d35bc94319c6fa267e973e533f4591568289ee96)

rbd-mirror: fix potential race between image sync and shut down

Fixes: http://tracker.ceph.com/issues/24008
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4f2dbcefc5e3547926b8d34135a808266bd059b8)

rbd-mirror: ensure instance replayer is stoped before destructing

Fixes: http://tracker.ceph.com/issues/24008
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9ba94f303072374b5c7c9265c3a493fddce2f8ca)

Merge pull request #21831 from tchaikov/wip-23966

osdc: invoke notify finish context on linger commit failure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21898 from adamemerson/wip-fedora28

src: Eliminate new warnings in Fedora 28

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21861 from adamemerson/wip-clangtasm

osd,mon,pybind: Make able to compile with Clang

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21912 from dillaman/wip-24064

rbd: disk-usage can now optionally compute exact on-disk usage

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21910 from dillaman/wip-rbd-help

rbd: don't load config overrides from monitor initially

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21823 from dillaman/wip-24009

rbd-mirror: ensure remote demotion is replayed locally

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21817 from dillaman/wip-24008

rbd-mirror: fixed potential crashes during shut down

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21853 from rishabh-d-dave/volclient-keyargs-for-all-exceptions

rados.pyx: make all exceptions accept keyword arguments

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21886 from jcsp/wip-no-install-hello

cmake: do not install `hello` demo module

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21782 from wido/mgr-telegraf

mgr/telegraf: Telegraf module for Ceph Mgr

Reveiwed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: set could_consume to false when no purge queue item actually executed

Fixes: http://tracker.ceph.com/issues/24073
Signed-off-by: Xuehan Xu <xuxuehan@360.cn>

Merge pull request #21918 from smithfarm/wip-24003-mimic

script: build-integration-branch: avoid Unicode error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21891 from cbodley/wip-cmake-crypto-plugins

cmake: move crypto_plugins target

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Revert "stop.sh: use --no-mon-config when trying to unmap rbd devices"

This reverts commit 6c722cd5f3f829ccf14280e7e36387bafd819c7a.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21770 from zy751713126/remove-redundant-code

rbd: add the judgment of resizing the image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21914 from cbodley/wip-common-async

common: add a generic async Completion for use with boost::asio

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

script: build-integration-branch: avoid Unicode error

When run on PRs that have non-ASCII characters in their titles, the script
fails like this when run in a non UTF-8 environment:

UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2026' in
position 651: ordinal not in range(256)

Since failing like that is not useful, avoid the error by forcing ASCII
encoding and filtering out any non-ASCII characters.

Fixes: http://tracker.ceph.com/issues/24003
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 6dd31fdbec43e37bece24c0de886d2c4083901c8)

Merge pull request #21909 from neha-ojha/wip-fix-powercycle

qa/suites: set osd_pg_log_dups_tracked in cfuse_workunit_suites_fsync.yaml

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: Silence sign comparison warnings

Since we're iterating with a size_t, we may as well compare against one.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: add an async SharedMutex

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: forward_handler() for handlers with move-only args

executors will always call the handler's lvalue overload of operator(),
so forward_handler() is required to wrap calls to bind_handler() with
move-only argument types. Completion uses this to forward all of its
CompletionHandlers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add a generic async Completion interface

a generic Completion for storing asio completion handlers for deferred
execution

Signed-off-by: Casey Bodley <cbodley@redhat.com>

build: Remove -Wno-invalid-offsetof

Since C++17 relaxed the rules for using offsetof, this is no longer needed.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

tools: Do not memset uuid_d

It is perfectly able to default initialize itself.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/rbd/fsx: Disable overflow-format warning

No return type or other way to react to it.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/libcephfs: Expand buffer slightly

So the snprintf output isn't truncated.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: There is no point in const return values

Pointer-to-const or reference-to-const yes. Const value, no.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Catch exceptions at const references

/Never/ by value.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove extra 'const' from cast

Casting to pointer-to-const makes sense. Casting to a const
value (even a const pointer) doesn't.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rdma: Silence qualifier warnings

No point in reinterpret_casting something const anyway, you can always
assign non-const to const.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

msg/rdma/infiniband: Silence memset warning

Though it may be better in the long-run to replace the malloc-like
interface with an Allocator type interface.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Exceptions should be caught by const reference

/Not/ by value.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

FuseStore: Expand buffers to prevent overflow

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

perfglue: Silence truncation warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mount: Enlarge buffer to fix overflow warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mon: Remove unused variable

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

blkdev: Check return of snprintf and error on truncation

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

blkdev: Get rid of get_device_by_uuid

This function's interface is bad, in that it takes char*s with no
sizes. Also nobody calls it.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Enlarge a few buffers

To get rid of overflow warnings.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mds: Do not memset fnode_t

fnode_t can initialize itself.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

auto: Catch exceptions by const reference

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

client: Do not memset frag_info_t and inode_t

They do a perfectly good job of initializing themselves.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Increase buffer to prevent possible truncation

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix uninitialized value warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

doc: fix iscsi target name when configuring target

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 3b2d739e4c12120d27fdaf6b0af0e0de5bb972f3)

Merge pull request #21906 from vshankar/iscsi-target-cli-typo

doc: fix iscsi target name when configuring target

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: disk-usage can now optionally compute exact on-disk usage

Fixes: http://tracker.ceph.com/issues/24064
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: don't load config overrides from monitor initially

The overrides will be loaded when we connect to the cluster via librados
and the current approach prevents running 'rbd help' without a running
cluster.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20974 from XiangxiangMeng/rgw-policy-keyword-modify

rgw: policy: modify some operation permission keyword

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mon: pool-ops against non-existent pools should return error

Fixes: http://tracker.ceph.com/issues/24000
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc: self-managed snapshot helper should catch decode exception

Fixes: http://tracker.ceph.com/issues/24000
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: fix iscsi target name when configuring target

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #21788 from cbodley/wip-22914

rgw: bucket sync status improvements, part 1

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #21855 from votdev/fix_rgw_backend_mimic

mimic: mgr/dashboard: Refactor RGW backend

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #21851 from votdev/improve_error_panel

mgr/dashboard: Improve error panel

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

ceph-disk: get_partition_dev() should fail until get_dev_path(partname) is populated

get_partition_dev() is a function called to get a block device path.

The @retry decorator implements a timeout mecanism as it exist a race
between the partition creation command and the actual presence of the
partition.

This function have a semantic issue.
The function is asked if /dev/sda6 exists but check if /sys/block/sda/sda6 exists.
Once /sys/block/sda/sda6 is populated within the timeout, the function returns "/dev/sda6".
But this doesn't check if "/dev/sda6" really exists, so a function
using this value fall into a race condition and fails too.

We do have this case inside the CI where timing are very variable
regarding the workload and VM's performance and even sometimes nasted VMs.

This patch offer to double-check before returning that the actual /dev/
entry actually exist once the BLOCKDIR (/sys) is populated. If not, the
@retry is triggered to let some time resolution this temporary situation.

If this state is permanent, an explicit error message is reported like :
    /dev/sdb1 is not populated while /sys/block/sdb has it

Signed-off-by: Erwan Velu <erwan@redhat.com>

osd: write "bench" output to stdout

...not stderr.  Seems more reasonable as this
command is essentially feeding back a measurement result,
rather than just commenting on its status

Fixes: http://tracker.ceph.com/issues/24022
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #21835 from tchaikov/mimic-23921

mimic: crush, osd: handle multiple parents properly when applying pg upmaps

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/telegraf: Telegraf module for Ceph Mgr

Telegraf is a agent for collecting and reporting metrics.

It has multiple inputs and can send data to various outputs like
for example InfluxDB or ElasticSearch.

This module works by using the socket_listener of Telegraf and can
send data over UDP, TCP and a local Unix Socket.

Signed-off-by: Wido den Hollander <wido@42on.com>

rbd-mirror: ensure remote demotion is replayed locally

The bootstrap process cannot immediately quit if it notices the remote
image is not primary. Instead, it needs to continue if the local image is
still chained to the remote.

Fixes: http://tracker.ceph.com/issues/24009
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cmake: move crypto_plugins target

the crypto_plugins target was defined in
src/crypto/isa-l/CMakeLists.txt, but this is only included
if(HAVE_INTEL AND HAVE_BETTER_YASM_ELF64 AND (NOT APPLE))

moving it out of the if() block allows the os target to depend on it
even if no plugins are built

Signed-off-by: Casey Bodley <cbodley@redhat.com>

osdc: guard op->on_notify_finish with lock

Fixes: http://tracker.ceph.com/issues/23966
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c16371965cc785f2e2b87d8c6684854fa2de65e1)

Merge pull request #21834 from tchaikov/mimic-23966

mimic: osdc: guard op->on_notify_finish with lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>