mds: fix encoded version of mds_info_t

Introduced-by: 1f5892f82046ebaac3b66dbd8e686825ebb1775c
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26734 into master

* refs/pull/26734/head:
doc: correct vstart.sh file location

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26682 from yuvalif/add_rgw_xml_unit_tests

Add rgw xml unit tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26680 from yuvalif/add_kill_9_to_mstop

mstop.sh allow kill -9 after failing to kill procs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #26704 into master

* refs/pull/26704/head:
msg/async, v2: drop alloc_aligned_buffer().
msg/async, v2: introduce frame late abort facility.
Revert "msg/async, v2: move ceph_msg_header2 to last frame segment."
msg, msg/async, v2: introduce late message abort facility.
msg/async, v2: failure of msg decode doesn't block throtlles.
msg/async, v2: move ceph_msg_header2 to last frame segment.
fixup: use frame epilogue for crc32 integrity checking.
msg/async, v2: epilogue size is variable in secure mode.
msg/async, v2: drop support for the buggy rx_buffers mechanism.
Revert "msg/async, v2: add flags field to frame's epilogue."
msg/async, v2: add flags field to frame's epilogue.
msg/async, v2: drop onwire_segment_t as epilogue had derogated it.
msg, msg/async, v2: drop crc fields from ceph_msg_header2.
msg/async, v2: use frame epilogue for crc32 integrity checking.
msg/async, v2: clean the ProtocolV2::{front,middle,data} up.
msg/async, v2: clean the ProtocolV2::epilogue up.
msg/async, v2: move crypto processing to segment reader.
msg/async, v2: handle epilogue separately from payload/data.
msg/async, v2: dissect decryption from SignedEncryptedFrame.
msg/async, v2: unify WAIT frames with other payload frames.
msg/async, v2: implement epilogue handling in secure mode.
msg/async, v2: message frames are pre-dispatched now.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26696 into master

* refs/pull/26696/head:
osdc/Objecter: preserve read-into-existing-buffer behavior
osdc/Objecter: don't use post_rx_buffer or revoke_rx_buffer at all
msg/Connection: disable {post,revoke}_rx_buffer
msg/async/ProtocolV1: disable rx_buffers behavior
msg/simple/Pipe: disable rx_buffer code

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #26727 into master

* refs/pull/26727/head:
qa/standalone/osd/pg-split-merge: fix import_after_merge_and_gap
osd: add 'send_beacon' tell/asok command(s)

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: correct vstart.sh file location

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

Merge pull request #26683 from dillaman/wip-38500

librbd: race condition possible when validating RBD pool

Reviewed-by: Mykola Golub <mgolub@suse.com>

msg/async, v2: drop alloc_aligned_buffer().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: introduce frame late abort facility.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #26698 into master

* refs/pull/26698/head:
common/str_map: better trim() impl using boost
common/str_map: fix trim() on empty string

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osdc/Objecter: preserve read-into-existing-buffer behavior

The libradosstriper::RadosStriperImpl::aio_read populates the target
outbl with a static buffer and relies on us reading into it.  This was
actually not reliable in the past (it could fail if the rx_buffers
optimization failed due to a retransmit or something else) but nevertheless
libradosstriper requires it to work *at all*.

Resolve this by modifying Objecter to copy the result into any provided
buffer at the lowest layer.  This should capture any other such user who
needed this behavior.

On the other hand, it will break any user who inadvertantly reads into a
non-empty bufferlist.  Given that any such user would already previously
have seen bad behavior due to the rx_buffers optimization, we expect
there to be 0 such instances.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/osd/pg-split-merge: fix import_after_merge_and_gap

This test introduces a map gap.  What *should* happen is that when there is
such a gap, we cannot import.  Previously, the test didn't reliably produce
a map gap at all, and didn't check that import failed--it verified that it
passed.

Fix the test so that it reliably produces a gap *and* reports
min_last_epoch_clean to the mon so we can trim.  Then verify we fail to
import, but can with --force.  But remove the pg again, because if we
force an import with a map gap the osd will refuse to start.

Fixes: http://tracker.ceph.com/issues/38525
Signed-off-by: Sage Weil <sage@redhat.com>

osd: add 'send_beacon' tell/asok command(s)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25538 from mkogan1/wip-rgw-reshard-status

rgw: change the "rgw admin status" 'num_shards' output to signed int

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add xml attributes test, override xml_handle_data

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #26668 from xiexingguo/wip-38493

msg/async/Protocol*: send keep alive if existing wins

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26723 from liewegas/wip-38537

mgr/BaseMgrModule: drop GIL for ceph_send_command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Revert "msg/async, v2: move ceph_msg_header2 to last frame segment."

This reverts commit 69dbc92ba8af844be0e3d3fc1b01287082753e56.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #26726 into master

* refs/pull/26726/head:
mds: Fix use of auto prototype, it is a non std extension

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26720 into master

* refs/pull/26720/head:
ceph-osd: infer objectstore type of 'type' file is missing

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

msg, msg/async, v2: introduce late message abort facility.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: failure of msg decode doesn't block throtlles.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #26705 from dzafman/wip-23999

Improve docs osd_recovery_priority, osd_recovery_op_priority and related

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26666 from badone/wip-kvstore_tool-ice

tools/ceph_kvstore_tool: Move summary output to print_summary

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

common, doc: Improve docs osd_recovery_priority, osd_recovery_op_priority and related

Add option desciptions for osd_recovery_priority and osd_recovery_op_priority

Fixes: https://tracker.ceph.com/issues/23999
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osd: infer objectstore type of 'type' file is missing

The current value of osd_objectstore should have no bearing--we should
rely on the state in front of us.

- If there is a directory called current/, assume filestore.
- If there is a symlink called block, assume bluestore.
- If we see none of those, fail.

Fixes: http://tracker.ceph.com/issues/38517
Signed-off-by: Sage Weil <sage@redhat.com>

msg/async, v2: move ceph_msg_header2 to last frame segment.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #26692 into master

* refs/pull/26692/head:
test/test_ipaddr: added test for pick_address when ipv4 and ipv6 are enabled
common/pick_address: improves log message when no IP is found
pick_address: error out in dual stack mode if both addr types cannot be provided

Reviewed-by: Sage Weil <sage@redhat.com>

mds: Fix use of auto prototype, it is a non std extension

Clang complains:
In file included from /home/jenkins/workspace/ceph-master/src/msg/Message.cc:128:
In file included from /home/jenkins/workspace/ceph-master/src/messages/MFSMap.h:20:
/home/jenkins/workspace/ceph-master/src/mds/FSMap.h:330:49: error: 'auto' not allowed in function prototype
  void modify_filesystem(fs_cluster_id_t fscid, auto&& fn)
                                                ^~~~
/home/jenkins/workspace/ceph-master/src/mds/FSMap.h:341:37: error: 'auto' not allowed in function prototype
  void modify_daemon(mds_gid_t who, auto&& fn)
                                    ^~~~

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

osdc/Objecter: don't use post_rx_buffer or revoke_rx_buffer at all

Signed-off-by: Sage Weil <sage@redhat.com>

fixup: use frame epilogue for crc32 integrity checking.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: epilogue size is variable in secure mode.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop support for the buggy rx_buffers mechanism.

See:
  * https://github.com/ceph/ceph/pull/26696,
  * http://tracker.ceph.com/issues/22480.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/BaseMgrModule: drop GIL for ceph_send_command

Otherwise, we can easily deadlock, since other bits of the code hold the
objecter lock and then take the GIL.

Fixes: http://tracker.ceph.com/issues/38537
Signed-off-by: Sage Weil <sage@redhat.com>

Revert "msg/async, v2: add flags field to frame's epilogue."

This reverts commit e18d65ae9e7113831480f5c3efad015e83bc1200.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #26688 from yangdongsheng/librbd_fix_all_throttled

librbd: fix were_all_throttled() to avoid incorrect ret-value

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #26659 into master

* refs/pull/26659/head:
qa: remove libcephfs-java tests

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #26695 from smithfarm/wip-make-check-amqp-endpoint

do_cmake.sh: SUSE builds need WITH_RADOSGW_AMQP_ENDPOINT=OFF

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #26699 into master

* refs/pull/26699/head:
qa: unmount clients before deleting fs

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #26700 into master

* refs/pull/26700/head:
qa: bump fsstress timeout to 6h

Reviewed-by: Zheng Yan <zyan@redhat.com>

librbd: fix were_all_throttled() to avoid incorrect ret-value

Fixes: http://tracker.ceph.com/issues/38504
Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: pool_init API now validates the pool

This offloads the validation process from the image creation
state machine under normal conditions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: create state machine uses new validate pool state machine

Fixes: http://tracker.ceph.com/issues/38500
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: separate pool validation into a standalone state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26482 from votdev/show_success_notification_rgw

mgr/dashboard: Show success notification in RGW forms

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26507 from votdev/yesno_pipe

mgr/dashboard: Introduce pipe to convert bool to text

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Introduce pipe to convert bool to text

Convert boolean values to 'Yes' or 'No' or any other value.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #26714 from cyx1231st/wip-async-bannercheck

msg/async: fix banner_v1 check in ProtocolV2

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26626 from LenzGr/master-documentation

doc: Updated Ceph Dashboard documentation

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26711 from cyx1231st/wip-async-frame-header

msg/async: fix include in frames_v2.h

Reviewed-by: Ricardo Dias <rdias@suse.com>

msg/async: fix banner_v1 check in ProtocolV2

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

common/str_map: better trim() impl using boost

Thanks Kefu!

Signed-off-by: Sage Weil <sage@redhat.com>

common/str_map: fix trim() on empty string

If was pass an empty string (e.g., with something like
get_str_map("M= T= P=")) we end up with a (size_t)-1 for end.

Fixes: http://tracker.ceph.com/issues/38329
Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: fix include in frames_v2.h

`ceph_clock_now()` needs to include common/Clock.h.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

Merge PR #26638 into master

* refs/pull/26638/head:
doc: update documentation for standby-replay
qa: update discontinous map test to use mds freezing
mon: add freeze MDS command
qa: update testing for standby-replay
mon: add setting for fs to enable standby-replay
ceph-mds: obsolete hot-standby option
fs: obsolete standby_for config options
messages/MMDSBeacon: use inline init
mds: avoid unnecessary copy of entity_addrvec_t
mds: use inline init for mds_info_t
mds: use rank from MDSMap always
mds: remove obsolete comment
qa: use SIGTERM when stopping vstart service

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

msg/async, v2: add flags field to frame's epilogue.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #26605 from xiexingguo/wip-misc-cleanups

osd/OSDMap: drop local pool filter in calc_pg_upmaps

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async, v2: drop onwire_segment_t as epilogue had derogated it.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg, msg/async, v2: drop crc fields from ceph_msg_header2.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: use frame epilogue for crc32 integrity checking.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async/ProtocolV2: send_keepalive under the scope of lock

Dropping lock is dangerous and could be race-prone.
Any time the lock is dropped we need to reverify the connection state.

send_keepalive under the scope of lock should be just fine since
it does not introduce any extra lock dependencies.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

msg/async/Protocol*: send keep alive if existing wins

This is a follow-up fix of https://github.com/ceph/ceph/pull/25754.

It turns out the existing connection would win the connection race
and then be stuck in the __replacing__ stage forever without being
aware of that the underlying Pipe is actually broken.

Fix by forcing existing sending keepalive periodically __too__.

Fixes: http://tracker.ceph.com/issues/38493
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26614 into master

* refs/pull/26614/head:
osd: Implement lazy omap usage statistics

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26453 into master

* refs/pull/26453/head:
rpm: build with lttng on ppc64le (SUSE)
rpm: no lttng build on openSUSE s390x

Reviewed-by: Tim Serong <tserong@suse.com>

Merge PR #26667 into master

* refs/pull/26667/head:
cmake: link rgw tests with ${rgw_libs}
cmake: link ${rgw_libs} not rgw_a and rabbitmq
cmake: move rgw tests targets down to end

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #26681 into master

* refs/pull/26681/head:
mon/ConfigMonitor: add 'config ls'

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #26693 from trociny/wip-rbd-bench-interrupt

rbd: close image when bench is interrupted

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26639 from cbodley/wip-38479

rgw: data sync drains lease stack on lease failure

qa: remove libcephfs-java tests

These have bit-rotted and no longer work. No cycles from interested parties
available to fix.

Fixes: https://tracker.ceph.com/issues/38487
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/async, v2: clean the ProtocolV2::{front,middle,data} up.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: clean the ProtocolV2::epilogue up.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: move crypto processing to segment reader.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: handle epilogue separately from payload/data.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: dissect decryption from SignedEncryptedFrame.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: unify WAIT frames with other payload frames.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: implement epilogue handling in secure mode.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: message frames are pre-dispatched now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: Update some of the priority item descriptions

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Minor fixes to monitoring-osd-pg.rst

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Remove osd disk thread items that no longer exist

Caused by: 35a4b5072f36df92391b7514eafdea498e42f1dd

Signed-off-by: David Zafman <dzafman@redhat.com>

common: Fix typo in description

Signed-off-by: David Zafman <dzafman@redhat.com>

msg/Connection: disable {post,revoke}_rx_buffer

Broken by design.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV1: disable rx_buffers behavior

This is broken by design.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple/Pipe: disable rx_buffer code

This reproducibly crashes. Although we don't have a complete understanding
of the exact sequence leading to the crash, we can reproduce, and we have
multiple theoretical cases where it appears to be broken by design.

Fixes: http://tracker.ceph.com/issues/22480
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26685 from ceph/wip-testinfra-ceph-volume

ceph-volume: use our own testinfra suite for functional testing

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #26650 from LenzGr/typo-fix

mgr/dashboard: Fixed typos in environment.build.js

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26588 from votdev/relocate_empty_pipe

mgr/dashboard: Relocate empty pipe

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26447 from votdev/fix_notify_show

mgr/dashboard: Remove param when calling notificationService.show

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

qa: bump fsstress timeout to 6h

When run with valgrind, it takes a significant amount of time to complete.

Fixes: http://tracker.ceph.com/issues/38520
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: unmount clients before deleting fs

Client unmount during test cleanup will hang if the file system was deleted.

Fixes: http://tracker.ceph.com/issues/38518
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/test_ipaddr: added test for pick_address when ipv4 and ipv6 are enabled

Signed-off-by: Ricardo Dias <rdias@suse.com>

common/pick_address: improves log message when no IP is found

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Fixed typo in environment.build.js

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

pick_address: error out in dual stack mode if both addr types cannot be provided

When ms_bind_ipv4 and ms_bind_ipv6 options are enabled, if the
public_network or cluster_network settings only contain networks of a
single type (either IPv4 or IPv6) then pick_addresses function must
fail.

Fixes: http://tracker.ceph.com/issues/38307
Signed-off-by: Ricardo Dias <rdias@suse.com>

doc: Updated Ceph Dashboard documentation

Renamed "Ceph Manager Dashboard" to "Ceph Dashboard" in
various locations, replaced "plugin" with "module".
Overhauled the feature list, added more references to configuration
instructions and related components.
Fixed capitalization of subchapters.
Added note about mixed content blocking to the Grafana section.
Removed duplicate content from the dashboard's `README.rst`,
moved some parts (supported browser list) into the documentation
instead.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

do_cmake.sh: SUSE builds need WITH_RADOSGW_AMQP_ENDPOINT=OFF

Without this patch, "make check" fails on SUSE systems because
WITH_RADOS_AMQP_ENDPOINT defaults to "ON".

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #26630 from sebastian-philipp/ssh-simple-type-error

mgr/ssh: fix type and doc errors

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

rbd: close image when bench is interrupted

Previously when `rbd bench` was interrupted it left the lock and
watcher and one had to wait for them to expire before reusing or
removing the image.

As a bonus now it prints the benchmark summary on exit even if it
is interrupted.

Signed-off-by: Mykola Golub <mgolub@suse.com>