rgw admin: implement a lc fix option

An radosgw-admin lc fix --bucket <> option is added which checks if the bucket
entry exists in the corresponding lc shard and creates it if not. In case of
resharded buckets not running a fixed rgw that writes/compares the marker this
would write a new entry with the marker as the old entry would've already been
deleted by a LC process. We currently don't cleanup the stale entry as it is
assumed this would be picked up by the LC processor already or would be picked
up in the next cycle.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cls rgw: implement a method to get a single LC entry

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cls_rgw: alias the LC entries as  rgw_lc_entry_t

instead of a naked std::pair<std::string, int> everywhere

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cls_rgw: implement a read_omap_entry method

Also refactor other methods that just read a single omap entry to use this
method instead

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw lc: use marker for the shard id

Since buckets can undergo resharding which changes the bucket id, using the
bucket marker in the shard id can help prevent the need to rewrite the entry as
the buckets get resharded. This also helps detect the exit criteria when the
bucket gets deleted.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: drop entries only if the markers do not match.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #26683 from dillaman/wip-38500

librbd: race condition possible when validating RBD pool

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #26698 into master

* refs/pull/26698/head:
common/str_map: better trim() impl using boost
common/str_map: fix trim() on empty string

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25538 from mkogan1/wip-rgw-reshard-status

rgw: change the "rgw admin status" 'num_shards' output to signed int

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #26668 from xiexingguo/wip-38493

msg/async/Protocol*: send keep alive if existing wins

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26723 from liewegas/wip-38537

mgr/BaseMgrModule: drop GIL for ceph_send_command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26726 into master

* refs/pull/26726/head:
mds: Fix use of auto prototype, it is a non std extension

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26720 into master

* refs/pull/26720/head:
ceph-osd: infer objectstore type of 'type' file is missing

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26705 from dzafman/wip-23999

Improve docs osd_recovery_priority, osd_recovery_op_priority and related

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26666 from badone/wip-kvstore_tool-ice

tools/ceph_kvstore_tool: Move summary output to print_summary

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

common, doc: Improve docs osd_recovery_priority, osd_recovery_op_priority and related

Add option desciptions for osd_recovery_priority and osd_recovery_op_priority

Fixes: https://tracker.ceph.com/issues/23999
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osd: infer objectstore type of 'type' file is missing

The current value of osd_objectstore should have no bearing--we should
rely on the state in front of us.

- If there is a directory called current/, assume filestore.
- If there is a symlink called block, assume bluestore.
- If we see none of those, fail.

Fixes: http://tracker.ceph.com/issues/38517
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26692 into master

* refs/pull/26692/head:
test/test_ipaddr: added test for pick_address when ipv4 and ipv6 are enabled
common/pick_address: improves log message when no IP is found
pick_address: error out in dual stack mode if both addr types cannot be provided

Reviewed-by: Sage Weil <sage@redhat.com>

mds: Fix use of auto prototype, it is a non std extension

Clang complains:
In file included from /home/jenkins/workspace/ceph-master/src/msg/Message.cc:128:
In file included from /home/jenkins/workspace/ceph-master/src/messages/MFSMap.h:20:
/home/jenkins/workspace/ceph-master/src/mds/FSMap.h:330:49: error: 'auto' not allowed in function prototype
  void modify_filesystem(fs_cluster_id_t fscid, auto&& fn)
                                                ^~~~
/home/jenkins/workspace/ceph-master/src/mds/FSMap.h:341:37: error: 'auto' not allowed in function prototype
  void modify_daemon(mds_gid_t who, auto&& fn)
                                    ^~~~

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/BaseMgrModule: drop GIL for ceph_send_command

Otherwise, we can easily deadlock, since other bits of the code hold the
objecter lock and then take the GIL.

Fixes: http://tracker.ceph.com/issues/38537
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26688 from yangdongsheng/librbd_fix_all_throttled

librbd: fix were_all_throttled() to avoid incorrect ret-value

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #26659 into master

* refs/pull/26659/head:
qa: remove libcephfs-java tests

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #26695 from smithfarm/wip-make-check-amqp-endpoint

do_cmake.sh: SUSE builds need WITH_RADOSGW_AMQP_ENDPOINT=OFF

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #26699 into master

* refs/pull/26699/head:
qa: unmount clients before deleting fs

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #26700 into master

* refs/pull/26700/head:
qa: bump fsstress timeout to 6h

Reviewed-by: Zheng Yan <zyan@redhat.com>

librbd: fix were_all_throttled() to avoid incorrect ret-value

Fixes: http://tracker.ceph.com/issues/38504
Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: pool_init API now validates the pool

This offloads the validation process from the image creation
state machine under normal conditions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: create state machine uses new validate pool state machine

Fixes: http://tracker.ceph.com/issues/38500
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: separate pool validation into a standalone state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26482 from votdev/show_success_notification_rgw

mgr/dashboard: Show success notification in RGW forms

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26507 from votdev/yesno_pipe

mgr/dashboard: Introduce pipe to convert bool to text

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Introduce pipe to convert bool to text

Convert boolean values to 'Yes' or 'No' or any other value.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #26714 from cyx1231st/wip-async-bannercheck

msg/async: fix banner_v1 check in ProtocolV2

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26626 from LenzGr/master-documentation

doc: Updated Ceph Dashboard documentation

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26711 from cyx1231st/wip-async-frame-header

msg/async: fix include in frames_v2.h

Reviewed-by: Ricardo Dias <rdias@suse.com>

msg/async: fix banner_v1 check in ProtocolV2

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

common/str_map: better trim() impl using boost

Thanks Kefu!

Signed-off-by: Sage Weil <sage@redhat.com>

common/str_map: fix trim() on empty string

If was pass an empty string (e.g., with something like
get_str_map("M= T= P=")) we end up with a (size_t)-1 for end.

Fixes: http://tracker.ceph.com/issues/38329
Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: fix include in frames_v2.h

`ceph_clock_now()` needs to include common/Clock.h.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

Merge PR #26638 into master

* refs/pull/26638/head:
doc: update documentation for standby-replay
qa: update discontinous map test to use mds freezing
mon: add freeze MDS command
qa: update testing for standby-replay
mon: add setting for fs to enable standby-replay
ceph-mds: obsolete hot-standby option
fs: obsolete standby_for config options
messages/MMDSBeacon: use inline init
mds: avoid unnecessary copy of entity_addrvec_t
mds: use inline init for mds_info_t
mds: use rank from MDSMap always
mds: remove obsolete comment
qa: use SIGTERM when stopping vstart service

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #26605 from xiexingguo/wip-misc-cleanups

osd/OSDMap: drop local pool filter in calc_pg_upmaps

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV2: send_keepalive under the scope of lock

Dropping lock is dangerous and could be race-prone.
Any time the lock is dropped we need to reverify the connection state.

send_keepalive under the scope of lock should be just fine since
it does not introduce any extra lock dependencies.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

msg/async/Protocol*: send keep alive if existing wins

This is a follow-up fix of https://github.com/ceph/ceph/pull/25754.

It turns out the existing connection would win the connection race
and then be stuck in the __replacing__ stage forever without being
aware of that the underlying Pipe is actually broken.

Fix by forcing existing sending keepalive periodically __too__.

Fixes: http://tracker.ceph.com/issues/38493
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26614 into master

* refs/pull/26614/head:
osd: Implement lazy omap usage statistics

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26453 into master

* refs/pull/26453/head:
rpm: build with lttng on ppc64le (SUSE)
rpm: no lttng build on openSUSE s390x

Reviewed-by: Tim Serong <tserong@suse.com>

Merge PR #26667 into master

* refs/pull/26667/head:
cmake: link rgw tests with ${rgw_libs}
cmake: link ${rgw_libs} not rgw_a and rabbitmq
cmake: move rgw tests targets down to end

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #26681 into master

* refs/pull/26681/head:
mon/ConfigMonitor: add 'config ls'

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #26693 from trociny/wip-rbd-bench-interrupt

rbd: close image when bench is interrupted

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26639 from cbodley/wip-38479

rgw: data sync drains lease stack on lease failure

qa: remove libcephfs-java tests

These have bit-rotted and no longer work. No cycles from interested parties
available to fix.

Fixes: https://tracker.ceph.com/issues/38487
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: Update some of the priority item descriptions

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Minor fixes to monitoring-osd-pg.rst

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Remove osd disk thread items that no longer exist

Caused by: 35a4b5072f36df92391b7514eafdea498e42f1dd

Signed-off-by: David Zafman <dzafman@redhat.com>

common: Fix typo in description

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #26685 from ceph/wip-testinfra-ceph-volume

ceph-volume: use our own testinfra suite for functional testing

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #26650 from LenzGr/typo-fix

mgr/dashboard: Fixed typos in environment.build.js

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26588 from votdev/relocate_empty_pipe

mgr/dashboard: Relocate empty pipe

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26447 from votdev/fix_notify_show

mgr/dashboard: Remove param when calling notificationService.show

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

qa: bump fsstress timeout to 6h

When run with valgrind, it takes a significant amount of time to complete.

Fixes: http://tracker.ceph.com/issues/38520
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: unmount clients before deleting fs

Client unmount during test cleanup will hang if the file system was deleted.

Fixes: http://tracker.ceph.com/issues/38518
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/test_ipaddr: added test for pick_address when ipv4 and ipv6 are enabled

Signed-off-by: Ricardo Dias <rdias@suse.com>

common/pick_address: improves log message when no IP is found

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Fixed typo in environment.build.js

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

pick_address: error out in dual stack mode if both addr types cannot be provided

When ms_bind_ipv4 and ms_bind_ipv6 options are enabled, if the
public_network or cluster_network settings only contain networks of a
single type (either IPv4 or IPv6) then pick_addresses function must
fail.

Fixes: http://tracker.ceph.com/issues/38307
Signed-off-by: Ricardo Dias <rdias@suse.com>

doc: Updated Ceph Dashboard documentation

Renamed "Ceph Manager Dashboard" to "Ceph Dashboard" in
various locations, replaced "plugin" with "module".
Overhauled the feature list, added more references to configuration
instructions and related components.
Fixed capitalization of subchapters.
Added note about mixed content blocking to the Grafana section.
Removed duplicate content from the dashboard's `README.rst`,
moved some parts (supported browser list) into the documentation
instead.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

do_cmake.sh: SUSE builds need WITH_RADOSGW_AMQP_ENDPOINT=OFF

Without this patch, "make check" fails on SUSE systems because
WITH_RADOS_AMQP_ENDPOINT defaults to "ON".

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #26630 from sebastian-philipp/ssh-simple-type-error

mgr/ssh: fix type and doc errors

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

rbd: close image when bench is interrupted

Previously when `rbd bench` was interrupted it left the lock and
watcher and one had to wait for them to expire before reusing or
removing the image.

As a bonus now it prints the benchmark summary on exit even if it
is interrupted.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26596 from smithfarm/wip-release-notes-12_2_11

doc: add missing 12.2.11 release note

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: update documentation for standby-replay

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: update discontinous map test to use mds freezing

and stop using standby_for_*.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon: add freeze MDS command

This is a new hidden command that allows us to do certain testing for race
conditions. A frozen MDS cannot change change state or be replaced by a
standby.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: update testing for standby-replay

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon: add setting for fs to enable standby-replay

Operators now simply mark a file system as enabling standby-replay via

    ceph fs set <name> allow_standby_replay true

The MDSMonitor will assign available standbys to take on this role.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-mds: obsolete hot-standby option

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

fs: obsolete standby_for config options

The operator can no longer configure which rank/fscid/name an MDS wants to
follow or standby for. This was an unfortunately confusing set of config
options as ultimately the MDSMonitor (by default) would set a standby to follow
any fscid/rank if no standby is available that explicitly follows the failed
rank. It is suggested that operators instead use the `standby_count_wanted`
setting on each fs to ensure that sufficient standbys are available.

The temporary effect of this commit is that the MDSMonitor no longer assigns
any standby to standby-replay, to be fixed in the following commits.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

messages/MMDSBeacon: use inline init

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: avoid unnecessary copy of entity_addrvec_t

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use inline init for mds_info_t

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use rank from MDSMap always

This old bit of code was using standby_for_rank to determine what rank it
should follow but, for the last few releases, the Monitor always sets the the
rank to follow.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove obsolete comment

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use SIGTERM when stopping vstart service

Otherwise MDS will not send down:dne to mons when restarting.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26663 into master

* refs/pull/26663/head:
doc: change ruleset to id in crush map file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26595 into master

* refs/pull/26595/head:
osd/PG: log tracked pg refs under debug_refs
osd/PG: include tag in get/put debug lines
osd: clean up oncommit contexts in _process shutdown

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26637 into master

* refs/pull/26637/head:
common/ceph_context: experimental kludge for crimson
auth/AuthRegistry: mark 'secure' con mode experimental for now
msg/async: set ms_*_mode to crc only (no secure)

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26640 into master

* refs/pull/26640/head:
mon/OSDMonitor: remove old v1 addrs when converting blacklist to new TYPE_ANY form

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26645 into master

* refs/pull/26645/head:
rpm: use %_python_buildid to specify python-remoto dep

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26678 into master

* refs/pull/26678/head:
mon: handle racing ms_handle_reset and handle_auth_request

Reviewed-by: Ricardo Dias <rdias@suse.com>

ceph-volume: use our own testinfra suite for functional testing

This allows us to run a smaller subset of tests from the ones provided
by ceph-ansible and test only the things that pertain to ceph-volume.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #26648 into master

* refs/pull/26648/head:
msg/async: msgr2: cleanup frame classes hacks

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #26523 from dillaman/wip-38387

librbd: add missing shutdown states to managed lock helper

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #26598 into master

* refs/pull/26598/head:
doc/releases/nautilus: dashboard tweaks
doc/releases/nautilus: more rados highlights
doc/release/nautilus: added RBD major features
doc/releases/nautilus: add msgr2 enablement steps
doc/releases/nautilus: add dashboard items

Reviewed-by: Kai Wagner <kwagner@suse.com>

Merge pull request #26570 from jtlayton/wip-rook

mgr/orchestrator/rook: allow the creation of OSDs in directories

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

doc/releases/nautilus: dashboard tweaks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26671 into master

* refs/pull/26671/head:
doc: Replaced "plugin" with "module" in the Mgr documentation

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: add 'config ls'

Signed-off-by: Sage Weil <sage@redhat.com>

mon: handle racing ms_handle_reset and handle_auth_request

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: log tracked pg refs under debug_refs

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: include tag in get/put debug lines

Signed-off-by: Sage Weil <sage@redhat.com>

osd: clean up oncommit contexts in _process shutdown

If we have queued PG completions in the workqueue and have to shut down,
delete them, so that the PG refs get cleaned up.

Fixes: http://tracker.ceph.com/issues/38431
Signed-off-by: Sage Weil <sage@redhat.com>