mon: implement hiding commands in ceph tool

Otherwise ceph.in doing get_command_descriptions sees the command does not
exist and will print an error.

Fixes: https://tracker.ceph.com/issues/37956
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25998 into master

* refs/pull/25998/head:
msg/Dispatcher: remove force_new arg from ms_get_authorizer()
crimson/net: drop authenticator retry
msg/simple: remove forced authorizer refresh

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #25937 into master

* refs/pull/25937/head:
qa: specify filestore for misc tests
osd/objectstore: change osd_objectstore default to bluestore

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25693 from shun-s/silence-mon-load-metadata

MDSMonitor: silence unable to load metadata

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge PR #25849 into master

* refs/pull/25849/head:
qa/suites/rados/upgrade: one mon per node, and enable-msgr2 at end
qa/rados/thrash-old-clients: avoid msgr2
mon: make bootstrap rank check more robust
mon: clean up probe debug output a bit
msg/async: use v1 for v1 <-> [v2,v1] peers
msg/async/AsyncMessenger: drop single-use _send_to
mon/HealthMonitor: raise MON_MSGR2_NOT_ENABLED if mons not bound to msgr2
doc/rados/operations/health-checks: document MON_* health warnings
mon/MonMapMonitor: add 'mon enable-msgr2' command
mon: respawn if rank addr changes
mon/MonMap: calc_addr_mons() after setting rank addrvec

Reviewed-by: Ricardo Dias <rdias@suse.com>

qa/suites/rados/upgrade: one mon per node, and enable-msgr2 at end

Signed-off-by: Sage Weil <sage@redhat.com>

qa/rados/thrash-old-clients: avoid msgr2

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25654 from tspmelo/wip-missing-i18n

mgr/dashboard: Add missing frontend I18N

Reviewed-by: Sebastian Krah <skrah@suse.com>

Merge pull request #25997 from zmc/wip-db-graf-validate

mgr/dashboard: Fix /api/grafana/validation

Reviewed-by: Volker Theile <vtheile@suse.com>

msg/Dispatcher: remove force_new arg from ms_get_authorizer()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25870 from votdev/validate_ip_addr

mgr/dashboard: Improve RGW address parser

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #25959 from xiexingguo/wip-memory-leak

crush: fix memory leak

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25941 from xiexingguo/wip-warn-enable-upmap

mgr/balancer: blame if upmap won't actually work

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25836 from votdev/improve_rgw_status_endpoint

mgr/dashboard: Improve exception handling in /api/rgw/status

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25950 from gregsfortytwo/wip-githubmap

Wip githubmap

Merge pull request #25751 from batrick/stackstringstream-reset

common: reinit StackStringStream on clear

Reviewed-by: Kefu Chai <kchai@redhat.com>

MDSMon: silence unable to load metadata

  silence unable to load metadata

Signed-off-by: Song Shun <song.shun3@zte.com.cn>

crimson/net: drop authenticator retry

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple: remove forced authorizer refresh

This synchronous check has always been kludgey; remove it and just fault
instead, just like we did with 794a8f9cf51cf176636d114ccfbbf68fbc304083
in AsyncMessenger.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25994 into master

* refs/pull/25994/head:
qa: silence read-only WRN for damage testing

Reviewed-by: Jeff Layton <jlayton@redhat.com>

mgr/dashboard: Fix /api/grafana/validation

If the Grafana URL contains a trailing slash, this endpoint ends up
returning a 404, disabling the integration entirely. Fix that.

Signed-off-by: Zack Cerza <zack@redhat.com>

qa: silence read-only WRN for damage testing

Fixes: http://tracker.ceph.com/issues/37944
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25934 into master

* refs/pull/25934/head:
msg/msg_type: entity_addr_t: fix legacy decode
msg/msg_types: make set_sockaddr() work with AF_UNSPEC (i.e., zeroed)
msg/msg_types: make set_sockaddr() a bit more robust
msg/async: fix IP inference

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: specify filestore for misc tests

Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osd/objectstore: change osd_objectstore default to bluestore
Fixes: http://tracker.ceph.com/issues/36494
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

Merge PR #25975 into master

* refs/pull/25975/head:
doc: complete documentation for ceph fs administration commands

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25964 from jmolmo/get_config_replaced

mgr/ansible: Replace deprecated <get_config> calls

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #25966 into master

* refs/pull/25966/head:
doc: add mimic 13.2.3 and 13.2.4 to releases chart

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: complete documentation for ceph fs administration commands

Fixes: http://tracker.ceph.com/issues/24580
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/msg_type: entity_addr_t: fix legacy decode

If we decode a zeroed sockaddr, we should end up with a TYPE_NONE
entity_addr_t, not v1::/0.

This was obscured by unit test TestAddrvecEncodeAddrDecode3, which
took an addrvec with all v2 addrs, decoded to an addr variable that
previously had v1:1.2.3.4:/0, and asserted the result was not v1::/0.
The test passed before because the set_sockaddr() failed on AF_UNSPEC
and the addr kept v1:1.2.3.4, but with the previous commit it failed
because it equaled v1::/0.  In reality, addr should get - (addr TYPE_NONE).

The TestEmptyAddrvecEncodeAddrDecode test case is similarly adjusted.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25970 from jtlayton/wip-rook

Fix rook spec and have service_describe provide rados_config_location field for nfs services

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Add missing frontend I18N

Fixes: http://tracker.ceph.com/issues/36719
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25376 from rhcs-dashboard/about-modal

mgr/dashboard: About modal proposed changes

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #25978 from falcon78921/wip-submitting-patches-doc

SubmittingPatches: added inline markup to important references

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25865 from Devp00l/wip-issue-37841

mgr/dashboard: RbdMirroringService test suite fails in dev mode

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Improve exception handling in /api/rgw/status

In some cases the exception message was not forwarded to the caller.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: About modal proposed changes

Fixes: http://tracker.ceph.com/issues/35693
Signed-off-by: Kanika Murarka <kmurarka@redhat.com>

Merge pull request #25939 from cyx1231st/wip-crimson-msgr-fix-segfault

crimson/net: fix misc issues for segment-fault and test-failures

Reviewed-by: Kefu Chai <kchai@redhat.com>

SubmittingPatches: added inline markup to important references

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

mgr/balancer: blame if upmap won't actually work

With automatic balancing on, and if mode is set to upmap,
balancer will fail silently if min_compat_client is lower than
luminous.
You can't figure out that unless you take a closer look at the
mgr log, which is super annoying..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25940 from xiexingguo/wip-balancer-pools-support

mgr/balancer: auto balance a list of pools

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/rook: populate rados_config_location field

The dashboard ganesha management code requires the rados:// config
URL for the nfs-ganesha daemons. Ensure that we populate that.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/rook: lowercase the "RADOS" dict label

rook is lowercasing this tag in the CRD in response to review comments.
Change the orchestrator accordingly.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

doc: add mimic 13.2.3 and 13.2.4 to releases chart

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/ansible: Replace deprecated <get_config> calls

Changes needed to replace the obsolete get_config by the new get_module_option manager module method

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

mon: make bootstrap rank check more robust

If we search for myaddrs, then we will detect our rank properly if we were
[a,b] and the a addr got removed.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: clean up probe debug output a bit

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: use v1 for v1 <-> [v2,v1] peers

If *peers* are communicating, i.e. there may be bidirectional connection
attempts, we must use the same protocol version from both ends or else
we will get very confused.

Fix this by forcing the use of v1 when we

- are bound to a v1 endpoint only (people can't connect to us via v2)
- we are connecting to a *peer*

If it is a non-peer, then connections are uni-directional.  If we both
have v2, we will both use v2.

If we ever switch to [v2,v1], it will be as part of a restart.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/AsyncMessenger: drop single-use _send_to

Signed-off-by: Sage Weil <sage@redhat.com>

mon/HealthMonitor: raise MON_MSGR2_NOT_ENABLED if mons not bound to msgr2

If the ms_bind_msgr2 option is enabled, and all mons are nautilus,
raise a health alert if any mons aren't bound to msgr2 addresses.

Whitelist tests that mon_bind_addrvec=false or mon_bind_msgr2=false.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/health-checks: document MON_* health warnings

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMapMonitor: add 'mon enable-msgr2' command

This switches any monitor listening on v1:...:6789 only to v2 (on
3300) + v1 (on 6789).  It should be run after upgrading all monitors
to nautilus.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: respawn if rank addr changes

If the address for our rank change, then respawn!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25947 into master

* refs/pull/25947/head:
msg/Messenger: remove get_connection() and send_message()
ceph_test_msgr: remove final get_connection() user

Reviewed-by: Kefu Chai <kchai@redhat.com>

common: restore constructor fmtflags on clear

This is to prevent fmtflags changes from a previous use (e.g. debug message)
leaking into the the next (i.e. the next debug message).

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: use better name reset for clear method

Reset is more analogous to re-initilization which is the intent.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/msg_types: make set_sockaddr() work with AF_UNSPEC (i.e., zeroed)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25649 from SUSE/wip-orchestrator-service-op

mgr/orchestrator: Add service_action method

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #25669 from sebastian-philipp/orchestrator-type-hinting

mgr/orchestrator: Improve docstrings, add type hinting

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #25543 from rjfd/wip-mgr-cli-commands

mgr: modules CLI commands declaration using @CLICommand decorator

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #25426 from Tinatinak/fix_bug25094_2

mgr/dashboard: Filter out tasks depending on permissions

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crush: fix memory leak

If we remove the last item of bucket, there should still be
one final entry in the __weights__ field of __weight_set__.
Free the corresponding memory before we __null__ the pointer.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/orchestrator_cli: Add service and service-instance start/stop/reload

Signed-off-by: Tim Serong <tserong@suse.com>

mgr/orchestrator: Add service_action method

This is to facilitate service start/stop/reload (for example,
after altering NFS Ganesha configuration, we need to trigger
a service reload).

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #25953 from falcon78921/wip-doc-24924

doc: fixed typo in CRUSH map docs

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

crimson/net: fix potential parallel write in SocketConnection

All the write/flush in SocketConnection should be serialized, or nasty
segment-fault could happen in seastar which is hard to diagnose.

This fix serializes the writes in `handle_keepalive2()` with other
writes in the open state.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

common/pick_address: #include "ceph_context.h"

pick_address.cc reference cct->_conf, so it should include the
definition of of CephContext explicitly instead of relying on the fact
that some headers it includes include its definition.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ip_addr: do not support numa preference if WITH_SEASTAR

in pick_address.cc, set_val_or_die() is called to set the "public_addr"
and/or "cluster_addr". but since the sharded ConfigProxy cannot
set_val_or_die() synchronously, we cannot do it this way.

we need to fix pick_addresses() so it works with seastar later. either
by making the involved function return future<>, or making the set_val()
call non-blocking or synchronous if we can go with 1:1 mapping
threading model.

but at this moment, let's just disable this feature in crimson-osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: enable unittest_seastar_messenger

so we can at least compile it, even it fails sometimes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: fix unittest_seastar_messenger failure

unittest_seastar_messenger is failed since updated the seastar repo.
std::system_error (error system:22, Invalid argument) will be raised
from `listener->accept()`, which will be re-raised and not be captured.

This fix is to print error logs for the unexpected exceptions in
`SocketMessenger::start()` and not raise them again.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

doc: fixed typo in CRUSH map docs

Fixes: https://tracker.ceph.com/issues/24924
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge PR #22612 into master

* refs/pull/22612/head:
kv/RocksDBStore: Don't use sync mode when disableWAL is set.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25915 into master

* refs/pull/25915/head:
mon/ConfigMonitor: add 'config generate-minimal-conf' command
common/options: add FLAG_MINIMAL_CONF

Reviewed-by: Neha Ojha <nojha@redhat.com>

msg/msg_types: make set_sockaddr() a bit more robust

We should keep the unused parts of u zeroed.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: fix IP inference

socket_addr is *our* socket address; target_addr is the remote peer's
socket address.  We are inferring the remote's IP address here.

Fixes: http://tracker.ceph.com/issues/37882
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25879 into master

* refs/pull/25879/head:
client: define template to declare ref
client: use message smart ptr instead of raw ptr
client: make map lookup optimizations
client: do not clear message payload
client: clean up variable name
*: use std::string_view instead of char ptr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25921 into master

* refs/pull/25921/head:
mds: convert const char ptr to string_view
common/TrackedOp: use string_view interface

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25948 from liewegas/wip-fix-locker-addr

cls/lock: always store v1 addr in locker_info_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25881 from cbodley/wip-rgw-sanitize-v4

rgw: sanitize customer encryption keys from log output in v4 auth

.githubmap: remove erroneous jlayton entry

He's jtlayton on github!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

.githubmap: switch Gregory Farnum to Greg

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

cls/lock: always store v1 addr in locker_info_t

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24446 from vico-research-and-consulting/master

common: fix for broken rbdmap parameter parsing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/Messenger: remove get_connection() and send_message()

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_msgr: remove final get_connection() user

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25754 into master

* refs/pull/25754/head:
msg/async/Protocol: send keepalive on connection race winner

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25932 into master

* refs/pull/25932/head:
qa/tasks/ceph: stop any split/merge activity before scrubbing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #25888 into master

* refs/pull/25888/head:
common/HeartbeatMap: no health check if heartbeat_file is empty

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25843 into master

* refs/pull/25843/head:
qa: fix damage expectation setting
qa: fix loop variable reference

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25814 into master

* refs/pull/25814/head:
common/StackStringStream: optimize xsputn() and overflow()

Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25633 into master

* refs/pull/25633/head:
mgr/rook: allow service describe and service ls to display nfs services
mgr/rook: add support for adding NFS gateways
mgr/orchestrator_cli: add support for adding NFS gateways
mgr/orchestrator_cli: convert service add/rm commands to be type-specific
mgr/rook: add decorator for add_stateless_service
mgr/rook: add a context manager instead of open-coding exception handling

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #25787 into master

* refs/pull/25787/head:
mount.fuse.ceph: Fix ambiguous shebang

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common: fix for broken rbdmap parameter parsing

This improves the broken parameter parsing discussed in #36327

Test expression for the parsing function:

PARAMS="options='lock_on_read,queue_depth=1024',id=archiv,keyring=/etc/ceph/ceph.client.archiv.keyring"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS="id=archiv,keyring=/etc/ceph/ceph.client.archiv.keyring,options='lock_on_read,queue_depth=1024'"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS=""
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS=",keyring=/etc/ceph/ceph.client.archiv.keyring,options='lock_on_read,queue_depth=1024',id=archiv,"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS="'keyring'=/etc/ceph/ceph.client.archiv.keyring,options='lock_on_read,queue_depth=1024',id=archiv,"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS="--keyring=/etc/ceph/ceph.client.archiv.keyring,options='lock_on_read,queue_depth=1024',id=archiv, # a comment"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

Fixes: https://tracker.ceph.com/issues/36327
Signed-off-by: Marc Schoechlin <ms@256bit.org]>

Merge pull request #25935 from trociny/wip-37883

test/rbd_mirror: wait for release leader lock fully complete

Merge pull request #25925 from markhpc/wip-rocksdb-pricache

kv/KeyValueDB: Move PriCache implementation to ShardedCache.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/rook: allow service describe and service ls to display nfs services

Extend the _service_ls routine and the rook backend so that it can
properly display ceph NFS gateways as well.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/rook: add support for adding NFS gateways

Add the necessary machinery to the rook orchestrator to allow it to
request a cluster of nfs-ganesha gateways.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #25876 from s0nea/wip-dashboard-rename-config-option-types

mgr/dashboard: config option type names update

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #25943 from rhcs-dashboard/update-sso-doc

mgr/dashboard: updated SSO documentation

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: updated SSO documentation

Added info about SAML request issuer value.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>