qa: silence read-only WRN for damage testing

Fixes: http://tracker.ceph.com/issues/37944
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25975 into master

* refs/pull/25975/head:
doc: complete documentation for ceph fs administration commands

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25964 from jmolmo/get_config_replaced

mgr/ansible: Replace deprecated <get_config> calls

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #25966 into master

* refs/pull/25966/head:
doc: add mimic 13.2.3 and 13.2.4 to releases chart

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: complete documentation for ceph fs administration commands

Fixes: http://tracker.ceph.com/issues/24580
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25970 from jtlayton/wip-rook

Fix rook spec and have service_describe provide rados_config_location field for nfs services

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25376 from rhcs-dashboard/about-modal

mgr/dashboard: About modal proposed changes

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #25978 from falcon78921/wip-submitting-patches-doc

SubmittingPatches: added inline markup to important references

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25865 from Devp00l/wip-issue-37841

mgr/dashboard: RbdMirroringService test suite fails in dev mode

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: About modal proposed changes

Fixes: http://tracker.ceph.com/issues/35693
Signed-off-by: Kanika Murarka <kmurarka@redhat.com>

Merge pull request #25939 from cyx1231st/wip-crimson-msgr-fix-segfault

crimson/net: fix misc issues for segment-fault and test-failures

Reviewed-by: Kefu Chai <kchai@redhat.com>

SubmittingPatches: added inline markup to important references

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #25940 from xiexingguo/wip-balancer-pools-support

mgr/balancer: auto balance a list of pools

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/rook: populate rados_config_location field

The dashboard ganesha management code requires the rados:// config
URL for the nfs-ganesha daemons. Ensure that we populate that.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/rook: lowercase the "RADOS" dict label

rook is lowercasing this tag in the CRD in response to review comments.
Change the orchestrator accordingly.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

doc: add mimic 13.2.3 and 13.2.4 to releases chart

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/ansible: Replace deprecated <get_config> calls

Changes needed to replace the obsolete get_config by the new get_module_option manager module method

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge PR #25947 into master

* refs/pull/25947/head:
msg/Messenger: remove get_connection() and send_message()
ceph_test_msgr: remove final get_connection() user

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25649 from SUSE/wip-orchestrator-service-op

mgr/orchestrator: Add service_action method

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #25669 from sebastian-philipp/orchestrator-type-hinting

mgr/orchestrator: Improve docstrings, add type hinting

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #25543 from rjfd/wip-mgr-cli-commands

mgr: modules CLI commands declaration using @CLICommand decorator

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #25426 from Tinatinak/fix_bug25094_2

mgr/dashboard: Filter out tasks depending on permissions

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/orchestrator_cli: Add service and service-instance start/stop/reload

Signed-off-by: Tim Serong <tserong@suse.com>

mgr/orchestrator: Add service_action method

This is to facilitate service start/stop/reload (for example,
after altering NFS Ganesha configuration, we need to trigger
a service reload).

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #25953 from falcon78921/wip-doc-24924

doc: fixed typo in CRUSH map docs

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

crimson/net: fix potential parallel write in SocketConnection

All the write/flush in SocketConnection should be serialized, or nasty
segment-fault could happen in seastar which is hard to diagnose.

This fix serializes the writes in `handle_keepalive2()` with other
writes in the open state.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

common/pick_address: #include "ceph_context.h"

pick_address.cc reference cct->_conf, so it should include the
definition of of CephContext explicitly instead of relying on the fact
that some headers it includes include its definition.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ip_addr: do not support numa preference if WITH_SEASTAR

in pick_address.cc, set_val_or_die() is called to set the "public_addr"
and/or "cluster_addr". but since the sharded ConfigProxy cannot
set_val_or_die() synchronously, we cannot do it this way.

we need to fix pick_addresses() so it works with seastar later. either
by making the involved function return future<>, or making the set_val()
call non-blocking or synchronous if we can go with 1:1 mapping
threading model.

but at this moment, let's just disable this feature in crimson-osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: enable unittest_seastar_messenger

so we can at least compile it, even it fails sometimes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: fix unittest_seastar_messenger failure

unittest_seastar_messenger is failed since updated the seastar repo.
std::system_error (error system:22, Invalid argument) will be raised
from `listener->accept()`, which will be re-raised and not be captured.

This fix is to print error logs for the unexpected exceptions in
`SocketMessenger::start()` and not raise them again.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

doc: fixed typo in CRUSH map docs

Fixes: https://tracker.ceph.com/issues/24924
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge PR #22612 into master

* refs/pull/22612/head:
kv/RocksDBStore: Don't use sync mode when disableWAL is set.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25915 into master

* refs/pull/25915/head:
mon/ConfigMonitor: add 'config generate-minimal-conf' command
common/options: add FLAG_MINIMAL_CONF

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #25879 into master

* refs/pull/25879/head:
client: define template to declare ref
client: use message smart ptr instead of raw ptr
client: make map lookup optimizations
client: do not clear message payload
client: clean up variable name
*: use std::string_view instead of char ptr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25921 into master

* refs/pull/25921/head:
mds: convert const char ptr to string_view
common/TrackedOp: use string_view interface

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25948 from liewegas/wip-fix-locker-addr

cls/lock: always store v1 addr in locker_info_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25881 from cbodley/wip-rgw-sanitize-v4

rgw: sanitize customer encryption keys from log output in v4 auth

cls/lock: always store v1 addr in locker_info_t

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24446 from vico-research-and-consulting/master

common: fix for broken rbdmap parameter parsing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/Messenger: remove get_connection() and send_message()

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_msgr: remove final get_connection() user

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25754 into master

* refs/pull/25754/head:
msg/async/Protocol: send keepalive on connection race winner

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25932 into master

* refs/pull/25932/head:
qa/tasks/ceph: stop any split/merge activity before scrubbing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #25888 into master

* refs/pull/25888/head:
common/HeartbeatMap: no health check if heartbeat_file is empty

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25843 into master

* refs/pull/25843/head:
qa: fix damage expectation setting
qa: fix loop variable reference

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25814 into master

* refs/pull/25814/head:
common/StackStringStream: optimize xsputn() and overflow()

Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25633 into master

* refs/pull/25633/head:
mgr/rook: allow service describe and service ls to display nfs services
mgr/rook: add support for adding NFS gateways
mgr/orchestrator_cli: add support for adding NFS gateways
mgr/orchestrator_cli: convert service add/rm commands to be type-specific
mgr/rook: add decorator for add_stateless_service
mgr/rook: add a context manager instead of open-coding exception handling

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #25787 into master

* refs/pull/25787/head:
mount.fuse.ceph: Fix ambiguous shebang

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common: fix for broken rbdmap parameter parsing

This improves the broken parameter parsing discussed in #36327

Test expression for the parsing function:

PARAMS="options='lock_on_read,queue_depth=1024',id=archiv,keyring=/etc/ceph/ceph.client.archiv.keyring"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS="id=archiv,keyring=/etc/ceph/ceph.client.archiv.keyring,options='lock_on_read,queue_depth=1024'"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS=""
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS=",keyring=/etc/ceph/ceph.client.archiv.keyring,options='lock_on_read,queue_depth=1024',id=archiv,"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS="'keyring'=/etc/ceph/ceph.client.archiv.keyring,options='lock_on_read,queue_depth=1024',id=archiv,"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

PARAMS="--keyring=/etc/ceph/ceph.client.archiv.keyring,options='lock_on_read,queue_depth=1024',id=archiv, # a comment"
echo -e " ORIGINAL PARAM: >>>$PARAMS<<<\nCONVERTED PARAM: >>>$(create_cmd_params "$PARAMS")<<<"

Fixes: https://tracker.ceph.com/issues/36327
Signed-off-by: Marc Schoechlin <ms@256bit.org]>

Merge pull request #25935 from trociny/wip-37883

test/rbd_mirror: wait for release leader lock fully complete

Merge pull request #25925 from markhpc/wip-rocksdb-pricache

kv/KeyValueDB: Move PriCache implementation to ShardedCache.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/rook: allow service describe and service ls to display nfs services

Extend the _service_ls routine and the rook backend so that it can
properly display ceph NFS gateways as well.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/rook: add support for adding NFS gateways

Add the necessary machinery to the rook orchestrator to allow it to
request a cluster of nfs-ganesha gateways.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #25876 from s0nea/wip-dashboard-rename-config-option-types

mgr/dashboard: config option type names update

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #25943 from rhcs-dashboard/update-sso-doc

mgr/dashboard: updated SSO documentation

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: updated SSO documentation

Added info about SAML request issuer value.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

qa/tasks/ceph: stop any split/merge activity before scrubbing

If there are leftover merges at the end of the run they can take a long
time to get through, blowing our timeout for (waiting for pgs to become
active and to stop splitting/merge) and scrubbing pgs.  Stop all of that
at the end of the run so that we don't have to wait so long.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: add support for adding NFS gateways

Add necessary commands to orchestrator_cli to allow it to deploy a
cluster of NFS servers. Note that we have to be able to specify the pool
and an optional namespace for the recovery backend objects.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/orchestrator_cli: convert service add/rm commands to be type-specific

When adding services, we often need different sets of arguments
depending on the service type. Convert the "service add" and "service
rm" commands into separate commands for each service type.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #25927 from dillaman/wip-37867

mgr/dashboard: incorporate RBD overall performance grafana dashboard

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/rook: add decorator for add_stateless_service

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/rook: add a context manager instead of open-coding exception handling

Suggested-by: Sebastian Wagner <sebastian.wagner@suse.com>
Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/orchestrator: Improve docstrings, add type hinting

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25492 from sebastian-philipp/orchestrator-common-metaclass

pybind/mgr: Unified bits of volumes and orchestrator

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

mgr/dashboard: access_control: change CLI commands declaration

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/mgr_module: @CLICommand decorator for declaring CLI commands

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/balancer: auto balance a list of pools

Add support which limits the auto balancer to a comma-separated list of pools.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25909 from votdev/issue_37862

mgr/dashboard: Confusing tilted time stamps in the CephFS performance graph

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #23729 from daikk115/master

docs: Fixed swift client authentication fail

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25913 from s0nea/wip-dashboard-test_full_health-fix

mgr/dashboard: Fix test_full_health test

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #25905 from s0nea/wip-dashboard-osd-add-flag-fix

mgr/dashboard: Provide all four 'mandatory' OSD flags

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #25859 from xiexingguo/wip-async-no-force-renew-auth

msg/async: do not force updating rotating keys inline

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25930 from xiexingguo/wip-invalid-upmap-access

osd/OSDMap: potential access violation fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25933 into master

* refs/pull/25933/head:
common/numa: remove stray debug output

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/OSDMap: potential access violation fix

Seems we'll continue to access the iterator after it is invalidated
by the __erase__ method.
Also this is more efficient considering there could be some extreme
ec-pool (e.g., 8 + 2) consumers..

Fixes: http://tracker.ceph.com/issues/37881
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25931 from yangdongsheng/journal_import

rbd: fix a typo in error output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/numa: remove stray debug output

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25902 into master

* refs/pull/25902/head:
qa/standalone/mon/mon-handle-forward: fix grep path and check return results
mon/MonClient: be less aggressive/paranoid about reopening mon connections

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25884 into master

* refs/pull/25884/head:
msg/async: better debug output

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #25917 into master

* refs/pull/25917/head:
qa/suites/rados/multimon/tasks/mon_recovery: whitelist PG_AVAILABILITY

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/rbd_mirror: wait for release leader lock fully complete

Previously the wait context was completed at post_release_lock
stage, which still was not the end of the lock releasing process.

Fixes: https://tracker.ceph.com/issues/37883
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #25273 into master

* refs/pull/25273/head:
doc/rados/operations/health-checks: Add LARGE_OMAP_OBJECTS

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25828 into master

* refs/pull/25828/head:
osd, mgr, mds: make timeout of updating rotating keys configurable

Reviewed-by: yanjun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25886 into master

* refs/pull/25886/head:
qa/tasks/rebuild_mondb: use monmap to properly name the mons
ceph-monstore-tool: add --monmap <path> argument to rebuild command
ceph-mon: dump monmap to debug logs
Revert "mon/MonMap: no noname- mon name prefix when for_mkfs"
mon/MonMap: improve/fix initial monmap generation for mkfs
ceph-mon: make membership check with public_addr more robust
ceph-mon: make mkfs check whether I am in monmap more robust

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #25806 into master

* refs/pull/25806/head:
mgr: diskprediction_cloud Add nvme devcie support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25662 from yangdongsheng/abort_on_full

rbd: introduce abort_on_full option for rbd map

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rbd: introduce abort_on_full option for rbd map

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: fix a typo in error output

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

doc/rados/operations/health-checks: Add LARGE_OMAP_OBJECTS

Document LARGE_OMAP_OBJECTS health check

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

mds: convert const char ptr to string_view

Trivial refactor.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25924 from neha-ojha/wip-fix-scrub-doc

doc/rados/configuration: add osd scrub {begin,end} week day

Reviewed-by: Sage Weil <sage@redhat.com>

common/TrackedOp: use string_view interface

Also, don't store pointers to strings outside. The interface was written to
optimize for string literals but this is structurally unsafe programming.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/rebuild_mondb: use monmap to properly name the mons

We used to rely on the monmap bootstrap code to magically create a valid
monmap with named mons because our old-style ceph.conf had mon_addr
values in each mon.foo section.  Instead, just feed it a real monmap
from pre-destruction.

In practice, a user can manually generate this monmap, or rename the
mons after the fact with --inject-monmap, or whatever.  Out of scope
for this test, so we just do the simplest thing to make the rebuild test
work.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-monstore-tool: add --monmap <path> argument to rebuild command

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-mon: dump monmap to debug logs

Verbosely during mkfs, at level 10 on normal startup.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: incorporate RBD overall performance grafana dashboard

Fixes: http://tracker.ceph.com/issues/37867
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

monitoring/grafana: new RBD overview dashboard page

This page pulls RBD stats from the Natuatilus prometheus exporter.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/rados/configuration: add osd scrub {begin,end} week day

Signed-off-by: Neha Ojha <nojha@redhat.com>

rgw: sanitize customer encryption keys from log output in v4 auth

Fixes: http://tracker.ceph.com/issues/37847
CVE-2018-16889

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24427 from thmour/thmour_api_purge

librbd: add trash purge api calls

Reviewed-by: Jason Dillaman <dillaman@redhat.com>