qa: unmount clients before deleting fs

Client unmount during test cleanup will hang if the file system was deleted.

Fixes: http://tracker.ceph.com/issues/38518
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26663 into master

* refs/pull/26663/head:
doc: change ruleset to id in crush map file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26595 into master

* refs/pull/26595/head:
osd/PG: log tracked pg refs under debug_refs
osd/PG: include tag in get/put debug lines
osd: clean up oncommit contexts in _process shutdown

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26637 into master

* refs/pull/26637/head:
common/ceph_context: experimental kludge for crimson
auth/AuthRegistry: mark 'secure' con mode experimental for now
msg/async: set ms_*_mode to crc only (no secure)

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26640 into master

* refs/pull/26640/head:
mon/OSDMonitor: remove old v1 addrs when converting blacklist to new TYPE_ANY form

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26645 into master

* refs/pull/26645/head:
rpm: use %_python_buildid to specify python-remoto dep

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26678 into master

* refs/pull/26678/head:
mon: handle racing ms_handle_reset and handle_auth_request

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #26648 into master

* refs/pull/26648/head:
msg/async: msgr2: cleanup frame classes hacks

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #26523 from dillaman/wip-38387

librbd: add missing shutdown states to managed lock helper

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #26598 into master

* refs/pull/26598/head:
doc/releases/nautilus: dashboard tweaks
doc/releases/nautilus: more rados highlights
doc/release/nautilus: added RBD major features
doc/releases/nautilus: add msgr2 enablement steps
doc/releases/nautilus: add dashboard items

Reviewed-by: Kai Wagner <kwagner@suse.com>

Merge pull request #26570 from jtlayton/wip-rook

mgr/orchestrator/rook: allow the creation of OSDs in directories

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

doc/releases/nautilus: dashboard tweaks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26671 into master

* refs/pull/26671/head:
doc: Replaced "plugin" with "module" in the Mgr documentation

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mon: handle racing ms_handle_reset and handle_auth_request

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: log tracked pg refs under debug_refs

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: include tag in get/put debug lines

Signed-off-by: Sage Weil <sage@redhat.com>

osd: clean up oncommit contexts in _process shutdown

If we have queued PG completions in the workqueue and have to shut down,
delete them, so that the PG refs get cleaned up.

Fixes: http://tracker.ceph.com/issues/38431
Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: msgr2: cleanup frame classes hacks

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge PR #26628 into master

* refs/pull/26628/head:
msg/async: msgr2: expected tags validation
msg/async: msgr2: template specialized write function for frames
msg/async: msgr2: extract frame classes into its own header

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26227 from Tinatinak/api_documentation

mgr/dashboard: Improved support for generating OpenAPI Spec documentation

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26575 from tspmelo/wip-multi-backstore-ui

mgr/dashboard: Add support for iSCSI's multi backstores (UI)

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #26622 from Songweibin/wip-fix-rbd-group

rbd: fix segmentation fault when rbd_group_image_list() getting -ENOENT

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26432 from dillaman/wip-librbd-discard

librbd: skip small, unaligned discard extents by default

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

doc: Replaced "plugin" with "module" in the Mgr documentation

The documentation currently refers to Ceph Manager Modules as
"plugins" in many places, while the command line interface uses
"module" to enable/disable modules. Replaced all occurences
of "plugin" with "module" in the docs, to avoid confusion and to
be in alignment with the CLI. Also fixed the capitalizations of some
module chapters.

Fixes: https://tracker.ceph.com/issues/38481
Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

mgr/dashboard: Add support for iSCSI's multi backstores (UI)

Fixes: https://tracker.ceph.com/issues/38286
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26586 from votdev/change_notify_icon_color

mgr/dashboard: Change icon color in notifications

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26649 from tspmelo/wip-i18n

mgr/dashboard: Update I18N translation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26589 from Devp00l/issue-38382

mgr/dashboard: PGs will update as expected

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #26116 from votdev/feature_37934

mgr/dashboard: Configure all mgr modules in UI

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26592 from nathan-weinberg/fs-breadcrumb

mgr/dashboard: Added breadcrumb tests to Filesystems menu

Merge pull request #26647 from tspmelo/wip-npm-update-19-03

mgr/dashboard: Update npm packages

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #26219 from yaozongyou/mon-cleanups

mon: some tiny cleanups related class forward declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonell@redhat.com>

doc: change ruleset to id in crush map file

ruleset is not used after merging below patch
   commit f9a095deb1e74642d88bc9015684b0be113cc5f9
       crush: s/ruleset/id/ in decompiled output
       Moving away from the 'ruleset' terminology.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

Merge pull request #26527 from iotcg/doc

use preferred commands for ceph config-key

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26558 from k0ste/prometheus_metadata

mgr/prometheus: get osd_objectstore once instead twice.

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common/ceph_context: experimental kludge for crimson

Signed-off-by: Sage Weil <sage@redhat.com>

auth/AuthRegistry: mark 'secure' con mode experimental for now

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26646 into master

* refs/pull/26646/head:
doc: fix typo in objectops.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/releases/nautilus: more rados highlights

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26627 from jan--f/c-v-38472

ceph-volume: don't create osd['block.db'] by default

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

msg/async: msgr2: expected tags validation

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26225 from jan--f/c-v-simple-activate-all

ceph-volume: add --all flag to simple activate

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #26481 from votdev/gitignore_wheelhouse

mgr: Ignore __pycache__ and wheelhouse dirs

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: Update I18N translation

Add support for Chinese (Simplified), Czech, French, Indonesian and Spanish.

Fixed Portuguese (PT) locale source.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26338 from rhcs-dashboard/ts-cleanup

mgr/dashboard: typescript cleanup

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Introduce upperFirst pipe

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Configure all mgr modules in UI

Fixes: https://tracker.ceph.com/issues/37934
Signed-off-by: Volker Theile <vtheile@suse.com>

qa/suites/rbd: enable partial discard test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: align discard requests to default bluestore allocation size

This will match the new krbd behavior of aligning discards to the
default bluestore minimum allocation size. Requests that are too
small (or the misaligned head / tail of the request) will be
ignored since it will not result in freed space.

Fixes: http://tracker.ceph.com/issues/38146
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #26636 into master

* refs/pull/26636/head:
messages/MMonMgrDigest: use set_data() to avoid screwing up byte_throttler

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26130 from minshenglin/wip-37855

rgw_file: use correct secret key to check auth

mgr/dashboard: Update npm packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

doc: fix typo in objectops.rst

Signed-off-by: Lan Liu <liulan@umcloud.com>

rpm: use %_python_buildid to specify python-remoto dep

Without this patch, the Python 2 version of python-remoto is always installed,
even on Python 3 systems.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

librbd: set *image_size to 0 when getting ENOENT in rbd_group_image_list()

Fixes: http://tracker.ceph.com/issues/38468
Signed-off-by: songweibin <song.weibin@zte.com.cn>

doc: use preferred commands for ceph config-key

'del' & 'list' & 'put' are deprecated. It's preferred to
using 'rm' & 'ls' & 'set' for ceph config-key

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

Merge pull request #26611 from falcon78921/wip-doc-log-based-pg

doc: updated reference link for log based PG

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: remove old v1 addrs when converting blacklist to new TYPE_ANY form

Fixes: http://tracker.ceph.com/issues/38477
Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: set ms_*_mode to crc only (no secure)

'secure' mode will be experimental in the initial nautilus release.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26617 into master

* refs/pull/26617/head:
rpm: Fix pyOpenSSL dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge PR #26623 into master

* refs/pull/26623/head:
mgr/PyModule: put mgr_module_path first in sys.path

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26543 into master

* refs/pull/26543/head:
Revert "ceph.spec.in: python-kubernetes broken on rhel"

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #26625 into master

* refs/pull/26625/head:
rpm,cmake: additional fixes to make amqp support optional

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22755 from zhangsw/feature-rgw-appendobj

rgw: Add append object api.

Merge PR #26610 into master

* refs/pull/26610/head:
doc: updated governance link for Ceph project leads

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: skip partial discards by default

Avoid the IO expense of writing zeros mid-object when a discard
is attempting to free space.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: image replay should retry asok registration upon failure

If the asok registration fails (perhaps due to a race condition with
a deleted and recreated image of the same name), periodically attempt
to register the missing asok hook.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: failure to initialize pool replayer should stop leader

The leader watcher should not start processing requests if it failed
to initialize.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: fixed potential bootstrap cancel race condition

If the image replay was canceled prior to the start of the bootstrap
stage, the image replayer would be stuck attempting to shut down if
the bootstrap is paused behind an image sync.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: complete pool watcher initialization if object missing

If the mirroring object is missing, complete the initialization and
continue to retry in the background. This is useful for cases where
the remote doesn't (yet) have mirroring enabled but the remote
pool watcher initialization is delaying the leader watcher promotion
to the point where the leader is blacklisted by its peers.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: abort trash watcher initialization if create blacklisted

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26603 from dillaman/wip-38453

rbd-mirror: use active/active policy by default

Reviewed-by: Mykola Golub <mgolub@suse.com>

messages/MMonMgrDigest: use set_data() to avoid screwing up byte_throttler

Use the existing set_data() helper to avoid breaking the byte_throttler
accounting.

Fixes e4ae368ff7a5396194f8bdd5692429af5457998b

Fixes: http://tracker.ceph.com/issues/38295
Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: avoid attempting to reacquire lock if blacklisted

The pool replayer state machine will automatically be restarted
when a blacklist is detected.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: don't delete global perf counters before shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add missing shutdown states to managed lock helper

The PRE_SHUTTING_DOWN and SHUTTING_DOWN states were missed
in the 'is_state_shutdown' helper method. This resulted in
rbd-mirror potentially entering an infinite loop during
shutdown.

http://tracker.ceph.com/issues/38387
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: delete pools before stopping rbd-mirror

This better mimics the behavior of teuthology and tests rbd-mirror
daemon's ability to handle a pool deletion.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/rook: have it create bluestore OSDs in data_directories

When the data_directories field is populated, use that to populate the
spec.storage.nodes.<host>.directories field in the CephCluster object.
That will cue the rook operator to create an OSD in using an area under
that directory for storage.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/orchestrator: add new list of directories to DriveGroupSpec

Add a new data_directories list to DriveGroupSpec that is intended to
hold a list of directories in which OSDs should be created, as opposed
to the raw devices in data_devices.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

doc/release/nautilus: added RBD major features

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #26632 into master

* refs/pull/26632/head:
14.1.0

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

msg/async: msgr2: template specialized write function for frames

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: extract frame classes into its own header

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge PR #26609 into master

* refs/pull/26609/head:
ceph_test_msgr: skip injection test for simplemessenger

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26600 from jtlayton/wip-dashboard-rook

mgr/dashboard: special casing for minikube in run-backend-rook-api-request.sh

rbd-mirror: use active/active policy by default

Fixes: http://tracker.ceph.com/issues/38453
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26522 from ashishkumsingh/wip-doc-38310

doc: Fix incorrect mention of 'osd_deep_mon_scrub_interval'

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #26598 into master

* refs/pull/26598/head:
doc/releases/nautilus: note about msgr2
doc/releases/nautilus: rgw highlights
doc/releases/nautilus: draft notes

Reviewed-by: Kai Wagner <kwagner@suse.com>

doc/releases/nautilus: add msgr2 enablement steps

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases/nautilus: add dashboard items

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Change icon color in notifications

- Use CSS filters to convert the existing icons to white.
- Add new Prometheus SVG image, the previous PNG image has used the color white instead or transparent to fill some areas.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #26280 into master

* refs/pull/26280/head:
mgr: Remove _ceph_(g|s)et_module_option_ex() method
mgr: remove _ceph_get_module_option_ex() method

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #26487 into master

* refs/pull/26487/head:
CLI: ability to change file ownership

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26591 into master

* refs/pull/26591/head:
osd/PrimaryLogPG: handle object !exists in handle_watch_timeout

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/releases/nautilus: note about msgr2

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases/nautilus: rgw highlights

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases/nautilus: draft notes

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_msgr: skip injection test for simplemessenger

The interceptor is only implemented in asyncmessenger.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26539 from tspmelo/wip-nfs-status

mgr/dashboard: Add NFS status endpoint

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume: don't create osd['block.db'] by default

Signed-off-by: Jan Fajerski <jfajerski@suse.com>