Merge PR #27141 into master

* refs/pull/27141/head:
mon/OSDMonitor: fix osd boot feature vs require_osd_release check
include/ceph_features: retire 7 other old features
include/ceph_features: retire ERASURE_CODE_PLUGINS_V2
include/ceph_features: retire OSD_ERASURE_CODES
include/ceph_features: update comment to align with N+2 upgrades
include/ceph_features: adjust whitespace for retired and now usable features
mon: remove check for jewel mons
mds/FSMap: remove support for encoding jewel FSMap
include/ceph_features: enable SERVER_OCTOPUS
test/cli/osdmaptool/feature-set-unset-list: add octopus to output
test/cli/osdmaptool/feature-set-unset-list: change unknown feature bit
qa/releases/octopus.yaml: add octopus upgrade final step
osd/OSDMap: octopus encoding features
mon/OSDMonitor: add mon_debug_no_require_octopus
mon/OSDMonitor: allow 'osd require-osd-release octopus'
mon: add ondisk incompat octopus feature
mon/mon_types: add mon feature for octopus
include/ceph_features: SERVER_O -> SERVER_OCTOPUS

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27291 into master

* refs/pull/27291/head:
qa/suites/upgrade/nautilus-x-singleton: upgrade mgrs early
qa/suites/upgrade/mimic-x-singleton: upgrade mgrs later
qa/suites: add upgrade/nautilus-x-singleton and symlink from rados/upgrade
qa/suites: move rados/upgrade/mimic-x-singleton to upgrade/ and symlink

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27146 into master

* refs/pull/27146/head:
mon/MonMap: add min_quorum_size() helper
mon/MDSMonitor: add 'mds ok-to-stop' command
mon: add 'mon ok-to-{stop,add-offline,rm}' commands

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26572 from rhcs-dashboard/wip-37337-master

mgr/dashboard: unify button/URL actions naming

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #26183 from letterwuyu/wip-rgw-get-detail-user-info

rgw: Make rgw admin ops api get user info consistent with the command line

Merge pull request #26412 from theanalyst/rgw-orphan-fixes

rgw: orphan fixes

Merge PR #27343 into master

* refs/pull/27343/head:
mon/LogMonitor: add mon_cluster_log_to_file bool option

Reviewed-by: Sébastien Han <seb@redhat.com>

mgr/dashboard: add wording section to hacking.rst

Fixes: http://tracker.ceph.com/issues/37337
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: unify button/URL actions naming

- Mappings (actually an Enum) created for actions (buttons and other UI elements) and URLs: ActionLabels and URLVerbs.
  - An alternative would be to fix/improve the current i18n-polyfill, which only works with literal strings (not even with 'const enums' which become literals after Typescript transpiling).
  - Additionally having a predefined file with some strings to translate (actions, verbs, etc) could improve on the 1st of the 2-stage i18n process (as extraction tool has a lot of limitations).
- A corresponding ActionLabelsI18n service with translated labels (it's a service as I haven't found the way to either translate no-const strings (ngx-translate/AST parser failure) or get a static translator).
  - This services could/should be extended to cover all strings that are defined in static/globally scoped objects before any I18n provider has been initialized.
- Breadcrumbs are not translated (neither were they before this change). This part remains untackled: using 'proxy' static objects and performing live translation could deal with the issue.
- New URLBuilder service created (following a established pattern in the Java/.NET world) . This should avoid the need of messing with literal URLs and string composition/parsing, and while the front-end is not meant to be consumed by anyone, Angular does not provide any other way for the app to navigate between components, so the URLs are a de-facto interface contract. Unlike this approach is not flawless, it's easier to enforce, while issues coming from free-from strings are really hard to catch.
  - This could be further improved by using a router registry/dynamic routing. Most of the routes are trivial.
- As a side effect of these changes, routing module has been refactored and some routes moved to their specific modules (pool, rbd, rgw), via loadChildren and routes.forChild() magic. Now the above mentioned components are lazy-loaded/pre-loaded (it means right after the main code is loaded). This should also decrease the loading time (though probably this is not biggest time eater here).
  - As now modules can be loaded multiple times, not only from App module by means of lazy loading, but also from other ones (as PoolModule loads BlockModule to get QoS widgets in Pool windows), now lazy loaded modules include 2 NgModules (one with imports: RouterModule.forChild(routes), meant for lazy-loading, and another without routes).
- Caveat: Some parts might not be (fully) translated (NFS, iSCSI, mirroring), as there's been ongoing work on them and it's hard to keep up with the new code.
These changes will be a waste of time if the new code does not take benefit from/adheres to it, so I'm still figuring out how to spread this (nothing really fancy to demo). Maybe adding some checks/harnessing to enforce the new naming convention (ideas greatly welcome here).

Fixes: http://tracker.ceph.com/issues/37337
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mon/LogMonitor: add mon_cluster_log_to_file bool option

Allow cluster logging to a file to be disabled via a boolean.  Default
to true to avoid any change in behavior.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27280 from xiexingguo/wip-39039

mgr: more GIL fixes

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27245 from tspmelo/wip-nfs-typo

mgr/dashboard: Fix typo in NFS form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27333 from tchaikov/wip-rpm-devtoolsset-8-on-aarch64

rpm: enable devtoolset-8 on aarch64 also

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

install-deps,rpm: enable devtoolset-8 on aarch64 also

in 5ae3b06e, we left aarch64 with devtoolset-7, but aarch64 builders are
also suffering from http://tracker.ceph.com/issues/38892. so i installed
devtoolset-8 on all aarch64 builders manually from the rpm packages
downloaded from koji builder:

- https://cbs.centos.org/koji/buildinfo?buildID=24923
- https://cbs.centos.org/koji/buildinfo?buildID=24931

libasan5, libubsan1 are not installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27328 into master

* refs/pull/27328/head:
qa/tests: reduce number of jobs for `kcephfs`

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #27310 from tchaikov/wip-crimson-cache-ss-oi

crimson/osd: cache object_info and snapset in PGBackend

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27307 from tchaikov/wip-crimson-osd-hb

crimson/osd: do not add whoami as hb peer and cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tests: reduce number of jobs for `kcephfs`

we need it on `master` as well
see https://github.com/ceph/ceph/pull/27165

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites/upgrade/nautilus-x-singleton: upgrade mgrs early

The mgrmap may change when the mons upgrade, causing the mgr daemons to
respawn.  Restart mgr daemons before mons to avoid a mismatch between
the running version and the ondisk binary.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimic-x-singleton: upgrade mgrs later

The mgrmap may change when the mons upgrade, causing the mgr daemons to
respawn.  Delay the upgrade until later so that the ondisk binary does
not change out from underneath them.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites: add upgrade/nautilus-x-singleton and symlink from rados/upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites: move rados/upgrade/mimic-x-singleton to upgrade/ and symlink

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27316 into master

* refs/pull/27316/head:
doc/rados/operations: document BLUEFS_SPILLOVER

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mon/OSDMonitor: fix osd boot feature vs require_osd_release check

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: retire 7 other old features

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: retire ERASURE_CODE_PLUGINS_V2

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: retire OSD_ERASURE_CODES

We can use this again in Q.

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: update comment to align with N+2 upgrades

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: adjust whitespace for retired and now usable features

The blank line suggests that the feature is now reusable.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: remove check for jewel mons

Signed-off-by: Sage Weil <sage@redhat.com>

mds/FSMap: remove support for encoding jewel FSMap

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: enable SERVER_OCTOPUS

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/osdmaptool/feature-set-unset-list: add octopus to output

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/osdmaptool/feature-set-unset-list: change unknown feature bit

Use something we won't consume anytime soon.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/releases/octopus.yaml: add octopus upgrade final step

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: octopus encoding features

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: add mon_debug_no_require_octopus

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: allow 'osd require-osd-release octopus'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27223 from ivancich/wip-fix-resharding-issue

rgw: race condition between resharding and ops waiting on resharding

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

doc/rados/operations: document BLUEFS_SPILLOVER

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Fix typo in NFS form

Fixes: http://tracker.ceph.com/issues/39067
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #27265 into master

* refs/pull/27265/head:
valgrind.supp: add clauses for uninitialised gcm.Xi in OpenSSL.
src/valgrind.supp: replace with the teuthology's whitelist.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27313 into master

* refs/pull/27313/head:
mon: quiet devname log noise

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26966 from SUSE/wip-orchestrator-deepsea-fix-device-ls

mgr/deepsea: use ceph_volume output in get_inventory()

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mon: quiet devname log noise

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27308 from tchaikov/wip-init-ceph-patience-is-a-virtue

init-ceph: wait longer before resending $signal

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27255 from adamemerson/wip-names-common-through-objecter

Remove dependence on `using namespace`: Build of 'common' through osdc/Objecter.cc

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: cache object_info and snapset in PGBackend

for better performance

Signed-off-by: Kefu Chai <kchai@redhat.com>

init-ceph: wait longer before resending $signal

if ceph-osd takes longer than 1 seconds to unmount the objectstore, it
won't get a chance to fully persist all of its stuff to disk before
init-ceph kills it again.

in my case, `MemStore::_save()` fails to write $path/collections before
it gets killed, if it has more than 512 PGs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not add whoami as hb peer

it does not make sense to say hello to myself.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not store whoami or nonce in Heartbeat

we don't create messengers in the class which consumes them anymore, so
no need to have these member variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: lower debug level of i/o to "debug"

for better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/deepsea: use ceph_volume output in get_inventory()

DeepSea is being updated to use ceph_volume internally (see
https://github.com/SUSE/DeepSea/pull/1517).  Once this is done,
the mgr_orch.get_inventory runner will just be returning the
raw ceph_volume output, so this commit updates the DeepSea mgr
module to match.

Signed-off-by: Tim Serong <tserong@suse.com>

mgr/deepsea: log salt job failures

If a request to salt succeeds, but the event comes back with some internal
failure (e.g.: an exception was thrown inside the salt runner), we need to
log this so the admin can find out what on earth went wrong.

Signed-off-by: Tim Serong <tserong@suse.com>

mgr/deepsea: use empty strings rather than None for option defaults

This makes the _config_valid() function work properly; the option
defaults are being strinified somewhere, so we need to use empty
strings rather than None (which becomes "None"), in order to check
if they're set or not.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge PR #27301 into master

* refs/pull/27301/head:
qa: remove requirement on simple msgr

Reviewed-by: Sage Weil <sage@redhat.com>

qa: remove requirement on simple msgr

Fixes: http://tracker.ceph.com/issues/39079
Introduced-by: 28b4392a7191c5ed59c03cdd113050107a8109fc
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26970 into master

* refs/pull/26970/head:
mds: initialize cap_revoke_eviction_timeout with conf

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26992 into master

* refs/pull/26992/head:
mount.ceph.c: do not pass nofail to the kernel

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #27087 into master

* refs/pull/27087/head:
mds: remove duplicated check on balance amount

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27063 into master

* refs/pull/27063/head:
mds: there is an assertion when calling Beacon::shutdown()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #27294 from tspmelo/wip-nfs-squash

mgr/dashboard: Fix NFS squash default value

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mon/MonMap: add min_quorum_size() helper

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MDSMonitor: add 'mds ok-to-stop' command

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25353 from cbodley/wip-rgw-sysobj-yield

rgw: add optional_yield to SysObj service interfaces

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #25355 from cbodley/wip-rgw-http-yield

rgw: add optional_yield to http client interface

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #27276 from cbodley/wip-39033

rgw multisite: data sync checks empty next_marker for datalog

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mon: add ondisk incompat octopus feature

Signed-off-by: Sage Weil <sage@redhat.com>

mon/mon_types: add mon feature for octopus

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: SERVER_O -> SERVER_OCTOPUS

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add 'mon ok-to-{stop,add-offline,rm}' commands

Helpers to decide when it is safe to stop a mon, add a mon that is
not started, or remove a mon.  (Adding and start a mon would always
be safe, but it takes time to sync, so it's not really possible to do
quickly.)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27244 from tspmelo/wip-nfsv3-prop

mgr/dashboard: Fix deletion of NFS protocol properties

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #27130 from cbodley/wip-38700

rgw: decrypt filter does not cross multipart boundaries

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

mgr/dashboard: Fix NFS squash default value

Fixes: http://tracker.ceph.com/issues/39064
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #27206 into master

* refs/pull/27206/head:
osd: shutdown recovery_request_timer earlier

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27074 from LenzGr/master-documentation

doc: Updated dashboard iSCSI configuration, added labels

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27164 from Devp00l/issue-38937

mgr/dashboard: Back button component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27269 from rhcs-dashboard/wip-39030-master

mgr/dashboard: update vstart to use new ssl port

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27203 from falcon78921/wip-doc-38721

doc: removed OpenStack Kilo references in Keystone docs

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #27264 from rouming/fio-ceph-messenger--dummy-auth-for-v2

fio_ceph_messenger: catch up v2 proto changes by using dummy auth

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/BaseMgrModule: drop GIL in ceph_store_set

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/BaseMgrModule: drop GIL in set_config

Holding GIL and run mon_command turns out to be easily deadlock-prone.
https://tracker.ceph.com/issues/35985 is a typical report, and here is
another.

Fix by using PyModuleConfig's own lock for protection of concurrent
accesses, which should be sufficient enough.

Fixes: http://tracker.ceph.com/issues/39040
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Updated dashboard iSCSI configuration, added labels

Added note about the requirement for the latest ceph-iscsi version
3 to the dashboard documentation. Added some doc references
and replaced some URLs in the iSCSI docs with reST labels instead.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #27205 from xiexingguo/wip-38784

osd/PG: introduce all_missing_unfound helper

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27182 from xiexingguo/wip-38931

osd/PG: move down peers out from peer_purged

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PG: move down peers out from peer_purged

In purge_strays(), we'll aggressively clear stray_set and
add all related peers into peer_purged.

However, if the corrsponding peer is down and becomes
up again, (unconditionally) adding it to peer_purged
will prevent primary from re-purging it.
(See Active::react(const MNotifyRec& notevt))

On consuming a new osdmap, let's move any down peers out from
peer_purged simutaneously. This way we can lower the risk
of leaving any leftover PGs behind.

Related-to: http://tracker.ceph.com/issues/38931
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #27249 into master

* refs/pull/27249/head:
qa/suites/rados/thrash-old-clients: fix up v1/v2 options

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27133 into master

* refs/pull/27133/head:
log,global: do not start flusher thread until after we have our mon config
log: buffer log entries until flusher thread starts
log: open log file from flusher thread
common/ceph_context: fix log_to_file observer

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26976 into master

* refs/pull/26976/head:
doc: add doc for cephfs lazyio

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27199 into master

* refs/pull/27199/head:
mds: fix dump routine for session_info_t.used_inos
mds: fix some misleading log messages
client: comment fix in _lookup
mds: fix comment over MDRequestImpl
mds: use stdbool.h instead of hand-rolling our own bool type

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

mgr/dashboard: udpate vstart to use new ssl port

Fixes: https://tracker.ceph.com/issues/39030
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #27242 from theanalyst/beast-fe-ports

rgw: beast port parsing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: beast: set a default port for endpoints

if no port is specified we don't set one(or set to 0), and don't exactly error
out, so rgw keeps running without being bound to a port. Set the port to default
in these cases

Fixes: http://tracker.ceph.com/issues/39000
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: callers pass null_yield to RGWSI_SysObj

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to RGWSI_Notify

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to RGWSI_SysObj

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to RGWSI_SysObj_Cache

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to RGWSI_SysObj_Core interface

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSI_RADOS::Obj::notify() takes optional_yield

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_rados_notify() with optional_yield

Signed-off-by: Casey Bodley <cbodley@redhat.com>