Merge PR #26435 into master

* refs/pull/26435/head:
msg/async/ProtocolV2: assume peer has SERVER_NAUTILUS for encoding v2 frame

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26275 from trociny/wip-mksnap-errmsg

rados: provide user with more meaningful error message

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #26454 from batrick/i37955-followup

ceph.in: use correct module for cmd flags

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26446 from tchaikov/wip-mgr-plugins-are-arch-indep

cmake,rpm,deb: install mgr plugins into /usr/share/ceph/mgr

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #26449 from tchaikov/wip-monc-cleanup

mon/MonClient: drop my_addr

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rpm: mark mgr plugins "noarch"

as they are architecture independent.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: s|lib/ceph/mgr|share/ceph/mgr|

as the mgr plugins are installed into $prefix/share/ceph/mgr now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: s|lib/ceph/mgr|share/ceph/mgr|

as the mgr plugins are installed into $prefix/share/ceph/mgr now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename CMAKE_INSTALL_{FULL_}?PKGLIBDIR

rename CMAKE_INSTALL_{FULL_}?PKGLIBDIR to
CEPH_INSTALL_{FULL_}?PKGLIBDIR. in order to differentiate them from the
paths not ended with "ceph".

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install mgr plugins into $prefix/share/ceph

as mgr plugins are architecture independent source files. see also
/usr/share/cmake-*/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #26455 into master

* refs/pull/26455/head:
qa/suites/upgrade/mimic-x/stress-split: drop pglog_hardlimit test
qa/suites/upgrade/mimic-x/stress-split: update for msgr2
qa/suites/upgrade/mimic-x/parallel: update for msgr v2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26416 from dzafman/wip-fix-test-loops

test: Limit loops waiting for force-backfill/force-recovery to happen

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26448 into master

* refs/pull/26448/head:
osd: do not send peers really old maps
osd: build_incremental_map_msg: recover if we are missing an incremental
osd: build_incremental_map_msg: behave if we have latest full but not incremental

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26345 into master

* refs/pull/26345/head:
ceph-osd: be helpful about upgrade gate
mon: record 'min_mon_release' file and prevent startup if it's too old
mon/Monitor: do not join cluster that is >2 releases old
mon/Elector: respect monmap min_mon_release
mon: check min_mon_release during probe
mon/MonmapMonitor: increase min_mon_release when full quorum is upgraded
mon: maintain quorum_min_mon_release
mon/MonMapMonitor: set initial min_mon_release
monmaptool: add --set-min-mon-release
monmaptool: don't spew usage on any error
mon/MonMap: add min_mon_release field

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #26456 into master

* refs/pull/26456/head:
mgr/PyModuleRegistry: avoid null deref from get_progress_events

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa/suites/upgrade/mimic-x/stress-split: drop pglog_hardlimit test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimic-x/stress-split: update for msgr2

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimic-x/parallel: update for msgr v2

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26212 from linuxbox2/wip-rgw-master-nosuch

rgw: check for non-existent bucket in RGWGetACLs

Merge PR #25739 into master

* refs/pull/25739/head:
mds: handle leftover reconnect information
client: split large reconnect into multiple messages
mds: update MClientReconnect encoding

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26423 into master

* refs/pull/26423/head:
mgr/volumes: create/delete fs subvolumes
pybind/ceph_volume_client: allow volume_client to

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26434 into master

* refs/pull/26434/head:
mds: return string_view for type
mds: reorg LogEvent def
mds: pass LogEvent as unique_ptr

Reviewed-by: Zheng Yan <zyan@redhat.com>

ceph.in: use correct module for cmd flags

argparse != ceph_argparse

Introduced-by: 19fbdb3ffde5bcb73792b1d822fe3f8d0407d490
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/PyModuleRegistry: avoid null deref from get_progress_events

active_modules may be null.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26445 into master

* refs/pull/26445/head:
mgr: hold GIL while generating a typed option value
mgr/ActivePyModules: do not touch PyFormatter without the GIL

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mon/MonClient: drop my_addr

the only consumer of this member variable is MgrStandby, and MgrStandby
does have access to the messenger. so let it get the IP address
directly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: do not send peers really old maps

We may receive a message that sat in a queue for a while with a low
priority and is tagged with an older epoch.  Don't send a bunch of old
maps that we have already sent the peer.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: build_incremental_map_msg: recover if we are missing an incremental

If we are building a history and are missing an incremental, continue with
a full map in its place.

Generally speaking, an OSD should *always* have a full map stored, but
may not have the corresponding incremental for various reasons.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: build_incremental_map_msg: behave if we have latest full but not incremental

Fixes 47a3f63ef71522d6b9572e81674598dd3e19df9d and
follow-on 57a13adc8d0e34b4bb1a4022eacbb3de2636df53

Fixes: http://tracker.ceph.com/issues/38330
Signed-off-by: Sage Weil <sage@redhat.com>

mgr: hold GIL while generating a typed option value

Drop the GIL while looking up the module and option value.  Retake
it before calling get_typed_option_value(), where we generate the
Python objects for the value.

Fixes: http://tracker.ceph.com/issues/38123
Fixes: http://tracker.ceph.com/issues/38204
Fixes: http://tracker.ceph.com/issues/38292
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ActivePyModules: do not touch PyFormatter without the GIL

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: move install(.. ./mgr) down to mgr

this helps to modularize the CMakeLists.txt

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26437 from tspmelo/wip-npm-update-19-2

mgr/dashboard: Update npm packages

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26429 from rhcs-dashboard/fix-toast-notifications

mgr/dashboard: fix: toast notifications hiding utility menu

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26334 from votdev/fix_typo

mgr/dashboard: Fix typo in NoOrchesrtatorConfiguredException class name

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Update prettier to 1.16

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update npm packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

mds: return string_view for type

To avoid unnecessary heap allocation.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: reorg LogEvent def

To follow code-style guidelines and protect some members.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: pass LogEvent as unique_ptr

To avoid leaks.

Fixes: http://tracker.ceph.com/issues/38324
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/async/ProtocolV2: assume peer has SERVER_NAUTILUS for encoding v2 frame

The v2 frames may contain entity_addr_t and entity_addrvec_t.  Those have
been understood since the MSG_ADDR2 feature, but with nautilus we've
additionally added the TYPE_ANY, and we encode TYPE_ANY addrs as
TYPE_LEGACY when the peer lacks SERVER_NAUTILUS.

For the purposes of the v2 protocol, we assume the peer understands
TYPE_ANY.

No other complex types are used in the v2 frame protocol, so this has no
effect on, e.g., what features we assume the peer has for the purposes
of encoding actual messages.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26375 from dzafman/wip-38034

pg stuck in backfill_wait with plenty of disk space

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26238 into master

* refs/pull/26238/head:
mds: resolve memory leak of recall gather

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #26374 into master

* refs/pull/26374/head:
mds: fix potential re-evaluate stray dentry in _unlink_local_finish

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #26394 into master

* refs/pull/26394/head:
mds: remove superfluous cap sequence increment
mds: fix MDCache::finish_snaprealm_reconnect()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd: Fix the problem by rechecking backfill status each time
a RemoteReservationRevoked message is received, so we can
restart the backfill process in a more safe and less-likely raced way.

Fixes: http://tracker.ceph.com/issues/38034
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Revert "osd: Handle getting RecoveryDone while in RepWaitBackfillReserved"

This reverts commit 83c2521a5316c922875746a40b95100b62865e67.

This original fix handled the race by accepting the RecoveryDone
while in RepWaitBackfillReserved by going to RepNotRecovering.
Once we weren't crashing OSDs, none of the tests at the time
detected that backfill was actually hung.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #26371 from dzafman/wip-38262

test: run-standalone.sh: point LD_LIBRARY_PATH to $(pwd)/lib

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26297 from s0nea/wip-dashboard-readonly-config-options

mgr/dashboard: disallow editing read-only config options

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26316 from wjwithagen/wjw-fix-FindStdFilesystem

cmake: Make the tests for finding Filesystem with more serious functions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26063 into master

* refs/pull/26063/head:
ceph: omit hidden commands from suggestions

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26413 from liewegas/wip-38282

osd: fix build_incremental_map_msg

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26420 from tchaikov/wip-pgmap-more-includes

mon/PGMap: add more #include

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: fix: toast notifications hiding utility menu

* Fixed margin-top taking into account responsiveness.
* dropdown-menu class: set z-index to avoid notification
  hiding dropdown menus when menu item clicked.

Fixes: https://tracker.ceph.com/issues/38313
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

cmake: Make the tests for finding Filesystem with more serious functions

Otherwise it is linked agains lc++ on FreeBSD,a dn that will satisfy both
tested libraries.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #26085 from tspmelo/wip-dashboard-nfs

mgr/dashboard: NFS management UI

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26149 from votdev/improve_get_typed_config

mgr: Improve ActivePyModules::get_typed_config implementation

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26407 from votdev/rgw_column_style

mgr/dashboard: Sync column style with the rest of the UI

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #25963 from smithfarm/wip-python-dep-cleanup

rpm: make Python dependencies somewhat less confusing

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26421 from b-ranto/wip-rest-fix

restful: Fix regression when traversing leaf nodes

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Add UI for NFS Form

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add UI for NFS List and Detail

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add NFS frontend services

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/volumes: create/delete fs subvolumes

... within the specified fs volume.

Previously, subvolumes (fs subdirectories) weren't created within
the desired volumes (fs filestyems).

Fixes: https://tracker.ceph.com/issues/38297
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #26410 from tspmelo/wip-bootstrap-3.4.1

mgr/dashboard: Update bootstrap to v3.4.1

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #26409 from rhcs-dashboard/fix-tox-ini

mgr/dashboard: fix: tox not detecting deps changes

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Fix typo in NoOrchesrtatorConfiguredException class name

Signed-off-by: Volker Theile <vtheile@suse.com>

pybind/ceph_volume_client: allow volume_client to

... mount a specificied ceph filesystem.

Signed-off-by: Ramana Raja <rraja@redhat.com>

restful: Fix regression when traversing leaf nodes

The commit 23b6c90 introduced a regression when traversing leaf nodes.
The issue is that it traverses the keys of a `dict` returned by
`nodes_by_id`, not the actual `items` of the node. That resulted in an
500 error because it tried to treat `str` as a `dict` and failed.

Signed-off-by: Boris Ranto <branto@redhat.com>

rpm: move Python deps out of distro-conditional blocks

The python%{_python_buildid}-bcrypt and python%{_python_buildid}-requests RPMs
are identically named across all the RPM distros, so move them out of the
distro conditional blocks.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: make Python build deps somewhat less confusing

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mon/PGMap: add more #include

* common/Clock.h for ceph_clock_now()
* global/global_context.h for g_conf()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26210 from falcon78921/wip-mgr-restful-python3

mgr/restful: updated string formatting to str.format()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26315 from tchaikov/wip-fix-Waddress-of-packed-member

msg/msg_types.h: do not cast `ceph_entity_name` to `entity_name_t` for printing

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26380 from tchaikov/wip-split-dashboard-package

deb,rpm,qa: split dashboard package

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26419 from cyx1231st/wip-fix-echo

test/crimson: fixes of unittest_seastar_echo

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25989 from votdev/feature_36488

mgr/dashboard: Add UI to configure the telemetry mgr plugin

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26398 from SUSE/wip-fix-spec-badness

ceph.spec.in: Don't use noarch for mgr module subpackages, fix /usr/lib64/ceph/mgr dir ownership

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: handle leftover reconnect information

previous commit allows client to split reconnect into multiple
messages. mds may evict client after getting incomplete recconect.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

client: split large reconnect into multiple messages

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #26393 from zmc/wip-rhel7-deps

install-deps.sh: Fixes for RHEL 7

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: update MClientReconnect encoding

The old encoding assumes that snaprealms are encoded at the tail of
message payload. So it does not allow adding new fields to the message.
The patch introduce new encoding for MClientReconnect, the new encoding
allows us to extend MClientReconnect.

The new encoding is not compatible with the old encoding. If mds does
not understand the new encoding. client needs to use the old encoding
to encode MClientReconnect.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

test: Limit loops waiting for force-backfill/force-recovery to happen

Fixes: http://tracker.ceph.com/issues/38309
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #26411 from liewegas/wip-remove-smart

mgr/smart: remove obsolete smart module

Merge PR #26389 into master

* refs/pull/26389/head:
message/MMonMgrReport: conditionally reencode PGMapDigest
qa/suites/upgrade/luminous-x/parallel: enable all classes
qa/suites/upgrade/luminous-x/parallel/5-final-workload/rados_mon_thrash: use x branch
qa/suites/upgade/luminous-x: pglog_hardlimit succeeds now on luminous due to backport
mgr/DaemonServer: use a luminous-compatible 'mgr metadata' command
mgr/Mgr: print bad (non-object) json
qa/suites/upgrade/luminous-x/stress-split: mons on separate hosts, enable msgr2
qa/suites/upgrade/luminous-x/parallel: mon per host, msgr2
qa/suites/upgrade/luminous-x: whitelist 'slow request'
mon/HealthMonitor: add mon_warn_on_msgr2_not_enabled

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

osd: fix build_incremental_map_msg

We need to fall back to an old map if since (the peer's epoch) is *older*
than our oldest.  If it's newer, we have it, and can just send
incrementals.

Fixes: http://tracker.ceph.com/issues/38282
Signed-off-by: Sage Weil <sage@redhat.com>

message/MMonMgrReport: conditionally reencode PGMapDigest

The encoding changed between luminous and mimic.  Reencode the
structure if the target doesn't have all the features.

Fixes: http://tracker.ceph.com/issues/38295
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/luminous-x/parallel: enable all classes

Otherwise it's annoying because the class list changes between luminous and nautilus,
and we don't want to futz around with changing this setting during the upgrade.

The problematic classes are 'cas' (added) and 'sdk' (not enabled by default but
included by the cls/ workunit.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/luminous-x/parallel/5-final-workload/rados_mon_thrash: use x branch

The luminous version is (1) not what we want and (2) will fail because
ceph_test_rados_api_tier no longer exists in master.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26397 from guits/guits-fix_ceph-volume

ceph-volume: update testing playbook 'deploy.yml'

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/smart: remove

This is obsolete and replaced by the devicehealth module.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/restful: updated string formatting to str.format()

Made changes per @tchaikov's request

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

ceph-osd: be helpful about upgrade gate

Signed-off-by: Sage Weil <sage@redhat.com>

mon: record 'min_mon_release' file and prevent startup if it's too old

Signed-off-by: Sage Weil <sage@redhat.com>

mon/Monitor: do not join cluster that is >2 releases old

This enforces the N+2 upgrade rule from the mon's perspective.

Note that this safety check is not as safe as the OSDs.  Notably, we
start up our backend store (rocksdb) *before* we probe other monitors
and discover any newer monmap that tells us we shouldn't join.  If there
is a *rocksdb* backward-compatibility problem it is too late by this
point.  Unfortunately, I don't see an easy way to get this far before
rocksdb is read-write--not without a lot more code, at least!

However, we'll still protect against a whole class of other potential
problems by not getting involved in a cluster that is too old.  :)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/Elector: respect monmap min_mon_release

Signed-off-by: Sage Weil <sage@redhat.com>

mon: check min_mon_release during probe

This will give us more informative errors messages than the feature bit
strings.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgade/luminous-x: pglog_hardlimit succeeds now on luminous due to backport

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: use a luminous-compatible 'mgr metadata' command

In luminous it was 'id', in mimic it changed to 'who'.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/Mgr: print bad (non-object) json

Signed-off-by: Sage Weil <sage@redhat.com>