mgr/orch/ssh: add qa test

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

qa: install/exclude ceph-mgr-ssh package

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

debian: package ssh orchestrator

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

spec.ceph: package ssh orchestrator

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

ssh/orch: add ssh orchestrator

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #26272 from ivancich/wip-log-excl-lock-busy-info

rgw: when exclusive lock fails due existing lock, log add'l info

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25926 from linuxbox2/wip-rgw-lc-prefix

rgw: lifcycle: don't reject compound rules with empty prefix

Merge pull request #26386 from votdev/improve_dashboard_module_opts

mgr/dashboard: Set MODULE_OPTIONS types and defaults

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26226 from trociny/wip-rbd-re-sparsify

rbd: online re-sparsify of images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26242 from p-na/wip-pna-osd-purge

mgr/dashboard: Implement OSD purge

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26431 from wjwithagen/wjw-fix-src_os_bluestore_KernelDevice.cc

os/bluestore: Only use F_SET_FILE_RW_HINT when available

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Implement OSD purge

Removes the `remove` functionality from the UI.

Fixes: http://tracker.ceph.com/issues/35811
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #26440 from xiexingguo/wip-balancer-week

mgr/balancer: restrict automatic balancing to specific weekdays

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26425 from trociny/wip-38315

journal: fix potential race when closing object recorder

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/balancer: restrict automatic balancing to specific weekdays

We have implemented similar semantics for the scrub behaviour.
Let's try to keep pace with that.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26353 into master

* refs/pull/26353/head:
osd/PG: print get/put under debug_refs
osd: clear priv on heartbeat_reset

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PG: print get/put under debug_refs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24805 into master

* refs/pull/24805/head:
qa/suite: add dedup test
src/tools: fix compile error (master version issue)
src/tools: add stats (fixed objects,total objects)
src/tools: make room for cdc
src/tools: make enhacned stats and interface class
src/tools: set timelimit and add signal handler to check progress
src/tools: use the slice thing and make parallel (chunk_scrub)
src/test: add max-thread test in test_dedup_tool.sh
src/tools: use the slice thing and make parallel
src/test: add chunk-scrub test in test_dedup_tool.sh
src/tools: add chunk-scrub op in dedup tool
src/cls/cas: add has_chunk op
src/test: add test_dedup_tool.sh
src/tools: initial works for dedup tool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26462 into master

* refs/pull/26462/head:
options: update some descriptions and see_also fields.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26187 into master

* refs/pull/26187/head:
mgr/balancer: balance pools with same crush_rule in batch
osd/OSDMap: more improvements to upmap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26435 into master

* refs/pull/26435/head:
msg/async/ProtocolV2: assume peer has SERVER_NAUTILUS for encoding v2 frame

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26275 from trociny/wip-mksnap-errmsg

rados: provide user with more meaningful error message

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #26454 from batrick/i37955-followup

ceph.in: use correct module for cmd flags

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26446 from tchaikov/wip-mgr-plugins-are-arch-indep

cmake,rpm,deb: install mgr plugins into /usr/share/ceph/mgr

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #26449 from tchaikov/wip-monc-cleanup

mon/MonClient: drop my_addr

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

options: update some descriptions and see_also fields.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

rpm: mark mgr plugins "noarch"

as they are architecture independent.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: s|lib/ceph/mgr|share/ceph/mgr|

as the mgr plugins are installed into $prefix/share/ceph/mgr now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: s|lib/ceph/mgr|share/ceph/mgr|

as the mgr plugins are installed into $prefix/share/ceph/mgr now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename CMAKE_INSTALL_{FULL_}?PKGLIBDIR

rename CMAKE_INSTALL_{FULL_}?PKGLIBDIR to
CEPH_INSTALL_{FULL_}?PKGLIBDIR. in order to differentiate them from the
paths not ended with "ceph".

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install mgr plugins into $prefix/share/ceph

as mgr plugins are architecture independent source files. see also
/usr/share/cmake-*/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #26455 into master

* refs/pull/26455/head:
qa/suites/upgrade/mimic-x/stress-split: drop pglog_hardlimit test
qa/suites/upgrade/mimic-x/stress-split: update for msgr2
qa/suites/upgrade/mimic-x/parallel: update for msgr v2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26416 from dzafman/wip-fix-test-loops

test: Limit loops waiting for force-backfill/force-recovery to happen

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26448 into master

* refs/pull/26448/head:
osd: do not send peers really old maps
osd: build_incremental_map_msg: recover if we are missing an incremental
osd: build_incremental_map_msg: behave if we have latest full but not incremental

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26345 into master

* refs/pull/26345/head:
ceph-osd: be helpful about upgrade gate
mon: record 'min_mon_release' file and prevent startup if it's too old
mon/Monitor: do not join cluster that is >2 releases old
mon/Elector: respect monmap min_mon_release
mon: check min_mon_release during probe
mon/MonmapMonitor: increase min_mon_release when full quorum is upgraded
mon: maintain quorum_min_mon_release
mon/MonMapMonitor: set initial min_mon_release
monmaptool: add --set-min-mon-release
monmaptool: don't spew usage on any error
mon/MonMap: add min_mon_release field

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #26456 into master

* refs/pull/26456/head:
mgr/PyModuleRegistry: avoid null deref from get_progress_events

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa/suites/upgrade/mimic-x/stress-split: drop pglog_hardlimit test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimic-x/stress-split: update for msgr2

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimic-x/parallel: update for msgr v2

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26212 from linuxbox2/wip-rgw-master-nosuch

rgw: check for non-existent bucket in RGWGetACLs

osd: clear priv on heartbeat_reset

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25739 into master

* refs/pull/25739/head:
mds: handle leftover reconnect information
client: split large reconnect into multiple messages
mds: update MClientReconnect encoding

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26423 into master

* refs/pull/26423/head:
mgr/volumes: create/delete fs subvolumes
pybind/ceph_volume_client: allow volume_client to

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26434 into master

* refs/pull/26434/head:
mds: return string_view for type
mds: reorg LogEvent def
mds: pass LogEvent as unique_ptr

Reviewed-by: Zheng Yan <zyan@redhat.com>

ceph.in: use correct module for cmd flags

argparse != ceph_argparse

Introduced-by: 19fbdb3ffde5bcb73792b1d822fe3f8d0407d490
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/PyModuleRegistry: avoid null deref from get_progress_events

active_modules may be null.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26445 into master

* refs/pull/26445/head:
mgr: hold GIL while generating a typed option value
mgr/ActivePyModules: do not touch PyFormatter without the GIL

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mon/MonClient: drop my_addr

the only consumer of this member variable is MgrStandby, and MgrStandby
does have access to the messenger. so let it get the IP address
directly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: do not send peers really old maps

We may receive a message that sat in a queue for a while with a low
priority and is tagged with an older epoch.  Don't send a bunch of old
maps that we have already sent the peer.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: build_incremental_map_msg: recover if we are missing an incremental

If we are building a history and are missing an incremental, continue with
a full map in its place.

Generally speaking, an OSD should *always* have a full map stored, but
may not have the corresponding incremental for various reasons.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: build_incremental_map_msg: behave if we have latest full but not incremental

Fixes 47a3f63ef71522d6b9572e81674598dd3e19df9d and
follow-on 57a13adc8d0e34b4bb1a4022eacbb3de2636df53

Fixes: http://tracker.ceph.com/issues/38330
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Set MODULE_OPTIONS types and defaults

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr: hold GIL while generating a typed option value

Drop the GIL while looking up the module and option value.  Retake
it before calling get_typed_option_value(), where we generate the
Python objects for the value.

Fixes: http://tracker.ceph.com/issues/38123
Fixes: http://tracker.ceph.com/issues/38204
Fixes: http://tracker.ceph.com/issues/38292
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ActivePyModules: do not touch PyFormatter without the GIL

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: move install(.. ./mgr) down to mgr

this helps to modularize the CMakeLists.txt

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26437 from tspmelo/wip-npm-update-19-2

mgr/dashboard: Update npm packages

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

rbd: online re-sparsify of images

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/rbd: add image sparsify method

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: online re-sparsify of images

Signed-off-by: Mykola Golub <mgolub@suse.com>

librados_test_stub: enable sparse_read

Although it actually does not do sparse read, enabling it makes
possible to link the same test code with both librados_test_stub
and real librados.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librados_test_stub: add cls_cxx_replace

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: add sparsify method

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26429 from rhcs-dashboard/fix-toast-notifications

mgr/dashboard: fix: toast notifications hiding utility menu

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26334 from votdev/fix_typo

mgr/dashboard: Fix typo in NoOrchesrtatorConfiguredException class name

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Update prettier to 1.16

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update npm packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

mds: return string_view for type

To avoid unnecessary heap allocation.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: reorg LogEvent def

To follow code-style guidelines and protect some members.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: pass LogEvent as unique_ptr

To avoid leaks.

Fixes: http://tracker.ceph.com/issues/38324
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/async/ProtocolV2: assume peer has SERVER_NAUTILUS for encoding v2 frame

The v2 frames may contain entity_addr_t and entity_addrvec_t.  Those have
been understood since the MSG_ADDR2 feature, but with nautilus we've
additionally added the TYPE_ANY, and we encode TYPE_ANY addrs as
TYPE_LEGACY when the peer lacks SERVER_NAUTILUS.

For the purposes of the v2 protocol, we assume the peer understands
TYPE_ANY.

No other complex types are used in the v2 frame protocol, so this has no
effect on, e.g., what features we assume the peer has for the purposes
of encoding actual messages.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26375 from dzafman/wip-38034

pg stuck in backfill_wait with plenty of disk space

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26238 into master

* refs/pull/26238/head:
mds: resolve memory leak of recall gather

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #26374 into master

* refs/pull/26374/head:
mds: fix potential re-evaluate stray dentry in _unlink_local_finish

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #26394 into master

* refs/pull/26394/head:
mds: remove superfluous cap sequence increment
mds: fix MDCache::finish_snaprealm_reconnect()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd: Fix the problem by rechecking backfill status each time
a RemoteReservationRevoked message is received, so we can
restart the backfill process in a more safe and less-likely raced way.

Fixes: http://tracker.ceph.com/issues/38034
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Revert "osd: Handle getting RecoveryDone while in RepWaitBackfillReserved"

This reverts commit 83c2521a5316c922875746a40b95100b62865e67.

This original fix handled the race by accepting the RecoveryDone
while in RepWaitBackfillReserved by going to RepNotRecovering.
Once we weren't crashing OSDs, none of the tests at the time
detected that backfill was actually hung.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #26371 from dzafman/wip-38262

test: run-standalone.sh: point LD_LIBRARY_PATH to $(pwd)/lib

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26297 from s0nea/wip-dashboard-readonly-config-options

mgr/dashboard: disallow editing read-only config options

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26316 from wjwithagen/wjw-fix-FindStdFilesystem

cmake: Make the tests for finding Filesystem with more serious functions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Bluestore: Only use F_SET_FILE_RW_HINT when available

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #26063 into master

* refs/pull/26063/head:
ceph: omit hidden commands from suggestions

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26413 from liewegas/wip-38282

osd: fix build_incremental_map_msg

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26420 from tchaikov/wip-pgmap-more-includes

mon/PGMap: add more #include

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: fix: toast notifications hiding utility menu

* Fixed margin-top taking into account responsiveness.
* dropdown-menu class: set z-index to avoid notification
  hiding dropdown menus when menu item clicked.

Fixes: https://tracker.ceph.com/issues/38313
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

cmake: Make the tests for finding Filesystem with more serious functions

Otherwise it is linked agains lc++ on FreeBSD,a dn that will satisfy both
tested libraries.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #26085 from tspmelo/wip-dashboard-nfs

mgr/dashboard: NFS management UI

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26149 from votdev/improve_get_typed_config

mgr: Improve ActivePyModules::get_typed_config implementation

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26407 from votdev/rgw_column_style

mgr/dashboard: Sync column style with the rest of the UI

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #25963 from smithfarm/wip-python-dep-cleanup

rpm: make Python dependencies somewhat less confusing

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26421 from b-ranto/wip-rest-fix

restful: Fix regression when traversing leaf nodes

Reviewed-by: Kefu Chai <kchai@redhat.com>

journal: fix potential race when closing object recorder

Calls `send_appends_aio` only if m_pending_buffers is not
empty. It was supposed it was ok to call it even for empty
buffers, because `send_appends_aio` just returned in this
case. But the problem is caused by m_aio_scheduled flag, which is
set before releasing the lock and cleared in `send_appends_aio`,
after reacquiring the lock. If during this time window `close` is
called it will return false due to m_aio_scheduled flag set, and
the caller will expect "closed" notification, which is never
fired in this case.

Fixes: https://tracker.ceph.com/issues/38315
Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Add UI for NFS Form

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add UI for NFS List and Detail

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add NFS frontend services

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/volumes: create/delete fs subvolumes

... within the specified fs volume.

Previously, subvolumes (fs subdirectories) weren't created within
the desired volumes (fs filestyems).

Fixes: https://tracker.ceph.com/issues/38297
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #26410 from tspmelo/wip-bootstrap-3.4.1

mgr/dashboard: Update bootstrap to v3.4.1

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #26409 from rhcs-dashboard/fix-tox-ini

mgr/dashboard: fix: tox not detecting deps changes

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Fix typo in NoOrchesrtatorConfiguredException class name

Signed-off-by: Volker Theile <vtheile@suse.com>

pybind/ceph_volume_client: allow volume_client to

... mount a specificied ceph filesystem.

Signed-off-by: Ramana Raja <rraja@redhat.com>

restful: Fix regression when traversing leaf nodes

The commit 23b6c90 introduced a regression when traversing leaf nodes.
The issue is that it traverses the keys of a `dict` returned by
`nodes_by_id`, not the actual `items` of the node. That resulted in an
500 error because it tried to treat `str` as a `dict` and failed.

Signed-off-by: Boris Ranto <branto@redhat.com>