mgr/dashboard: Infrastructure for multiple backstores (backend)

Fixes: https://tracker.ceph.com/issues/38090
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #26405 from jtlayton/wip-jlayton-orch

Allow rook to scale the mon count

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #23221 from baixueyu/wip-rgw-cleanup-regex

rgw: delete some unused code about std::regex

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #26254 from tchaikov/wip-crimson-cache

crimson: cache osdmap using LRU cache

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #26378 from votdev/issue_38267

mgr/dashboard: Module dashboard.services.ganesha has several lint issues

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

rgw: delete some unused code about std::regex

Signed-off-by: Xueyu Bai <baixueyu@inspur.com>

cmake: add PGMap.cc to crimson-common

otherwise we have failures like:

/usr/bin/ld: ../../../lib/libcrimson-common.a(Message.cc.o): in function
`PGMapDigest::~PGMapDigest()':
/var/ssd/ceph/src/mon/PGMap.h:37: undefined reference to `vtable for
PGMapDigest'
/usr/bin/ld: ../../../lib/libcrimson-common.a(Message.cc.o): in function
`PGMapDigest::~PGMapDigest()':

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: set_callback in ctor

there is no need to do so in `start()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: lookup using unordered_map first

assuming the unordered_map in LRU cache is faster.

inspired by #25480 by Radoslaw Zarzynski

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: cache osdmaps using LRU cache

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25570 from cbodley/wip-rgw-cls-bi-list-log-level

cls/rgw: raise debug level of bi_log_iterate_entries output

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #26491 from ceph/revert-25926-wip-rgw-lc-prefix

Revert "rgw: lifcycle: don't reject compound rules with empty prefix"

Revert "rgw: lifcycle: don't reject compound rules with empty prefix"

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26399 from jan--f/c-v-fix-hashable-py3

ceph-volume: make Device hashable to allow set of Device list in py3

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #26401 from jan--f/c-v-fix-siez-format-py3

ceph-volume: implement __format__ in Size to format sizes in py3

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #26403 from jan--f/c-v-no-iteritems-py3

ceph-volume: remove iteritems instances

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #26463 into master

* refs/pull/26463/head:
script/ptl-tool: allow invocation within a subdirectory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26272 from ivancich/wip-log-excl-lock-busy-info

rgw: when exclusive lock fails due existing lock, log add'l info

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25926 from linuxbox2/wip-rgw-lc-prefix

rgw: lifcycle: don't reject compound rules with empty prefix

Merge pull request #26386 from votdev/improve_dashboard_module_opts

mgr/dashboard: Set MODULE_OPTIONS types and defaults

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26226 from trociny/wip-rbd-re-sparsify

rbd: online re-sparsify of images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26242 from p-na/wip-pna-osd-purge

mgr/dashboard: Implement OSD purge

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #26431 from wjwithagen/wjw-fix-src_os_bluestore_KernelDevice.cc

os/bluestore: Only use F_SET_FILE_RW_HINT when available

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Implement OSD purge

Removes the `remove` functionality from the UI.

Fixes: http://tracker.ceph.com/issues/35811
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #26440 from xiexingguo/wip-balancer-week

mgr/balancer: restrict automatic balancing to specific weekdays

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26425 from trociny/wip-38315

journal: fix potential race when closing object recorder

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/balancer: restrict automatic balancing to specific weekdays

We have implemented similar semantics for the scrub behaviour.
Let's try to keep pace with that.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26353 into master

* refs/pull/26353/head:
osd/PG: print get/put under debug_refs
osd: clear priv on heartbeat_reset

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PG: print get/put under debug_refs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24805 into master

* refs/pull/24805/head:
qa/suite: add dedup test
src/tools: fix compile error (master version issue)
src/tools: add stats (fixed objects,total objects)
src/tools: make room for cdc
src/tools: make enhacned stats and interface class
src/tools: set timelimit and add signal handler to check progress
src/tools: use the slice thing and make parallel (chunk_scrub)
src/test: add max-thread test in test_dedup_tool.sh
src/tools: use the slice thing and make parallel
src/test: add chunk-scrub test in test_dedup_tool.sh
src/tools: add chunk-scrub op in dedup tool
src/cls/cas: add has_chunk op
src/test: add test_dedup_tool.sh
src/tools: initial works for dedup tool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26462 into master

* refs/pull/26462/head:
options: update some descriptions and see_also fields.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

script/ptl-tool: allow invocation within a subdirectory

Make ptl-tool.py work from a subdirectory inside the repo (e.g., build/).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26187 into master

* refs/pull/26187/head:
mgr/balancer: balance pools with same crush_rule in batch
osd/OSDMap: more improvements to upmap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26435 into master

* refs/pull/26435/head:
msg/async/ProtocolV2: assume peer has SERVER_NAUTILUS for encoding v2 frame

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26275 from trociny/wip-mksnap-errmsg

rados: provide user with more meaningful error message

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #26454 from batrick/i37955-followup

ceph.in: use correct module for cmd flags

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26446 from tchaikov/wip-mgr-plugins-are-arch-indep

cmake,rpm,deb: install mgr plugins into /usr/share/ceph/mgr

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #26449 from tchaikov/wip-monc-cleanup

mon/MonClient: drop my_addr

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

options: update some descriptions and see_also fields.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

rpm: mark mgr plugins "noarch"

as they are architecture independent.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: s|lib/ceph/mgr|share/ceph/mgr|

as the mgr plugins are installed into $prefix/share/ceph/mgr now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: s|lib/ceph/mgr|share/ceph/mgr|

as the mgr plugins are installed into $prefix/share/ceph/mgr now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename CMAKE_INSTALL_{FULL_}?PKGLIBDIR

rename CMAKE_INSTALL_{FULL_}?PKGLIBDIR to
CEPH_INSTALL_{FULL_}?PKGLIBDIR. in order to differentiate them from the
paths not ended with "ceph".

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install mgr plugins into $prefix/share/ceph

as mgr plugins are architecture independent source files. see also
/usr/share/cmake-*/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #26455 into master

* refs/pull/26455/head:
qa/suites/upgrade/mimic-x/stress-split: drop pglog_hardlimit test
qa/suites/upgrade/mimic-x/stress-split: update for msgr2
qa/suites/upgrade/mimic-x/parallel: update for msgr v2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26416 from dzafman/wip-fix-test-loops

test: Limit loops waiting for force-backfill/force-recovery to happen

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26448 into master

* refs/pull/26448/head:
osd: do not send peers really old maps
osd: build_incremental_map_msg: recover if we are missing an incremental
osd: build_incremental_map_msg: behave if we have latest full but not incremental

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26345 into master

* refs/pull/26345/head:
ceph-osd: be helpful about upgrade gate
mon: record 'min_mon_release' file and prevent startup if it's too old
mon/Monitor: do not join cluster that is >2 releases old
mon/Elector: respect monmap min_mon_release
mon: check min_mon_release during probe
mon/MonmapMonitor: increase min_mon_release when full quorum is upgraded
mon: maintain quorum_min_mon_release
mon/MonMapMonitor: set initial min_mon_release
monmaptool: add --set-min-mon-release
monmaptool: don't spew usage on any error
mon/MonMap: add min_mon_release field

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #26456 into master

* refs/pull/26456/head:
mgr/PyModuleRegistry: avoid null deref from get_progress_events

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa/suites/upgrade/mimic-x/stress-split: drop pglog_hardlimit test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimic-x/stress-split: update for msgr2

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimic-x/parallel: update for msgr v2

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26212 from linuxbox2/wip-rgw-master-nosuch

rgw: check for non-existent bucket in RGWGetACLs

osd: clear priv on heartbeat_reset

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25739 into master

* refs/pull/25739/head:
mds: handle leftover reconnect information
client: split large reconnect into multiple messages
mds: update MClientReconnect encoding

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26423 into master

* refs/pull/26423/head:
mgr/volumes: create/delete fs subvolumes
pybind/ceph_volume_client: allow volume_client to

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26434 into master

* refs/pull/26434/head:
mds: return string_view for type
mds: reorg LogEvent def
mds: pass LogEvent as unique_ptr

Reviewed-by: Zheng Yan <zyan@redhat.com>

ceph.in: use correct module for cmd flags

argparse != ceph_argparse

Introduced-by: 19fbdb3ffde5bcb73792b1d822fe3f8d0407d490
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/PyModuleRegistry: avoid null deref from get_progress_events

active_modules may be null.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26445 into master

* refs/pull/26445/head:
mgr: hold GIL while generating a typed option value
mgr/ActivePyModules: do not touch PyFormatter without the GIL

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mon/MonClient: drop my_addr

the only consumer of this member variable is MgrStandby, and MgrStandby
does have access to the messenger. so let it get the IP address
directly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: do not send peers really old maps

We may receive a message that sat in a queue for a while with a low
priority and is tagged with an older epoch.  Don't send a bunch of old
maps that we have already sent the peer.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: build_incremental_map_msg: recover if we are missing an incremental

If we are building a history and are missing an incremental, continue with
a full map in its place.

Generally speaking, an OSD should *always* have a full map stored, but
may not have the corresponding incremental for various reasons.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: build_incremental_map_msg: behave if we have latest full but not incremental

Fixes 47a3f63ef71522d6b9572e81674598dd3e19df9d and
follow-on 57a13adc8d0e34b4bb1a4022eacbb3de2636df53

Fixes: http://tracker.ceph.com/issues/38330
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Set MODULE_OPTIONS types and defaults

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr: hold GIL while generating a typed option value

Drop the GIL while looking up the module and option value.  Retake
it before calling get_typed_option_value(), where we generate the
Python objects for the value.

Fixes: http://tracker.ceph.com/issues/38123
Fixes: http://tracker.ceph.com/issues/38204
Fixes: http://tracker.ceph.com/issues/38292
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ActivePyModules: do not touch PyFormatter without the GIL

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: move install(.. ./mgr) down to mgr

this helps to modularize the CMakeLists.txt

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26437 from tspmelo/wip-npm-update-19-2

mgr/dashboard: Update npm packages

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

rbd: online re-sparsify of images

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/rbd: add image sparsify method

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: online re-sparsify of images

Signed-off-by: Mykola Golub <mgolub@suse.com>

librados_test_stub: enable sparse_read

Although it actually does not do sparse read, enabling it makes
possible to link the same test code with both librados_test_stub
and real librados.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librados_test_stub: add cls_cxx_replace

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: add sparsify method

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26429 from rhcs-dashboard/fix-toast-notifications

mgr/dashboard: fix: toast notifications hiding utility menu

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26334 from votdev/fix_typo

mgr/dashboard: Fix typo in NoOrchesrtatorConfiguredException class name

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Update prettier to 1.16

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update npm packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

mds: return string_view for type

To avoid unnecessary heap allocation.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: reorg LogEvent def

To follow code-style guidelines and protect some members.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: pass LogEvent as unique_ptr

To avoid leaks.

Fixes: http://tracker.ceph.com/issues/38324
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/async/ProtocolV2: assume peer has SERVER_NAUTILUS for encoding v2 frame

The v2 frames may contain entity_addr_t and entity_addrvec_t.  Those have
been understood since the MSG_ADDR2 feature, but with nautilus we've
additionally added the TYPE_ANY, and we encode TYPE_ANY addrs as
TYPE_LEGACY when the peer lacks SERVER_NAUTILUS.

For the purposes of the v2 protocol, we assume the peer understands
TYPE_ANY.

No other complex types are used in the v2 frame protocol, so this has no
effect on, e.g., what features we assume the peer has for the purposes
of encoding actual messages.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26375 from dzafman/wip-38034

pg stuck in backfill_wait with plenty of disk space

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26238 into master

* refs/pull/26238/head:
mds: resolve memory leak of recall gather

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #26374 into master

* refs/pull/26374/head:
mds: fix potential re-evaluate stray dentry in _unlink_local_finish

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #26394 into master

* refs/pull/26394/head:
mds: remove superfluous cap sequence increment
mds: fix MDCache::finish_snaprealm_reconnect()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd: Fix the problem by rechecking backfill status each time
a RemoteReservationRevoked message is received, so we can
restart the backfill process in a more safe and less-likely raced way.

Fixes: http://tracker.ceph.com/issues/38034
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Revert "osd: Handle getting RecoveryDone while in RepWaitBackfillReserved"

This reverts commit 83c2521a5316c922875746a40b95100b62865e67.

This original fix handled the race by accepting the RecoveryDone
while in RepWaitBackfillReserved by going to RepNotRecovering.
Once we weren't crashing OSDs, none of the tests at the time
detected that backfill was actually hung.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #26371 from dzafman/wip-38262

test: run-standalone.sh: point LD_LIBRARY_PATH to $(pwd)/lib

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26297 from s0nea/wip-dashboard-readonly-config-options

mgr/dashboard: disallow editing read-only config options

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26316 from wjwithagen/wjw-fix-FindStdFilesystem

cmake: Make the tests for finding Filesystem with more serious functions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Bluestore: Only use F_SET_FILE_RW_HINT when available

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

crimson/common: added templated LRU cache

the implementation is modeled after src/common/shared_cache.hpp, with
following changes/improvements

* split the implementation into two parts:
  - simple_lru.h: for the basic LRU cache. which tracks the least recently
    used (LRU) cache using a list, and uses a map for lookup. in
    future, we could use boost's multiple-index-container to replace
    this combination.
  - shared_lru.h: for the improved version of LRU cache, it also tries
    to track the evcited but still alive objects using a customized
    deleter of shared_ptr, and it keeps a map of weak_ptr pointing to
    these objects. the customized deleter will remove the tracked
    item in weak_ptr map once all shared_ptrs are destroyed.
* specialize LRUCache for ordered and non-ordered lookup. in existing
  OSD, `SimpleLRU` is used for caching the encoded osdmaps, and we
  don't use `lower_bound()`, `upper_bound()` or `get_next()` accessors
  for accessing these maps, so no need to store them with `std::map`.
  and in crimson-osd, it would be great if we can ditch `SimpleLRU`,
  and reuse `LRUCache` directly if `SharedLRU` is overkill. so let's
  prepare for this by specializing for the `unordered_map<>`.
* remove (not yet) unused methods from SharedLRU, like `purge()`,
  dump_weak_refs()
* change some function signatures to resemble those in `std::map`,
  for instance,
  - change `lookup()` to `find()`
  - change `lookup_or_create()` to `operator[]()`
* add comment to document `SharedLRU` class
* add test for SharedLRU

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #26063 into master

* refs/pull/26063/head:
ceph: omit hidden commands from suggestions

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26413 from liewegas/wip-38282

osd: fix build_incremental_map_msg

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26420 from tchaikov/wip-pgmap-more-includes

mon/PGMap: add more #include

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: fix: toast notifications hiding utility menu

* Fixed margin-top taking into account responsiveness.
* dropdown-menu class: set z-index to avoid notification
  hiding dropdown menus when menu item clicked.

Fixes: https://tracker.ceph.com/issues/38313
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

cmake: Make the tests for finding Filesystem with more serious functions

Otherwise it is linked agains lc++ on FreeBSD,a dn that will satisfy both
tested libraries.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #26085 from tspmelo/wip-dashboard-nfs

mgr/dashboard: NFS management UI

Reviewed-by: Ricardo Dias <rdias@suse.com>