pybind/mirroring: introduce peer_bootstrap {create|import} commands

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: use peer cluster monitor address (and key) if available

This allows connecting to the peer cluster without having the cluster
configuration file on the primary cluster.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mon: peer_add should accept Ceph file system UUID

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mon: introduce "profile cephfs-mirror" cap constrained to "config-get cephfs/mirror/peer"

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add test for failed filesystem mirror instances

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: restart failed mirror filesystem instances

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40021 from xxhdx1985126/wip-io-sequencer-bug-fix

crimson/osd: retrive client_requests' prev_op_id right before "start_op"

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: retrive client_requests' prev_op_id right before "start_op"

ClientRequest::prev_op_id should record its immediate predecessor in the
pipeline. If we capture sequencer's last_issued when creating the client
request, it may not represent that predecessor

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39859 from jhonxue/master

osd: do not dump an osd multiple times

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39989 from athanatos/sjust/wip-fix-seastore-gc

crimson: btree_lba_manager and segment_cleaner fixes

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: fix lookup

lower_bound() here is wrong, use bound instead.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add releasing state for segments pending close

This should fix a bug by which we might start scanning a segment a second
time as it is released and possibly even reused resulting in nonsensical
behavior.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #39677 into master

* refs/pull/39677/head:
mgr/cephadm: add iscsi and nfs to upgrade
mgr/cephadm: update caps if necessary when getting keyring

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: add iscsi and nfs to upgrade

Fixes: https://tracker.ceph.com/issues/49462
Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: update caps if necessary when getting keyring

If the caps change from the old version to the new one it causes
issues in the upgrade. This allows the caps to be updated. Currently
only seeing this with iscsi but changing it for other as a precaution

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #39640 from sebastian-philipp/cephadm-cephfs-mirror

cephadm: cephfs-mirror

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39800 from MahatiC/wip-ssd-fixes

librbd/cache/pwl: Fix IO completion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #38913 into master

* refs/pull/38913/head:
qa/tasks/cephfs/nfs: Add tests for updating fs exports
mgr/volumes/nfs: Handle rook restart error
doc/cephfs/nfs: Add about update export interface
mgr/volumes/nfs: Add command to update cephfs exports
pybind/volumes/nfs: set mds caps according to user specified access type
mgr/volumes/module: Remove unused json module

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39953 into master

* refs/pull/39953/head:
debian/ceph-common.postinst: do not chown cephadm log dirs

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: add cephfs-mirror to CEPH_UPGRADE_ORDER

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: Add cephfs-mirror

Fixes: https://tracker.ceph.com/issues/47261
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/cephadm: Add cephfs-mirror test

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/tasks: some type annotations

Mostly for making my IDE aware of things

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: Add cephfs-mirror to enum

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add CephfsMirrorService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: replace def add_{type}(...) with generic add_daemon()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: drop `create_func` arg from _add_daemon

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move CephadmExporter to new module

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #39860 into master

* refs/pull/39860/head:
vstart,init-ceph: include build bin/ in $PATH

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39619 from sebastian-philipp/rm_injected_argv

cephadm: remove injected_args

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39877 from liewegas/cephadm-rgw-simplification

mgr/cephadm: simplify handling for rgw

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #39991 from tchaikov/wip-ninja-again

ceph.spec: include ninja as BuildRequires

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #39878 from batrick/centos-8.3

qa/distros: add centos 8.3

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39793 from rhcs-dashboard/replace-inventory-devices-title

mgr/dashboard: cluster > hosts: replace inventory->devices table title

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #39882 from tchaikov/wip-osd-watch

osd/Watch: use vector<> instead of list<>

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39897 from Rethan/clean-msgr-type

msg/Messenger: remove unused variable r

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39937 from tchaikov/wip-49661

mgr: add mon metada using type of "mon"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39541 from pcuzner/config-checker

mgr/cephadm: Provide an integrated configuration validation feature

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39905 from tchaikov/wip-lock-deps

common/lockdep: increase MAX_LOCKS to 128k

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #39503 from rhcs-dashboard/fix-ssl-certificate-code

mgr/dashboard: fix dashboard instance ssl certificate functionality

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39599 from pcuzner/cephadm-with-lsm

mgr/cephadm:Enable cephadm device scan to use libstoragemgmt

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #39924 from Daniel-Pivonka/osdrmstoptraceback

mgr/cephadm: prevent traceback when invalid osd id passed to 'orch osd rm stop'

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph.spec: include ninja as BuildRequires

to ease the build for developers using SUSE, Fedora, CentOS or RHEL.
so install-deps.sh can install ninja for them.

Fixes: https://tracker.ceph.com/issues/49694
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39828 from athanatos/sjust/wip-seastore-omap2

crimson/seastore: start wiring up collection_manager, omap_manager, onode_manager

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/.../staged-fltree/value.cc: return ValueDeltaRecorder for ONODE

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/seastore/onode_tree: add FLTreeOnodeManager unittest

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../fltree_onode_manager: fill in delta recorder

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore.cc: wire up omap_manager

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/omap_manager: generalize omap_list to support lower_bound

The iterator interface needs both lower and upper bound, generalize list()
to support both.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os: remove FuturizedStore::OmapIterator::tail_key

This doesn't actually make sense for FuturzizedStore implementations other
than BlueStore, and then only in a diagnostic capacity.  It also doesn't
have any actual users at the moment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: remove create_ephemeral

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: wire up FLTreeOnodeManager and FlatCollectionManager

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: introduce FLTreeOnodeManager

Adjusts OnodeManager interface and introduces FLTreeOnodeManager
implementation.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: remove simple-fltree

Doesn't actually have a TransactionManager implementation, and
it needs substantial changes to adapt to changes in Onode.
We can resurrect it in the future if we want to.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39982 from tchaikov/wip-doc-theme

doc: fine tune the theme, fix formatting in mount.ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/man/8/mount.ceph: fix the formatting

* fix the indent of item
* add an empty line before code block

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/start: fix the link pointing to ceph doc

use https instead of http, the latter cannot be redirected to the valid
docs. also, we don't have "next" branch, so remove the document
mentioning it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_themes: fine tune styling

* fix the "dl" (definition list) display in manpage. before this
  change, if an item in dl is composed of strings with different fonts,
  it is cluttered into a single string. after this change, they are
  separated with proper spacing as before. see the "WORKLOAD" section in
  https://docs.ceph.com/en/latest/man/8/ceph-syn/ and
  https://docs.ceph.com/en/octopus/man/8/ceph-syn/
* reduce the spacing in "ul" (unordered list) in table. before
  this change, we have large spacing at end of a list in table. after
  this change, the spacing is reduced. see
  https://docs.ceph.com/en/latest/start/hardware-recommendations/#ram
  https://docs.ceph.com/en/octopus/start/hardware-recommendations/#ram
* increase the spacing between two "ul"s in table. see
  https://docs.ceph.com/en/latest/start/hardware-recommendations/#Processor
  https://docs.ceph.com/en/octopus/start/hardware-recommendations/#ram

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39981 from xxhdx1985126/wip-obc-tri_mutex-bug

crimson/osd: fix two obc lock related bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: use excl_lock_from_excl when promoting tri_mutex from excl to excl

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: acquire locks strictly by OSDOp type

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39826 from tchaikov/wip-ninja

run-make: switch over to ninja-build

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39946 from TheJJ/fix-ceph-crash-caps

doc/mgr/crash: explain needed crash upload permissions

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #39955 into master

* refs/pull/39955/head:
mgr/cephadm: do not prime service cache on reconfig

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #37711 into master

* refs/pull/37711/head:
qa/cephfs: create class LocalCephFSMount
vstart_runner: reuse FuseMount methods in LocalFuseMount
vstart_runner: reuse KernelMount methods in LocalKernelMount
qa/cephfs: move common and generic methods mount.py
qa/cephfs: refactor FuseMount.mount method
qa/cephfs: refactor KernelMount.mount method

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37945 into master

* refs/pull/37945/head:
mds: add opened files/inodes and pinned i_caps metric payload support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39408 into master

* refs/pull/39408/head:
test: add test for validating cephfs-mirror daemon service status
pybind/mirroring: interface to fetch mirror daemon status
cephfs-mirror: register mirror daemon with service manager
cephfs-mirror: remove unneeded json_spirit header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39502 into master

* refs/pull/39502/head:
qa: add sleep for blocklisting to take effect

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #39687 into master

* refs/pull/39687/head:
tools/cephfs-shell: continue file listing even on error
tools/cephfs-shell: refactoring of code related listing
tools/cephfs-shell: fix listing of symbolic links

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39814 into master

* refs/pull/39814/head:
qa/cephfs: move is_blocklisted() to filesystem.CephCluster

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

python-common: continue to allow RGWSpec(realm=r,zone=z)

This is for backward compatibility: an octopus spec yaml can still be
applied to an existing cluster.  Note that it might not work on a new
cluster, since cephadm no longer tries to create the realm or zone if
they don't exist.

Signed-off-by: Sage Weil <sage@newdream.net>

PendingReleaseNodes: note changes in cephadm rgw behavior

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: drop realm.zone convention for rgw

Note that cephadm.py will no longer do anything with rgw realms and
zones.  That means that the setup of rgw roles here is only useful
for the default zone and a non-multisite config.

Signed-off-by: Sage Weil <sage@newdream.net>

doc: update docs

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: do not prime service cache on reconfig

Ceph daemon reconfig does not need any daemon state refresh since we don't
do a restart--we just rewrite the ceph.conf.  This also avoids priming
our cache with a 'starting' state when the daemon wasn't touched.

Fixes: https://tracker.ceph.com/issues/49675
Signed-off-by: Sage Weil <sage@newdream.net>

debian/ceph-common.postinst: do not chown cephadm log dirs

The container uid/gid is different than the debian uid/gid (because the
container is centos-based and we got a different uid/gid allocation there).

Fixes: https://tracker.ceph.com/issues/49677
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39948 from tchaikov/wip-clang-tidy

crimson,common: cleanups inspired by clang-tidy

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

python-common/ceph/deployment: RGWSpec: accept (and drop) subcluster arg

This is just so we can load up a stored spec after upgrade.  We'll silently
drop it, since we have the service_id, and this was only used to generate
that anyway.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/orchestrator: drop $realm.$zone naming convention

- Let users name the rgw service(s) whatever they like
- Make the rgw_zone and rgw_realm arguments optional, so that they can
  omit them and let radosgw start up in the generic single-cluster
  configuration (whichk, notably, has no realm).
- Continue to set the rgw_realm and rgw_zone options for the daemon(s),
  but only when those values are specified.
- Adjust the CLI accordingly.  Drop the subcluster argument, which was
  only used to generate a service_id.
- Adjust rook. This is actually a simplification and improved mapping onto
  the rook CRD, *except* that for octopus we enforced the realm.zone
  naming *and* realm==zone.  I doubt a single user actually did this
  so it is not be worth dealing with, but we need a special case for
  where there is a . in the service name.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: rgw: do not mess with realm configuration

It is simpler to consider this out of scope for the orchestrator.  The
user should set up their multisite realms/zones before deploying the
daemons (or the daemons will not start).  In the future we can wrap this
with a more friendly tool, perhaps.

Signed-off-by: Sage Weil <sage@newdream.net>

crimson: pass by `const` or `const&` if possible

more constness, and better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39930 from zdover23/wip-doc-cephadm-convert-adoption-process-2021-Mar-09

doc/cephadm: rewrite "adoption process"

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/mon: comment unused parameter

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not rely on implicit conversion to bool

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: use `auto*` instead of `auto` for pointer

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: use moved string for constructing vector<>

it is the last consumer of cmdline, so move it away.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: avoid using C-style cast

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/str_list: remove unused "using"

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: rewrite "adoption process"

This rewrites the "Adoption Process" section
of the "Converting and Existing Cluster to
cephadm" chapter of the Cephadm guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/mgr/crash: explain needed crash upload permissions

Signed-off-by: Jonas Jelten <jj@sft.lol>

mgr: add mon metada using type of "mon"

this change addresses a regression introduced by
c037f4cb5d7436879d58c34748ef516b5269781f

also remove the "P" before the json command.

see also: https://tracker.ceph.com/issues/48905

Fixes: https://tracker.ceph.com/issues/49661
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: silence "You are in 'detached HEAD' state" warning

git warns us when the repo is in 'detached HEAD' state, like:

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in
this state without impacting any branches by switching back to a branch.
...
Turn off this advice by setting config variable advice.detachedHead to
false

Signed-off-by: Kefu Chai <kchai@redhat.com>

nasm-wrapper: use += for appending string to a variable

less repeating, better readablity

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not pass "-fdiagnostics-color" to as

this option can be understood by GCC. see
https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Message-Formatting-Options.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

nasm-wrapper: swallow -MD option

see also

- https://discourse.cmake.org/t/cmake-generating-invalid-ninja-build-rules-for-asm/1389
- https://gitlab.kitware.com/cmake/cmake/-/merge_requests/4460
- https://gitlab.kitware.com/cmake/cmake/-/merge_requests/4838

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: define BUILD_BYPRODUCTS for Boost

so if a cmake generator needs any one of them, it knowns how to build
it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not specify CMAKE_GENERATOR for external projects

as per CMake document,

> Without this option, the same generator as the main build will be
> used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use "" for no-op INSTALL_COMMAND

as suggested by CMake document, see
https://cmake.org/cmake/help/latest/module/ExternalProject.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: define BUILD_BYPRODUCTS for spdk-ext

so if a cmake generator needs any one of them, it knowns how to build
it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

do_cmake: always pass -DWITH_PYTHON3 to cmake

do not pretend that we support python2 anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>