Merge pull request #39991 from tchaikov/wip-ninja-again

ceph.spec: include ninja as BuildRequires

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #39878 from batrick/centos-8.3

qa/distros: add centos 8.3

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39793 from rhcs-dashboard/replace-inventory-devices-title

mgr/dashboard: cluster > hosts: replace inventory->devices table title

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #39882 from tchaikov/wip-osd-watch

osd/Watch: use vector<> instead of list<>

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39897 from Rethan/clean-msgr-type

msg/Messenger: remove unused variable r

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39937 from tchaikov/wip-49661

mgr: add mon metada using type of "mon"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39541 from pcuzner/config-checker

mgr/cephadm: Provide an integrated configuration validation feature

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39905 from tchaikov/wip-lock-deps

common/lockdep: increase MAX_LOCKS to 128k

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #39503 from rhcs-dashboard/fix-ssl-certificate-code

mgr/dashboard: fix dashboard instance ssl certificate functionality

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39599 from pcuzner/cephadm-with-lsm

mgr/cephadm:Enable cephadm device scan to use libstoragemgmt

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #39924 from Daniel-Pivonka/osdrmstoptraceback

mgr/cephadm: prevent traceback when invalid osd id passed to 'orch osd rm stop'

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph.spec: include ninja as BuildRequires

to ease the build for developers using SUSE, Fedora, CentOS or RHEL.
so install-deps.sh can install ninja for them.

Fixes: https://tracker.ceph.com/issues/49694
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39828 from athanatos/sjust/wip-seastore-omap2

crimson/seastore: start wiring up collection_manager, omap_manager, onode_manager

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/.../staged-fltree/value.cc: return ValueDeltaRecorder for ONODE

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/seastore/onode_tree: add FLTreeOnodeManager unittest

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../fltree_onode_manager: fill in delta recorder

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore.cc: wire up omap_manager

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/omap_manager: generalize omap_list to support lower_bound

The iterator interface needs both lower and upper bound, generalize list()
to support both.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os: remove FuturizedStore::OmapIterator::tail_key

This doesn't actually make sense for FuturzizedStore implementations other
than BlueStore, and then only in a diagnostic capacity.  It also doesn't
have any actual users at the moment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: remove create_ephemeral

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: wire up FLTreeOnodeManager and FlatCollectionManager

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: introduce FLTreeOnodeManager

Adjusts OnodeManager interface and introduces FLTreeOnodeManager
implementation.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: remove simple-fltree

Doesn't actually have a TransactionManager implementation, and
it needs substantial changes to adapt to changes in Onode.
We can resurrect it in the future if we want to.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39982 from tchaikov/wip-doc-theme

doc: fine tune the theme, fix formatting in mount.ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/man/8/mount.ceph: fix the formatting

* fix the indent of item
* add an empty line before code block

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/start: fix the link pointing to ceph doc

use https instead of http, the latter cannot be redirected to the valid
docs. also, we don't have "next" branch, so remove the document
mentioning it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_themes: fine tune styling

* fix the "dl" (definition list) display in manpage. before this
  change, if an item in dl is composed of strings with different fonts,
  it is cluttered into a single string. after this change, they are
  separated with proper spacing as before. see the "WORKLOAD" section in
  https://docs.ceph.com/en/latest/man/8/ceph-syn/ and
  https://docs.ceph.com/en/octopus/man/8/ceph-syn/
* reduce the spacing in "ul" (unordered list) in table. before
  this change, we have large spacing at end of a list in table. after
  this change, the spacing is reduced. see
  https://docs.ceph.com/en/latest/start/hardware-recommendations/#ram
  https://docs.ceph.com/en/octopus/start/hardware-recommendations/#ram
* increase the spacing between two "ul"s in table. see
  https://docs.ceph.com/en/latest/start/hardware-recommendations/#Processor
  https://docs.ceph.com/en/octopus/start/hardware-recommendations/#ram

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39981 from xxhdx1985126/wip-obc-tri_mutex-bug

crimson/osd: fix two obc lock related bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: use excl_lock_from_excl when promoting tri_mutex from excl to excl

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: acquire locks strictly by OSDOp type

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39826 from tchaikov/wip-ninja

run-make: switch over to ninja-build

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39946 from TheJJ/fix-ceph-crash-caps

doc/mgr/crash: explain needed crash upload permissions

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #39955 into master

* refs/pull/39955/head:
mgr/cephadm: do not prime service cache on reconfig

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #37711 into master

* refs/pull/37711/head:
qa/cephfs: create class LocalCephFSMount
vstart_runner: reuse FuseMount methods in LocalFuseMount
vstart_runner: reuse KernelMount methods in LocalKernelMount
qa/cephfs: move common and generic methods mount.py
qa/cephfs: refactor FuseMount.mount method
qa/cephfs: refactor KernelMount.mount method

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37945 into master

* refs/pull/37945/head:
mds: add opened files/inodes and pinned i_caps metric payload support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39408 into master

* refs/pull/39408/head:
test: add test for validating cephfs-mirror daemon service status
pybind/mirroring: interface to fetch mirror daemon status
cephfs-mirror: register mirror daemon with service manager
cephfs-mirror: remove unneeded json_spirit header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39502 into master

* refs/pull/39502/head:
qa: add sleep for blocklisting to take effect

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #39687 into master

* refs/pull/39687/head:
tools/cephfs-shell: continue file listing even on error
tools/cephfs-shell: refactoring of code related listing
tools/cephfs-shell: fix listing of symbolic links

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39814 into master

* refs/pull/39814/head:
qa/cephfs: move is_blocklisted() to filesystem.CephCluster

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/cephadm: do not prime service cache on reconfig

Ceph daemon reconfig does not need any daemon state refresh since we don't
do a restart--we just rewrite the ceph.conf.  This also avoids priming
our cache with a 'starting' state when the daemon wasn't touched.

Fixes: https://tracker.ceph.com/issues/49675
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39948 from tchaikov/wip-clang-tidy

crimson,common: cleanups inspired by clang-tidy

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: pass by `const` or `const&` if possible

more constness, and better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39930 from zdover23/wip-doc-cephadm-convert-adoption-process-2021-Mar-09

doc/cephadm: rewrite "adoption process"

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/mon: comment unused parameter

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not rely on implicit conversion to bool

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: use `auto*` instead of `auto` for pointer

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: use moved string for constructing vector<>

it is the last consumer of cmdline, so move it away.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: avoid using C-style cast

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/str_list: remove unused "using"

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: rewrite "adoption process"

This rewrites the "Adoption Process" section
of the "Converting and Existing Cluster to
cephadm" chapter of the Cephadm guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/mgr/crash: explain needed crash upload permissions

Signed-off-by: Jonas Jelten <jj@sft.lol>

mgr: add mon metada using type of "mon"

this change addresses a regression introduced by
c037f4cb5d7436879d58c34748ef516b5269781f

also remove the "P" before the json command.

see also: https://tracker.ceph.com/issues/48905

Fixes: https://tracker.ceph.com/issues/49661
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: silence "You are in 'detached HEAD' state" warning

git warns us when the repo is in 'detached HEAD' state, like:

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in
this state without impacting any branches by switching back to a branch.
...
Turn off this advice by setting config variable advice.detachedHead to
false

Signed-off-by: Kefu Chai <kchai@redhat.com>

nasm-wrapper: use += for appending string to a variable

less repeating, better readablity

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not pass "-fdiagnostics-color" to as

this option can be understood by GCC. see
https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Message-Formatting-Options.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

nasm-wrapper: swallow -MD option

see also

- https://discourse.cmake.org/t/cmake-generating-invalid-ninja-build-rules-for-asm/1389
- https://gitlab.kitware.com/cmake/cmake/-/merge_requests/4460
- https://gitlab.kitware.com/cmake/cmake/-/merge_requests/4838

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: define BUILD_BYPRODUCTS for Boost

so if a cmake generator needs any one of them, it knowns how to build
it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not specify CMAKE_GENERATOR for external projects

as per CMake document,

> Without this option, the same generator as the main build will be
> used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use "" for no-op INSTALL_COMMAND

as suggested by CMake document, see
https://cmake.org/cmake/help/latest/module/ExternalProject.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: define BUILD_BYPRODUCTS for spdk-ext

so if a cmake generator needs any one of them, it knowns how to build
it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

do_cmake: always pass -DWITH_PYTHON3 to cmake

do not pretend that we support python2 anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

do_cmake,run-make: switch to ninja

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec: prepare for out-of-source build

see also
https://fedoraproject.org/wiki/Changes/CMake_to_do_out-of-source_builds

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39226 from rhcs-dashboard/host-maintenance-feature

mgr/dashboard: Host Maintenance Feature

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39909 from smithfarm/wip-revert-make-build-on-suse

rpm: refrain from using %make_build macro on SUSE

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39936 from tchaikov/wip-workflow-use-known-sha1

.github/workflows: use a released sha1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

.github/workflows: use a released sha1

otherwise the action cannot be found

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39892 from tchaikov/wip-github-workflow-milestone

.github: correct the regex in milestone workflow

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/cephadm: prevent traceback when invalid osd id passed to 'orch osd rm stop'

orch osd rm exepcts a str that can be converted to an int passed to it
if the user passed something that cant be converted it shows a traceback
catching the ValueError prevents this traceback.

Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>

.github/workflows: rename labeler.yml to pr-triage.yml

to reflect the change in the actions in it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

.github/workflows: use @{sha1} for actions

more secure this way. see also https://julienrenaux.fr/2019/12/20/github-actions-security-risk/

point the sha1 to

* labeler@v3
* milestone@main HEAD

Signed-off-by: Kefu Chai <kchai@redhat.com>

.github/workflows: move milestone.yml into labeler.yml

no need to have two actions triggered by the same event.

we could backport this labeler.yml to LTS branches.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39246 from rhcs-dashboard/fix-45415-master

mgr/dashboard: test prometheus rules through promtool

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: p-se <NOT@FOUND>

mgr/dashboard: Host Maintenance Feature

In Cluster -> Hosts, I've added additional button to put the selected host on maintenance or exit out of the maintenance mode. Also for some hosts the ok-to-stop tests may trigger some warnings which requires a --force command to pass along with the maintenance enter command to enter a host into maintenance. In UI this is achieved using a confirmation Modal. In addition to this if the check error is It is NOT safe to stop the host then the host wont be able to put into maintenance mode.

Fixes: https://tracker.ceph.com/issues/49101
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #38605 from mdw-at-linuxbox/wip-master-transit

rgw: new vault transit logic for SSE-C

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37536 from xxhdx1985126/wip-crimson-interruptible-future-final

crimson: add continuation chain interruption capabilities

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39876 from dang/wip-dang-tempest

RGW - Fix TempURL user handling

Merge pull request #39476 from rhcs-dashboard/fix-host-label

mgr/dashboard: fix PUT - /api/host/{hostname} while adding labels

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

common/lockdep: increase MAX_LOCKS to 128k

initialize follows and follows_bt with 4k, and resize them when the
size of locks exceeds the initial number, but with the upper bound of
MAX_LOCKS which is fixed at compile time.

this change allows us to use 128k locks in the same time.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/lockdep: introduce MAX_FOLLOWERS

before this change we use a matrix for tracking the locks and those
which are locked after acquiring the formers. where follows[a][b]
sigifies that b is acquired after a is acquired, and a and b are both
lock ids allocated in the lock_ids registry.

but since we might need to have a much large set of locks which are
acquired at the same time. but the numbers of their follower of
each of them are always much smaller. because we are not likely
to hold, for instance, 4096 locks when we still holding a lock.
actually, this might be a bug, if we actually do this.

so we can use smaller vectors for tracking the followers. and the
size of those vectors can be one or more orders of magnitude smaller
than the number of locks.

in this change, MAX_FOLLOWERS in introduced for the max number
of followers.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/lockdeps: use vector to allocate follows and follows_bt on heap

this allows us to increase MAX_LOCKS without being limited by the size
of .bss segment. as cephfs plans to use a large number of locks for
finer grained inode lock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/lockdeps: use std::bitset for tracking free_ids

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/lockdep: use std::bitset for tracking follows

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39825 from mgfritch/cephadm-rgw-nfs-health

mgr/cephadm: alias rgw-nfs -> nfs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38908 from smithfarm/wip-script-agpl3

COPYING: add entries for backport scripts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39593 from mychoxin/fix_promote_dead_loop

osd: avoid two copy with same src cancel each other

Reviewed-by: Sage Weil <sage@redhat.com>

rpm: refrain from using %make_build macro on SUSE

The recently merged commit 0e511973f7b6b0e5dbb3c4caf98f1290f81829e9 replaced

    make %{_smp_mflags}

with

    %make_build

for the stated purpose of hiding the %_smp_mflags macro in a higher-level macro.

But, on SUSE, the higher-level macro (%make_build) expands to:

    make -O %{_smp_mflags}

The addition of the -O flag makes the build considerably slower and increases
the memory requirement. The exact reason for this is unknown - possibly it's due
to a bug in make, although the same slowness was observed with ninja as well.
In any event, this is a deal-breaker when building in the OBS, because the build
infrastructure there is optimized for builds that do not require huge amounts of
memory and we would rather have a fast build with mixed up compiler messages
than a very slow one with synced compiler messages.

Fixes: 0e511973f7b6b0e5dbb3c4caf98f1290f81829e9
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #39821 from dillaman/wip-qa-objectstore-logs

qa: reduce debug log levels for bluestore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39900 from tchaikov/wip-crimson-watch

crimson/osd: cleanup Notify::send_completion()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard:test prometheus rules through promtool

This PR intends to add unit testing for prometheus rules using promtool. To run the tests run 'run-promtool-unittests.sh' file.

Fixes: https://tracker.ceph.com/issues/45415
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

osd/Watch: use vector<> instead of list<>

there is no need to use a list<> here. and this change does not
change the encoding schema here. also, as a fact, the user of the
watch-notify feature is decoding the bufferlist like:

 std::map<std::pair<uint64_t,uint64_t>, bufferlist> reply_map;
 std::set<std::pair<uint64_t,uint64_t> > missed_map;
 decode(reply_map, reply_p);
 decode(missed_map, reply_p);

so, in this change, change list<> to vector<> for both smaller memory
footprint and for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39729 from aclamk/wip-rocksdb-aware-bluefs-buffered-io

Allows RocksDB to distinguish between buffered/direct_io files

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #38997 from gerald-yang/patch-48946

common: Fix assertion when disabling and re-enabling clog_to_monitors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39775 from rishabh-d-dave/fs-qa-accept-cmds-as-str

qa/ceph_manager: accepts commands as str too

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39690 from rishabh-d-dave/qa-raw_cluster_cmd

qa/ceph_manger: fixes bugs in CephManager

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39712 from tchaikov/wip-prometheus-ann

mgr/prometheus: add type annotations

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #39740 from tchaikov/wip-pybind-rtype

pybind/{rados,cephfs}: use type annotation for rtype docstring

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/osd: add constness to Notify

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: cleanup Notify::send_completion()

Notify::send_completion() should just send the completion message. so,
move irrelevant code out of Notify::send_completion().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: differentiate Notify with and without watchers

we don't need to arm the timer, if there is no watchers.

Signed-off-by: Kefu Chai <kchai@redhat.com>