test/librbd: s/RWLock/share_lock/

to consolidate the code base further, we don't need to use the
homebrewed shared lock unless the std::shared_mutex cannot fufill our
needs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/rgw_common: silence Wsign-compare warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librados_test_stub: do not include unused RWLock.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd,rbd_mirror: do not include RWLock.h unless it is used

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: do not include RWLock.h unless it is used

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: do not include RWLock.h unless it is used

Signed-off-by: Kefu Chai <kchai@redhat.com>

os: do not include RWLock.h unless it is used

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: do not include RWLock.h unless RWLock is used

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39828 from athanatos/sjust/wip-seastore-omap2

crimson/seastore: start wiring up collection_manager, omap_manager, onode_manager

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/.../staged-fltree/value.cc: return ValueDeltaRecorder for ONODE

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/seastore/onode_tree: add FLTreeOnodeManager unittest

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../fltree_onode_manager: fill in delta recorder

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore.cc: wire up omap_manager

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/omap_manager: generalize omap_list to support lower_bound

The iterator interface needs both lower and upper bound, generalize list()
to support both.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os: remove FuturizedStore::OmapIterator::tail_key

This doesn't actually make sense for FuturzizedStore implementations other
than BlueStore, and then only in a diagnostic capacity.  It also doesn't
have any actual users at the moment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: remove create_ephemeral

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: wire up FLTreeOnodeManager and FlatCollectionManager

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: introduce FLTreeOnodeManager

Adjusts OnodeManager interface and introduces FLTreeOnodeManager
implementation.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: remove simple-fltree

Doesn't actually have a TransactionManager implementation, and
it needs substantial changes to adapt to changes in Onode.
We can resurrect it in the future if we want to.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39982 from tchaikov/wip-doc-theme

doc: fine tune the theme, fix formatting in mount.ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/man/8/mount.ceph: fix the formatting

* fix the indent of item
* add an empty line before code block

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/start: fix the link pointing to ceph doc

use https instead of http, the latter cannot be redirected to the valid
docs. also, we don't have "next" branch, so remove the document
mentioning it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_themes: fine tune styling

* fix the "dl" (definition list) display in manpage. before this
  change, if an item in dl is composed of strings with different fonts,
  it is cluttered into a single string. after this change, they are
  separated with proper spacing as before. see the "WORKLOAD" section in
  https://docs.ceph.com/en/latest/man/8/ceph-syn/ and
  https://docs.ceph.com/en/octopus/man/8/ceph-syn/
* reduce the spacing in "ul" (unordered list) in table. before
  this change, we have large spacing at end of a list in table. after
  this change, the spacing is reduced. see
  https://docs.ceph.com/en/latest/start/hardware-recommendations/#ram
  https://docs.ceph.com/en/octopus/start/hardware-recommendations/#ram
* increase the spacing between two "ul"s in table. see
  https://docs.ceph.com/en/latest/start/hardware-recommendations/#Processor
  https://docs.ceph.com/en/octopus/start/hardware-recommendations/#ram

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39981 from xxhdx1985126/wip-obc-tri_mutex-bug

crimson/osd: fix two obc lock related bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: use excl_lock_from_excl when promoting tri_mutex from excl to excl

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: acquire locks strictly by OSDOp type

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39826 from tchaikov/wip-ninja

run-make: switch over to ninja-build

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39946 from TheJJ/fix-ceph-crash-caps

doc/mgr/crash: explain needed crash upload permissions

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #39955 into master

* refs/pull/39955/head:
mgr/cephadm: do not prime service cache on reconfig

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #37711 into master

* refs/pull/37711/head:
qa/cephfs: create class LocalCephFSMount
vstart_runner: reuse FuseMount methods in LocalFuseMount
vstart_runner: reuse KernelMount methods in LocalKernelMount
qa/cephfs: move common and generic methods mount.py
qa/cephfs: refactor FuseMount.mount method
qa/cephfs: refactor KernelMount.mount method

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37945 into master

* refs/pull/37945/head:
mds: add opened files/inodes and pinned i_caps metric payload support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39408 into master

* refs/pull/39408/head:
test: add test for validating cephfs-mirror daemon service status
pybind/mirroring: interface to fetch mirror daemon status
cephfs-mirror: register mirror daemon with service manager
cephfs-mirror: remove unneeded json_spirit header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39502 into master

* refs/pull/39502/head:
qa: add sleep for blocklisting to take effect

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #39687 into master

* refs/pull/39687/head:
tools/cephfs-shell: continue file listing even on error
tools/cephfs-shell: refactoring of code related listing
tools/cephfs-shell: fix listing of symbolic links

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39814 into master

* refs/pull/39814/head:
qa/cephfs: move is_blocklisted() to filesystem.CephCluster

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/cephadm: do not prime service cache on reconfig

Ceph daemon reconfig does not need any daemon state refresh since we don't
do a restart--we just rewrite the ceph.conf.  This also avoids priming
our cache with a 'starting' state when the daemon wasn't touched.

Fixes: https://tracker.ceph.com/issues/49675
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39948 from tchaikov/wip-clang-tidy

crimson,common: cleanups inspired by clang-tidy

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: pass by `const` or `const&` if possible

more constness, and better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39930 from zdover23/wip-doc-cephadm-convert-adoption-process-2021-Mar-09

doc/cephadm: rewrite "adoption process"

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/mon: comment unused parameter

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not rely on implicit conversion to bool

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: use `auto*` instead of `auto` for pointer

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: use moved string for constructing vector<>

it is the last consumer of cmdline, so move it away.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: avoid using C-style cast

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/str_list: remove unused "using"

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: rewrite "adoption process"

This rewrites the "Adoption Process" section
of the "Converting and Existing Cluster to
cephadm" chapter of the Cephadm guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/mgr/crash: explain needed crash upload permissions

Signed-off-by: Jonas Jelten <jj@sft.lol>

cmake: silence "You are in 'detached HEAD' state" warning

git warns us when the repo is in 'detached HEAD' state, like:

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in
this state without impacting any branches by switching back to a branch.
...
Turn off this advice by setting config variable advice.detachedHead to
false

Signed-off-by: Kefu Chai <kchai@redhat.com>

nasm-wrapper: use += for appending string to a variable

less repeating, better readablity

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not pass "-fdiagnostics-color" to as

this option can be understood by GCC. see
https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Message-Formatting-Options.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

nasm-wrapper: swallow -MD option

see also

- https://discourse.cmake.org/t/cmake-generating-invalid-ninja-build-rules-for-asm/1389
- https://gitlab.kitware.com/cmake/cmake/-/merge_requests/4460
- https://gitlab.kitware.com/cmake/cmake/-/merge_requests/4838

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: define BUILD_BYPRODUCTS for Boost

so if a cmake generator needs any one of them, it knowns how to build
it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not specify CMAKE_GENERATOR for external projects

as per CMake document,

> Without this option, the same generator as the main build will be
> used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use "" for no-op INSTALL_COMMAND

as suggested by CMake document, see
https://cmake.org/cmake/help/latest/module/ExternalProject.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: define BUILD_BYPRODUCTS for spdk-ext

so if a cmake generator needs any one of them, it knowns how to build
it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

do_cmake: always pass -DWITH_PYTHON3 to cmake

do not pretend that we support python2 anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

do_cmake,run-make: switch to ninja

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec: prepare for out-of-source build

see also
https://fedoraproject.org/wiki/Changes/CMake_to_do_out-of-source_builds

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39226 from rhcs-dashboard/host-maintenance-feature

mgr/dashboard: Host Maintenance Feature

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39909 from smithfarm/wip-revert-make-build-on-suse

rpm: refrain from using %make_build macro on SUSE

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39936 from tchaikov/wip-workflow-use-known-sha1

.github/workflows: use a released sha1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

.github/workflows: use a released sha1

otherwise the action cannot be found

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39892 from tchaikov/wip-github-workflow-milestone

.github: correct the regex in milestone workflow

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

.github/workflows: rename labeler.yml to pr-triage.yml

to reflect the change in the actions in it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

.github/workflows: use @{sha1} for actions

more secure this way. see also https://julienrenaux.fr/2019/12/20/github-actions-security-risk/

point the sha1 to

* labeler@v3
* milestone@main HEAD

Signed-off-by: Kefu Chai <kchai@redhat.com>

.github/workflows: move milestone.yml into labeler.yml

no need to have two actions triggered by the same event.

we could backport this labeler.yml to LTS branches.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39246 from rhcs-dashboard/fix-45415-master

mgr/dashboard: test prometheus rules through promtool

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: p-se <NOT@FOUND>

mgr/dashboard: Host Maintenance Feature

In Cluster -> Hosts, I've added additional button to put the selected host on maintenance or exit out of the maintenance mode. Also for some hosts the ok-to-stop tests may trigger some warnings which requires a --force command to pass along with the maintenance enter command to enter a host into maintenance. In UI this is achieved using a confirmation Modal. In addition to this if the check error is It is NOT safe to stop the host then the host wont be able to put into maintenance mode.

Fixes: https://tracker.ceph.com/issues/49101
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #38605 from mdw-at-linuxbox/wip-master-transit

rgw: new vault transit logic for SSE-C

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37536 from xxhdx1985126/wip-crimson-interruptible-future-final

crimson: add continuation chain interruption capabilities

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39876 from dang/wip-dang-tempest

RGW - Fix TempURL user handling

Merge pull request #39476 from rhcs-dashboard/fix-host-label

mgr/dashboard: fix PUT - /api/host/{hostname} while adding labels

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39825 from mgfritch/cephadm-rgw-nfs-health

mgr/cephadm: alias rgw-nfs -> nfs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38908 from smithfarm/wip-script-agpl3

COPYING: add entries for backport scripts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39593 from mychoxin/fix_promote_dead_loop

osd: avoid two copy with same src cancel each other

Reviewed-by: Sage Weil <sage@redhat.com>

rpm: refrain from using %make_build macro on SUSE

The recently merged commit 0e511973f7b6b0e5dbb3c4caf98f1290f81829e9 replaced

    make %{_smp_mflags}

with

    %make_build

for the stated purpose of hiding the %_smp_mflags macro in a higher-level macro.

But, on SUSE, the higher-level macro (%make_build) expands to:

    make -O %{_smp_mflags}

The addition of the -O flag makes the build considerably slower and increases
the memory requirement. The exact reason for this is unknown - possibly it's due
to a bug in make, although the same slowness was observed with ninja as well.
In any event, this is a deal-breaker when building in the OBS, because the build
infrastructure there is optimized for builds that do not require huge amounts of
memory and we would rather have a fast build with mixed up compiler messages
than a very slow one with synced compiler messages.

Fixes: 0e511973f7b6b0e5dbb3c4caf98f1290f81829e9
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #39821 from dillaman/wip-qa-objectstore-logs

qa: reduce debug log levels for bluestore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39900 from tchaikov/wip-crimson-watch

crimson/osd: cleanup Notify::send_completion()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard:test prometheus rules through promtool

This PR intends to add unit testing for prometheus rules using promtool. To run the tests run 'run-promtool-unittests.sh' file.

Fixes: https://tracker.ceph.com/issues/45415
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #39729 from aclamk/wip-rocksdb-aware-bluefs-buffered-io

Allows RocksDB to distinguish between buffered/direct_io files

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #38997 from gerald-yang/patch-48946

common: Fix assertion when disabling and re-enabling clog_to_monitors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39775 from rishabh-d-dave/fs-qa-accept-cmds-as-str

qa/ceph_manager: accepts commands as str too

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39690 from rishabh-d-dave/qa-raw_cluster_cmd

qa/ceph_manger: fixes bugs in CephManager

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39712 from tchaikov/wip-prometheus-ann

mgr/prometheus: add type annotations

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #39740 from tchaikov/wip-pybind-rtype

pybind/{rados,cephfs}: use type annotation for rtype docstring

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/osd: add constness to Notify

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: cleanup Notify::send_completion()

Notify::send_completion() should just send the completion message. so,
move irrelevant code out of Notify::send_completion().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: differentiate Notify with and without watchers

we don't need to arm the timer, if there is no watchers.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: factor out "maybe" from maybe_send_completion()

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39896 from bolapara/patch-1

docs: Ubuntu uses 'apt' not 'dnf'

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39895 from tchaikov/wip-crimson-watch

crimson/osd: watch-notify related cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/cephadm: use 'apt' to install cephadm on Ubuntu

Adjusted so Ubuntu command uses 'apt' and added Fedora since that uses 'dnf'.

crimson/common: deal with seastar::thread in interruptible_future

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/test: add interruptible future unit test

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: use interruptible future in the PG layer

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson: add interruptible future facilities

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: no need to clear watchers beforehand

as long as we don't yield, we can assume the atomicity.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use transparent comparator

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39879 from rzarzynski/wip-crimson-notify-timeout

crimson/osd: implement timeout for notify propagation.

Reviewed-by: Kefu Chai <kchai@redhat.com>

.github: correct the regex in mileston workflow

also use pull_request_target event so the action is run in the
context of the base of the pull request. this helps us to overcome
the "Resource not accessible by integration" issue where the action
is run in the context of the pull request.

Signed-off-by: Kefu Chai <kchai@redhat.com>