doc/_ext: ignore desc if it is unavailable

there is chance that we don't have desc, desc_long or fmt_desc, in that
case, we should just skip desc before checking its length. so, just use
'if desc' which is able to check for None or empty string.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: check "default" for None

default could be None and we don't reference the default value using
`opt.default`, so this variable is always defined. to avoid printing
a None default value, check for a None value. see
https://jinja.palletsprojects.com/en/2.11.x/templates/#none

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: print 0B if option value is 0

before this change, the rendered Default is "0Ei" if the default size is 0,
would be more readable if it is "0B" or "0".

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephfs: render options using confval directive

less repeating, better maintainability this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41085 from idryomov/wip-rbd-nuke-source-spec-file

rbd: --source-spec-file should be --source-spec-path

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #41116 from tchaikov/wip-in_addr_t

common/pick_address: define in_addr_t if it is not defined

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41117 from tchaikov/wip-doc-cephfs-conf

doc/cephfs/mds-config-ref: remove duplicated confval

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common/pick_address: define in_addr_t if it is not defined

neither mingw not not have in_addr_t defined, see
https://docs.microsoft.com/en-us/windows/win32/api/winsock2/ns-winsock2-in_addr,
so define it if it is not defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephfs/mds-config-ref: remove duplicated confval

this change addresses the warning of:

/home/jenkins-build/build/workspace/ceph-pr-docs/doc/cephfs/mds-config-ref.rst:2: WARNING: duplicate confval_option description of mds_cache_memory_limit, other instance in cephfs/cache-configuration
/home/jenkins-build/build/workspace/ceph-pr-docs/doc/cephfs/mds-config-ref.rst:2: WARNING: duplicate confval_option description of mds_cache_reservation, other instance in cephfs/cache-configuration

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41043 from ifed01/wip-ifed-fix-use-ratio

osd: compute OSD's space usage ratio via raw space utilization

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41012 into master

* refs/pull/41012/head:
doc/cephfs/cache-config: use config item macros

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40961 from tchaikov/wip-50456

common/pick_address: filter out loopback addresses

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41071 from tchaikov/wip-cmake-disable-vta

cmake: disable "variable tracking" when building rados python binding

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #41091 into master

* refs/pull/41091/head:
doc/cephfs/nfs: add section about ganesha logs
doc/cephfs/nfs: Replace volume/nfs with nfs
doc/cephfs/nfs: add note about export management with volume/nfs interface only

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41004 from sseshasa/wip-mclock-max-backfill-misc-fix

osd: Override recovery, backfill and sleep related config options during OSD and mclock scheduler initialization.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>

Merge pull request #40967 from dang/wip-dang-zipper-user

RGW Zipper consistent load/store APIs for User

Merge pull request #40946 from ifed01/wip-ifed-dump-pool-statfs

osd: admin cmd to dump pools statfs for a specific OSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41014 from smithfarm/wip-mempool-cacheline-49781

qa: verify the benefits of mempool cacheline optimization

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40998 from cyx1231st/wip-seastore-onode-tree-delete

crimson/onode-staged-tree: implement/validate features to erase values from tree

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/cephfs/nfs: add section about ganesha logs

Fixes: https://tracker.ceph.com/issues/50008
Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephfs/nfs: Replace volume/nfs with nfs

Recently, nfs related code was moved out of volumes plugin[1]. So using the
name volume/nfs for the interface is not appropriate.

[1] https://github.com/ceph/ceph/pull/40526

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephfs/nfs: add note about export management with volume/nfs interface only

Signed-off-by: Varsha Rao <varao@redhat.com>

qa: verify the benefits of mempool cacheline optimization

There already is a test to verify the mempool sharding works, in the sense that
it uses at least half of the variables available to count the number of
allocated objects and their total size. This new test verifies that, with
sharding, object counting is at least twice faster than without sharding. It
also collects cacheline contention data with the perf c2c tool. The manual
analysis of this data shows the optimization gain is indeed related to cacheline
contention.

Fixes: https://tracker.ceph.com/issues/49896
Signed-off-by: Loïc Dachary <loic@dachary.org>

Merge pull request #40502 from zdover23/wip-doc-podman-version-octopus-note-2021-30-Mar

doc/cephadm: add podman version note to install

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #40072 from wornet-mwo/dashboard--grafana-hostname-corrections

mgr/dashboard: Fixed name clash when hostname similar to another

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: p-se <NOT@FOUND>

RGW Zipper - Permenant User storage API

The load/store/remove APIs for User are needed, so make them consistent
and permenant.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #41040 from tchaikov/wip-doc-mds-conf

doc/cephfs/mds-config-ref: render options using confval directive

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rbd: --source-spec-file should be --source-spec-path

It was renamed during development, get rid of the leftovers.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41028 from mkogan1/wip-rgw-tracker-50467

rgw: fix segfault related to explicit object manifest handling

Merge pull request #41083 from rhcs-dashboard/fix-50580-master

mgr/dashboard: OSDs placement text is unreadable

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41082 from tchaikov/wip-50558

os/bluestore: Revert" Optimizing the lock of bluestore writing process"

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #41064 from petrutlucian94/win_at_removedir

include: define AT_REMOVEDIR on Windows

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41038 from tchaikov/wip-cmake-liburing

cmake: pass "CC" using configure when building liburing

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

mgr/dashboard:OSDs placement text is unreadable

While displaying the host pattern in the OSDs placement tab, it gets splited with semi-colons. Also adjusted the column size of Container Image ID and Placement columns.

Fixes: https://tracker.ceph.com/issues/50580
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

os/bluestore: Revert" Optimizing the lock of bluestore writing process"

This reverts commit e80d3f7a891af99ba0d4cfe99e5530016ca867ac.

Fixes: https://tracker.ceph.com/issues/50571
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: implement erase for perf-staged-tree

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement synthetic tests for value erase

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix ownership issues at node-level

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: allow to insert item to an empty node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: move try_downgrade_root() into fix_index()

Specifically, apply_children_merge() doesn't know whether it will be
retired due to fix_parent_index().

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: don't validate child when it's outdated

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: pass origin_left_addr to apply_children_merge()

rebuild_extent() may change the address of the left child, we need to
pass the origin_left_addr in order to validate if it is correct.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/.../node_extent_manager: replay extents may be dirty

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/onode-staged-tree: cleanup tree_utils.h

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: test fix_index() in internal node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: test merge in internal node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement staged::get_prev_slot()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: test merge in leaf node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement layout-level merge

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: test erase in leaf node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement staged::erase()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix container_t::update_size() to accept negative value

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement staged::is_keys_one()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement node-level erase/merge logic

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement delta encode/decode for erase ops

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: improve logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement fast Node::get_name()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix get_pivot_index() for leaf node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: be explicit about node num-key/value invariants

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup node interfaces

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement states for NodeExtentAccessorT

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement NodeExtentManager::retire_extent()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix log

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: remove implemented TODO

Already implemented in c32c1032de1223169a1223d1b8995856e8763df5

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement tree-level erase interfaces

Define and implement erase related interfaces from BTree, BTree::Cursor,
tree_cursor_t, Node to LeafNode.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add invalid state to tree_cursor_t

If a key-value pair is erased from the tree, the according tree_cursor_t
is untracked and becomes invalid.

From Cursor point-of-view, it becomes an end Cursor if the internal
cursor is invalidated.

From Value point-of-view, it becomes untracked if the value is erased.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup replace_track/track_split()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: disable concurrent gc

Fix test/perf hang according to 947b103eed0720bc6276274b1b707b221b895a62

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #40493 from CongMinYin/fix-rwl-fragment

librbd/cache/pwl: Fix pmem cache fragment issue

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: Fix pmem cache fragment issue

I/O may hang due to pmem cache fragment issue when blocks are diffrent
in size. Call pmdk API(pmemobj_defrag) to solve.

Fixes: https://tracker.ceph.com/issues/49879
Signed-off-by: Yin Congmin <congmin.yin@intel.com>

Merge PR #41072 into master

* refs/pull/41072/head:
doc/releases/nautilus: fix a braino in 14.2.20 release note

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #40924 into master

* refs/pull/40924/head:
mgr/cephadm: check hostname resolution before adding host
cephadm: provide a way to checkhost connection without /etc/hosts passed the shell
doc/cephadm: remove /etc/hosts from list of hostname resoltion methods

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #40593 from ideepika/wip-new-testing-params

qa/config/rados: add dispatch delay testing params

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sam Just <sjust@redhat.com>

Merge PR #40941 into master

* refs/pull/40941/head:
qa/suites/rados/cephadm/smoke-roleless: test client-keyring
qa/tasks/cephadm.py: adjust client.admin key mode; place on all hosts
cephadm: distribute client.admin keyring+conf to label:_admin on bootstrap
doc/cephadm: document the default 'admin' label
mgr/cephadm: 'ceph orch client-keyring ...' commands to manage keyring files
mgr/cephadm: reimplement ceph.conf pushing
mgr/cephadm: use _write_remote_file for ceph.conf
mgr/cephadm: _write_remote_file helper
mgr/cephadm: add placementspec for which hosts get ceph.conf

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge PR #40943 into master

* refs/pull/40943/head:
mgr/cephadm: skip ok-to-stop for mons in upgrade if < 3 mons
mgr/cephadm: don't allow upgrade start with less than 2 mgrs

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #41049 into master

* refs/pull/41049/head:
cephadm: re-assimilate user provided conf after mgr created

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #41055 into master

* refs/pull/41055/head:
cephadm: allow several public networks be matched

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #41059 from tchaikov/wip-crimson-clang-cleanup

crimson: clean up issues spotted by the clang compiler

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: disable "variable tracking" when building rados python binding

to silence the warning like:

/home/jenkins-build/build/workspace/ceph-pull-requests/build/src/pybind/rados/rados.c: In function '__pyx_pymod_exec_rados':
/home/jenkins-build/build/workspace/ceph-pull-requests/build/src/pybind/rados/rados.c:78903:30: note: variable tracking size limit exceeded with '-fvar-tracking-assignments', retrying without
78903 | static CYTHON_SMALL_CODE int __pyx_pymod_exec_rados(PyObject *__pyx_pyinit_module)
      |                              ^~~~~~~~~~~~~~~~~~~~~~
cc: warning: /usr/bin/cc: linker input file unused because linking not done

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not pass ${PYTHON_VERSION} to distutils_add_cython_module()

the python2 support was dropped in
5fc657b40dc7d27e84d4b62b5f37d9c771d74fcd, so no need to pass the python
version anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/nautilus: fix a braino in 14.2.20 release note

Setting auth_expose_insecure_global_id_reclaim to false disables the
"disconnect right after authentication" behavior, not the warning.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41063 from tchaikov/wip-rgw-move

rgw/rgw_rest_sts: return local variable w/o std::move()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

RGW Zipper - move attrs into User

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #41067 from zdover23/wip-doc-crismon-misspelling-fix-2021-Apr-28

doc/crimson: s/Crismon/Crimson/

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: add podman version note to install

This PR adds a note to the cephadm installation
guide that informs users that only podman version
2.0.0 and higher work with Ceph, with the
exception of podman version 2.2.1, which does
not work with Ceph. There is also a note regarding
kubic stable 3.0.1 working, but only with newer
kernels.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41048 from zdover23/wip-doc-rados-config-cluster-names-2021-Apr-27

doc/rados: rename "Running Multiple Clusters"

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/crimson: s/Crismon/Crimson/

This misspelling occurred twice in this
document.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/os/seastore: do not capture unused variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: do not capture unused variable

this change silences the warning from clang, like:

 ../src/crimson/common/interruptible_future.h:276:8: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
       [this, func=std::move(func),
        ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: mark final class "final"

this change silences warning like:

 ../src/crimson/os/seastore/onode_manager/staged-fltree/fltree_onode_manager.h:80:18: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
   ~FLTreeOnode() final {}
                  ^
 ../src/crimson/os/seastore/onode_manager/staged-fltree/fltree_onode_manager.h:13:8: note: mark 'crimson::os::seastore::onode::FLTreeOnode' as 'final' to silence this warning
 struct FLTreeOnode : Onode, Value {
        ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson/seastore: do not put braces around scalar initializer

this change silences the warning from clang like:

src/test/crimson/seastore/onode_tree/test_value.h:152:12: warning: braces around scalar initializer [-Wbraced-scalar-init]
     return {static_cast<value_size_t>(size - sizeof(value_header_t))};
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: do not reference non-template with template

this change addresses the build failure with clang:

src/crimson/os/seastore/onode_manager/staged-fltree/node_extent_accessor.h:352:29: error: 'encode_split' following the 'template' keyword does not refer to a template
    test_recorder->template encode_split(split_at, read().p_start());
                   ~~~~~~~~ ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: do not capture label using &

structured binding does not define variables. unlike GCC, Clang does
not allow this. this change address following build failures with clang:

../src/crimson/os/seastore/omap_manager/btree/omap_btree_node_impl.cc:210:24:
note: 'result' declared here
      auto &[complete, result] = ret;
                       ^
../src/crimson/os/seastore/omap_manager/btree/omap_btree_node_impl.cc:214:6:
error: reference to local binding 'complete' declared in enclosing
lambda expression
            complete = biter == eiter;
            ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados: rename "Running Multiple Clusters"

This PR corrects a title so that the title
more accurately describes the content in the
section it names.

Fixes: https://tracker.ceph.com/issues/50517 (partially)
Signed-off-by: Zac Dover <zac.dover@gmail.com>

include: define AT_REMOVEDIR on Windows

The AT_REMOVEDIR flag is used by a recent change [1] but isn't
defined on Windows.

This commit will add the missing definition. Note that it won't be
passed to OS functions, only being used at the CephFS level.

[1] https://github.com/ceph/ceph/pull/40810

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

rgw/rgw_rest_sts: return local variable w/o std::move()

in C++17, the C++ standard ensures that the return value in this case
can be optimized with copy elision, adding std::move() is not necessary,
and prevents copy elision from happening.

this change also silences the warning of:

../src/rgw/rgw_rest_sts.cc: In member function 'std::unique_ptr<rgw::sal::RGWOIDCProvider> rgw::auth::sts::WebTokenEngine::get_provider(const DoutPrefixProvider*, const string&, const string&) const':
../src/rgw/rgw_rest_sts.cc:92:19: warning: moving a local object in a return statement prevents copy elision [-Wpessimizing-move]
   92 |   return std::move(provider);
      |          ~~~~~~~~~^~~~~~~~~~
../src/rgw/rgw_rest_sts.cc:92:19: note: remove 'std::move' call

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: allow several public networks be matched

Fixes: https://tracker.ceph.com/issues/50548
Signed-off-by: Stanislav Datskevych <me@nek0.net>

cmake: bump to PMDK v1.10

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Yin Congmin <congmin.yin@intel.com>

Merge pull request #41032 from leseb/raw-list-json

ceph-volume: fix "device" output

Merge pull request #41002 from strenuous-life/wip-cephadm-daemon-restart-mgr-loop

mgr/cephadm: The command of 'ceph orch daemon restart mgr.xxx' may ca…

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Adam King <adking@redhat.com>