doc/man: build and install cephfs-shell manpage

otherwise the manpage of cephfs-shell is not built and installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/man/8/cephfs-shell: reformat options

* format global options using option directive
* fix the header, so man/conf.py is able to parse
  the description
* define "Synopsis" section to be consistent with other manpages.
* drop reference to glossary using "term" as manapge does not have
  reference to glossary entries.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #41160 into master

* refs/pull/41160/head:
Revert "Merge PR #41134 into master"

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert "Merge PR #41134 into master"

This reverts commit 93bca2d0e9d771e3019af5e7228af7986c20ab61, reversing
changes made to 7e70b15c437a2fbe207b515a98b8b21880980867.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41152 from linuxbox2/wip-restore-rgw-lua

cmake: restore Lua scripting support for RGW

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41024 from rhcs-dashboard/generate-manifest

mgr/dashboard: Generate NPM dependencies manifest

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41062 from rhcs-dashboard/50519-fix-ssl-cmd

mgr/dashboard: fix set-ssl-certificate{,-key} commands

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

cmake: restore Lua scripting support for RGW

RGW linkage with Lua was broken by the following commit:

commit 7471fbfa4bf015c972c0e577da2a1ee87f62adfd
Author: Kefu Chai <kchai@redhat.com>
Date:   Mon Apr 26 22:44:16 2021 +0800

    cmake: require lua if WITH_CEPHFS is enabled

    in src/mds/Mantle.cc, the lua C binding APIs are used, so we should only
    require lua if WITH_CEPHFS is enabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #41134 into master

* refs/pull/41134/head:
doc/man: build and install cephfs-shell manpage
doc/man/8/cephfs-shell: reformat options

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40926 from ivancich/wip-orphan-list-pools

rgw: allow rgw-orphan-list to process multiple data pools

Reviewed-by: Michael Kidd <linuxkidd@gmail.com>

Merge pull request #40631 from cbodley/wip-49780

rgw/sts: read_obj_policy() consults iam_user_policies on ENOENT

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #41033 from tchaikov/wip-osx-build

cmake,common: various cleanup for building on MacOS

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41045 from jmolmo/fix_cv_lvm_lock

mgr/cephadm: ceph-volume verbose only when fails

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #40313 from jmolmo/purge_iscsi_config

mgr/cephadm: Remove gateway.conf from iscsi pool when service is removed

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sage Weil <sage@newdream.net>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #40799 from myoungwon/wip-fix-null-promote-obc

osd: fix not set promote_obc when manifest object is rollbacked

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/man: build and install cephfs-shell manpage

otherwise the manpage of cephfs-shell is not built and installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/man/8/cephfs-shell: reformat options

* format global options using option directive
* fix the header, so man/conf.py is able to parse
  the description
* define "Synopsis" section to be consistent with other manpages.
* drop reference to glossary using "term" as manapge does not have
  reference to glossary entries.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #41066 into master

* refs/pull/41066/head:
mds: session dumps num_completed_flushes
mds: completed_requests -> num_completed_requests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40963 into master

* refs/pull/40963/head:
mds: do not trim the inodes from the lru list in standby_replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40962 into master

* refs/pull/40962/head:
test: add test to validate snap synchronization with parent directory snapshots
cephfs-mirror: ignore parent directory snapshots when building snap map

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40927 into master

* refs/pull/40927/head:
mds: reset the return value for heap command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40903 into master

* refs/pull/40903/head:
test: add test case for MDS privated inos accessing
mds: make the lost+found dir accessible from clients
mds: move the inos 1,2 and 3 macros to ceph_fs.h

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40882 into master

* refs/pull/40882/head:
client: Fix executeable access check for the root user

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #41120 into master

* refs/pull/41120/head:
doc/_ext: ignore desc if it is unavailable
doc/_ext: check "default" for None
doc/_ext: print 0B if option value is 0
doc/cephfs: render options using confval directive

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41102 from soumyakoduri/set_acl_fix

rgw/acl: set_acl shouldn't overwrite bucket attrs

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: fix set-ssl-certificate{,-key} commands

- Now create-self-signed-cert command relies on set-ssl-certificate{,-key} commands.
- Simplify the command testing and increase the test coverage.

Fixes: https://tracker.ceph.com/issues/50519
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #41093 from idryomov/wip-rbd-pwl-ssd-boundary

librbd/cache/pwl/ssd/WriteLog: don't crash on split log entries

Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge PR #41105 into master

* refs/pull/41105/head:
qa/workunits/cephadm/test_cephadm: test zap-osds
cephadm: add --zap-osds argument to rm_cluster
cephadm: implement zap-osds --fsid ... command

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

librbd/cache/pwl/ssd/WriteLog: don't crash on split log entries

write_log_entries() will split a log entry at the end of the log, the
remainder is written to the beginning at DATA_RING_BUFFER_OFFSET.  On
the read side aio_read_data_block() doesn't handle this case and just
crashes.  Unless the workload in use is <= 4K, the image is rendered
unusable sooner or later.

Fixes: https://tracker.ceph.com/issues/50589
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41068 from idryomov/wip-rbd-pwl-ssd-capacity

librbd/cache/pwl/ssd/WriteLog: decrement m_bytes_allocated when retiring

Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

osd: fix not set promote_obc when manifest object is rollbacked

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #41042 from tchaikov/wip-rbd-cls-sparse-size

librbd: 32-bit and libc++ portability fixups

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rgw/acl: set_acl shouldn't overwrite bucket attrs

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

doc/_ext: ignore desc if it is unavailable

there is chance that we don't have desc, desc_long or fmt_desc, in that
case, we should just skip desc before checking its length. so, just use
'if desc' which is able to check for None or empty string.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: check "default" for None

default could be None and we don't reference the default value using
`opt.default`, so this variable is always defined. to avoid printing
a None default value, check for a None value. see
https://jinja.palletsprojects.com/en/2.11.x/templates/#none

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: print 0B if option value is 0

before this change, the rendered Default is "0Ei" if the default size is 0,
would be more readable if it is "0B" or "0".

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephfs: render options using confval directive

less repeating, better maintainability this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41085 from idryomov/wip-rbd-nuke-source-spec-file

rbd: --source-spec-file should be --source-spec-path

Reviewed-by: Mykola Golub <mgolub@suse.com>

librados/librados_c: check .symver support using cmake

the __asm__(".asmver ..") is a support provided by the compiler, so
would be better to detect it by either checking the compiler identifer
or just try it out.

in this change, instead of checking the building platform, we check this
feature using check_c_source_compiles().

in future, we could support versioned symbols using function attriubte
or symbol tables or version-script.

on platform where symbol versioning is not supported, we might need to
go with a different approach.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/util: enable cgroup only on linux

cgroups, i.e. control groups, are a linux kernel feature. this is not
available on FreeBSD, Win32 or MacOS. furthermore, we are reading the
information of it by accessing files exposed using a tmpfs filesystem
mounted at /sys/fs/cgroup, which is not available on other systems.

in this change, instead of guarding it using `#ifndef _WIN32`, it is now
guraded using `defined(__linux__)`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/compat.h: declare cpu_set_t for MacOS

to appease the compiler as it needs to compile numa.cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Throttle: remove unused member variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/compat: detect memset_s() using check_function_exists()

more robust this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ceph_time: do not include non-existent header file

i have following error when building on MacOS

ceph/src/common/ceph_time.cc:29:10: fatal error: 'ostringstream' file not found
         ^~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/StackStringStream: make override method with "final"

this change also silences the warnings like:

src/common/StackStringStream.h:75:7: warning: 'overflow' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
  int overflow(int c)
      ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: detect suseconds_t using CHECK_TYPE_SIZE

instead assuming it's a macro use CHECK_TYPE_SIZE to check it.

on Darwin, it's defined as `__darwin_suseconds_t`

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not build libblk if libblk_srcs is empty

if we don't or just not able to buld libblk, let's skip it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: require lua if WITH_CEPHFS is enabled

in src/mds/Mantle.cc, the lua C binding APIs are used, so we should only
require lua if WITH_CEPHFS is enabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for empty string by its value

the behavior of cmake 3.17 is a little bit different as it consider

NOT _library STREQUAL ""

as true, if ${_library} is "".

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41116 from tchaikov/wip-in_addr_t

common/pick_address: define in_addr_t if it is not defined

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41117 from tchaikov/wip-doc-cephfs-conf

doc/cephfs/mds-config-ref: remove duplicated confval

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common/pick_address: define in_addr_t if it is not defined

neither mingw not not have in_addr_t defined, see
https://docs.microsoft.com/en-us/windows/win32/api/winsock2/ns-winsock2-in_addr,
so define it if it is not defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephfs/mds-config-ref: remove duplicated confval

this change addresses the warning of:

/home/jenkins-build/build/workspace/ceph-pr-docs/doc/cephfs/mds-config-ref.rst:2: WARNING: duplicate confval_option description of mds_cache_memory_limit, other instance in cephfs/cache-configuration
/home/jenkins-build/build/workspace/ceph-pr-docs/doc/cephfs/mds-config-ref.rst:2: WARNING: duplicate confval_option description of mds_cache_reservation, other instance in cephfs/cache-configuration

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41043 from ifed01/wip-ifed-fix-use-ratio

osd: compute OSD's space usage ratio via raw space utilization

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41012 into master

* refs/pull/41012/head:
doc/cephfs/cache-config: use config item macros

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40961 from tchaikov/wip-50456

common/pick_address: filter out loopback addresses

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41071 from tchaikov/wip-cmake-disable-vta

cmake: disable "variable tracking" when building rados python binding

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #41091 into master

* refs/pull/41091/head:
doc/cephfs/nfs: add section about ganesha logs
doc/cephfs/nfs: Replace volume/nfs with nfs
doc/cephfs/nfs: add note about export management with volume/nfs interface only

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41004 from sseshasa/wip-mclock-max-backfill-misc-fix

osd: Override recovery, backfill and sleep related config options during OSD and mclock scheduler initialization.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>

qa/workunits/cephadm/test_cephadm: test zap-osds

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: add --zap-osds argument to rm_cluster

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: implement zap-osds --fsid ... command

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #40967 from dang/wip-dang-zipper-user

RGW Zipper consistent load/store APIs for User

Merge pull request #40946 from ifed01/wip-ifed-dump-pool-statfs

osd: admin cmd to dump pools statfs for a specific OSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41014 from smithfarm/wip-mempool-cacheline-49781

qa: verify the benefits of mempool cacheline optimization

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40998 from cyx1231st/wip-seastore-onode-tree-delete

crimson/onode-staged-tree: implement/validate features to erase values from tree

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/cephfs/nfs: add section about ganesha logs

Fixes: https://tracker.ceph.com/issues/50008
Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephfs/nfs: Replace volume/nfs with nfs

Recently, nfs related code was moved out of volumes plugin[1]. So using the
name volume/nfs for the interface is not appropriate.

[1] https://github.com/ceph/ceph/pull/40526

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephfs/nfs: add note about export management with volume/nfs interface only

Signed-off-by: Varsha Rao <varao@redhat.com>

qa: verify the benefits of mempool cacheline optimization

There already is a test to verify the mempool sharding works, in the sense that
it uses at least half of the variables available to count the number of
allocated objects and their total size. This new test verifies that, with
sharding, object counting is at least twice faster than without sharding. It
also collects cacheline contention data with the perf c2c tool. The manual
analysis of this data shows the optimization gain is indeed related to cacheline
contention.

Fixes: https://tracker.ceph.com/issues/49896
Signed-off-by: Loïc Dachary <loic@dachary.org>

librbd/cache/pwl: include head and tail pointers in STATS

While at it, reduce the number of calls to operator<< and drop
the trailing comma.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: bump "Waiting for allocation" and "Retiring" dout level

Bump "Waiting for allocation" to 5.

"Retiring" is at 20 for rwl and 1 for ssd.  Bump the latter to 20 as
well.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: use m_bytes_allocated_cap for both rwl and ssd

Follow rwl mode and use AbstractWriteLog::m_bytes_allocated_cap
instead of m_log_pool_ring_buffer_size specific to ssd.  This fixes
"bytes available" calculation in STATS output.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd/WriteLog: decrement m_bytes_allocated when retiring

Currently if ssd cache is filled to capacity, all future I/O hangs
indefinitely because even though the cache eventually becomes clean
and retires enough entries to get back under RETIRE_HIGH_WATER, this
isn't communicated to AbstractWriteLog::check_allocation().

Fixes: https://tracker.ceph.com/issues/50560
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd/WriteLog: fix free()/delete mismatch

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd/WriteLog: fix indentation

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #40502 from zdover23/wip-doc-podman-version-octopus-note-2021-30-Mar

doc/cephadm: add podman version note to install

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #40072 from wornet-mwo/dashboard--grafana-hostname-corrections

mgr/dashboard: Fixed name clash when hostname similar to another

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: p-se <NOT@FOUND>

RGW Zipper - Permenant User storage API

The load/store/remove APIs for User are needed, so make them consistent
and permenant.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #41040 from tchaikov/wip-doc-mds-conf

doc/cephfs/mds-config-ref: render options using confval directive

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rbd: --source-spec-file should be --source-spec-path

It was renamed during development, get rid of the leftovers.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41028 from mkogan1/wip-rgw-tracker-50467

rgw: fix segfault related to explicit object manifest handling

Merge pull request #41083 from rhcs-dashboard/fix-50580-master

mgr/dashboard: OSDs placement text is unreadable

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41082 from tchaikov/wip-50558

os/bluestore: Revert" Optimizing the lock of bluestore writing process"

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #41064 from petrutlucian94/win_at_removedir

include: define AT_REMOVEDIR on Windows

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: ceph-volume verbose only when fails

Resolves: https://tracker.ceph.com/issues/50526

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #41038 from tchaikov/wip-cmake-liburing

cmake: pass "CC" using configure when building liburing

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

mgr/dashboard:OSDs placement text is unreadable

While displaying the host pattern in the OSDs placement tab, it gets splited with semi-colons. Also adjusted the column size of Container Image ID and Placement columns.

Fixes: https://tracker.ceph.com/issues/50580
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

os/bluestore: Revert" Optimizing the lock of bluestore writing process"

This reverts commit e80d3f7a891af99ba0d4cfe99e5530016ca867ac.

Fixes: https://tracker.ceph.com/issues/50571
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: implement erase for perf-staged-tree

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement synthetic tests for value erase

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix ownership issues at node-level

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: allow to insert item to an empty node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: move try_downgrade_root() into fix_index()

Specifically, apply_children_merge() doesn't know whether it will be
retired due to fix_parent_index().

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: don't validate child when it's outdated

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: pass origin_left_addr to apply_children_merge()

rebuild_extent() may change the address of the left child, we need to
pass the origin_left_addr in order to validate if it is correct.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/.../node_extent_manager: replay extents may be dirty

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/onode-staged-tree: cleanup tree_utils.h

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: test fix_index() in internal node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: test merge in internal node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>