common/options.h: use uint64_t for TYPE_SIZE options regardless of CPU architecture

On 32 bit systems TYPE_SIZE should remain as 64 bit value as large number of default options of TYPE_SIZE easily overflow 32 bit word. Also cluster wide options from 64 bit systems should fit unmodified into options structures on 32 bit systems.

Signed-off-by: Vladimir Bashkirtsev <vladimir@bashkirtsev.com>

Merge pull request #38796 from MahatiC/wip-ssd-read

librbd/cache: Implement aio_read operation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38272 from vbashkirtsev/master

test/test_arch.cc: search "pni" in CPU flags also when detecting SSE3

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/test_arch.cc: search "pni" in CPU flags also when detecting SSE3

as sometimes a SSE-3 CPU reports "pni" (Prescott New Instructions) in /proc/cpuinfo.

Signed-off-by: Vladimir Bashkirtsev <vladimir@bashkirtsev.com>

do_cmake.sh: do not perform 'git submodule update' if building from tarball

Signed-off-by: Vladimir Bashkirtsev <vladimir@bashkirtsev.com>

Merge pull request #38753 from Rethan/style-ceph-df-output

mon/PGMap: align to same side when output ceph df / ceph df detail

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38775 into master

* refs/pull/38775/head:
qa: remove ceph file systems on completion

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38791 into master

* refs/pull/38791/head:
mon/MDSMonitor: add missing frozen checks

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #38778 into master

* refs/pull/38778/head:
mds: don't use dentry_key_t in C_IO_Dir_Commit_Ops

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #38773 into master

* refs/pull/38773/head:
mds: avoid spurious sleeps

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #38698 into master

* refs/pull/38698/head:
client: fix the cache size check
client: add more detail info for the _put_inode debug log
client: skip putting inodes if map is empty

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38683 into master

* refs/pull/38683/head:
mds: add scrub perf counters support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38642 into master

* refs/pull/38642/head:
rpm/deb/cmake: install mount.fuse.ceph man page

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37882 from cbodley/wip-rgw-data-sync-notification-fix

rgw: data sync: fix obligation check

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #38720 from bk201/wip-48717

mgr/dashboard: fix Reads/Writes ratio of Clients IOPS donut chart

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #38776 into master

* refs/pull/38776/head:
qa: get pool holding file data from file layout

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38664 into master

* refs/pull/38664/head:
qa: bump scrub timeout
qa: move cephfs_ec_profile under cephfs
qa: do not use ec pools for default data pool
qa: skip check-counters for light workloads
qa: remove empty multimds suite
qa: merge multimds:verify with fs:verify
qa: merge multimds:thrash to fs:thrash
qa: remove dead multimds:basic
qa: move functional multimds tests to fs:functional
qa: migrate multimds workloads to fs:workloads
qa: only run valgrind on cephfs daemons
qa: stop testing filestore on cephfs suites
qa: load data pools before deleting fs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Tested-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38693 into master

* refs/pull/38693/head:
qa: execute scrubs only on rank 0
client: print debug information about resolved MDS
qa: let Client::resolve_mds lookup the rank

Reviewed-by: Xiubo Li <xiubli@redhat.com>

qa: remove ceph file systems on completion

So that we can avoid MDS replacement warnings.

Fixes: https://tracker.ceph.com/issues/48757
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38510 into master

* refs/pull/38510/head:
PendingReleaseNotes: Update about volume/nfs cluster id changes
mgr/volumes/nfs: remove 'ganesha-' prefix from cluster id

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38537 from IlsooByun/fix_ret_invalid_err

rgw: return error when trying to copy encrypted object without key

Merge pull request #38283 from dvanders/dvanders_throttle

rgw: add an outstanding_requests perf counter

Merge pull request #38242 from theanalyst/rgw/reset-stats-1-fix

rgw: cls/user: set from_index for reset stats calls

Merge pull request #38725 from CongMinYin/immutable-obejct-cache-doc

doc/rbd: Immutable object cache doc

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38667 from pcuzner/fix-cephadm-stdout

cephadm: make host add failure message more friendly

Merge pull request #37920 from votdev/issue_48068_timezone

cephadm: Various properties like 'last_refresh' do not contain timezone

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38807 from tchaikov/wip-mon-cmd-doc

doc: build mon_command_api.rst using a homebrew extension

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

PendingReleaseNotes: Update about volume/nfs cluster id changes

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: remove 'ganesha-' prefix from cluster id

Orchestrator defines service name as "<service_type>.<service_id>". The ganesha
common config object name in orchestrator is "conf-<service_name>". Volume's
nfs plugin deploys nfs-ganesha clusters with 'ganesha' prefixed to cluster id
and common config object.  It can cause unecessary issues in rook and cephadm.
So let's remove the prefix.

Fixes: https://tracker.ceph.com/issues/48514
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #38754 from tchaikov/wip-confutil-string-view

common/ConfUtils: pass string_view instead of string

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/_ext: accept params separated with ", "

"mon enable_stretch_mode" command's arg descriptors are delimited with
", ", so we should ignore empty kv when split the params with ",".

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: build mon_command_api.rst using a homebrew extension

so we can build the command doc using ReadTheDoc infra, without adding
the generated rst file to the source repo.

before this change, all the commands are ordered alphabetically. after
this change, command docs are generated by two directives, and are
ordered separately. we could restructure the directives and merge them.

but let's leave it for a future change if this is important.

for more details on writing sphinx directives, see
https://www.sphinx-doc.org/en/master/extdev/markupapi.html and
https://docutils.sourceforge.io/docs/howto/rst-directives.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: bump scrub timeout

To avoid timeouts for large workloads.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: move cephfs_ec_profile under cephfs

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: do not use ec pools for default data pool

This configuration unnecessarily slows down backtrace writes and bloats
the used disk space.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: skip check-counters for light workloads

None of these are likely to generate exports.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove empty multimds suite

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/48121

qa: merge multimds:verify with fs:verify

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/48121

qa: merge multimds:thrash to fs:thrash

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/48121

qa: remove dead multimds:basic

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/48121

qa: move functional multimds tests to fs:functional

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/48121

qa: migrate multimds workloads to fs:workloads

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/48121

qa: only run valgrind on cephfs daemons

OSD valgrind slows things down too much to the point where some tasks
fail to complete.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: stop testing filestore on cephfs suites

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: load data pools before deleting fs

Otherwise current data pools at time of deletion may not be known.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38794 into master

* refs/pull/38794/head:
qa: add debug vstart_runner option

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #38377 from yanghonggang/master

rgw/rgw_http_client.cc: Introduce a new configuration item - rgw_curl_buffersize

Merge pull request #38800 from sebastian-philipp/doc-cephadm-rm-link-to-de

doc/cephadm: some minor updates

Reviewed-by: Varsha Rao <varao@redhat.com>

doc/cephadm: /etc/ceph is created automaticlly now

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: CephFS: Encourage users to call `ceph fs volume create`

Manually deploying MDS daemons is an advanced topic.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: update stability notes

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Rm doc/dev/cephadm from user docs

We should not link the dev guide form the user documentation. The user docs are already convoluted and hard to follow. Adding the dev guide is not helpful.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38784 from sebastian-philipp/resurect-doc-dev-cephadm

doc/dev: resurrect hidden dev/cephadm.rst

Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #38560 from pcuzner/cephadm-maintenance-mode

cephadm: add host-maintenance subcommand

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #38681 from pcuzner/enable-exporter

cephadm: enable exporter deployment at bootstrap

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38704 from varshar16/wip-fix-rook-rm-service

mgr/rook: Remove service_type param from remove_service()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

librbd/cache: Implement aio_read operation

... and retire entries i.e. flush to OSD.
Support writesame, compare_and_write, discard
and invalidate IO operations with tests.

Signed-off-by: Lisa Li <xiaoyan.li@intel.com>
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

Merge pull request #38025 from MahatiC/wip-ssd-write

librbd/cache: Implement aio_write operation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mds: add scrub perf counters support

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #38795 from yuriw/wip-yuriw-octopus-x-master

qa/tests: changed the dirs structure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #38792 into master

* refs/pull/38792/head:
.github: add MDSMonitor to cephfs label

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #38587 into master

* refs/pull/38587/head:
doc: add explanatory note about cache trim rate setting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tests: changed the dirs structure

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38787 from yuriw/wip-yuriw-octopus-x-master

qa/tests: octopus-x suite added more workloads

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38789 from neha-ojha/wip-fix-cephadm-task

qa/tasks/cephadm.py: do not create rbd pool by default

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa: add debug vstart_runner option

The default was changed to INFO recently but there was no way to restore
visibility of DEBUG messages.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

.github: add MDSMonitor to cephfs label

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/MDSMonitor: add missing frozen checks

This has no known impact on testing.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tests: octopus-x suite added more workloads

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/tasks/cephadm.py: do not create rbd pool by default

rados/cephadm/smoke* does not use the install task and the adjust-ulimits
dependency is met as a part of it. create_rbd_pool needs adjust-ulimits,
so for now we will disable create_rbd_pool by default and only set it
to true for the upgrade suite.

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: get pool holding file data from file layout

Otherwise it always looks at the default data pool. For ec pools, this
may not be where the file data is.

Fixes: https://tracker.ceph.com/issues/48756
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38714 from tchaikov/wip-better-vstart

vstart: format the customized options in ceph.conf

Reviewed-by: luo rixin <luorixin@huawei.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38769 into master

* refs/pull/38769/head:
doc/cephfs: add data pool-MDS instructions link

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

mds: avoid spurious sleeps

Like trim_interval, don't sleep for small amounts of time. This avoids
spurious sleeps like:

    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s
    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s
    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s
    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s
    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s
    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s
    2020-12-25T00:14:22.242+0000 7f6a95884700 20 mds.0.cache upkeep thread waiting interval 0.000000108s

Also, fix the same issue in the Client.

Fixes: https://tracker.ceph.com/issues/48753
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd/cache: Implement aio_write operation

...along with load_existing_entries and shutdown cache
operation

Signed-off-by: Lisa Li <xiaoyan.li@intel.com>
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

doc/dev: resurrect hidden dev/cephadm.rst

some time ago, a cephadm folder was added that did hide the
old dev documentation page.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephfs: add data pool-MDS instructions link

- This commit adds a link to the "Create a Ceph
File System" page. The link that it adds is to the
"Adding a data pool to the MDS" subsection of the
file layouts page.
- s/mds/file system/

Fixes: https://tracker.ceph.com/issues/48531
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38713 from tchaikov/wip-ceph-annotations

ceph.in: return bytes as outbuf in error path, add type annotations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: don't use dentry_key_t in C_IO_Dir_Commit_Ops

dentry_key_t uses std::string_view to access corresponding dentry's name.
C_IO_Dir_Commit_Ops() is executed by worker thread. It's possible dentry
gets freed before C_IO_Dir_Commit_Ops() gets executed.

Signed-off-by: "Yan, Zheng" <ukernel@gmail.com>

tools: update immutable object cache water mark

Signed-off-by: Yin Congmin <congmin.yin@intel.com>

doc/rbd: add the configuration description of immutable object cache throttle

Signed-off-by: Yin Congmin <congmin.yin@intel.com>

Merge pull request #38777 from neha-ojha/wip-cephadm-rbd

qa: allow the cephadm task to create an RBD pool and add workload to use it

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/suites/upgrade/octopus-x: add rados_loadgenbig workload

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/tasks/cephadm.py: create RBD pool

The ceph task already does this and a bunch of tests rely on this pool
being already present. Can be disabled by setting create_rbd_pool to False.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #38742 into master

* refs/pull/38742/head:
doc/cephfs/nfs: Add more details about path and pseudo path
doc/cephfs/nfs: Add note about how to fetch port details with rook
doc/cephfs/nfs: Add warning note about management of dashboard created exports

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #38735 from changchengx/pwl_load_existing_cache

librbd/cache/pwl: correct update previous_sync_point_entry

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38520 from lixiaoy1/wip-doc-pwl

doc: add RBD persistent write-back cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: add RBD persistent write-back cache

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

doc/cephfs/nfs: Add more details about path and pseudo path

Fixes: https://tracker.ceph.com/issues/48731
Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephfs/nfs: Add note about how to fetch port details with rook

Fixes: https://tracker.ceph.com/issues/48731
Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephfs/nfs: Add warning note about management of dashboard created exports

Fixes: https://tracker.ceph.com/issues/48731
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge PR #32776 into master

* refs/pull/32776/head:
win32*.sh: fetch WNBD dependency
win32*.sh: bump win32 boost dependency
cmake: skip osd, Lua and neorados on Windows
win32*.sh: improve lib handling
win32*.sh: update openssl location
win32*.sh: cleanup dependency build dir when targetting Windows
win32*.sh: Strip binaries individually
win32*.sh: Cleanup zip archive creation
cmake,win32*.sh: Add OS=(ubuntu|suse) flag, for distro settings
win32*.sh: Use nproc for CPU count when targeting win32
test: Include compat.h to find uid_t/gid_t/gidlist when cross compiling
win32*.sh: improve win32 dependency check
win32*.sh: isolate network operations when building win32 dependencies
cmake: drop Python dependency when targeting Windows
win32*.sh: cleanup Windows build script
win32*.sh: fix patch command
win32*.sh: update Windows build scripts and readme

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cephadm: Various properties like 'last_refresh' do not contain timezone

Fixes: https://tracker.ceph.com/issues/48068
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #38699 from rosinL/boost-paralllel

do_cmake: add a default value to BOOST_J

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38745 from zdover23/wip-doc-jan-2021-dev-guide-basic-workflow-reorg

doc/dev: restructure basic-workflow.rst

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38749 from tchaikov/wip-cmake-check-nasm

cmake: only display the result of checking nasm once

Reviewed-by: Changcheng Liu <changcheng.liu@aliyun.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #38703 into master

* refs/pull/38703/head:
test: move snap delete operation to non-root directory
client: optionally check client permission on rmsnap()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38758 from tchaikov/wip-crimson-cleanup

crimson/os: do not let omap_get_values() return input_output_error

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38757 from rosinL/wip-remove-unused-options

common/options: remove ununsed options

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not let omap_get_values() return input_output_error

BlueStore only returns 0 or -ENOENT. so no need to handle the cases
where the return value is neither of them.

but Seastore might return EIO, so keep the interface unchanged.

Signed-off-by: Kefu Chai <kchai@redhat.com>