qa: add debug vstart_runner option

The default was changed to INFO recently but there was no way to restore
visibility of DEBUG messages.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38714 from tchaikov/wip-better-vstart

vstart: format the customized options in ceph.conf

Reviewed-by: luo rixin <luorixin@huawei.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38769 into master

* refs/pull/38769/head:
doc/cephfs: add data pool-MDS instructions link

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

doc/cephfs: add data pool-MDS instructions link

- This commit adds a link to the "Create a Ceph
File System" page. The link that it adds is to the
"Adding a data pool to the MDS" subsection of the
file layouts page.
- s/mds/file system/

Fixes: https://tracker.ceph.com/issues/48531
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38713 from tchaikov/wip-ceph-annotations

ceph.in: return bytes as outbuf in error path, add type annotations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38777 from neha-ojha/wip-cephadm-rbd

qa: allow the cephadm task to create an RBD pool and add workload to use it

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/suites/upgrade/octopus-x: add rados_loadgenbig workload

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/tasks/cephadm.py: create RBD pool

The ceph task already does this and a bunch of tests rely on this pool
being already present. Can be disabled by setting create_rbd_pool to False.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #38742 into master

* refs/pull/38742/head:
doc/cephfs/nfs: Add more details about path and pseudo path
doc/cephfs/nfs: Add note about how to fetch port details with rook
doc/cephfs/nfs: Add warning note about management of dashboard created exports

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #38735 from changchengx/pwl_load_existing_cache

librbd/cache/pwl: correct update previous_sync_point_entry

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38520 from lixiaoy1/wip-doc-pwl

doc: add RBD persistent write-back cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: add RBD persistent write-back cache

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

doc/cephfs/nfs: Add more details about path and pseudo path

Fixes: https://tracker.ceph.com/issues/48731
Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephfs/nfs: Add note about how to fetch port details with rook

Fixes: https://tracker.ceph.com/issues/48731
Signed-off-by: Varsha Rao <varao@redhat.com>

doc/cephfs/nfs: Add warning note about management of dashboard created exports

Fixes: https://tracker.ceph.com/issues/48731
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge PR #32776 into master

* refs/pull/32776/head:
win32*.sh: fetch WNBD dependency
win32*.sh: bump win32 boost dependency
cmake: skip osd, Lua and neorados on Windows
win32*.sh: improve lib handling
win32*.sh: update openssl location
win32*.sh: cleanup dependency build dir when targetting Windows
win32*.sh: Strip binaries individually
win32*.sh: Cleanup zip archive creation
cmake,win32*.sh: Add OS=(ubuntu|suse) flag, for distro settings
win32*.sh: Use nproc for CPU count when targeting win32
test: Include compat.h to find uid_t/gid_t/gidlist when cross compiling
win32*.sh: improve win32 dependency check
win32*.sh: isolate network operations when building win32 dependencies
cmake: drop Python dependency when targeting Windows
win32*.sh: cleanup Windows build script
win32*.sh: fix patch command
win32*.sh: update Windows build scripts and readme

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38699 from rosinL/boost-paralllel

do_cmake: add a default value to BOOST_J

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38745 from zdover23/wip-doc-jan-2021-dev-guide-basic-workflow-reorg

doc/dev: restructure basic-workflow.rst

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38749 from tchaikov/wip-cmake-check-nasm

cmake: only display the result of checking nasm once

Reviewed-by: Changcheng Liu <changcheng.liu@aliyun.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #38703 into master

* refs/pull/38703/head:
test: move snap delete operation to non-root directory
client: optionally check client permission on rmsnap()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38758 from tchaikov/wip-crimson-cleanup

crimson/os: do not let omap_get_values() return input_output_error

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38757 from rosinL/wip-remove-unused-options

common/options: remove ununsed options

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not let omap_get_values() return input_output_error

BlueStore only returns 0 or -ENOENT. so no need to handle the cases
where the return value is neither of them.

but Seastore might return EIO, so keep the interface unchanged.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38616 from ofriedma/wip-ofriedma-dist-cache

rgw: distribute cache for exclusive put

Merge pull request #38663 from tchaikov/wip-crimson-recovery-cleanups

crimson/osd: fixes and cleanups

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/options: remove ununsed options

Signed-off-by: luo rixin <luorixin@huawei.com>

Merge pull request #38676 from ronen-fr/wip-48690

qa/standalone/scrub/osd-scrub-snaps.sh: fix Python print syntax

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38603 from smithfarm/wip-48604

rpm: require smartmontools on SUSE

Reviewed-by: Volker Theile <vtheile@suse.com>

vstart: format the customized options in ceph.conf

better formatted in ceph.conf, no extra spaces and tabs anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: only display the result of checking nasm once

if nasm is able to emit AVX512 instructions, we can assume that it's
able to generate AVX2 instructions as well. so no need to print "Found
nasm" multiple times.

and by chaining mutual exclusive branches with "elseif", we can have
better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: drop duplicated variable

not_arch_x32 replicates ${_support_x64}. and a variable starts with
"not" is just difficult to parse. so drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38694 from dillaman/wip-migration-import

librbd/migration: tweaks and initial set of documentation

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/rbd: incorporate external data source support for live-migration

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: fixed missing return statements on stream error path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: optionally pull S3 keys from MON config store

This allows the S3 keys to be better protected since the MON can
be configured to restrict access to keys by user and the results
can be encrypted in transit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper utility to retrieve config from the MON config store

A special "config://" URI prefix can be used to denote configuration
settings that should be (securely) pulled from the MON config store.
This will be first used in a follow-up commit to support storing
the S3 access and secret keys in the MON config store.

This "config://" syntax is already in-use by RGW and ceph-iscsi for
pulling secrets when deployed via cephadm.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: require snapshot when importing from native source

Since we cannot mark the source image read-only when running in import-only
migration mode, we should require the user to provide a snapshot to ensure
that data cannot change while the migration is running.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph.in: remove unused indent

no need to put `if True`. it was used when removing unused variables.
but it'd be better to have less indent for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in: do not apply rados.cstr() to a bytes

since the parameter is already a bytes instance, there is no need to encode it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: correct annotation of {mon,pg,osd}_command

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: no need to import print()

we are already in a python3 world.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: PEP8 related cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in: return bytes as outbuf in error path

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in,ceph_argparse: add type annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30027 from majianpeng/optimize-mutex-contention

os/bluestore: optimize mutex contention

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #36088 from rhcs-dashboard/fix-45822-master

mgr/dashboard:minimize console log traces of Ceph backend API test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38651 from ifed01/wip-ifed-new-omap-naming

os/bluestore: introduce new "per-pg" naming scheme for omap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38680 from ifed01/wip-ifed-rocksdb-options-annex

os/bluestore: introduce bluestore_rocksdb_options_annex config parame…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38738 from ronen-fr/wip-ronenf-rawformat

librbd/migration: removing an unused capture

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38709 from wuhongsong/aios_be_truncated

blk: add upper bound of bluestore_deferred_batch_ops* options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/dev: restructure basic-workflow.rst

- Rewrote a sentence about forking.
- s/exactly how to proceed/how to proceed/
- Changed titles to present participles
- Rewrote "Fixing the Bug Locally" to improve its elegance.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38655 from dillaman/wip-47828

librbd: ensure stripe unit is at least 512 bytes

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Mykola Golub <mgolub@suse.com>

librbd/migration: removing an unused capture

...silencing a compiler warning.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

librbd/cache/pwl: correct update previous_sync_point_entry

Always update previous_sync_point_entry value whatever its old
value.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #38733 from ukernel/wip-yanzheng-email

mailmap: update affiliation for Yan, Zheng

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38237 from myoungwon/wip-fix-clone-size-mismatch

osd: fix clone size mismatch when deduped object is evicted

Reviewed-by: Samuel Just <sjust@redhat.com>

mailmap: update affiliation for Yan, Zheng

Signed-off-by: "Yan, Zheng" <ukernel@gmail.com>

Merge pull request #37704 from Huber-ming/log_error

radosgw-admin: fix error message when datalog list

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

blk: add upper bound of bluestore_deferred_batch_ops* options

so the number of pending io does not overflow when being passed to submit_batch().

Signed-off-by: hzwuhongsong <hzwuhongsong@corp.netease.com>

Merge pull request #38728 from xxhdx1985126/wip-crimson-osd_op-no-splice

crimson/osd: do not splice osd_op indata

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make sure PGBackend's modification operations not change osd_op state

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: do not splice osd_op indata

As each osd_op's indata holds only its own data, there's no need to drop their it.
On the other hand, indata is still needed when osd_op needs to be redone in case of
acting set change.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #38633 from rhcs-dashboard/fix-logs-copy-button

mgr/dashboard: Fix for copy2Clipboard failing to copy on the first time

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: Fix for copy2Clipboard failing to copy on the first time.

Fixes: https://tracker.ceph.com/issues/48601
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Fix for the copy2Clipboard button to work on the first click itself.

Merge pull request #37267 from TheJJ/fix-up-acting-docs

doc: fix up and acting set explanation

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix and improve the explainations of up and acting osd sets

acting = where clients requests are processed
up = where pgs were planned to be by crush+upmap

Fixes: https://tracker.ceph.com/issues/48718
Signed-off-by: Jonas Jelten <jj@sft.lol>

mailmap: update mail for Jonas Jelten

Signed-off-by: Jonas Jelten <jj@sft.lol>

Merge pull request #38719 from fengchunsong/master

test/perf_local: Add div32,prefetch,memory barrier bench for arm64

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38568 from batrick/i48594

pybind/mgr/cephadm: limit rgw osd caps

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38644 from pcuzner/fix-missing-py3-error

mgr/cephadm: correct connection failure message

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38654 from Daniel-Pivonka/osddryrunbug

mgr/cephadm:fix bug in orch apply osd --dry-run where empty table printed

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

test/perf_local: Add div32,prefetch,memory barrier bench for arm64

arm64 support div32,prefetch,memory barrier, so add them to benchmark.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>

Merge pull request #38717 from swinds24/fix_compile_error

kv/LevelDBStore: fix incorrect ceph_abort_msg() usage

Reviewed-by: Kefu Chai <kchai@redhat.com>

kv/LevelDBStore: fix incorrect ceph_abort_msg() usage

Compile error is encountered when HAVE_LEVELDB_FILTER_POLICY is not
defined.

/ceph/src/kv/LevelDBStore.cc: In member function 'int LevelDBStore::load_leveldb_options(bool, leveldb::Options&)':
/ceph/src/kv/LevelDBStore.cc:96:22: error: invalid initialization of reference of type 'const string& {aka const std::__cxx11::basic_string<char>&}' from expression of type 'bool'ceph_abort_msg(0 == "bloom size set but installed leveldb doesn't support bloom filters");
                    ~~^~~~
/ceph/src/include/ceph_assert.h:90:69: note: in definition of macro 'ceph_abort_msg'
   ::ceph::__ceph_abort( __FILE__, __LINE__, __CEPH_ASSERT_FUNCTION, msg)
                                                                     ^~~
/ceph/src/include/ceph_assert.h:62:19: note: in passing argument 4 of 'void ceph::__ceph_abort(const char*, int, const char*, const string&)'
 [[noreturn]] void __ceph_abort(const char *file, int line, const char *func,
                   ^~~~~~~~~~~~
src/kv/CMakeFiles/kv.dir/build.make:130: recipe for target 'src/kv/CMakeFiles/kv.dir/LevelDBStore.cc.o' failed

Signed-off-by: Jerry Lee <leisurelysw24@gmail.com>

Merge pull request #38702 from mkogan1/wip-fix-vstart-o

vstart: fix -o ceph.conf options processing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38711 from Rethan/fix-osdmap-message-max-bytes-comments

option: update comments of option osd_map_message_max_bytes

option: update comments of option osd_map_message_max_bytes

Option osd_map_message_max_bytes indicates the max bytes of MOSDMap message, not maps count
as osd_map_message_max.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

Merge pull request #38700 from tchaikov/wip-seastar-msgr-test-timeout

test/crimson: add abort timer to messenger test

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38576 from myoungwon/wip-fix-seg-fault-manifest-count

osd : prevent accessing deleted reference

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38662 from Songweibin/wip-121918

osd/ReplicatedBackend: handle tailing zeros of recovering objects nicer

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38665 from tchaikov/wip-arm32-cleanups

erasure-code, test, pybind: various arm32 related cleanups.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38677 from ilanddev/mgr-deadlock-fix

mgr/ActivePyModules.cc: always release GIL before attempting to acquire a lock

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38678 from tchaikov/wip-denc

include/denc: use pair<const K,V> in range-based for loop

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #37428 into master

* refs/pull/37428/head:
mds: move duplicated inner loop out
mds: try to replicate hot dir to restarted MDS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #36597 into master

* refs/pull/36597/head:
client: make the send_to_auth display of mds_requests more accurate

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38037 into master

* refs/pull/38037/head:
qa: test_readahead add kernel client support

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38631 into master

* refs/pull/38631/head:
MDSLogContext: fix typo in MDSLogContextBase and add more comments

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38672 into master

* refs/pull/38672/head:
mds: increase the l_mds_openino_backtrace_fetch when fetching the backtrace

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38692 into master

* refs/pull/38692/head:
pybind/cephfs: pass command as size-1 array of strings

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38668 into master

* refs/pull/38668/head:
osdc: Fix return type of map waiting functions
monc: Use post rather than defer
common/async: Hold lock in constructor of blocked_result
osdc: Don't keep wrapping completion in wait_for_latest_osdmap
neorados: Replace unsafe uses of dispatch with post

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore: upgrade omap naming scheme to be per-pg

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38707 from ceph/wip-yuriw-octopus-x-4-master

qa/tests: initial check in for octopus-x/parallel tests (try 2)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/cephadm: include first mgr host when applying mgrs

This prevents the first mgr from being shut down due to lack of
appropriate placements.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw: distribute cache for exclusive put

Distribute cache for exclusive put

Fixes: https://tracker.ceph.com/issues/48632
Thanks to Casey Bodley for finding the solution

Signed-off-by: Or Friedmann <ofriedma@redhat.com>

test: move snap delete operation to non-root directory

In teuthology, root directory in a Ceph Filesystem is owned
by root (0:0) and has sticky-bit enabled. Snapshots created
on the directory carry its UID/GID and mode.

This causes tests that remove snapshots on root directory to
fail. Sticky bit on a directory gives permission to delete a
file in that directory only if you own the file.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

client: optionally check client permission on rmsnap()

By default, Client::rmsnap() skips checking client permissions.
This was ok till now since it was not a user facing API (via
libcephfs).

With PR #37721 ("mds: store custom metadata on snapshot creation"),
rmsnap can be invoked via ceph_rmsnap() libcephfs call. Therefore,
the usual client side permission checks should be performed before
calling out to the MDS.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

vstart: fix -o ceph.conf options processing

options passed to vstart.sh via the -o param and bunched on the same
line without line termination between them
hence not being parsed.

ex:
```
cat ./ceph.conf
...
debug_ms=0 debug_objecter=0 bluestore_debug_enforce_settings=ssd
bluestore_block_size=214748364800
...
```

Fixes: https://tracker.ceph.com/issues/48708
Signed-off-by: Mark Kogan <mkogan@redhat.com>

test/crimson: add abort timer to messenger test

Use a 120s timer to force abort the messenger test which should normally
succeeds in 60s, in order to:
* Fail the make check to print logs about where it is blocking;
* Free the listening IP address for the next make check;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

do_cmake: add a default value to BOOST_J

Signed-off-by: luo rixin <luorixin@huawei.com>

mgr/dashboard:minimize console log traces of Dashboard backend API tests

Changed log levels in some files to reduce console log traces

Fixes: https://tracker.ceph.com/issues/45822
Signed-off-by: Aashish Sharma <aasharma@redhat.com>