Merge pull request #38304 from sebastian-philipp/cephadm-inventory-cache-docstring

mgr/cephadm: add docstring describing the HostCache

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #38078 from zdover23/wip-doc-docs-style-guide-prompts-nov-2020

doc/start: update prompts in documenting-ceph.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38303 from aclamk/wip-bs-types-test-leak

os/bluestore: Fixed memory leak in test

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38058 from lxbsz/rbd_expand11

global: reexpand the config meta just after the fork() is done

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38240 from ifed01/wip-ifed-is-valid-io-log

blk: log is_valid_io() parameters when unsuccessful.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38262 from tchaikov/wip-librbd-gcc-10

neorados: use lambda instead of std::mem_fn()

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38278 from votdev/fix_cephadm_typo

pybind/mgr/cephadm: Fix various typos

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #38211 from ricardoasmarques/sort-host-device-ls

mgr/orchestrator: Sort 'ceph orch device ls' by host

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #38138 from batrick/centos-8.2

qa: update centos release to 8.2

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38224 from Rethan/fix_pg_dump_json

mgr: fix "ceph pg dump_json/dump_pools_json" format error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38206 from dvanders/dvanders_upmap

mgr/balancer: fix available pgs sent to calc_pg_upmaps

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #38264 from tchaikov/wip-mgr-prometheus-cleanups

pybind/mgr/prometheus: cleanups

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #38241 from Rethan/update_url_from_master_to_lastest

doc/dev: use http://docs.ceph.com/en/latest/ instead of /docs/master/ for docs

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #38306 from wjwithagen/wjw-fix-uring-functype

blk: fix parameters for non native uring

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38297 from badone/wip-make-sure--e-actually-works

run-make-check.sh: Make sure a build failure will exit

Reviewed-by: Kefu Chai <kchai@redhat.com>

blk: fix parameters for non native uring

The parameterlist used when uring is availble needs
to be equal to the case where no uring is available.

fixes: https://github.com/ceph/ceph/pull/38257
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #38265 from Rethan/doc-prometheus-config-key

doc/mgr/prometheus: update 'config-key set' to 'config set' for consistency

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38257 from lnsyyj/wip-iouring-poll

blk/kernel: expose IORING_SETUP_{IOPOLL,SQPOLL} as options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38263 from tchaikov/wip-boost-1.74-asio

src,cmake: define BOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT for Boost.Asio users and cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mgr/cephadm: add docstring describing the HostCache

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

os/bluestore: Fixed memory leak in test

This is only unittest issue. This (potential) problems do not appear is BlueStore itself.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge PR #38149 into master

* refs/pull/38149/head:
ceph-volume inventory: make libstoragemgmt data retrieval optional

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

doc/mgr/prometheus: update 'config-key set' to 'config set' for consistency

Elsewere in mgr/prometheus use the 'config set' way, so update the only 'config-key set'
to make it consistent.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

run-make-check.sh: Make sure a build failure will exit

We 'set -e' but that is ignored because 'build tests' is executed in a
'&&' list (see 'man set') so move the echo to the following line.

Follow-up to 03ff2146f95

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38292 from badone/wip-dont-run-tests-if-build-fails

run-make-check.sh: Don't run tests if build fails

Reviewed-by: Kefu Chai <kchai@redhat.com>

run-make-check.sh: Don't run tests if build fails

When run-make was taken out we lost the 'set -e' call and therefore
continue after an error.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #38121 into master

* refs/pull/38121/head:
client: do not unset the client_debug_inject_tick_delay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38109 into master

* refs/pull/38109/head:
mds: dir->mark_new() should together with dir->mark_dirty()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38031 into master

* refs/pull/38031/head:
client: check rdonly file handle on truncate
test/libcephfs: test truncate on rdonly fd

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38104 into master

* refs/pull/38104/head:
client: dump which fs is used by client for multiple-fs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume inventory: make libstoragemgmt data retrieval optional

Default to not retrieving libstoragemgmt data since it seems this can
cause serious issues on older hardware. Safest way is to only retrieve
lsm data when the user opts in..

Fixes: https://tracker.ceph.com/issues/48270
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #38068 from dillaman/wip-librbd-migration-4

librbd: S3-based migration source

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38195 from anthonyeleven/anthonyeleven/docs-14730

doc/rados/troubleshooting: description of upgrade procedure - stopping/starting OSDs

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38276 from tchaikov/wip-labeler

github/labeler: add "rgw" label to rgw related changes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38091 from anthonyeleven/anthonyeleven/docs-46367

doc/dev/developer_guide: verb disagreement on "Testing: Unit Tests" page

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38143 from anthonyeleven/anthonyeleven/docs-46364

doc/dev/developer_guide: What Is Merged and When?" could be less colloquial

Reviewed-by: Zac Dover <zac.dover@gmail.com>

mgr: Fix various typos

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #38079 from anthonyeleven/anthonyeleven/docs-24143

doc/rados/configuration/common.rst:  enhance the running multiple clusters section

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38088 from anthonyeleven/anthonyeleven/docs-47627

doc/mgr: document dashboard configuration unsetting

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #38156 into master

* refs/pull/38156/head:
ceph-volume batch: reject partitions in argparser

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

github/labeler: label src/test/** with "tests"

Signed-off-by: Kefu Chai <kchai@redhat.com>

github/labeler: add "rgw" label to rgw related changes

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/prometheus: use get_module_option() wrapper

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/prometheus: catch ImportError

silences following warning from flake8

E722 do not use bare 'except'

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: have two lines before class definition

* have two lines before class definition, to silence E302 warning from flake8
* have single line before function definition, to silence E303 warning from flake8
* remove trailing space, to silence W291 warning from flake8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38274 from dillaman/wip-github-labeler-rbd

github/labeler: add rbd label to src/tools/rbd_[ggate/wnbd]

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

github/labeler: add rbd label to src/tools/rbd_[ggate/wnbd]

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38213 from petrutlucian94/rbd_exit

rbd: minor rbd-wnbd improvements

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'master' into anthonyeleven/docs-46367

Merge pull request #38254 from neha-ojha/wip-48299

doc/releases/nautilus.rst: add release notes for 14.2.15

Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38144 from anthonyeleven/anthonyeleven/docs-46365

doc/dev/developer_guide: prefer Latinate verbs to English phrasal verbs

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38260 from bk201/wip-48338

mgr/dashboard: fix tox py3 environment error

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/mgr: document dashboard configuration unsetting

Fixes: https://tracker.ceph.com/issues/47627
Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #38253 from cbodley/wip-rgw-gc-disable-defer

rgw: temporarily disable calls to defer_gc() in RGWGetObj

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

doc/releases/nautilus.rst: add release notes for 14.2.15

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38142 from theanalyst/doc/releases/v14.2.14

doc: release notes for octopus & nautilus backports

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #38247 from dillaman/wip-librbd-test-rename

test/librbd: rename tests to match proper namespaces

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc: releases: v15.2.6 Octopus release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
Co-authored-by: Ilya Dryomov <idryomov@gmail.com>

doc: releases: v14.2.14 Nautilus release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
Co-authored-by: Ilya Dryomov <idryomov@gmail.com>

blk/kernel: expose IORING_SETUP_{IOPOLL,SQPOLL} as options

Signed-off-by: JiangYu <lnsyyj@hotmail.com>

Merge pull request #38261 from tchaikov/wip-45919

pybind/mgr/dashboard/run-backend-api-tests: print more log from mgr

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #38217 from ideepika/wip-fix-lua-failure

cmake: require Lua 5.3 and up

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38204 from lnsyyj/wip-iouring

blk/kernel/io_uring: do not guard liburing backend with __x86_64__ anymore

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: fix "ceph pg dump_json/dump_pools_json" format error

When execute ceph pg dump_json, the expected result is json format,
but now is plain. Same to ceph pg dump_pools_json. So check args to
show right format result.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>
Co-authored-by: Kefu Chai <kefu@redhat.com>

Merge pull request #38256 from badone/wip-dont-build-io_uring.c-every-time

cmake: Stop 'configure' running every time

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/*: drop backward compatibility with boost<1.72

boost 1.72 is required to build Ceph since
713efb66e8d7b2585dfd6783f8f23ce409b50871. so there is no need to be
backward compatible with older Boost libraries.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: define BOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT for Boost.Asio users

see also
https://www.boost.org/doc/libs/1_74_0/doc/html/boost_asio/std_executors.html#boost_asio.std_executors.polymorphic_i_o_executor

we could use `asio::any_io_executor` later on though for better
performance.

also, define CMP0093, so FindBoost reports Boost_VERSION in x.y.z
format. it is simpler to use `VERSION_GREATER_EQUAL` to compare its
version with 1.74 instead of its C macro version ("107000").

Signed-off-by: Kefu Chai <kchai@redhat.com>

neorados: use lambda instead of std::mem_fn()

this addresses the FTBFS with GCC-10.2.0. FWIW, i cannot create a minimal
reproducer.

the error message was:

/usr/bin/g++-10 -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__
-Isrc/include -I../src -isystem /opt/ceph/include -isystem include -isystem\
 ../src/xxHash -isystem ../src/rapidjson/include -fno-omit-frame-pointer -g -fPIC -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security
-Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifie\
rs -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wstrict-null-sentinel -Woverloaded-virtual -fno-new-ttp-matching -DCEPH_DEBUG_MUTEX -fstack-protector-strong -D_GLIBCXX_ASSERTIONS
-fdiagnostics-color=auto -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-\
realloc -fno-builtin-free -std=c++17 -MD -MT src/neorados/CMakeFiles/neorados_api_obj.dir/RADOS.cc.o -MF src/neorados/CMakeFiles/neorados_api_obj.dir/RADOS.cc.o.d -o
src/neorados/CMakeFiles/neorados_api_obj.dir/RADOS.cc.o -c ../src/neorados/RADOS.cc
In file included from /usr/include/c++/10/bits/move.h:57,
                 from /usr/include/c++/10/bits/stl_pair.h:59,
                 from /usr/include/c++/10/utility:70,
                 from /usr/include/c++/10/optional:36,
                 from ../src/neorados/RADOS.cc:17:
/usr/include/c++/10/type_traits: In substitution of ‘template<class _Fp, class _Tp1, class ... _Args> static std::__result_of_success<decltype
((declval<_Tp1>)().*(declval<_Fp>)()((declval<_Args>)()...)), std::__invoke_memfun_ref> std::__result_of_memfun_ref_impl::_S_tes\
t(int) [with _Fp = long int (OSDMap::*)(std::basic_string_view<char>) const; _Tp1 = OSDMap&; _Args = {std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&}]’:
/usr/include/c++/10/type_traits:2377:56:   required from ‘struct std::__result_of_memfun_ref<long int (OSDMap::*)(std::basic_string_view<char>) const, OSDMap&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >&>’
/usr/include/c++/10/type_traits:2463:21:   required from ‘struct std::__result_of_memfun<long int (OSDMap::*)(std::basic_string_view<char>) const, OSDMap&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >&>’
/usr/include/c++/10/type_traits:2496:12:   required from ‘struct std::__result_of_impl<false, true, long int (OSDMap::* const&)(std::basic_string_view<char>) const, OSDMap&,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>’
/usr/include/c++/10/type_traits:2522:12:   required from ‘struct std::__invoke_result<long int (OSDMap::* const&)(std::basic_string_view<char>) const, OSDMap&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >&>’
/usr/include/c++/10/bits/invoke.h:89:5:   required by substitution of ‘template<class _Callable, class ... _Args> constexpr typename std::__invoke_result<_Functor, _ArgTypes>::type
std::__invoke(_Callable&&, _Args&& ...) [with _Callable = long int (OSDMap::* const&)(std:\
:basic_string_view<char>) const; _Args = {OSDMap&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&}]’
/usr/include/c++/10/functional:121:27:   required by substitution of ‘template<class ... _Args> decltype (std::__invoke(((const std::_Mem_fn_base<long int (OSDMap::*)(std::basic_string_view<char>)
const, true>*)this)->std::_Mem_fn_base<long int (OSDMap::*)(std::basic_str\
ing_view<char>) const, true>::_M_pmf, (forward<_Args>)(std::_Mem_fn_base<_MemFunPtr, __is_mem_fn>::operator()::__args)...)) std::_Mem_fn_base<long int (OSDMap::*)(std::basic_string_view<char>) const,
true>::operator()<_Args ...>(_Args&& ...) const [with _Args = {OSDMap&,\
 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&}]’
../src/osdc/Objecter.h:2600:38:   required from ‘decltype(auto) Objecter::with_osdmap(Callback&&, Args&& ...) [with Callback = std::_Mem_fn<long int (OSDMap::*)(std::basic_string_view<char>) const>;
Args = {std::__cxx11::basic_string<char, std::char_traits<char>, std::al\
locator<char> >&}]’
../src/neorados/RADOS.cc:903:9:   required from here
/usr/include/c++/10/type_traits:2366:50: internal compiler error: in build_over_call, at cp/call.c:8976
 2366 |       (std::declval<_Tp1>().*std::declval<_Fp>())(std::declval<_Args>()...)
      |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
0x5de01a build_over_call
    ../../src/gcc/cp/call.c:8976
0xaa30fe build_new_method_call_1
    ../../src/gcc/cp/call.c:10385
0xaa429e build_new_method_call(tree_node*, tree_node*, vec<tree_node*, va_gc, vl_embed>**, tree_node*, int, tree_node**, int)
    ../../src/gcc/cp/call.c:10460
0xaa429e build_special_member_call(tree_node*, tree_node*, vec<tree_node*, va_gc, vl_embed>**, tree_node*, int, int)
    ../../src/gcc/cp/call.c:9861
0xa94458 build_temp
    ../../src/gcc/cp/call.c:7128
0xa94458 convert_like_real_1
    ../../src/gcc/cp/call.c:7705
0xa95b3d perform_implicit_conversion_flags(tree_node*, tree_node*, int, int)
    ../../src/gcc/cp/call.c:11921
0xca79f5 convert_arguments
    ../../src/gcc/cp/typeck.c:4197
0xca79f5 cp_build_function_call_vec(tree_node*, vec<tree_node*, va_gc, vl_embed>**, int, tree_node*)
    ../../src/gcc/cp/typeck.c:4031
0xb4c4d0 build_offset_ref_call_from_tree(tree_node*, vec<tree_node*, va_gc, vl_embed>**, int)
    ../../src/gcc/cp/decl2.c:5276
0xc28b46 tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
    ../../src/gcc/cp/pt.c:20073
0xc2b204 tsubst(tree_node*, tree_node*, int, tree_node*)
    ../../src/gcc/cp/pt.c:15946
0xc2db88 tsubst_template_args(tree_node*, tree_node*, int, tree_node*)
    ../../src/gcc/cp/pt.c:13193
0xc3331b tsubst_aggr_type
    ../../src/gcc/cp/pt.c:13396
0xc35911 tsubst_function_type
    ../../src/gcc/cp/pt.c:14942
0xc2ae00 tsubst(tree_node*, tree_node*, int, tree_node*)
    ../../src/gcc/cp/pt.c:15753
0xc3988c tsubst_function_decl
    ../../src/gcc/cp/pt.c:13787
0xc2316f tsubst_decl
    ../../src/gcc/cp/pt.c:14230
0xc33a29 instantiate_template_1
    ../../src/gcc/cp/pt.c:20871
0xc4321b instantiate_template(tree_node*, tree_node*, int)
    ../../src/gcc/cp/pt.c:20928
Please submit a full bug report,

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard/run-backend-api-tests: print more log from mgr

in hope to more context when debugging https://tracker.ceph.com/issues/45919

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix tox py3 environment error

Add `ceph_argparse` module to the base environment.

Fixes: https://tracker.ceph.com/issues/48338
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

doc/dev: use http://docs.ceph.com/en/latest/ instead of /docs/master/ for docs

Several links under http://docs.ceph.com/docs/master/ were unable to access.
Change them to http://docs.ceph.com/en/lastest so we can access them directly.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

common: do not dup the options when reexpanding

The old code will store all the options, which has `$pid` in them,
in may_reexpand_meta map. And when reexpanding later, the reexpand
code will dup them with a higher priority(CONF_OVERRIDE).

This will be a problem, if the default value has `$pid` and be
stored in the may_reexpand_meta map, and then the code set a new
different value, which may have no `$pid`, from CLI or config file.
The reexpand will override it with the default value always.

This will do not duplicate the options with CONF_OVERRIDE priority
when reexpanding, just refresh them and call the observers. And the
finalize_reexpand_meta() will always be called after the fork() is
done in child processes.

Fixes: https://tracker.ceph.com/issues/48240
Signed-off-by: Xiubo Li <xiubli@redhat.com>

cmake: Stop 'configure' running every time

Currently the configure script is run on 'liburing_ext' not only for the
initial build but for subsequent incremental builds.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38243 from idryomov/wip-relax-preauth-asserts-again

msg/async/ProtocolV2: allow rxbuf/txbuf get bigger in testing, again

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38252 from idryomov/wip-resurrect-log-compression

qa/tasks/ceph: resurrect log compression

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph: resurrect log compression

Commit 953662555840 ("qa/tasks/ceph: use Cluster.sh() and Remote.sh()
when appropriate") dropped run.wait(), which waits for all given
processes to exit.  This resulted in errors like

  INFO:teuthology.orchestra.run.smithi107.stderr:tar: ./objectstore_tool..log: file changed as we read it
  INFO:teuthology.orchestra.run.smithi107.stderr:tar: ./ceph-client.admin.175125.log: File removed before we read it

as the task moved on to archiving semi-corrupted and uncompressed logs,
filling up the lab cluster.

Revert that hunk, as Cluster.sh() is useless here -- we don't need
stdout or stderr, but very much need parallel execution and wait for
the compression to finish.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rgw: temporarily disable calls to defer_gc() in RGWGetObj

cls_rgw_gc_queue_update_entry() is known to cause data loss when called
on objects that have not actually been scheduled for garbage collection

RGWGetObj is the only caller, and uses defer_gc() when reads are taking
a long time compared to rgw_gc_obj_min_wait. if an object has since been
deleted and submitted for garbage collection, this allows RGWGetObj to
defer that gc until the entire read completes

by disabling these calls to defer_gc(), very long reads (longer than 1hr,
with default configuration) may fail if the object gets deleted, and a
retry will result in a 404 Not Found error as expected

Fixes: https://tracker.ceph.com/issues/47866
Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/dev/developer_guide: prefer Latinate verbs to English phrasal verbs

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>
Fixes: https://tracker.ceph.com/issues/46365

Merge branch 'master' into anthonyeleven/docs-14730

test/librbd: rename tests to match proper namespaces

Some of the new crypto tests use a generic test name that
doesn't include the full namespace of the class under test.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37875 from orozery/librbd-luks-format

librbd: add LUKS support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38201 from lixiaoy1/minor_fix_parent_cache

librbd/plugin: delete field m_image_ctx in ParentCache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: add LUKS support

This commit introduces internal (not yet part of the api) librbd functions for:
1. formating an RBD image in LUKS format
2. parsing an RBD image in LUKS format

 The actual implementation of the LUKS format is done via libcryptsetup, which is added as a new dependency.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38089 from anthonyeleven/anthonyeleven/docs-47976

doc/mgr: Fix msgr2 TCP port in a couple of examples

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38180 from anthonyeleven/anthonyeleven/docs-23442

doc/rados/operations: Add description of new auth profiles

Reviewed-by: Zac Dover <zac.dover@gmail.com>

librbd/api/Migration: fix potential double-close of image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: implement a basic S3 stream source

The new S3 stream requires a fully qualified URL to a bucket
(i.e. no virtual host names) along with the access and secret
keys to the bucket.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: simplify common http result code handling

Move the processing of common HTTP result codes (not found, permission
denied, etc) higher up the stack so it can be re-used between all
request paths.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: implement a callback hook for processing http requests

The S3 authentication scheme requires adding additional headers to the
outbound request as well as computing a secure signature for the request
using the secret key.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: move read processing into HttpClient

This will allow the logic to be re-used between the standard
HTTP stream and the future S3 stream.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38229 from wjwithagen/wjw-fix-librbd-HttpStream.cc

librbd: Fix variable declaration in capture list in lambda

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/async/ProtocolV2: allow rxbuf/txbuf get bigger in testing, again

With CEPHX_V2 authorizer challenges brought back in commit
4a82c72e3bdd, these need to be bumped again, as two authorizers
(without and then with the challenge) are transmitted and signed
instead of one (without the challenge).  See commit 94953dd9398a
("msg/async/ProtocolV2: allow rxbuf/txbuf get bigger in testing")
for details.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #38207 into master

* refs/pull/38207/head:
ceph-volume: fix test_setup_device_device_name_is_none

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

blk: log is_valid_io() parameters when unsuccessful.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38092 from anthonyeleven/anthonyeleven/docs-mixed-device-trick

doc/rados/operations: enhance primary affinity section

Reviewed-by: Zac Dover <zac.dover@gmail.com>

cmake: require Lua 5.3 and up

fixes: https://tracker.ceph.com/issues/48312
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

common/config_proxy: clear the values.changed in _gather_changes()

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #38181 from anthonyeleven/anthonyeleven/docs-23443

doc: object -> file -> disk is wrong for bluestore

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #34506 from ktdreyer/doc-rm-sysv-init

doc: remove SysV init reference in manual deployment walkthrough

Reviewed-by: Zac Dover <zac.dover@gmail.com>

global: reexpand the conf meta in all the child processes

Especially for the tools or the daemons whose config options need
to expand the '$pid', they will be always expanded with the parent
processes. We need to reexpand them in child processes just after
the fork is done.

Fixes: https://tracker.ceph.com/issues/48240
Signed-off-by: Xiubo Li <xiubli@redhat.com>