Merge pull request #32247 from dzafman/wip-43307

Remove use of rules batching for upmap balancer and default for upmap_max_deviation to 5

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32412 from tchaikov/wip-spec-more-deps-for-el8

ceph.spec.in: re-enable "make check" deps for el8

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #32483 from trociny/wip-pool_replayer-print_status

rbd-mirror: fix "pool replayer status" for case when init failed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32463 from trociny/wip-unlink-peer

librbd: when unlinking peer from mirror snaps do it in all namespaces

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32456 from trociny/wip-43428

rbd-mirror: make logrotate work

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32404 from kalebskeithley/master

rgw: build radosgw daemon as a shared lib + small executable

Merge pull request #32494 from ronen-fr/clang_3

crimson: fix lambda captures of non-variables

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32499 from sebastian-philipp/orchestrator-cli-mon-update-count-only

mgr/orchestrator_cli: _update_mons require host spec only

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: fix lambda captures of non-variables

One cannot just capture a structured binding "non-variable".
(From the C++ standard, $8.4.5.2:
"If a lambda-expression [...] captures a structured binding (explicitly or implicitly),
the program is ill-formed.")

Co-authored-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #32376 from idryomov/wip-krbd-fsx-object-map

qa/suites/krbd: fsx with object-map and fast-diff

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32518 from tchaikov/wip-orchestrator-service-action-tests

qa/tasks/mgr/test_orchestrator_cli: fix service action tests

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/rook: Add some more type annotations

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: _update_mons require host spec only

* `mgr/cephadm` requres a host spec right now
* `mgr/rook` only supports `spec.count` right now

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32496 from majianpeng/osd-remove-unused-func2

osd/OSD: remove unused func enqueue_peering_evt_front

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32484 from bzed/bluefs-use-uint64_t-for-len

os/bluestore/Blue(FS|Store): uint64_t alloc_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32473 from sseshasa/fix-mon-autotune-update

mon/OSDMonitor: Don't update mon cache settings if rocksdb is not used

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28921 from rzarzynski/wip-msgr-drop-0c

msg/async: drop zero_copy_read() & co from ConnectedSocket.

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32112 from hjwsm1989/wip-42706

test/librados: don't release handler if set_pg_num failed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32428 from tchaikov/wip-qa-mgr-mon-update

qa/tasks/mgr/test_orchestrator_cli: fix "mon update" test

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31929 from majianpeng/msg-async-bugfix

msg/async: update refcount and perf counter properly

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix service action tests

in b77f0c74a5213ee57de19d21c930b2f3c872a1c4, "reload"
service[-instance] action was dropped. so replace "reload" with
"restart" in the related tests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: Sort pool list because the order isn't guaranteed from "balancer pool ls"

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: Change default upmap_max_deviation to 5

Fixes: https://tracker.ceph.com/issues/43312
Signed-off-by: David Zafman <dzafman@redhat.com>

osdmaptool: Add --upmap-active to simulate active upmap balancing

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Add upmap options to osdmaptool man page and give example

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: osdmaptool document non-upmap options that were missing

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix test case for pool based balancing instead of rule batched

Signed-off-by: David Zafman <dzafman@redhat.com>

Revert "tools: osdmaptool sync with balancer module behavior"

This was the rules batching version of osdmaptool

This reverts commit f165d4ca39b1edef4bf22a7597af79339c95026f.

Signed-off-by: David Zafman <dzafman@redhat.com>

Partially revert "mgr/balancer: balance pools with same crush_rule in batch"

This partially reverts commit 3a730d751deff892e7a0bddba87eba3dbb829c3e

Fixes: https://tracker.ceph.com/issues/43307
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #32358 from idryomov/wip-krbd-py3-fixup

qa: krbd_exclusive_option.sh: fixup for json.tool ordering change

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32498 into master

* refs/pull/32498/head:
doc: mounting CephFS subdirectory and Persistent Mounts cleanup

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32480 into master

* refs/pull/32480/head:
qa: use -D_GNU_SOURCE when compiling fsync-tester.c
qa: add comment about fsync-tester distribution

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29995 into master

* refs/pull/29995/head:
mds: add config to require forward to auth MDS

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31494 into master

* refs/pull/31494/head:
qa: force creation of fs with EC default data pool
qa: add tests for adding EC data pools
mon/MDSMonitor: warn when creating fs with default EC data pool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32015 into master

* refs/pull/32015/head:
doc/cephfs/standby: document mds_join_fs
mgr/cephadm: map mds daemons to a particular fs
mon/MDSMonitor: respect mfs fscid preference
mon/MDSMonitor: assign standbys to their preferred fscid
mds/FSMap: track preferred fscid for standby daemons
mds: add mds_join_fs option; pass via MMDSBeacon

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32268 into master

* refs/pull/32268/head:
mds: only enable batch getattr/lookup for non-snap request.
mds: pin dentry when attaching batch lookup to it

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32307 into master

* refs/pull/32307/head:
mon: print FSMap if standbys exist

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32308 into master

* refs/pull/32308/head:
mon/MDSMonitor: cleanup check_subs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32467 into master

* refs/pull/32467/head:
qa: fix various py3 cephfs qa bugs

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32164 into master

* refs/pull/32164/head:
mount.ceph: give a hint message when no mds is up or cluster is laggy

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #32363 into master

* refs/pull/32363/head:
qa: add .qa link

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32426 from tchaikov/wip-qa-test_envlibrados_for_rocksdb.sh

qa: enable CRB repo for RHEL8

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa: enable CRB repo for RHEL8

instead of enabling PowerTools repo, we need to enable CodeReady Builder
repo for RHEL8

also, since we are moving to RHEL8, there is no need to install cmake3
specifically for CentOS. CentOS 8 comes with cmake3.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32242 from rishabh-d-dave/cv-methods-for-lvm-opt-S

ceph-volume: add methods to pass filters to pvs, vgs and lvs commands

qa: krbd_exclusive_option.sh: fixup for json.tool ordering change

In Python 3.5 json.tool was changed to produce unsorted output and
--sort-keys option was added to compensate.  This wasn't caught by
4fe245cc2f2d ("qa: update krbd tests for python3") because it raced
with 50933b863a1d ("qa: krbd_exclusive_option.sh: update for recent
kernel changes").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-volume: add methods to pass filters to pvs, vgs and lvs commands

Filters can be passed to these commands by using option '-S'.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/suites/krbd: fsx with object-map and fast-diff

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd/fsx: don't disable features for krbd

rbd_default_features is set in krbd suite, no need to hard-code this in
fsx (especially now when object-map, fast-diff and deep-flatten are all
supported).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: mounting CephFS subdirectory and Persistent Mounts cleanup

Fixes: https://tracker.ceph.com/issues/37746
Signed-off-by: Jos Collin <jcollin@redhat.com>

osd/OSD: remove unused func enqueue_peering_evt_front

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #32485 from tchaikov/wip-clang

common,crimson: fixes to compile with clang and libc++

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #31728 from luis-henrix/copy-from-notrunc

osd: add a 'copy-from2' operation that includes truncate_{seq,size} parameters

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #32488 from ronen-fr/clang_3

crimson: CLANG-related fixes to errorator.h

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32490 from rzarzynski/wip-crimson-missing_err_include

crimson: add missing include in common/errorator.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: add missing include in common/errorator.h

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: fix non-standard C++ constructs in errorator.h

Fixing use patterns accepted by gcc, but frowned upon by Clang:

Put noexcept specifier after attributes specifiers (as
per the C++ standard $11.3.5)

Limit the use of a not-completely-defined class names to
within function bodies, per the standard ("A class is considered
a completely-defined object type (or complete type) at the
closing } of the class-specifier.
Within the class member-specification, the class is regarded as
complete within function bodies, default arguments, noexcept-specifiers,
and default member initializers (including such things in nested classes).
Otherwise it is regarded as incomplete within its own class member-specification.")

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson: fix FTBFS on Clang induced by errorator.

Reported-by Ronen Friedman <rfriedma@redhat.com>
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: add missing #include

clang++-9.0.1 and libc++ require them to build

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not use auto in param list of function prototype

src/crimson/net/Socket.cc:114:57: error: 'auto' not allowed in function
prototype
static inline seastar::future<> close_and_handle_errors(auto& out) {
                                                        ^~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not capture unused variables

to silence warnings like

src/crimson/net/Socket.cc:93:41: warning: lambda capture 'this' is not
used [-Wunused-lambda-capture]
    return in.read_exactly(bytes).then([this](auto buf) {
                                        ^~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore/Blue(FS|Store): uint64_t alloc_size

Compiling on Debian's 32bit architectures failed with

<<PKGBUILDDIR>>/src/os/bluestore/BlueStore.cc:5539:37: error: no matching function for call to ‘p2align(size_t&, uint64_t&)’
 5539 |     total += p2align(len, alloc_size);
      |                                     ^
In file included from /<<PKGBUILDDIR>>/src/include/denc.h:42,
                 from /<<PKGBUILDDIR>>/src/include/encoding.h:40,
                 from /<<PKGBUILDDIR>>/src/include/compact_map.h:16,
                 from /<<PKGBUILDDIR>>/src/include/mempool.h:32,
                 from /<<PKGBUILDDIR>>/src/os/bluestore/BlueStore.h:36,
                 from /<<PKGBUILDDIR>>/src/os/bluestore/BlueStore.cc:25:
/<<PKGBUILDDIR>>/src/include/intarith.h:57:20: note: candidate: ‘template<class T> constexpr T p2align(T, T)’
   57 | constexpr inline T p2align(T x, T align) {
      |                    ^~~~~~~
/<<PKGBUILDDIR>>/src/include/intarith.h:57:20: note:   template argument deduction/substitution failed:
/<<PKGBUILDDIR>>/src/os/bluestore/BlueStore.cc:5539:37: note:   deduced conflicting types for parameter ‘T’ (‘unsigned int’ and ‘long long unsigned int’)
 5539 |     total += p2align(len, alloc_size);

As far as I understand it the available_freespace should at least be
able to return the same number as alloc_size, so we should use uint64_t
instead of size_t here, similar to
10a953afc8f803e50c96354470fb114b33e62599

Fixes: https://tracker.ceph.com/issues/43451
Signed-off-by: Bernd Zeimetz <bernd@bzed.de>

logrotate: also sighup rbd-mirror

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: reopen all contexts logs on SIGHUP

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: fix "pool replayer status" for case when init failed

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: delay local/remote rados initialization until context created

We rely on that if a rados ref is initialized it contains the
valid context.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32474 from kalebskeithley/fedora32

do_cmake.sh: fedora-32 (rawhide) build with python-3.8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32481 from tchaikov/wip=admin-build-doc-python3

admin/build-doc: always install python3-* for build deps

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

admin/build-doc: always install python3-* for build deps

since we've dropped the support of python36-*, i.e. python packages
provided by EPEL7 before RHEL7/CentOS7 included python3. as before the
inclusion of python3 as supported python3, python packages are named
python36-*. and they don't provide python3-*. so we had to install
python36-* explicitly. now that we are able to use the python3-*
packages, we can just install python3-*.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: fix various py3 cephfs qa bugs

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use -D_GNU_SOURCE when compiling fsync-tester.c

Otherwise loff_t is unavailable for some distributions.

Fixes: https://tracker.ceph.com/issues/43460
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add comment about fsync-tester distribution

I was initially confused why this would not be included in the QA suite.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32451 from rs-fabrica/doc_rados_operations_crush-map-edits

doc/rados/operations/crush-map-edits: recompile and set instructions

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32442 into master

* refs/pull/32442/head:
doc/developer: fix dev mailing list address

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/developer: fix dev mailing list address

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

do_cmake.sh: fedora-32 (rawhide) build with python-3.8

Fedora-32 now has python-3.8.

Fedora-29 is EOL. All non-EOL versions of Fedora have python-3.7, except
Fedora-32

(Unclear to me why PYBUILD="3" isn't used, like Ubuntu and FreeBSD. That
would eliminate breakage when pyhon versions change in Rawhide.)

Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

mon/OSDMonitor: Don't update mon cache settings if rocksdb is not used

Currently any update to mon cache config settings like 'mon_memory_target'
or 'rocksdb_cache_size' will cause a crash in case rocksdb and priority
cache manager is not being used.

If rocksdb (and therefore priority cache manager) is not being used, then
the update to the mon caches is ignored. Also, reduce the log levels of
some important log messages to help in debugging.

Fixes: https://tracker.ceph.com/issues/43454
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge PR #32457 into master

* refs/pull/32457/head:
doc: fix Getting Started with CephFS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31309 from matthewoliver/rgw_swift_auth_note

doc: Add a RGW swift auth note

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31432 from zhangsw/rgw-cleanup-cls-reshard-const

rgw:cleanup the "magic string" usage in cls_rgw_client.cc

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #31429 from zhangsw/rgw-cleanup-cls-guard-bucket-resharding

rgw: correct some error log about reshard in cls_rgw.cc.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31404 from zhangsw/rgw-cleanup-deletelc

rgw:cleanup:remove un-used class member in RGWDeleteLC

Merge pull request #31345 from zhangsw/rgw-cleanup-rgw-admin-check-instance

rgw:cleanup:remove un-used create_new_bucket_instance in rgw_admin.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: when unlinking peer from mirror snaps do it in all namespaces

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32351 from mykaul/osd_fsid

ceph-volume/lvm/activate.py: clarify error message: fsid refers to osd_fsid

Merge pull request #27792 from smithfarm/wip-make-dist

make-dist: drop Python 2/3 autoselect

Reviewed-by: Kefu Chai <kchai@redhat.com>

lvm/activate.py: clarify error message: fsid refers to osd_fsid

Users complained[1] the error message isn't clear, and they thought
it referred to the cluster fsid instead of the osd_fsid.
Made it clearer.

[1] rook/rook#4547

Fixes: https://tracker.ceph.com/issues/43442
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

doc: fix Getting Started with CephFS

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: Add a RGW swift auth note

Swift accounts are not like normal accounts, they are more akin to a
bank account that multile people could share. Or in the case of a cloud
it is usually mapped to the tenant.

Radosgw deals with this with a user and subuser, which is great, but a
little confusing. So this patch adds a note to those used to the Swift
API to make it more clear.

Signed-off-by: Matthew Oliver <moliver@suse.com>

msg/async: don't forget dec(Worker::references) when met error.

This will cause unbalance between workes.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

msg/async: dec(l_msgr_active_connections) when unregister anonymous connection

When do all in one test,  l_msgr_active_connection don't decrease when
client shutdown.
This bug introduced by 5b41e12fdb445.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

doc/rados/operations/crush-map-edits: including recompile and set crushmapo instructions

Including missing recompile and set crushmap sections in crush-map-edits

Signed-off-by: Rodrigo Severo <rodrigo@fabricadeideias.com>

Merge pull request #32433 from tchaikov/wip-43414

rgw: svc.bucket: assign to optional<> using "="

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #32415 into master

* refs/pull/32415/head:
include/cpp-btree: use inline variable to define btree_is_key_compare_to
include/cpp-btree: use std::conditional_t<>
include/cpp-btree: use static_assert()
include/interval_set: use interval_set when appropriate
include/interval_set: rename some types
include/cpp-btree: add emplace_hint()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32399 from dillaman/wip-41556-2

rbd-mirror: continue to isolate journal replay logic

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32427 from tchaikov/qa-devices-schema-with-path

qa: add "path" to "device" output schema

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32430 from tchaikov/wip-rgw-rgw_log_entry

rgw: initialize member variables of rgw_log_entry

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32438 from tchaikov/wip-vstart-runner-py3

qa/tasks/vstart_runner: write string to StringIO

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/vstart_runner: write string to StringIO

we use `six.StringIO` for stdout when writing output messages from
`self.subproc.communicate()`, and `six.StringIO` accepts strings, so we
need to decode the output before sending them to `six.StringIO`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

make-dist: drop Python 2/3 autoselect

virtualenv will prefer Python 3 over Python 2 if both are available.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: svc.bucket: assign to optional<> using "="

instead of using

*optional_val = value;

we should use:

optional_val = value;

otherwise we will have following runtime assertion failure like

/usr/include/c++/8/optional:945: constexpr _Tp& std::_Optional_base<_Tp, true, true>::_M_get() [with _Tp = RGWBucketInfo*]: Assertion 'this->_M_is_engaged()' failed.

when compiled with the `_GLIBCXX_ASSERTIONS` macro defined. and recent
`redhat-rpm-config` adds `-D_GLIBCXX_ASSERTIONS` to
`__global_compiler_flags`, which is in turn included by `optflags`. and
`optflags` is pratically used everywhere.

Fixes: https://tracker.ceph.com/issues/43414
Signed-off-by: Kefu Chai <kchai@redhat.com>