qa/rgw: add rgw_obj and throttle tests to rgw verify suite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32508 into master

* refs/pull/32508/head:
doc: remove reference to obsolete scrub command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32535 into master

* refs/pull/32535/head:
qa: define centos version for fs:verify

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32517 from tchaikov/wip-mon-tell-command

mon: should not take non-tell commands as tell ones

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32539 from tchaikov/wip-debian-librgw

debian/radosgw.install: correct path to libradosgw.so*

Reviewed-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

Merge pull request #32331 from smithfarm/wip-spec-drop-vim-header

rpm: drop vim-specific header

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32461 from s0nea/wip-dashboard-fix-user-test

mgr/dashboard: fix failing user test

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32537 into master

* refs/pull/32537/head:
Switch spelling of utilization

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

debian/radosgw.install: correct path to libradosgw.so*

it was a regression introduced by f528f173e4

Signed-off-by: Kefu Chai <kchai@redhat.com>

Switch spelling of utilization

Prefer the non-British spelling of utilization since that's what the majority
of the code base seems to use.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

doc: remove reference to obsolete scrub command

Fixes: e9a5ce0897efc6126caeebea9900bf05ec3d2174
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30802 into master

* refs/pull/30802/head:
cephfs-shell: Modify Quota Test
cephfs-shell: Fix permission errors on quota tests
cephfs-shell: Add tests for quota
cephfs-shell: Print max_bytes and max_files as string on get Quota
cephfs-shell: Remove extra length argument passed to setxattr()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #32347 into master

* refs/pull/32347/head:
cephfs-shell: set shell.exit_code to 1 when reading conf fails
cephfs-shell: cephfs-shell breaks when conf_get() returns None
cephfs-shell: use libcephfs for exceptions
cephfs-shell: fix incorrect use of str.strip

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32459 into master

* refs/pull/32459/head:
qa/tasks: Nothing to clean up if the volume was not created
mgr/volumes: drop obsolete comment in _cmd_fs_volume_create
mgr/volumes: cleanup on fs create error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32318 into master

* refs/pull/32318/head:
mds: don't add metadata to session close message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31250 into master

* refs/pull/31250/head:
mds: Reorganize class members in MDSTableServer header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31481 into master

* refs/pull/31481/head:
mds: Reorganize struct members in Mutation header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32452 into master

* refs/pull/32452/head:
cephfs-journal-tool: fix the usage
cephfs-journal-tool: fix crash with empty argv

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32510 into master

* refs/pull/32510/head:
mon: use non-obsolete mon scrub cmd

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: define centos version for fs:verify

Otherwise it uses the teuthology default of 7.6.

Fixes: https://tracker.ceph.com/issues/43516
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32247 from dzafman/wip-43307

Remove use of rules batching for upmap balancer and default for upmap_max_deviation to 5

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32412 from tchaikov/wip-spec-more-deps-for-el8

ceph.spec.in: re-enable "make check" deps for el8

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #32483 from trociny/wip-pool_replayer-print_status

rbd-mirror: fix "pool replayer status" for case when init failed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32463 from trociny/wip-unlink-peer

librbd: when unlinking peer from mirror snaps do it in all namespaces

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32456 from trociny/wip-43428

rbd-mirror: make logrotate work

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32404 from kalebskeithley/master

rgw: build radosgw daemon as a shared lib + small executable

Merge pull request #32494 from ronen-fr/clang_3

crimson: fix lambda captures of non-variables

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix failing user tests

Choose another future_date_1 date (9 days instead of 10) to
make sure the expiration date will be between future_date_1
and future_date_2. Otherwise the expiration date might be
equal to future_date_1.
Also wait for 10 seconds before the password refresh in
test_pwd_expiration_date_update to prevent date equality.

Fixes: https://tracker.ceph.com/issues/43431
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #32499 from sebastian-philipp/orchestrator-cli-mon-update-count-only

mgr/orchestrator_cli: _update_mons require host spec only

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: fix lambda captures of non-variables

One cannot just capture a structured binding "non-variable".
(From the C++ standard, $8.4.5.2:
"If a lambda-expression [...] captures a structured binding (explicitly or implicitly),
the program is ill-formed.")

Co-authored-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #32376 from idryomov/wip-krbd-fsx-object-map

qa/suites/krbd: fsx with object-map and fast-diff

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32518 from tchaikov/wip-orchestrator-service-action-tests

qa/tasks/mgr/test_orchestrator_cli: fix service action tests

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/rook: Add some more type annotations

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: _update_mons require host spec only

* `mgr/cephadm` requres a host spec right now
* `mgr/rook` only supports `spec.count` right now

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32496 from majianpeng/osd-remove-unused-func2

osd/OSD: remove unused func enqueue_peering_evt_front

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32484 from bzed/bluefs-use-uint64_t-for-len

os/bluestore/Blue(FS|Store): uint64_t alloc_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32473 from sseshasa/fix-mon-autotune-update

mon/OSDMonitor: Don't update mon cache settings if rocksdb is not used

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28921 from rzarzynski/wip-msgr-drop-0c

msg/async: drop zero_copy_read() & co from ConnectedSocket.

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32112 from hjwsm1989/wip-42706

test/librados: don't release handler if set_pg_num failed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32428 from tchaikov/wip-qa-mgr-mon-update

qa/tasks/mgr/test_orchestrator_cli: fix "mon update" test

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31929 from majianpeng/msg-async-bugfix

msg/async: update refcount and perf counter properly

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix service action tests

in b77f0c74a5213ee57de19d21c930b2f3c872a1c4, "reload"
service[-instance] action was dropped. so replace "reload" with
"restart" in the related tests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: should not take non-tell commands as tell ones

this change addresses a regression introduced by a2c34794dc. in which, a
new flag, 'FLAG_TELL' was added. and it's used to check if a command is
"TELL" command. if it is, it's added to the tell/asok command registry
and monitor will handle the commands in this registry using asok hooks.

but there are some commands whose flag is "HIDDEN". and after
a2c34794dc, is_tell() takes HIDDEN commands as TELL command. that's why
`ceph_test_admin_socket_output --all` fails. because, "mds freeze" is
now wrongly considered as a TELL command. but monitor is not able to
handle is using the asok hooks.

after this change, is_tell() will not mistake "mds freeze" as a TELL
command anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: Sort pool list because the order isn't guaranteed from "balancer pool ls"

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: Change default upmap_max_deviation to 5

Fixes: https://tracker.ceph.com/issues/43312
Signed-off-by: David Zafman <dzafman@redhat.com>

osdmaptool: Add --upmap-active to simulate active upmap balancing

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Add upmap options to osdmaptool man page and give example

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: osdmaptool document non-upmap options that were missing

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix test case for pool based balancing instead of rule batched

Signed-off-by: David Zafman <dzafman@redhat.com>

Revert "tools: osdmaptool sync with balancer module behavior"

This was the rules batching version of osdmaptool

This reverts commit f165d4ca39b1edef4bf22a7597af79339c95026f.

Signed-off-by: David Zafman <dzafman@redhat.com>

Partially revert "mgr/balancer: balance pools with same crush_rule in batch"

This partially reverts commit 3a730d751deff892e7a0bddba87eba3dbb829c3e

Fixes: https://tracker.ceph.com/issues/43307
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #32358 from idryomov/wip-krbd-py3-fixup

qa: krbd_exclusive_option.sh: fixup for json.tool ordering change

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32498 into master

* refs/pull/32498/head:
doc: mounting CephFS subdirectory and Persistent Mounts cleanup

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32480 into master

* refs/pull/32480/head:
qa: use -D_GNU_SOURCE when compiling fsync-tester.c
qa: add comment about fsync-tester distribution

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29995 into master

* refs/pull/29995/head:
mds: add config to require forward to auth MDS

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31494 into master

* refs/pull/31494/head:
qa: force creation of fs with EC default data pool
qa: add tests for adding EC data pools
mon/MDSMonitor: warn when creating fs with default EC data pool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32015 into master

* refs/pull/32015/head:
doc/cephfs/standby: document mds_join_fs
mgr/cephadm: map mds daemons to a particular fs
mon/MDSMonitor: respect mfs fscid preference
mon/MDSMonitor: assign standbys to their preferred fscid
mds/FSMap: track preferred fscid for standby daemons
mds: add mds_join_fs option; pass via MMDSBeacon

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32268 into master

* refs/pull/32268/head:
mds: only enable batch getattr/lookup for non-snap request.
mds: pin dentry when attaching batch lookup to it

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32307 into master

* refs/pull/32307/head:
mon: print FSMap if standbys exist

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32308 into master

* refs/pull/32308/head:
mon/MDSMonitor: cleanup check_subs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32467 into master

* refs/pull/32467/head:
qa: fix various py3 cephfs qa bugs

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32164 into master

* refs/pull/32164/head:
mount.ceph: give a hint message when no mds is up or cluster is laggy

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #32363 into master

* refs/pull/32363/head:
qa: add .qa link

Reviewed-by: Sage Weil <sage@redhat.com>

mon: use non-obsolete mon scrub cmd

Fixes: e9a5ce0897efc6126caeebea9900bf05ec3d2174
Fixes: https://tracker.ceph.com/issues/43488
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32426 from tchaikov/wip-qa-test_envlibrados_for_rocksdb.sh

qa: enable CRB repo for RHEL8

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa: enable CRB repo for RHEL8

instead of enabling PowerTools repo, we need to enable CodeReady Builder
repo for RHEL8

also, since we are moving to RHEL8, there is no need to install cmake3
specifically for CentOS. CentOS 8 comes with cmake3.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32242 from rishabh-d-dave/cv-methods-for-lvm-opt-S

ceph-volume: add methods to pass filters to pvs, vgs and lvs commands

qa: krbd_exclusive_option.sh: fixup for json.tool ordering change

In Python 3.5 json.tool was changed to produce unsorted output and
--sort-keys option was added to compensate.  This wasn't caught by
4fe245cc2f2d ("qa: update krbd tests for python3") because it raced
with 50933b863a1d ("qa: krbd_exclusive_option.sh: update for recent
kernel changes").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-volume: add methods to pass filters to pvs, vgs and lvs commands

Filters can be passed to these commands by using option '-S'.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/suites/krbd: fsx with object-map and fast-diff

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd/fsx: don't disable features for krbd

rbd_default_features is set in krbd suite, no need to hard-code this in
fsx (especially now when object-map, fast-diff and deep-flatten are all
supported).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: mounting CephFS subdirectory and Persistent Mounts cleanup

Fixes: https://tracker.ceph.com/issues/37746
Signed-off-by: Jos Collin <jcollin@redhat.com>

osd/OSD: remove unused func enqueue_peering_evt_front

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #32485 from tchaikov/wip-clang

common,crimson: fixes to compile with clang and libc++

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #31728 from luis-henrix/copy-from-notrunc

osd: add a 'copy-from2' operation that includes truncate_{seq,size} parameters

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #32488 from ronen-fr/clang_3

crimson: CLANG-related fixes to errorator.h

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32490 from rzarzynski/wip-crimson-missing_err_include

crimson: add missing include in common/errorator.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: add missing include in common/errorator.h

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: fix non-standard C++ constructs in errorator.h

Fixing use patterns accepted by gcc, but frowned upon by Clang:

Put noexcept specifier after attributes specifiers (as
per the C++ standard $11.3.5)

Limit the use of a not-completely-defined class names to
within function bodies, per the standard ("A class is considered
a completely-defined object type (or complete type) at the
closing } of the class-specifier.
Within the class member-specification, the class is regarded as
complete within function bodies, default arguments, noexcept-specifiers,
and default member initializers (including such things in nested classes).
Otherwise it is regarded as incomplete within its own class member-specification.")

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson: fix FTBFS on Clang induced by errorator.

Reported-by Ronen Friedman <rfriedma@redhat.com>
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: add missing #include

clang++-9.0.1 and libc++ require them to build

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not use auto in param list of function prototype

src/crimson/net/Socket.cc:114:57: error: 'auto' not allowed in function
prototype
static inline seastar::future<> close_and_handle_errors(auto& out) {
                                                        ^~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not capture unused variables

to silence warnings like

src/crimson/net/Socket.cc:93:41: warning: lambda capture 'this' is not
used [-Wunused-lambda-capture]
    return in.read_exactly(bytes).then([this](auto buf) {
                                        ^~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore/Blue(FS|Store): uint64_t alloc_size

Compiling on Debian's 32bit architectures failed with

<<PKGBUILDDIR>>/src/os/bluestore/BlueStore.cc:5539:37: error: no matching function for call to ‘p2align(size_t&, uint64_t&)’
 5539 |     total += p2align(len, alloc_size);
      |                                     ^
In file included from /<<PKGBUILDDIR>>/src/include/denc.h:42,
                 from /<<PKGBUILDDIR>>/src/include/encoding.h:40,
                 from /<<PKGBUILDDIR>>/src/include/compact_map.h:16,
                 from /<<PKGBUILDDIR>>/src/include/mempool.h:32,
                 from /<<PKGBUILDDIR>>/src/os/bluestore/BlueStore.h:36,
                 from /<<PKGBUILDDIR>>/src/os/bluestore/BlueStore.cc:25:
/<<PKGBUILDDIR>>/src/include/intarith.h:57:20: note: candidate: ‘template<class T> constexpr T p2align(T, T)’
   57 | constexpr inline T p2align(T x, T align) {
      |                    ^~~~~~~
/<<PKGBUILDDIR>>/src/include/intarith.h:57:20: note:   template argument deduction/substitution failed:
/<<PKGBUILDDIR>>/src/os/bluestore/BlueStore.cc:5539:37: note:   deduced conflicting types for parameter ‘T’ (‘unsigned int’ and ‘long long unsigned int’)
 5539 |     total += p2align(len, alloc_size);

As far as I understand it the available_freespace should at least be
able to return the same number as alloc_size, so we should use uint64_t
instead of size_t here, similar to
10a953afc8f803e50c96354470fb114b33e62599

Fixes: https://tracker.ceph.com/issues/43451
Signed-off-by: Bernd Zeimetz <bernd@bzed.de>

logrotate: also sighup rbd-mirror

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: reopen all contexts logs on SIGHUP

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: fix "pool replayer status" for case when init failed

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: delay local/remote rados initialization until context created

We rely on that if a rados ref is initialized it contains the
valid context.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32474 from kalebskeithley/fedora32

do_cmake.sh: fedora-32 (rawhide) build with python-3.8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32481 from tchaikov/wip=admin-build-doc-python3

admin/build-doc: always install python3-* for build deps

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

admin/build-doc: always install python3-* for build deps

since we've dropped the support of python36-*, i.e. python packages
provided by EPEL7 before RHEL7/CentOS7 included python3. as before the
inclusion of python3 as supported python3, python packages are named
python36-*. and they don't provide python3-*. so we had to install
python36-* explicitly. now that we are able to use the python3-*
packages, we can just install python3-*.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: fix various py3 cephfs qa bugs

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use -D_GNU_SOURCE when compiling fsync-tester.c

Otherwise loff_t is unavailable for some distributions.

Fixes: https://tracker.ceph.com/issues/43460
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add comment about fsync-tester distribution

I was initially confused why this would not be included in the QA suite.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32451 from rs-fabrica/doc_rados_operations_crush-map-edits

doc/rados/operations/crush-map-edits: recompile and set instructions

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32442 into master

* refs/pull/32442/head:
doc/developer: fix dev mailing list address

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/developer: fix dev mailing list address

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mds: Reorganize struct members in Mutation header

Fixes: https://tracker.ceph.com/issues/42690
Signed-off-by: Varsha Rao <varao@redhat.com>

do_cmake.sh: fedora-32 (rawhide) build with python-3.8

Fedora-32 now has python-3.8.

Fedora-29 is EOL. All non-EOL versions of Fedora have python-3.7, except
Fedora-32

(Unclear to me why PYBUILD="3" isn't used, like Ubuntu and FreeBSD. That
would eliminate breakage when pyhon versions change in Rawhide.)

Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

qa/tasks: Nothing to clean up if the volume was not created

There are only 2 cases which needs cleanup:
1. The volume is successfully created
2. The volume is successfully created but create_mds fails

In either case, we could do a 'volume rm'.

Signed-off-by: Jos Collin <jcollin@redhat.com>