Merge PR #42137 into master

* refs/pull/42137/head:
msg/async/net_handler: do not define variable unless it is used
mds/cephfs_features: print size_t using "%zu"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42155 from yuvalif/wip-yuval-fix-51347

rgw/notifications: cache object size to avoid accessing invalid memoery

Merge PR #41966 into master

* refs/pull/41966/head:
mgr/volumes: use dedicated libcephfs handles for subvolume calls and async jobs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42174 from cyx1231st/wip-seastore-fix-transactional-retire

crimson/os/seastore: leverage RetiredExtentPlaceholder to detect transaction conflicts

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: cleanup query_cache_with_placeholders()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction: remove unused retired_uncached

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: leverage RetiredExtentPlaceholder to detect transaction conflicts

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: remove unused template in RetiredExtentPlaceholder

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup, consolidate extent lookup through query_cache_for_extent()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42187 from tchaikov/wip-crimson-cleanup

crimson/os/seastore: use discard_result() to discard futurized value

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42186 from lorenzbausch/fix_typo

doc/dev/crimson/osd: fix typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common/errorator: forward declare futurize as a struct

as it is defined as struct, otherwise clang complains on this
inconsistency.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: use discard_result() to discard futurized value

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: capture variables explicitly

less error prune this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: add discard_result() to errorator

so we can discard result explicitly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson/osd: fix typo

Signed-off-by: Lorenz Bausch <info@lorenzbausch.de>

Merge pull request #42184 from xxhdx1985126/wip-seastore-lambda

crimson/os/seastore: fix lambda capture issue in Journal::scan_extents()

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: fix lambda capture issue in Journal::scan_extents()

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #42178 from xxhdx1985126/wip-collection-manager-test-case

test/crimson/seastore: fix a bug in the collection manager test case

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson/seastore: fix bug in the collection manager test case

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #42171 from sharuzzaman/patch-1

doc/rados/configuration: fixing small typo for word "implemebnt"

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration: fixing small typo for word "implemebnt"

Signed-off-by: Sharuzzaman Ahmat Raslan <sharuzzaman@gmail.com>

Merge pull request #42147 from xxhdx1985126/wip-interruptible-errorator

crimson/common: allow interruptible parallel_for_each to handle error…

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: allow interruptible parallel_for_each to handle errorated future

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge PR #41938 into master

* refs/pull/41938/head:
mgr/cephadm: include addr in HOST_CHECK_FAILED alert detail

Reviewed-by: Adam King <adking@redhat.com>

rgw/notifications: cache object size to avoid accessing invalid memoery

Fixes: https://tracker.ceph.com/issues/51347
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge PR #42081 into master

* refs/pull/42081/head:
qa: use kclient xattr to lookup client id
qa: refactor reading debug file code
qa: get mount id before failing fs

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #42075 into master

* refs/pull/42075/head:
qa: avoid using sudo for regular test artifacts
qa: convert mount calls to mount_wait
qa: use run_shell_payload to avoid sudo

Reviewed-by: Rishabh Dave <ridave@redhat.com>

qa: use kclient xattr to lookup client id

Added relatively recently but in RHEL 8.4.

Fixes: https://tracker.ceph.com/issues/51183
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41926 from TRYTOBE8TME/wip-rgw-dpp-2

src/rgw: DPP addition 3

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #42105 from dvanders/dvanders_pgp_num_actual

mgr/DaemonServer: skip redundant update of pgp_num_actual

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41829 from SMIL-Infra/clean-cg

cephadm: workaround unit replace failure

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #41928 from sebastian-philipp/rm-qa-qa

qa: remove `qa/.qa` symlink

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #41943 from liewegas/qa-8-stream

qa/distro/supported: add centos 8.stream

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42146 from tchaikov/wip-crimson-cleanup

crimson/os/seastore: do not capture unused variable

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #42032 from sebastian-philipp/cephadm_shared_folder_bin_cephadm

cephadm: shared folder: Mount the cephadm

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #41906 from mgfritch/cephadm-test-monaddrv

cephadm: add `bootstrap --mon-addrv` test

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42017 from Daniel-Pivonka/cephadm-orch-host-drain

mgr/cephadm: add ceph orch host drain and limit host removal to empty hosts

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #41883 from rhcs-dashboard/run-dashboard-cephadm-e2e-tests

mgr/dashboard: run cephadm-backend e2e tests with KCLI

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42144 from cyx1231st/wip-fix-seastore-cache2

crimson/os/seastore/cache: fix retiring mutation-pending extents

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

mgr/dashboard: run cephadm-backend e2e tests with KCLI

Fixes: https://tracker.ceph.com/issues/51300
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

crimson/os/seastore/cache: fix retiring mutation-pending extents

Mark the retiring mutation-pending extent as INVALID, and add it's
prior-instance to retired-set in order to populate transaction
invalidation correctly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: use map::emplace(key, val) instead of map::emplace(pair<>)

simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: do not capture unused variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: read_set_t<> should be an alias of set<>

we don't extend it by inheriting from this class. what we need is to
have a convenient template alias here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42143 from athanatos/sjust/wip-cm-omap-int-future

crimson/os/seastore: convert omap and collection_manager to interruptible future

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: DPP addition 3

Following files have been covered in this PR:

1. rgw_period_puller.cc
2. rgw_reshard.cc
3. rgw_sal_rados.cc
4. services/svc_sync_modules.cc
5. services/svc_bucket_sobj.cc
6. services/svc_notify.cc
7. rgw_acl_swift.cc
8. rgw_acl.cc
9. rgw_sync_module_aws.cc
10. rgw_tools.cc
11. services/svc_sys_obj_cache.cc
12. rgw_bucket_sync.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

Merge pull request #42138 from rzarzynski/wip-crimson-alienstore-indent

crimson/os: fix indentation in alien_store.cc.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42128 from tchaikov/wip-crimson-asock-cleanup

crimson/asock: cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/seastore/omap_manager: convert to use interruptible future

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/collection_manager: convert to use interruptible_future

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: clean up transaction_manager based tests

Permits using either vanilla TransactionManager and
InterruptedTransactionManager, updates users to use submit_transaction
helpers.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #41980 from yuvalif/wip-yuval-fix-51320

rgw/notification: support version-id for all event types

Merge pull request #42127 from ivancich/wip-little-rgw-cleanup

rgw: a small set of clean-ups

Reviewed-by: Ali Maredia <amaredia@redhat.com>

qa: avoid using sudo for regular test artifacts

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: convert mount calls to mount_wait

These tests want to immediately use the mount anyway. But the main
problem is, without waiting for the mount to complete, the command:

    chmod 1777 /path/to/mount

is not run so the mount cannot be written to by normal users without
sudo.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use run_shell_payload to avoid sudo

"run_shell" adds 'sudo' which runs afoul of new security protections on
Ubuntu 20.04.

Fixes: https://tracker.ceph.com/issues/51417
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42012 from yuvalif/wip-yuval-fix-50039

rgw/notifications: support v4 auth for topics and notifications

rgw/notification: support version-id for all event types

including: object copy, multipart upload, delete marker on versioned bucket

Fixes: https://tracker.ceph.com/issues/51320
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #42134 from zdover23/wip-doc-cephadm-operations-2021-07-01

doc/cephadm: improve "Ceph Daemon Logs" (1 of x)

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42023 from TRYTOBE8TME/wip-rgw-notification-tests

rgw: Modification in the usage of force-branch

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #41945 from yuvalif/wip-yuval-fix-51261

rgw/notifications: support metadata filter in CompleteMultipartUpload and Copy events

Merge pull request #41953 from SMIL-Infra/fix-dashboard-osd-count

mgr/dashboard: fix OSD out count

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42116 from votdev/issue_51443_migrate_userdb

mgr/dashboard: User database migration has been cut out

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crimson/os: fix indentation in alien_store.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async/net_handler: do not define variable unless it is used

on WIN32, `flags` is not used at all, hence GCC complains like

../src/msg/async/net_handler.cc: In member function 'int ceph::NetHandler::set_nonblock(int)':
../src/msg/async/net_handler.cc:67:7: warning: unused variable 'flags' [-Wunused-variable]
   67 |   int flags;
      |       ^~~~~

this change silences the warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds/cephfs_features: print size_t using "%zu"

to silence the warning from GCC like:

../src/mds/cephfs_features.cc: In function 'void cephfs_dump_features(ceph::Formatter*, const feature_bitset_t&)':
../src/mds/cephfs_features.cc:71:39: warning: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'size_t' {aka 'long long unsigned int'} [-Wformat=]
   71 |     snprintf(s, sizeof(s), "feature_%lu", i);
      |                                     ~~^   ~
      |                                       |   |
      |                                       |   size_t {aka long long unsigned int}
      |                                       long unsigned int
      |                                     %llu

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #42107 into master

* refs/pull/42107/head:
mgr/ActivePyModules: streamline mgr_ips a bit
mgr/ActivePyModules: don't take unnecessary lock
pybind/mgr/mgr_module: use get("mgr_ips") for standby get_mgr_ip()
mgr: add get() for standby modules

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

doc/cephadm: improve "Ceph Daemon Logs" (1 of x)

This PR turned out to be a 3-in-1:

(1) improves syntax and formatting of "Logging to stdout"
(2) improves syntax and formatting of "Logging to files"
(3) replaces all carets with tildes in 3rd-level section
    headers in operations.rst (./build-doc was crying
    about inconsistency when I fed it tildes, but tildes
    and not carets are the RST standard according to
    https://docutils.sourceforge.io/ \
    docs/user/rst/quickstart.html#sections
    so the carets had to go.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42130 from tchaikov/wip-doc-install

doc/install/get-packages: s/el7/el8/ and cleanups

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42108 from zdover23/wip-doc-upgrading-ceph-cancelling-the-upgrade-2021-06-30

doc/cephadm: improve "Canceling an Upgrade"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/install/get-packages: do not mention additional 3rd party packages

since we are able to build and run on ubuntu focal without extra
dependencies of 3rd party packages, there is no need to mention them.
actually all of the removed packages are available in ubuntu focal.
some of them have different package names though.

the same applies to el8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/install/get-packages: s/el7/el8/

since we don't build for el7 anymore. replace all occurrences of el7 with
el8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: Modification in the usage of force-branch

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

crimson/admin: let AdminSocket::register_command() return void

instead of returning a seastar::future<>, it can just return void,
as it does not involve asynchronous operations.

also drop servers_tbl_rwlock, as we don't register hooks after OSD
starts, there is no need to protect hooks with a rw lock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: remove unused variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42080 from cyx1231st/wip-seastore-fix-cache

crimson/os/seastore/cache: misc fixes and cleanup

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

crimson/os/seastore/cache: rename try_construct_record to prepare_record

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: wait io when read from Cache::extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add the missing extent invalid check

see 2bc257beb2ef1253e38c254745f42b3ac5db52de

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add missing add_to_read_set()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup get_extent_if_cached()

And remove the query_cache_for_extent(t).

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: fix get_extent_by_type(t) to use Transaction::get_extent()

The original query_cache_for_extent() doesn't call wait_io() if the
extent is pending. And the following get_extent_by_type() already covers
the lookup to Cache::extents and also implements wait_io() correctly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add absent extent to read-set before retire

So that transaction can be invalidated if the retiring extent has
conflict.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: remove retired extents from cache

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: fix print_detail() of RetiredExtentPlaceholder

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup try_construct_record() optional return

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: check invalid transaction only

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: retire PRESENT extents and add to retired-set

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: invalidate transaction referencing retired extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42103 from athanatos/sjust/wip-seastore-collection-ordering

crimson/seastore: enforce per-collection transaction ordering

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

Merge pull request #41932 from tchaikov/wip-mgr-mock-osdmap

mgr,pybind/mgr: add OSDMap.build_simple() method

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41958 from ceph/wip-yuriw-notes-14.2.22_2

doc: 14.2.22 Release Notes

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases/nautilus,octopus: add a note for msgr2 interoperability fix

octopus 15.2.13 already shipped, let's add it retroactively.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases/nautilus: add notable changes for RADOS

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: 14.2.22 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #42118 from sandrobonazzola/patch-3

doc/rbd/iscsi-initiator-linux: set "product" in multipath.conf

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rbd/iscsi-initiator-linux: set "product" in multipath.conf

Adding missing parameter in multipath configuration.  See
https://bugzilla.redhat.com/show_bug.cgi?id=1769135.

Signed-off-by: Sandro Bonazzola <sbonazzo@redhat.com>

rgw: a small set of clean-ups

Small items noticed along the way and addressed collectively in this
small "omnibus" PR.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #42121 from ceph/wip-md

ceph-release-notes: Support for markdown for new ceph.io site

Reviewed-by: Josh Durgin <jdurgin@redhat.com>