Merge pull request #36594 from matthewoliver/ipv6_prometheus_module

prometheus: Properly split the port off IPv6 addresses

Merge pull request #34293 from yuvalif/notifications_persistent_delivery2

rgw/notifications: add persistent delivery to notifications

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #36847 from X0Ken/fix-journal-size

ceph-volume: fix journal size argument not work

Merge pull request #36545 from AmnonHanuhov/wip-seastar_metrics

crimson/admin: Add a hook for Seastar metrics

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36743 from cbodley/wip-rgw-sal-bucket-ctime

rgw: RGWBucket uses creation_time from RGWBucketInfo

ceph-volume: fix journal size argument not work

Journal size argument not work in batch prepare
filestore with SSD journal device.

Fixes: https://tracker.ceph.com/issues/41374
Signed-off-by: wanghongxu <wanghongxu@t2cloud.net>

Merge pull request #35802 from rhcs-dashboard/wip-46160-fix_cephfs_api_inconsistency

mgr/dashboard: Address API inconsistencies in Cephfs endpoint

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36873 from sebastian-philipp/cephadm-osd-activate-rm-storage

cephadm: properly remove `osd-activate` container

Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson/osd: Register Seastar metrics hook

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

crimson/admin: Add Seastar metrics hook

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

Merge pull request #36923 from rhcs-dashboard/fix-47232-master

mgr/dashboard/api: print logs when vstart fails

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36222 from ishanrai05/docs

doc: Create new chapter about the Dashboard's landing page

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34644 from rhcs-dashboard/26954-histogram

mgr/dashboard: Remove Histogram Component

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36869 from tchaikov/wip-mon-no-MMonMetadata

mon/Monitor: do not handle MMonMetadata

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #35697 into master

* refs/pull/35697/head:
doc: document cephfs mirroring developement status
cephfs-mirror: cephfs-mirror daemon
cephfs-mirror: FSMirror class to synchronize directory snaps
common: introduce configuration options for cephfs-mirror daemon
cephfs-mirror: InstanceWatcher class to register mirror instance
cephfs-mirror: MirrorWatcher class to watch cephfs-mirror object
cephfs-mirror: Watcher class to receive/ack watch notifications
cephfs-mirror: ClusterWatcher class for watching peer changes
librados: add API for fetching client addrs
common: add cephfs_mirror subsystem
cephfs-mirror: filesystem specification class

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35899 from myoungwon/wip-fix-manifest-tier

osd, test: refactoring manfiest-tier

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #36915 from yuriw/wip-yuriw-46529-master

qa/tests: use bionic only for old clients in rados/thrash-old-clients

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36700 from leseb/retry-on-lock

ceph-volume: retry when acquiring lock fails

Merge pull request #36889 from idryomov/wip-relax-preauth-asserts

msg/async/ProtocolV2: allow rxbuf/txbuf get bigger in testing

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard/api: print logs when vstart fails

Fixes: https://tracker.ceph.com/issues/47232
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #36241 from jan--f/c-v-mpath-support

ceph-volume: support for mpath devices

doc: document cephfs mirroring developement status

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: cephfs-mirror daemon

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: FSMirror class to synchronize directory snaps

Signed-off-by: Venky Shankar <vshankar@redhat.com>

common: introduce configuration options for cephfs-mirror daemon

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: InstanceWatcher class to register mirror instance

... and watch for notifications via instance object.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: MirrorWatcher class to watch cephfs-mirror object

... and ack details such as instance-id and address.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: Watcher class to receive/ack watch notifications

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: ClusterWatcher class for watching peer changes

Signed-off-by: Venky Shankar <vshankar@redhat.com>

librados: add API for fetching client addrs

Signed-off-by: Venky Shankar <vshankar@redhat.com>

common: add cephfs_mirror subsystem

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: filesystem specification class

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #36917 from zdover23/wip-doc-rados-init-mems-to-mon-host

doc/rados: s/initial members/mon host

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #36800 from adk3798/cephadm-46654

cephadm: Change service type to forking via systemd

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/rados: s/initial members/mon host

This changes an old string, "initial members"
to a current string, "mon host".

Fixes: https://tracker.ceph.com/issues/47116
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36879 from tchaikov/wip-doc-stable-release

doc: bump up sphinx to 3.2.1 and define "stable-release" global substitution

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36897 from tchaikov/wip-cmake-asan

cmake: expose Sanitizers_COMPILE_OPTIONS as a list

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

doc: fix broken hyper link

and always reference multisite.rst using tha label not the file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix broken hyper link

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: define label before section title

so we can see the title of section when following the link

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add sphinx.ext.mathjax for math-to-MathML rendering

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix broken hyper link

this link was broken in 1427905c473e352e7cac1a9ac209cddb82544b57

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: always refernce terms in case sensitive mode

to silence warning from sphinx like:

ceph/doc/install/ceph-deploy/quick-start-preflight.rst:5: WARNING: term node not found in case sensitive match.made a reference to Node instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/rados/librados.h: avoid redefinition of rados_object_list_item

doxygen complains at seeing rados_object_list_item gets defined twice.
so let's fix it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: replace luminous with |stable-release|

"luminous" is a magic number in these contexts, and we should use a
constant for representing it.

the "sphinx_substitution_extensions" sphinx extension is introduced for
performing the global subsitution.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36907 from yuvalif/fix-issue-47209

cmake: make crimson libraries static to prevent cmake error

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: fix reference leak

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

qa/tests: use bionic only for old clients in rados/thrash-old-clients

Fixes: https://tracker.ceph.com/issues/46529
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #36089 into master

* refs/pull/36089/head:
mds/OpenFileTable: add option that disables dirfrags prefetch
mds: use parent/d_name in cap_reconnect_t to speedup MDCache::open_ino()
client: encode inode's parent/d_name in cap_reconnect_t
mds: explicitly track files with client writeable ranges
mds: properly set CLIENTWRITEABLE flag for imported caps
mds: in-place update projected inode's client ranges
mds: only add inodes that clients want Frw or excl caps to open file table
mds/OpenFileTable: reduce anchor map lookup during prefetch
mds/OpenFileTable: merge dirfrag set into anchor map

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36887 into master

* refs/pull/36887/head:
qa: ignore expected mds failover message

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #36846 into master

* refs/pull/36846/head:
client: support getting ceph.dir.rsnaps vxattr
messages: fix decode version for snap_btime

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36777 into master

* refs/pull/36777/head:
osdc/Journaler: do not call onsafe->complete() if onsafe is 0

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36672 into master

* refs/pull/36672/head:
client: handle readdir reply without Fs cap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36631 into master

* refs/pull/36631/head:
client: dump frag's repmap if dir_replicated is true

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36885 into master

* refs/pull/36885/head:
mon: ignore statfs requests on invalid poolid
qa: test for statfs on deleted file system
qa: move run_shell args to kwargs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36881 from mhackett/wip-mhackett-47189

doc: document tuning of object gateway garbage collection

mgr/dashboard: Address API inconsistencies

Cephfs endpoints are inconsistencies with other endpoints.

* Change rm endpoints from GET to DELETE requests
* Change set_quotas from POST to PUT
* Renames endpoints accordingly
* Changes related angular files
* update a few lines related to mypy
* remove future import as the python2 compatibility is not needed

Fixes: https://tracker.ceph.com/issues/46160
Signed-off-by: Fabrizio D'Angelo <fdangelo@redhat.com>
(cherry picked from commit 222286b08bde3600d50a6deb25e7425c17e9e482)

ceph-volume: retry when acquiring lock fails

When preaparing the osd device with --mkfs, the ceph-osd binary tries to
acquire an exclusive lock on the device (soon to become an OSD).
Unfortunately, when running in containers, we have seen cases where
there is a race between ceph-osd and systemd-udevd to acquire a lock on
the device. Sometimes systemd-udevd gets the lock and releases it soon
so that the ceph-osd gets sometimes the lock is still held and because
ceph-osd uses LOCK_NB the command fails.

This commit retries if the lock cannot be acquired, up to 5 times for 5
seconds, this should be more than enough to acquire the lock and
proceed with the OSD mkfs.

Unfortunately, this is so transient that we cannot lock earlier from c-v,
this won't do anything.

Fixes: https://tracker.ceph.com/issues/47010
Signed-off-by: Sébastien Han <seb@redhat.com>

crimson/cmake: fix copy&paste error

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

crimson/cmake: make libraries static to prevent cmake error

Fixes: https://tracker.ceph.com/issues/47209
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

doc: create new chapter about the Dashboard's landing page

Fixes: https://tracker.ceph.com/issues/46415
Signed-off-by: Ishan Rai <ishanrai05@gmail.com>

Merge pull request #36826 from agayev/fix-hmsmr-compilation

os/bluestore: Restore HM-SMR functionality on BlueStore

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34653 from clwluvw/bucket-stats

rgw: Add bucket name to bucket stats error logging

Merge pull request #36857 from tchaikov/wip-crimson-conf

crimson/osd: support "dump_pgstate_history" command and update legacy conf vars

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: improve the const correctness

it should suffice to pass a `const OSD&` to the ctor of `OsdStatusHook`

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: support "dump_pgstate_history" command

it is required by the "ceph_test_admin_socket_output" test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: make injected_args optional

this option for `injectargs` command is optional, so we should not
crash if it does not exist.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: do not check for experimental/filesystem anymore

seastar is only compatible with C++17 and C++20, so no need to check for
<experimental/filesystem> anymore. the C++ standard library should be
C++17 compatible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: call config::update_legacy_vals()

so the legacy vals are populated with the changes received from conffile.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: make update_legacy_vals() public

so crimson can call it in `ConfigProxy::parse_config_files()` after calling
`get_config().parse_buffer()`. so the legacy vals are populated with the
changes received from conffile.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36891 from tchaikov/wip-less-warnings

crimson/mon,common: replace #warning with // TODO

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36892 from tchaikov/wip-crimson-cleanups

crimson/osd: return EAGAIN if !can_serve_replica_read() and cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36905 from sebastian-philipp/cephadm-pep8-test-migration

mgr/cephadm: PEP8tify test_migration.py

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36676 from matthewoliver/cephadm_orch_ipv6

cephadm: auto wrap and unwrap ipv6 addresses

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36896 from xxhdx1985126/wip-47133

crimson/pg: check acting set change before appending pglog

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: PEP8tify test_migration.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36877 from sebastian-philipp/mgr-module-fix-kwargs-cli-command

mgr/mgr_module.py: CLICommand: Fix parsing of kwargs arguments

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36859 from tspmelo/wip-fix-onselect

mgr/dashboard: Fix stale table actions

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

cmake: expose Sanitizers_COMPILE_OPTIONS as a list

* cmake/modules/FindSanitizers.cmake: do not pollute CMAKE_REQUIRED_FLAGS
* cmake/modules/FindSanitizers.cmake: expose Sanitizers_COMPILE_OPTIONS
  as a list
* CMakeLists.txt: append Sanitizers_COMPILE_OPTIONS to
  *_LINKER_FLAGS after replacing ";" with " " in it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36839 from dillaman/wip-46875

librbd: flush all incomplete in-flight IOs upon image close

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36819 from vumrao/wip-vumrao-46663

mon/PGMap: add pg count for pools in the ceph df command

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/pg: check acting set change before appending pglog

Fixes: https://tracker.ceph.com/issues/47133
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #33852 from rosinL/wip-fix-enqueue-dead-qp-deadlock

msg/async/rdma: avoid enqueue_dead_qp deadlock

Reviewed-by: Peng Liu <liupeng37@baidu.com>

Merge pull request #36767 from tchaikov/wip-47061

test/osd/RadosModel: check retval of read op

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: return EAGAIN if !can_serve_replica_read()

it's necessary for serving balanced read.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: check for primary out side of is_misdirected()

so we can check can_serve_replica_read() without checking primary again.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: encode OI with CEPH_FEATURES_ALL

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use logger() instead of crimson::get_logger(...)

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: call PeeringState::get_peer_missing(peer)

as PeeringState::get_peer_missing(const pg_shard_t &peer) already checks
for the non-existent shard, and because `get_replica_recovery_order()`
never returns primary osd, it's safe to use
`PeeringState::get_peer_missing(peer)` instead of
`pg->get_peering_state().get_peer_missing().find(peer)`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: eliminate a local variable

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd/RadosModel: assure that tid seq is contiguous

before this change:

4:  finishing write tid 1 to gen81094472-1
4:  finishing write tid 2 to gen81094472-1
4:  finishing write tid 3 to gen81094472-1
4:  finishing write tid 5 to gen81094472-1
4:  finishing write tid 6 to gen81094472-1

after this change:

1:  finishing write tid 1 to gen81117124-1
1:  finishing write tid 2 to gen81117124-1
1:  finishing write tid 3 to gen81117124-1
1:  finishing write tid 4 to gen81117124-1
1:  finishing write tid 5 to gen81117124-1

more pleasant when reading the logging messages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd/RadosModel: check retval of read op

* check racing read's return value
* check if racing read completes

Fixes: https://tracker.ceph.com/issues/47061
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd/RadosModel: do not set done

we don't reuse TestOp, and `done` should have been reset in the
constructor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd/RadosModel: use range-based for loop

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd/RadosModel: use in-class initiailization

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd: use stack for ObjectDesc::iterator::stack

better readability this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: replace #warning with TODO

less distracting when fixing compiling error / warnings this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/admin_socket: replace #warning with TODO

less distracting when fixing compiling error / warnings this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36815 from sebastian-philipp/cephadm-pep8-module

mgr/cephadm: PEP8tify module.py

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: PEP8tify module.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>