Merge pull request #36298 from p-se/mgr-prom-collect-time-metric

mgr/prometheus: introduce metric for collection time

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40100 from liewegas/bug-49788

osd: propagate base pool application_metadata to tiers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40041 from adk3798/helpcoms

pybind/ceph_argparse: print flag name before CephChoices in help descs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40062 from tchaikov/wip-no-lockdep-in-rel-builds

cmake: do not build debug_mutex or lockdep for Release build

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

osd: propagate base pool application_metadata to tiers

If there is application metadata on the base pool, it should be mirrored
to any other tiers in the set.  This aligns with the fact that the
'ceph osd pool application ...' commands refuse to operate on a non-base
pool.

This fixes problems with accessing tiers (e.g., cache tiers) when the
cephx cap is written in terms of application metadata.

Fixes: https://tracker.ceph.com/issues/49788
Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #40042 into master

* refs/pull/40042/head:
qa/suites/rados/cephadm: add back centos+rhel with kubic podman

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cmake: do not build lockdep for Release build

lockdep create large data structures on .bss and on heap for tracking
the locks and their dependencies. but we don't need to pay for this
if lockdep is not enabled.

lockdep helps us to track the lock dependencies related issue on Debug
build. and Release build, this feature hurts the performance and more
importantly, lockdeps is a feature only kicks in when using the
mutex_debug and friends. they are not used in Release build at all.

so, after this change, lockdep is not built in Release build. and
the static variables defined in lockdep.cc are not allocated anymore
in Release build.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/lockdep: disable lockdep if !CEPH_DEBUG_MUTEX

make lockdep a feature which can only be enabled at compile time.

we don't need to have lockdep when CEPH_DEBUG_MUTEX is OFF, so
just comment it out in Release builds.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40036 from tchaikov/wip-doc-mgr

doc/mgr: fix formatting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40044 from ceph/wip-1427-notes

doc: 14.2.17 Release Notes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge PR #40045 into master

* refs/pull/40045/head:
cephadm: use image id, not name, when inspecting for RepoDigests

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #40052 from tchaikov/wip-rpm-no-rbd-rwl-on-el8

ceph.spec: build with system libpmem on fedora and el8

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #40050 from dillaman/wip-49690

librbd/io: send alloc_hint when compression hint is set

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38853 from dzafman/wip-48843

osd: Try other PGs when reservation failures occur

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

common/ceph_context: disable lockdep if !CEPH_DEBUG_MUTEX

make lockdep a feature which can only be enabled at compile time.

we don't need to have lockdep when CEPH_DEBUG_MUTEX is OFF, so
just comment it out in Release builds.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not build mutex_debug.cc if !WITH_CEPH_DEBUG_MUTEX

there is no need to build shared_mutex_debug.cc and
mutex_debug.cc, if they are not used at all. in Release build
we just use the mutex primitives offered by C++ standard library and
the POSIX API offered by libc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/crimson: define crimson_alien_common_srcs

so we can add to this list if some option is enabled

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: print flag name before CephChoices in help descs

Fixes: https://tracker.ceph.com/issues/49757
Signed-off-by: Adam King <adking@redhat.com>

Merge PR #40046 into master

* refs/pull/40046/head:
qa/suites/rados/cephadm/upgrade: deploy a legacy r.z-style rgw

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #40010 from dillaman/wip-49716

librbd: allow interrupted trash move request to be restarted

Reviewed-by: Mykola Golub <mgolub@suse.com>

ceph.spec: build with system libpmem on fedora and el8

* build with WITH_SYSTEM_PMDK=ON on fedora, as f32 and f33 ship
  libpmem1.8 and libpmem1.9 respectively. and we need libpmem v1.7
* build with WITH_SYSTEM_PMDK=ON on el8, as el8 and CentOS8 AppStream
  ships libpmem v1.6,

quote from nvml.spec:

> By design, PMDK does not support any 32-bit architecture.
> Due to dependency on some inline assembly, PMDK can be compiled only
> on these architectures:
> - x86_64
> - ppc64le (experimental)
> - aarch64 (unmaintained, supporting hardware doesn't exist?)

so far, only x86_64 and ppc64le packages are built.

see also,
https://src.fedoraproject.org/rpms/nvml/blob/rawhide/f/nvml.spec

this change addresses a regression introduced by
a49d1dbb32e2436ff2836a85b2fa84418f0a5fff

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40068 from tchaikov/wip-readme-ninja

README.md: correct notes on `-j` option

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

README.md: correct notes on `-j` option

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40057 from tchaikov/wip-49726

Revert "osd: avoid two copy with same src cancel each other"

Reviewed-by: Samuel Just <sjust@redhat.com>

cmake: make "WITH_CEPH_DEBUG_MUTEX" depend on CMAKE_BUILD_TYPE

this option is available only if CMAKE_BUILD_TYPE is Debug.

this change helps us to unify the checks for WITH_CEPH_DEBUG_MUTEX,
without this change, we always have to check both WITH_CEPH_DEBUG_MUTEX
*and* CMAKE_BUILD_TYPE.

after this change, we only respect WITH_CEPH_DEBUG_MUTEX.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40004 from vshankar/wip-cephfs-mirror-symlink-overrun

cephfs-mirror: fix possible incorrect symbolic link synchronization

Merge pull request #38476 from aclamk/wip-bs-alloc-check

os/bluestore: add asserts for allocator regions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #39670 from myoungwon/wip-fix-missing-snap-manifest

osd: fix missing adjacent snaps when handling manifest object

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #39941 from tchaikov/wip-mgr-localpool-ann

mgr/localpool: add type annotations

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39994 from Rishabh4275/master

os/bluestore: Fix zoned code location in _do_truncate

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #39992 from tchaikov/wip-byteorder-endian

include/byteorder: drop init_le*() helpers

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #39998 from tchaikov/wip-rwlock

src/: do not include RWLock.h unless RWLock is used

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39410 from sebastian-philipp/drive-group-validate-data-devices-none

python-common: Verify data_devices is not None

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39857 from adk3798/dup-labels

mgr/cephadm: remove duplicate labels when adding a host

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

test: add symbolic link synchronization test for cephfs-mirror

Fixes: http://tracker.ceph.com/issues/49711
Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: null terminate buffer before synchronizing symbolc link

Signed-off-by: Venky Shankar <vshankar@redhat.com>

osd: Revert "osd: avoid two copy with same src cancel each other"

This reverts commit 617f7114e261595e2e85970e171dcad41f283599.

Fixes: https://tracker.ceph.com/issues/49726
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40053 from rishabh-d-dave/readme-update

README.md: replace "make" with "ninja"

Reviewed-by: Kefu Chai <kchai@redhat.com>

README.md: replace "make" by "ninja"

Update README.md since ninja is the default generator now.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

librbd/io: send alloc_hint when compression hint is set

Previously the hint would not be set if the object map indicated the
object may exist.

Fixes: https://tracker.ceph.com/issues/49690
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #40034 into master

* refs/pull/40034/head:
qa/suites/rados/cephadm/upgrade: start at 15.2.9 to test iscsi upgrade
qa/tasks/cephadm.py: don't set mgr count to +1

Reviewed-by: Sebastian Wagner <swagner@suse.com>

doc: 14.2.17 Release Notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

cephadm: use image id, not name, when inspecting for RepoDigests

The name is ambiguous, but the image_id is not!  This fixes problems
during upgrade where upgrade thinks the container is upgraded (due to
an incorrect digest) when in fact it is not.

Fixes: 0826c45e0cb5d60fcf8cd71cd14edd34a6997cd4
Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/rados/cephadm/upgrade: start at 15.2.9 to test iscsi upgrade

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/rados/cephadm: add back centos+rhel with kubic podman

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm.py: don't set mgr count to +1

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #40007 from neha-ojha/wip-cc-weight

pybind/mgr/balancer/module.py: assign weight-sets to all buckets before balancing

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #39845 into master

* refs/pull/39845/head:
doc: clarify mirror daemon user capability requirements
doc: doc changes for additional mirroring interfaces
pybind/mirroring: add interface to list file system mirror peers
pybind/mirroring: set libcephfs handle root uid/gid as 0:0
test: add tests for mirroring bootstrap interfaces
pybind/mirroring: introduce peer_bootstrap {create|import} commands
cephfs-mirror: use peer cluster monitor address (and key) if available
mon: peer_add should accept Ceph file system UUID
mon: introduce "profile cephfs-mirror" cap constrained to "config-get cephfs/mirror/peer"
test: add test for failed filesystem mirror instances
cephfs-mirror: restart failed mirror filesystem instances

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39671 into master

* refs/pull/39671/head:
doc: snap-schedule documentation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/mgr: fix formatting

a code block should start with "::".

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd: clarify trash remove error code from interrupted move

Fixes: https://tracker.ceph.com/issues/49716
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/trash: don't return -ENOENT error from move state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/api: trash remove/purge should indicate interrupted move

This will help the user self-diagnose that a trash move operation
was interrupted and therefore the state is invalid.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #40006 from dillaman/wip-qa-rbd-rwl-ssd

qa/suites/rbd: added SSD PWL cache mode to tests

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39875 from zdover23/wip-doc-cephadm-adoption-overview-and-limitations-rewrite-2021-Mar-06

doc/cephadm: rewrite part of adoption.rst

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #40008 from mgfritch/cephadm-nfsv3-deprecate

doc/cephadm: add note about deprecation of NFSv3

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39932 from mgfritch/custom-monitoring-j2

doc/cephadm: update section on applying custom config (j2)

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: clarify mirror daemon user capability requirements

Fixes: http://tracker.ceph.com/issues/49619
Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: doc changes for additional mirroring interfaces

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mirroring: add interface to list file system mirror peers

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mirroring: set libcephfs handle root uid/gid as 0:0

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add tests for mirroring bootstrap interfaces

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mirroring: introduce peer_bootstrap {create|import} commands

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: use peer cluster monitor address (and key) if available

This allows connecting to the peer cluster without having the cluster
configuration file on the primary cluster.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mon: peer_add should accept Ceph file system UUID

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mon: introduce "profile cephfs-mirror" cap constrained to "config-get cephfs/mirror/peer"

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add test for failed filesystem mirror instances

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: restart failed mirror filesystem instances

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40021 from xxhdx1985126/wip-io-sequencer-bug-fix

crimson/osd: retrive client_requests' prev_op_id right before "start_op"

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: retrive client_requests' prev_op_id right before "start_op"

ClientRequest::prev_op_id should record its immediate predecessor in the
pipeline. If we capture sequencer's last_issued when creating the client
request, it may not represent that predecessor

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39859 from jhonxue/master

osd: do not dump an osd multiple times

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: Synchronize AvlAllocator device_size and block_size with other Allocators.

Modify AvlAllocator to use device_size and block_size from Allocator.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluestore: Added asserts for allocator regions

Functions release/init_add_free/init_rm_free did not check its input against device size.
It is incorrect and had been a problem when you shrink device.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #39989 from athanatos/sjust/wip-fix-seastore-gc

crimson: btree_lba_manager and segment_cleaner fixes

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: fix lookup

lower_bound() here is wrong, use bound instead.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add releasing state for segments pending close

This should fix a bug by which we might start scanning a segment a second
time as it is released and possibly even reused resulting in nonsensical
behavior.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #39677 into master

* refs/pull/39677/head:
mgr/cephadm: add iscsi and nfs to upgrade
mgr/cephadm: update caps if necessary when getting keyring

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: add iscsi and nfs to upgrade

Fixes: https://tracker.ceph.com/issues/49462
Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: update caps if necessary when getting keyring

If the caps change from the old version to the new one it causes
issues in the upgrade. This allows the caps to be updated. Currently
only seeing this with iscsi but changing it for other as a precaution

Signed-off-by: Adam King <adking@redhat.com>

pybind/mgr/balancer/module.py: assign weight-sets to all buckets before balancing

Add an additional check to make sure that the choose_args section has the same
number of buckets as the crushmap. If not, ensure that
get_compat_weight_set_weights assigns weight-sets to all buckets.

Without this change, if we end up with an orig_ws, which has fewer buckets
than the crushmap, the mgr will crash due a KeyError in do_crush_compat().

Fixes: https://tracker.ceph.com/issues/49576
Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/cephadm/upgrade: deploy a legacy r.z-style rgw

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephadm: add note about deprecation of NFSv3

Signed-off-by: Michael Fritch <mfritch@suse.com>

librbd/api: allow an interrupted trash move to be restarted

Search the trash entries for a matching image name that is
still in the moving state and allow the operation to be
restarted.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: Fix zoned code location in _do_truncate

Signed-off-by: Rishabh Chawla <rishabhchawla1995@gmail.com>

librbd/api: helper method for natively listing the trash

The existing list method converts the native TrashImageSpec to the
API's rbd_trash_image_info_t which is missing the source field.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39640 from sebastian-philipp/cephadm-cephfs-mirror

cephadm: cephfs-mirror

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: snap-schedule documentation

Fixes: https://tracker.ceph.com/issues/48017
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

qa/suites/rbd: added SSD PWL cache mode to tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39800 from MahatiC/wip-ssd-fixes

librbd/cache/pwl: Fix IO completion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #38913 into master

* refs/pull/38913/head:
qa/tasks/cephfs/nfs: Add tests for updating fs exports
mgr/volumes/nfs: Handle rook restart error
doc/cephfs/nfs: Add about update export interface
mgr/volumes/nfs: Add command to update cephfs exports
pybind/volumes/nfs: set mds caps according to user specified access type
mgr/volumes/module: Remove unused json module

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39953 into master

* refs/pull/39953/head:
debian/ceph-common.postinst: do not chown cephadm log dirs

Reviewed-by: Kefu Chai <kchai@redhat.com>

include/byteorder: drop init_le*() helpers

they are not used anymore, so drop them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: use s/init_le*/ceph_le*/

for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/rgw, rgw: use s/init_le*/ceph_le*/

for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth,msg,common: test: use s/init_le*/ceph_le*/

for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: use s/init_le*/ceph_le*/

for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: switch to ceph_le*() constructors

instead of using init_le*() helpers, use ceph_le*() constructors for
better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: add cephfs-mirror to CEPH_UPGRADE_ORDER

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: Add cephfs-mirror

Fixes: https://tracker.ceph.com/issues/47261
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/cephadm: Add cephfs-mirror test

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>