mgr/cephadm: tolerate failure to update daemon caps

If we're upgrading from 15.2.0, we may fail to update caps.  Instead of
failing the upgrade hard, warn to the log and continue.  This is less
than ideal, but the caps will get corrected the next time the daemon is
redeployed on the next upgrade, and most likely the previous caps will
continue to work (given they were presumably working before the upgrade).

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: fix get_keyring_with_caps

1- Pass caps to 'auth get-or-create'
2- Only try 'auth caps' if the get-or-create failed

Note that the 'auth caps' step can fail if upgrading from 15.2.0 since
'profile mgr' didn't include 'auth caps' until 15.2.1.  We're not
addressing that for now...

Fixes: 7c0d532f3a4839f4199a13773fb5fa8b6fb3f183
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #40010 from dillaman/wip-49716

librbd: allow interrupted trash move request to be restarted

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #40068 from tchaikov/wip-readme-ninja

README.md: correct notes on `-j` option

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

README.md: correct notes on `-j` option

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40057 from tchaikov/wip-49726

Revert "osd: avoid two copy with same src cancel each other"

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #40004 from vshankar/wip-cephfs-mirror-symlink-overrun

cephfs-mirror: fix possible incorrect symbolic link synchronization

Merge pull request #38476 from aclamk/wip-bs-alloc-check

os/bluestore: add asserts for allocator regions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #39670 from myoungwon/wip-fix-missing-snap-manifest

osd: fix missing adjacent snaps when handling manifest object

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #39941 from tchaikov/wip-mgr-localpool-ann

mgr/localpool: add type annotations

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39994 from Rishabh4275/master

os/bluestore: Fix zoned code location in _do_truncate

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #39992 from tchaikov/wip-byteorder-endian

include/byteorder: drop init_le*() helpers

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #39998 from tchaikov/wip-rwlock

src/: do not include RWLock.h unless RWLock is used

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39410 from sebastian-philipp/drive-group-validate-data-devices-none

python-common: Verify data_devices is not None

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39857 from adk3798/dup-labels

mgr/cephadm: remove duplicate labels when adding a host

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

test: add symbolic link synchronization test for cephfs-mirror

Fixes: http://tracker.ceph.com/issues/49711
Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: null terminate buffer before synchronizing symbolc link

Signed-off-by: Venky Shankar <vshankar@redhat.com>

osd: Revert "osd: avoid two copy with same src cancel each other"

This reverts commit 617f7114e261595e2e85970e171dcad41f283599.

Fixes: https://tracker.ceph.com/issues/49726
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40053 from rishabh-d-dave/readme-update

README.md: replace "make" with "ninja"

Reviewed-by: Kefu Chai <kchai@redhat.com>

README.md: replace "make" by "ninja"

Update README.md since ninja is the default generator now.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #40034 into master

* refs/pull/40034/head:
qa/suites/rados/cephadm/upgrade: start at 15.2.9 to test iscsi upgrade
qa/tasks/cephadm.py: don't set mgr count to +1

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/suites/rados/cephadm/upgrade: start at 15.2.9 to test iscsi upgrade

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm.py: don't set mgr count to +1

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #40007 from neha-ojha/wip-cc-weight

pybind/mgr/balancer/module.py: assign weight-sets to all buckets before balancing

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #39845 into master

* refs/pull/39845/head:
doc: clarify mirror daemon user capability requirements
doc: doc changes for additional mirroring interfaces
pybind/mirroring: add interface to list file system mirror peers
pybind/mirroring: set libcephfs handle root uid/gid as 0:0
test: add tests for mirroring bootstrap interfaces
pybind/mirroring: introduce peer_bootstrap {create|import} commands
cephfs-mirror: use peer cluster monitor address (and key) if available
mon: peer_add should accept Ceph file system UUID
mon: introduce "profile cephfs-mirror" cap constrained to "config-get cephfs/mirror/peer"
test: add test for failed filesystem mirror instances
cephfs-mirror: restart failed mirror filesystem instances

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39671 into master

* refs/pull/39671/head:
doc: snap-schedule documentation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

rbd: clarify trash remove error code from interrupted move

Fixes: https://tracker.ceph.com/issues/49716
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/trash: don't return -ENOENT error from move state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/api: trash remove/purge should indicate interrupted move

This will help the user self-diagnose that a trash move operation
was interrupted and therefore the state is invalid.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #40006 from dillaman/wip-qa-rbd-rwl-ssd

qa/suites/rbd: added SSD PWL cache mode to tests

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39875 from zdover23/wip-doc-cephadm-adoption-overview-and-limitations-rewrite-2021-Mar-06

doc/cephadm: rewrite part of adoption.rst

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #40008 from mgfritch/cephadm-nfsv3-deprecate

doc/cephadm: add note about deprecation of NFSv3

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39932 from mgfritch/custom-monitoring-j2

doc/cephadm: update section on applying custom config (j2)

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: clarify mirror daemon user capability requirements

Fixes: http://tracker.ceph.com/issues/49619
Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: doc changes for additional mirroring interfaces

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mirroring: add interface to list file system mirror peers

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mirroring: set libcephfs handle root uid/gid as 0:0

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add tests for mirroring bootstrap interfaces

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mirroring: introduce peer_bootstrap {create|import} commands

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: use peer cluster monitor address (and key) if available

This allows connecting to the peer cluster without having the cluster
configuration file on the primary cluster.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mon: peer_add should accept Ceph file system UUID

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mon: introduce "profile cephfs-mirror" cap constrained to "config-get cephfs/mirror/peer"

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add test for failed filesystem mirror instances

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: restart failed mirror filesystem instances

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40021 from xxhdx1985126/wip-io-sequencer-bug-fix

crimson/osd: retrive client_requests' prev_op_id right before "start_op"

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: retrive client_requests' prev_op_id right before "start_op"

ClientRequest::prev_op_id should record its immediate predecessor in the
pipeline. If we capture sequencer's last_issued when creating the client
request, it may not represent that predecessor

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39859 from jhonxue/master

osd: do not dump an osd multiple times

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: Synchronize AvlAllocator device_size and block_size with other Allocators.

Modify AvlAllocator to use device_size and block_size from Allocator.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluestore: Added asserts for allocator regions

Functions release/init_add_free/init_rm_free did not check its input against device size.
It is incorrect and had been a problem when you shrink device.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #39989 from athanatos/sjust/wip-fix-seastore-gc

crimson: btree_lba_manager and segment_cleaner fixes

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: fix lookup

lower_bound() here is wrong, use bound instead.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add releasing state for segments pending close

This should fix a bug by which we might start scanning a segment a second
time as it is released and possibly even reused resulting in nonsensical
behavior.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #39677 into master

* refs/pull/39677/head:
mgr/cephadm: add iscsi and nfs to upgrade
mgr/cephadm: update caps if necessary when getting keyring

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: add iscsi and nfs to upgrade

Fixes: https://tracker.ceph.com/issues/49462
Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: update caps if necessary when getting keyring

If the caps change from the old version to the new one it causes
issues in the upgrade. This allows the caps to be updated. Currently
only seeing this with iscsi but changing it for other as a precaution

Signed-off-by: Adam King <adking@redhat.com>

pybind/mgr/balancer/module.py: assign weight-sets to all buckets before balancing

Add an additional check to make sure that the choose_args section has the same
number of buckets as the crushmap. If not, ensure that
get_compat_weight_set_weights assigns weight-sets to all buckets.

Without this change, if we end up with an orig_ws, which has fewer buckets
than the crushmap, the mgr will crash due a KeyError in do_crush_compat().

Fixes: https://tracker.ceph.com/issues/49576
Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/cephadm: add note about deprecation of NFSv3

Signed-off-by: Michael Fritch <mfritch@suse.com>

librbd/api: allow an interrupted trash move to be restarted

Search the trash entries for a matching image name that is
still in the moving state and allow the operation to be
restarted.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: Fix zoned code location in _do_truncate

Signed-off-by: Rishabh Chawla <rishabhchawla1995@gmail.com>

librbd/api: helper method for natively listing the trash

The existing list method converts the native TrashImageSpec to the
API's rbd_trash_image_info_t which is missing the source field.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39640 from sebastian-philipp/cephadm-cephfs-mirror

cephadm: cephfs-mirror

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: snap-schedule documentation

Fixes: https://tracker.ceph.com/issues/48017
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

qa/suites/rbd: added SSD PWL cache mode to tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39800 from MahatiC/wip-ssd-fixes

librbd/cache/pwl: Fix IO completion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #38913 into master

* refs/pull/38913/head:
qa/tasks/cephfs/nfs: Add tests for updating fs exports
mgr/volumes/nfs: Handle rook restart error
doc/cephfs/nfs: Add about update export interface
mgr/volumes/nfs: Add command to update cephfs exports
pybind/volumes/nfs: set mds caps according to user specified access type
mgr/volumes/module: Remove unused json module

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39953 into master

* refs/pull/39953/head:
debian/ceph-common.postinst: do not chown cephadm log dirs

Reviewed-by: Kefu Chai <kchai@redhat.com>

include/byteorder: drop init_le*() helpers

they are not used anymore, so drop them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: use s/init_le*/ceph_le*/

for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/rgw, rgw: use s/init_le*/ceph_le*/

for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth,msg,common: test: use s/init_le*/ceph_le*/

for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: use s/init_le*/ceph_le*/

for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: switch to ceph_le*() constructors

instead of using init_le*() helpers, use ceph_le*() constructors for
better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: add cephfs-mirror to CEPH_UPGRADE_ORDER

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: Add cephfs-mirror

Fixes: https://tracker.ceph.com/issues/47261
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/cephadm: Add cephfs-mirror test

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/tasks: some type annotations

Mostly for making my IDE aware of things

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: Add cephfs-mirror to enum

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add CephfsMirrorService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: replace def add_{type}(...) with generic add_daemon()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: drop `create_func` arg from _add_daemon

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move CephadmExporter to new module

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

test/librbd: s/RWLock/share_lock/

to consolidate the code base further, we don't need to use the
homebrewed shared lock unless the std::shared_mutex cannot fufill our
needs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #39860 into master

* refs/pull/39860/head:
vstart,init-ceph: include build bin/ in $PATH

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39619 from sebastian-philipp/rm_injected_argv

cephadm: remove injected_args

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39877 from liewegas/cephadm-rgw-simplification

mgr/cephadm: simplify handling for rgw

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #39991 from tchaikov/wip-ninja-again

ceph.spec: include ninja as BuildRequires

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #39878 from batrick/centos-8.3

qa/distros: add centos 8.3

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39793 from rhcs-dashboard/replace-inventory-devices-title

mgr/dashboard: cluster > hosts: replace inventory->devices table title

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #39882 from tchaikov/wip-osd-watch

osd/Watch: use vector<> instead of list<>

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39897 from Rethan/clean-msgr-type

msg/Messenger: remove unused variable r

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39937 from tchaikov/wip-49661

mgr: add mon metada using type of "mon"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39541 from pcuzner/config-checker

mgr/cephadm: Provide an integrated configuration validation feature

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39905 from tchaikov/wip-lock-deps

common/lockdep: increase MAX_LOCKS to 128k

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #39503 from rhcs-dashboard/fix-ssl-certificate-code

mgr/dashboard: fix dashboard instance ssl certificate functionality

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

rgw/rgw_common: silence Wsign-compare warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librados_test_stub: do not include unused RWLock.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd,rbd_mirror: do not include RWLock.h unless it is used

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: do not include RWLock.h unless it is used

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: do not include RWLock.h unless it is used

Signed-off-by: Kefu Chai <kchai@redhat.com>

os: do not include RWLock.h unless it is used

Signed-off-by: Kefu Chai <kchai@redhat.com>