osd: Revert "osd: avoid two copy with same src cancel each other"

This reverts commit 617f7114e261595e2e85970e171dcad41f283599.

Fixes: https://tracker.ceph.com/issues/49726
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #40034 into master

* refs/pull/40034/head:
qa/suites/rados/cephadm/upgrade: start at 15.2.9 to test iscsi upgrade
qa/tasks/cephadm.py: don't set mgr count to +1

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/suites/rados/cephadm/upgrade: start at 15.2.9 to test iscsi upgrade

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm.py: don't set mgr count to +1

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #40007 from neha-ojha/wip-cc-weight

pybind/mgr/balancer/module.py: assign weight-sets to all buckets before balancing

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #39845 into master

* refs/pull/39845/head:
doc: clarify mirror daemon user capability requirements
doc: doc changes for additional mirroring interfaces
pybind/mirroring: add interface to list file system mirror peers
pybind/mirroring: set libcephfs handle root uid/gid as 0:0
test: add tests for mirroring bootstrap interfaces
pybind/mirroring: introduce peer_bootstrap {create|import} commands
cephfs-mirror: use peer cluster monitor address (and key) if available
mon: peer_add should accept Ceph file system UUID
mon: introduce "profile cephfs-mirror" cap constrained to "config-get cephfs/mirror/peer"
test: add test for failed filesystem mirror instances
cephfs-mirror: restart failed mirror filesystem instances

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39671 into master

* refs/pull/39671/head:
doc: snap-schedule documentation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40006 from dillaman/wip-qa-rbd-rwl-ssd

qa/suites/rbd: added SSD PWL cache mode to tests

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39875 from zdover23/wip-doc-cephadm-adoption-overview-and-limitations-rewrite-2021-Mar-06

doc/cephadm: rewrite part of adoption.rst

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #40008 from mgfritch/cephadm-nfsv3-deprecate

doc/cephadm: add note about deprecation of NFSv3

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39932 from mgfritch/custom-monitoring-j2

doc/cephadm: update section on applying custom config (j2)

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: clarify mirror daemon user capability requirements

Fixes: http://tracker.ceph.com/issues/49619
Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: doc changes for additional mirroring interfaces

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mirroring: add interface to list file system mirror peers

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mirroring: set libcephfs handle root uid/gid as 0:0

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add tests for mirroring bootstrap interfaces

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mirroring: introduce peer_bootstrap {create|import} commands

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: use peer cluster monitor address (and key) if available

This allows connecting to the peer cluster without having the cluster
configuration file on the primary cluster.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mon: peer_add should accept Ceph file system UUID

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mon: introduce "profile cephfs-mirror" cap constrained to "config-get cephfs/mirror/peer"

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add test for failed filesystem mirror instances

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: restart failed mirror filesystem instances

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40021 from xxhdx1985126/wip-io-sequencer-bug-fix

crimson/osd: retrive client_requests' prev_op_id right before "start_op"

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: retrive client_requests' prev_op_id right before "start_op"

ClientRequest::prev_op_id should record its immediate predecessor in the
pipeline. If we capture sequencer's last_issued when creating the client
request, it may not represent that predecessor

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39859 from jhonxue/master

osd: do not dump an osd multiple times

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39989 from athanatos/sjust/wip-fix-seastore-gc

crimson: btree_lba_manager and segment_cleaner fixes

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: fix lookup

lower_bound() here is wrong, use bound instead.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add releasing state for segments pending close

This should fix a bug by which we might start scanning a segment a second
time as it is released and possibly even reused resulting in nonsensical
behavior.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #39677 into master

* refs/pull/39677/head:
mgr/cephadm: add iscsi and nfs to upgrade
mgr/cephadm: update caps if necessary when getting keyring

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: add iscsi and nfs to upgrade

Fixes: https://tracker.ceph.com/issues/49462
Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: update caps if necessary when getting keyring

If the caps change from the old version to the new one it causes
issues in the upgrade. This allows the caps to be updated. Currently
only seeing this with iscsi but changing it for other as a precaution

Signed-off-by: Adam King <adking@redhat.com>

pybind/mgr/balancer/module.py: assign weight-sets to all buckets before balancing

Add an additional check to make sure that the choose_args section has the same
number of buckets as the crushmap. If not, ensure that
get_compat_weight_set_weights assigns weight-sets to all buckets.

Without this change, if we end up with an orig_ws, which has fewer buckets
than the crushmap, the mgr will crash due a KeyError in do_crush_compat().

Fixes: https://tracker.ceph.com/issues/49576
Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/cephadm: add note about deprecation of NFSv3

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39640 from sebastian-philipp/cephadm-cephfs-mirror

cephadm: cephfs-mirror

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: snap-schedule documentation

Fixes: https://tracker.ceph.com/issues/48017
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

qa/suites/rbd: added SSD PWL cache mode to tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39800 from MahatiC/wip-ssd-fixes

librbd/cache/pwl: Fix IO completion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #38913 into master

* refs/pull/38913/head:
qa/tasks/cephfs/nfs: Add tests for updating fs exports
mgr/volumes/nfs: Handle rook restart error
doc/cephfs/nfs: Add about update export interface
mgr/volumes/nfs: Add command to update cephfs exports
pybind/volumes/nfs: set mds caps according to user specified access type
mgr/volumes/module: Remove unused json module

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39953 into master

* refs/pull/39953/head:
debian/ceph-common.postinst: do not chown cephadm log dirs

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: add cephfs-mirror to CEPH_UPGRADE_ORDER

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: Add cephfs-mirror

Fixes: https://tracker.ceph.com/issues/47261
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/cephadm: Add cephfs-mirror test

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/tasks: some type annotations

Mostly for making my IDE aware of things

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: Add cephfs-mirror to enum

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add CephfsMirrorService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: replace def add_{type}(...) with generic add_daemon()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: drop `create_func` arg from _add_daemon

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move CephadmExporter to new module

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #39860 into master

* refs/pull/39860/head:
vstart,init-ceph: include build bin/ in $PATH

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39619 from sebastian-philipp/rm_injected_argv

cephadm: remove injected_args

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39877 from liewegas/cephadm-rgw-simplification

mgr/cephadm: simplify handling for rgw

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #39991 from tchaikov/wip-ninja-again

ceph.spec: include ninja as BuildRequires

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #39878 from batrick/centos-8.3

qa/distros: add centos 8.3

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39793 from rhcs-dashboard/replace-inventory-devices-title

mgr/dashboard: cluster > hosts: replace inventory->devices table title

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #39882 from tchaikov/wip-osd-watch

osd/Watch: use vector<> instead of list<>

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39897 from Rethan/clean-msgr-type

msg/Messenger: remove unused variable r

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39937 from tchaikov/wip-49661

mgr: add mon metada using type of "mon"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39541 from pcuzner/config-checker

mgr/cephadm: Provide an integrated configuration validation feature

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39905 from tchaikov/wip-lock-deps

common/lockdep: increase MAX_LOCKS to 128k

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #39503 from rhcs-dashboard/fix-ssl-certificate-code

mgr/dashboard: fix dashboard instance ssl certificate functionality

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39599 from pcuzner/cephadm-with-lsm

mgr/cephadm:Enable cephadm device scan to use libstoragemgmt

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #39924 from Daniel-Pivonka/osdrmstoptraceback

mgr/cephadm: prevent traceback when invalid osd id passed to 'orch osd rm stop'

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph.spec: include ninja as BuildRequires

to ease the build for developers using SUSE, Fedora, CentOS or RHEL.
so install-deps.sh can install ninja for them.

Fixes: https://tracker.ceph.com/issues/49694
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39828 from athanatos/sjust/wip-seastore-omap2

crimson/seastore: start wiring up collection_manager, omap_manager, onode_manager

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/.../staged-fltree/value.cc: return ValueDeltaRecorder for ONODE

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/seastore/onode_tree: add FLTreeOnodeManager unittest

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../fltree_onode_manager: fill in delta recorder

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore.cc: wire up omap_manager

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/omap_manager: generalize omap_list to support lower_bound

The iterator interface needs both lower and upper bound, generalize list()
to support both.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os: remove FuturizedStore::OmapIterator::tail_key

This doesn't actually make sense for FuturzizedStore implementations other
than BlueStore, and then only in a diagnostic capacity.  It also doesn't
have any actual users at the moment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: remove create_ephemeral

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: wire up FLTreeOnodeManager and FlatCollectionManager

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: introduce FLTreeOnodeManager

Adjusts OnodeManager interface and introduces FLTreeOnodeManager
implementation.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: remove simple-fltree

Doesn't actually have a TransactionManager implementation, and
it needs substantial changes to adapt to changes in Onode.
We can resurrect it in the future if we want to.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39982 from tchaikov/wip-doc-theme

doc: fine tune the theme, fix formatting in mount.ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/man/8/mount.ceph: fix the formatting

* fix the indent of item
* add an empty line before code block

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/start: fix the link pointing to ceph doc

use https instead of http, the latter cannot be redirected to the valid
docs. also, we don't have "next" branch, so remove the document
mentioning it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_themes: fine tune styling

* fix the "dl" (definition list) display in manpage. before this
  change, if an item in dl is composed of strings with different fonts,
  it is cluttered into a single string. after this change, they are
  separated with proper spacing as before. see the "WORKLOAD" section in
  https://docs.ceph.com/en/latest/man/8/ceph-syn/ and
  https://docs.ceph.com/en/octopus/man/8/ceph-syn/
* reduce the spacing in "ul" (unordered list) in table. before
  this change, we have large spacing at end of a list in table. after
  this change, the spacing is reduced. see
  https://docs.ceph.com/en/latest/start/hardware-recommendations/#ram
  https://docs.ceph.com/en/octopus/start/hardware-recommendations/#ram
* increase the spacing between two "ul"s in table. see
  https://docs.ceph.com/en/latest/start/hardware-recommendations/#Processor
  https://docs.ceph.com/en/octopus/start/hardware-recommendations/#ram

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: remove step to restart the mgr

a restart of the mgr does not appear to be necessary
after a `ceph config-key set ...`

Signed-off-by: Michael Fritch <mfritch@suse.com>

doc/cephadm: use `reconfig` instead of `redeploy`

`reconfig` can be used to apply a change to either
the tls/ssl cert or a custom configuration file (j2)

Signed-off-by: Michael Fritch <mfritch@suse.com>

doc/cephadm: update custom j2 config-key name

introduced by:
cd79c9912ab35ee6296d613edc7830410a141e05

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39981 from xxhdx1985126/wip-obc-tri_mutex-bug

crimson/osd: fix two obc lock related bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: use excl_lock_from_excl when promoting tri_mutex from excl to excl

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: acquire locks strictly by OSDOp type

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39826 from tchaikov/wip-ninja

run-make: switch over to ninja-build

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39946 from TheJJ/fix-ceph-crash-caps

doc/mgr/crash: explain needed crash upload permissions

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #39955 into master

* refs/pull/39955/head:
mgr/cephadm: do not prime service cache on reconfig

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #37711 into master

* refs/pull/37711/head:
qa/cephfs: create class LocalCephFSMount
vstart_runner: reuse FuseMount methods in LocalFuseMount
vstart_runner: reuse KernelMount methods in LocalKernelMount
qa/cephfs: move common and generic methods mount.py
qa/cephfs: refactor FuseMount.mount method
qa/cephfs: refactor KernelMount.mount method

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37945 into master

* refs/pull/37945/head:
mds: add opened files/inodes and pinned i_caps metric payload support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39408 into master

* refs/pull/39408/head:
test: add test for validating cephfs-mirror daemon service status
pybind/mirroring: interface to fetch mirror daemon status
cephfs-mirror: register mirror daemon with service manager
cephfs-mirror: remove unneeded json_spirit header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39502 into master

* refs/pull/39502/head:
qa: add sleep for blocklisting to take effect

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #39687 into master

* refs/pull/39687/head:
tools/cephfs-shell: continue file listing even on error
tools/cephfs-shell: refactoring of code related listing
tools/cephfs-shell: fix listing of symbolic links

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39814 into master

* refs/pull/39814/head:
qa/cephfs: move is_blocklisted() to filesystem.CephCluster

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

python-common: continue to allow RGWSpec(realm=r,zone=z)

This is for backward compatibility: an octopus spec yaml can still be
applied to an existing cluster.  Note that it might not work on a new
cluster, since cephadm no longer tries to create the realm or zone if
they don't exist.

Signed-off-by: Sage Weil <sage@newdream.net>

PendingReleaseNodes: note changes in cephadm rgw behavior

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: drop realm.zone convention for rgw

Note that cephadm.py will no longer do anything with rgw realms and
zones.  That means that the setup of rgw roles here is only useful
for the default zone and a non-multisite config.

Signed-off-by: Sage Weil <sage@newdream.net>

doc: update docs

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: do not prime service cache on reconfig

Ceph daemon reconfig does not need any daemon state refresh since we don't
do a restart--we just rewrite the ceph.conf.  This also avoids priming
our cache with a 'starting' state when the daemon wasn't touched.

Fixes: https://tracker.ceph.com/issues/49675
Signed-off-by: Sage Weil <sage@newdream.net>

debian/ceph-common.postinst: do not chown cephadm log dirs

The container uid/gid is different than the debian uid/gid (because the
container is centos-based and we got a different uid/gid allocation there).

Fixes: https://tracker.ceph.com/issues/49677
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39948 from tchaikov/wip-clang-tidy

crimson,common: cleanups inspired by clang-tidy

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>