cmake/rgw: forward spawn's compile options to rgw_common object library

since rgw_common is an OBJECT library, we can't use
target_link_libraries() for its dependency on spawn. we add its
include directories manually already with
$<TARGET_PROPERTY:spawn,INTERFACE_INCLUDE_DIRECTORIES>, but this didn't
pull in the compile definitions. this ultimately prevented the
WITH_BOOST_VALGRIND option from passing the BOOST_USE_VALGRIND
definition attached to boost::context

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: remove unnecessary target_include_directories() for spawn

targets that depend on spawn via target_link_libraries() automatically
pull in their compile options and include directories

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39274 from yuriw/wip-yuriw-crontab-master

qa/tests: removed ceph-ansible (master and pacific)

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/tests: removed ceph-ansible (master and pacific)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #38687 into master

* refs/pull/38687/head:
ceph-volume: mock VGs with string properties in tests
ceph-volume: add some flexibility to bytes_to_extents

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #39186 from tchaikov/wip-mgr-telemetry-ann

pybind/mgr: add typing annotation

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #39260 from idryomov/wip-expand-randomized-suite-doc

qa: expand the documentation on the '$' suite feature

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37954 from joscollin/wip-F46226-log-dq-throttle-limit-new

msg,mon,common: log when DispatchQueue throttle limit is reached

Merge PR #39069 into master

* refs/pull/39069/head:
mgr/cephadm/upgrade: tolerate pre-pacific upgrade state
mgr/cephadm/upgrade: scale down MDS cluster(s) for major version upgrades
mgr/cephadm: fix capitalization, level; drop elipses of log msgs
mgr/cephadm/upgrade: match against any repo_digest, not image_id
cephadm: return repo_digests (plural) in pull/inspect output
mgr/cephadm: include container_image_digests in inventory
cephadm: include image_digests list in 'ls' output
vstart.sh: only extract first container digest
mgr/cephadm: move release -> major translation to helper
mgr/cephadm/upgrade: tolerate old upgrade_state.target_versoin
mgr/cephadm/upgrade: set require-osd-release when done with OSDs
mgr: add lookup_release_name(int) to mgr interface
mgr/cephadm: verify container image version after we pull it
mgr/cephadm: only save version portion of version string
cephadm: fix 'inspect' and 'pull'
mgr/cephadm/upgrade: implement N-2 version checks on upgrade start

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39234 from neha-ojha/wip-49064

qa/*/test_envlibrados_for_rocksdb: use osd_client_message_cap to prevent slow requests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: expand the documentation on the '$' suite feature

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

pybind/mgr/hello: use f-string when appropriate

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/crash: flake8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/crash: add type annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/crash: define options using Option

more type safe this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/crash: define commands using CLICommand

more type safe this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38955 from varshar16/wip-doc-update-nfs-ganesha-dynamic

doc/cephfs/nfs: Add rook pod restart note, export and log block example

pybind/mgr/alerts: define options using Option

more type safe this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: PEP8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/alerts: use CLIReadCommand to define commands

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: PEP8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/iostat: PEP8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/devicehealth: PEP8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/insights: PEP8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/prometheus: PEP8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: use CLICommand to define commands

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: silence flake8 warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: define substitution for autopep8 params

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: remove duplicated dict entry

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: remove duplicated method

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: use Option to define options

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/hello: add typing annotation

also, use Option and CLIReadCommand to define options and cli
commands. this module serves as a "hello world" example for
developers of mgr modules. so it's important to use the more
convenient and safer way to implement the module

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38911 from pcuzner/add-cluster-network-to-bootstrap

cephadm: Add cluster network to bootstrap

Merge pull request #39239 from athanatos/sjust/wip-49114

crimson/os/seastore: fix staged-fltree build problems

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/.../staged-fltree: propogate eagain in node, value

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../staged-fltree: fix tree_cursor_t::(trim|extend)_value

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #39206 into master

* refs/pull/39206/head:
mgr/MetricTypes: condition encoding on feature bits

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #38924 from cyx1231st/wip-seastar-onode-tree-onode_t

crimson/onode-staged-tree: implement an extensive Value framework

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/*/test_envlibrados_for_rocksdb: use osd_client_message_cap to prevent slow requests

EnvLibradosMutipoolTest.DBBulkLoadKeysInRandomOrder can overload OSDs and cause
heartbeat timeouts. Tests in test_envlibrados_for_rocksdb also generate slow
requests on OSDs. Use osd_client_message_cap to prevent this.
Since this option is disabled by default, this may be a good way to exercise it.

Fixes: https://tracker.ceph.com/issues/49064
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39209 from idryomov/wip-unbreak-rbd-task

qa/tasks/rbd: fix regressions introduced with rbd encryption support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39212 into master

* refs/pull/39212/head:
cpatch: python 3.6 -> 3.8
cpatch: new default base image

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39207 from zdover23/wip-doc-removing-workbench-docs

doc/dev: remove workbench sections

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39227 from tchaikov/wip-48825

mgr/zabbix: format ceph.[{#POOL},percent_used as float

Reviewed-by: Wido den Hollander <wido@widodh.nl>

Merge pull request #39225 from tchaikov/wip-crimson-seastore-dec-ref

crimson/os/seastore: don't use variadic template for dec_ref()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38050 from ifed01/wip-ifed-fsck-spanning-blob

os/bluestore: detect and fix "zombie" spanning blobs using fsck.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

doc/cephfs/nfs: Add rook pod restart note, export and log block example

Fixes: https://tracker.ceph.com/issues/48914
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/zabbix: format ceph.[{#POOL},percent_used as float

also, multiply it with 100, as we are using 1.0 for 100% here.

for more details regarding the mapping from value_type to its values.
see
https://www.zabbix.com/documentation/4.0/manual/api/reference/item/object

Fixes: https://tracker.ceph.com/issues/48825
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: detect and fix "zombie" spanning blobs using fsck.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

crimson/os/seastore: don't use variadic template for dec_ref()

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39041 from ifed01/wip-ifed-fix-48966

os/bluestore: fix a bug causing unexpected Onode's unpinned state.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38869 into master

* refs/pull/38869/head:
ceph-volume: Fix usage of is_lv

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #38975 from athanatos/sjust/wip-seastore-pipelines

seastore: update to support pipelined write operations

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix a bug causing unexpected Onode's unpinned state.

There could be a race for Onodes put() and get() methods:

put()(pinned, nref=3)
  int n = --nref; (nref = 2)
  if (n == 2) {
    ..
    std::lock_guard l(ocs->lock);
    ...
    pinned = pinned && nref > 2; (= false)
    ...                                     get()
    if (r) {                                ++nref; (=3)
      n = --nref; (nref = 2)                return;
    }
    ...
    return

As a result nref = 2, pinned = false which is wrong

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #39205 from rzarzynski/wip-crimson-configurable-alien-threads-num

crimson: make the number of alien threads configurable.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

cephadm:updates following bootstrap code clean up

Patch to align to new(ish) bootstrap code changes

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: updated doc text

The phrase "recommended"  has been removed.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: unit test updates for cluster-network handling

Tests updated to handle comma separated subnet lists

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: updated handling of cluster_network parameter

Parameter now supports a comma separated list of subnets and
detects the ipv4 or ipv6 setting on the cluster network to enable
osds to bind correctly.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: install doc updated to include cluster-network parameter

Install guide updated to include a description of the --cluster-network
parameter. The text also links to the complete definition for cluster-network
on the rados/configuration/network-config-ref page.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: unit test updates for new subnet function

Adds tests to validate the check_subnet function will work as
expected with various inputs

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: Add cluster network parameter to bootstrap cmd

Adds a --cluster-network parameter to the bootstrap cmd to
set the internal cluster network.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #38956 from zdover23/2021_Jan_19_release_notes_grammar

doc/PendingReleaseNotes: grammar and wording

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm/upgrade: tolerate pre-pacific upgrade state

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/upgrade: scale down MDS cluster(s) for major version upgrades

For octopus -> pacific, as with other recent releases, we need to scale
down the MDS cluster(s) to a single daemon before upgrading.  (This is
because the MDS intra-cluster protocols aren't fully versioned.)

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: fix capitalization, level; drop elipses of log msgs

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/upgrade: match against any repo_digest, not image_id

The image id can vary across hosts and (most notably) docker vs podman.
Instead, use the repo_digest as an image identifier.

Unfortunately, a single image may have multiple digests, even within the
same registry, so keep a list of the digests for the image we are
upgrading to, and ensure that each container has a digest that matches at
least one of them.

This allows upgrade to proceed in mixed docker+podman clusters.  However,
it does not yet address a cluster with mixed CPU architectures, because
the container image will have different digest(s) for each architecture
build.

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: return repo_digests (plural) in pull/inspect output

When we inspect a (pulled) image, return all of the repo digests.  Update
the mgr/cephadm code accordingly to match.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #39149 into master

* refs/pull/39149/head:
qa/workunits/cephtool/test.sh: tolerate some settling for osd info vs dump test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: include container_image_digests in inventory

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: include image_digests list in 'ls' output

Signed-off-by: Sage Weil <sage@newdream.net>

vstart.sh: only extract first container digest

Otherwise the container_image value includes the same hash twice, separated
by a newline, causing all sorts of confusion.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: move release -> major translation to helper

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/upgrade: tolerate old upgrade_state.target_versoin

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/upgrade: set require-osd-release when done with OSDs

After we've upgraded all of the osds, complete the major version upgrade
for core RADOS by setting the OSD cluster min.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr: add lookup_release_name(int) to mgr interface

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: verify container image version after we pull it

Apply the version checks to make sure this upgrade/downgrade is possible.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: only save version portion of version string

We don't want to 'ceph version ' prefix or sha1 suffix.  This matches us
up with 'orch upgrade start'

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: fix 'inspect' and 'pull'

With podman 2.2.1 I would get

ERRO[0000] Error printing inspect output: template: all inspect:1: function "json" not defined

Removing the 'json' portion of the query resolves it.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/upgrade: implement N-2 version checks on upgrade start

- Prevent major upgrades that span > 2 releases
- Prevent downgrades to an rc or dev release

Signed-off-by: Sage Weil <sage@newdream.net>

cpatch: python 3.6 -> 3.8

Signed-off-by: Sage Weil <sage@newdream.net>

cpatch: new default base image

Signed-off-by: Sage Weil <sage@newdream.net>

crimson/os/seastore/segment_cleaner: tolerate mark_space_used on closed segment

A journal segment can be closed while outstanding writes on that
segment have not yet updated metadata.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: tolerate journal rolls during commit

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add basic pipeline phases to TransactionManager

We need to ensure that the metadata preperation and completions happen
in order.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/operation: add OrderedConcurrentPhase

Signed-off-by: Samuel Just <sjust@redhat.com>

src/crimson/common: decouple handle from stage

We're going to introduce stages with other properties.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: fix find_hole

We actually need to propogate eagain errors now, fix to use do_until.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: annotate all assert_all instances

Signed-off-by: Samuel Just <sjust@redhat.com>

store-nbd: support concurrent operations

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: validate node immediately prior to adding pin

Mostly, operating on an invalid extent is harmless by construction --
it'll be caught by the read set check in submit_transaction.  However,
prior to adding an extent to the lba pin set, we need to check that it's
really still live, which we can do by checking that it and its immediate
parent are still valid.  Mutating the target extent would have
invalidated it, and replacing the target extent would necessarily have
involved a mutation to the parent, which would have invalidated it.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: expand error type for reads to include eagain

There are cases where reads may detect an invalid cached extent and require
the user to restart the transaction.

As a side effect, clean up errorators to assert cases where construction
is invalid (invalid refcounts, enoent or invarg on node read, etc).  In
such cases, we may eventually want to introduce an internal_error error
type, but asserting is fine for now as we do not expect users to handle
these cases.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/errorator: add extend_ertr helper for merging errorators

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: put root in t.mutated_extents and avoid special behavior

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/operation.h: permit moving Handle objects

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: add tests for concurrent IO

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent: add asserts for debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #36518 from yongseokoh/fix-dashboard

mgr/dashboard: Fix missing root path of each session for CephFS

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/MetricTypes: condition encoding on feature bits

Do not encode new metric|config types for older peers.

The combination of boost::optional and boost::variant make this super
awkward to condition on the features. :(

Fixes: https://tracker.ceph.com/issues/49069
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #38805 from kamoltat/wip-mgr-pg-autoscaler-improve-out-of-box-experience

pybind/mgr/pg_autoscaler: avoid scale-down until there is pressure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>