Merge PR #39212 into master

* refs/pull/39212/head:
cpatch: python 3.6 -> 3.8
cpatch: new default base image

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39207 from zdover23/wip-doc-removing-workbench-docs

doc/dev: remove workbench sections

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39227 from tchaikov/wip-48825

mgr/zabbix: format ceph.[{#POOL},percent_used as float

Reviewed-by: Wido den Hollander <wido@widodh.nl>

Merge pull request #39225 from tchaikov/wip-crimson-seastore-dec-ref

crimson/os/seastore: don't use variadic template for dec_ref()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38050 from ifed01/wip-ifed-fsck-spanning-blob

os/bluestore: detect and fix "zombie" spanning blobs using fsck.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

mgr/zabbix: format ceph.[{#POOL},percent_used as float

also, multiply it with 100, as we are using 1.0 for 100% here.

for more details regarding the mapping from value_type to its values.
see
https://www.zabbix.com/documentation/4.0/manual/api/reference/item/object

Fixes: https://tracker.ceph.com/issues/48825
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: detect and fix "zombie" spanning blobs using fsck.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

crimson/os/seastore: don't use variadic template for dec_ref()

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39041 from ifed01/wip-ifed-fix-48966

os/bluestore: fix a bug causing unexpected Onode's unpinned state.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38869 into master

* refs/pull/38869/head:
ceph-volume: Fix usage of is_lv

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #38975 from athanatos/sjust/wip-seastore-pipelines

seastore: update to support pipelined write operations

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix a bug causing unexpected Onode's unpinned state.

There could be a race for Onodes put() and get() methods:

put()(pinned, nref=3)
  int n = --nref; (nref = 2)
  if (n == 2) {
    ..
    std::lock_guard l(ocs->lock);
    ...
    pinned = pinned && nref > 2; (= false)
    ...                                     get()
    if (r) {                                ++nref; (=3)
      n = --nref; (nref = 2)                return;
    }
    ...
    return

As a result nref = 2, pinned = false which is wrong

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #39205 from rzarzynski/wip-crimson-configurable-alien-threads-num

crimson: make the number of alien threads configurable.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #38956 from zdover23/2021_Jan_19_release_notes_grammar

doc/PendingReleaseNotes: grammar and wording

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39149 into master

* refs/pull/39149/head:
qa/workunits/cephtool/test.sh: tolerate some settling for osd info vs dump test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cpatch: python 3.6 -> 3.8

Signed-off-by: Sage Weil <sage@newdream.net>

cpatch: new default base image

Signed-off-by: Sage Weil <sage@newdream.net>

crimson/os/seastore/segment_cleaner: tolerate mark_space_used on closed segment

A journal segment can be closed while outstanding writes on that
segment have not yet updated metadata.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: tolerate journal rolls during commit

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add basic pipeline phases to TransactionManager

We need to ensure that the metadata preperation and completions happen
in order.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/operation: add OrderedConcurrentPhase

Signed-off-by: Samuel Just <sjust@redhat.com>

src/crimson/common: decouple handle from stage

We're going to introduce stages with other properties.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: fix find_hole

We actually need to propogate eagain errors now, fix to use do_until.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: annotate all assert_all instances

Signed-off-by: Samuel Just <sjust@redhat.com>

store-nbd: support concurrent operations

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: validate node immediately prior to adding pin

Mostly, operating on an invalid extent is harmless by construction --
it'll be caught by the read set check in submit_transaction.  However,
prior to adding an extent to the lba pin set, we need to check that it's
really still live, which we can do by checking that it and its immediate
parent are still valid.  Mutating the target extent would have
invalidated it, and replacing the target extent would necessarily have
involved a mutation to the parent, which would have invalidated it.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: expand error type for reads to include eagain

There are cases where reads may detect an invalid cached extent and require
the user to restart the transaction.

As a side effect, clean up errorators to assert cases where construction
is invalid (invalid refcounts, enoent or invarg on node read, etc).  In
such cases, we may eventually want to introduce an internal_error error
type, but asserting is fine for now as we do not expect users to handle
these cases.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/errorator: add extend_ertr helper for merging errorators

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: put root in t.mutated_extents and avoid special behavior

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/operation.h: permit moving Handle objects

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: add tests for concurrent IO

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent: add asserts for debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #36518 from yongseokoh/fix-dashboard

mgr/dashboard: Fix missing root path of each session for CephFS

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38805 from kamoltat/wip-mgr-pg-autoscaler-improve-out-of-box-experience

pybind/mgr/pg_autoscaler: avoid scale-down until there is pressure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/PendingReleaseNotes: grammar and wording

This commit vets the PendingReleaseNotes for Octopus.
This might not be the only commit that updates the
release notes, but it is the first by me.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc:dev remove workbench sections

This PR removes the following two sections:

1. Getting Ceph Workbench
2. Linking Ceph Workbench With Your OpenStack Tenant

Fixes: https://tracker.ceph.com/issues/48942
Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson: make the number of alien threads configurable.

It's particularly important for reads to have large-enough
thread pool in `AlienStore` as they are synchronous; that
is, e.g. `BlueStore` blocks on IO when handling reads.

This commit introduces a configurable allowing operators
to increase the number from `1` which we were limited to
before the change.

Naming similarity with `osd_op_num_threads_per_shard` is
intentional.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/pg_autoscaler: avoid scale-down until there is pressure

The autoscaler will start out with scaling each
pools to have a full complements of pgs from the start
and will only decrease it when pools need more due to
increased usage.

Introduced a unit test that tests only the
function get_final_pg_target_and_ratio() which
deals with the distrubtion of pgs amongst the
pools

Edited workunit script to reflect the change
of how pgs are calculated and distrubted.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #39200 from mitsu-ko/mypatch

doc/mgr/orchestrator: Unify the content of command and yaml

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39201 from tchaikov/wip-sphinx-openapi

doc: silence warnings from openapi sphinx extension

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39194 from tchaikov/wip-dashboard-enum

pybind/mgr/dashboard: do not install enum34

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

doc/mgr/orchestrator: Unify the content of command and yaml

Signed-off-by: Mitsumasa KONDO <kondo.mitsumasa@gmail.com>

doc: silence warnings from openapi sphinx extension

it prints out

skipping non-JSON example generation.

if an API does not contains example while ":example:" is specified for
openapi extension, and if this API is not a GET request, openapi
extention complains.

see also
https://github.com/sphinx-contrib/openapi/blob/9dbae9c9a65291c44b27041c63ded1b20611a2fc/sphinxcontrib/openapi/openapi30.py#L191

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39198 from tchaikov/wip-mypy.ini

mypy.ini: fix the unresolve conflict

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39196 from badone/wip-doc-python3-api

doc/rados/api: Update code samples to python3

Reviewed-by: Zac Dover <zac.dover@gmail.com>

mypy.ini: fix the unresolved conflict

it was introduced by a0930a63106e9f0d556e18bd70ffca6566388742, i failed
to resolve this conflict when resolving the merge confliction.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/api: Update code samples to python3

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39106 from mgfritch/cephadm-apt-get

cephadm: use `apt-get` for package install/update

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39131 from sebastian-philipp/rook-disable-untyped

mgr/rook: disallow_untyped_defs = True

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

pybind/mgr/dashboard: do not install enum34

enum was introduced by python 3.4, but we require python3.6 and up.
so drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add rook_client._helper to mock_imports

This is due to `mgr/rook` now also includes `._helper`

Co-authored-by Kefu Chai <kchai@redhat.com>

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: disallow_untyped_defs = True

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Fix possibly infinite loop in _execute_blight_job

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39143 from adk3798/ha-rgw-data-dir

cephadm: fix get_data_dir calls for ha-rgw daemons

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge pull request #39081 from ifed01/wip-ifed-perpg-instrumentation

os/bluestore: be more verbose when fsck detects stray per-pg omaps

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39095 from tchaikov/wip-cephadm-fsid

cephadm: add fsid if --name is not specified

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39056 from sebastian-philipp/alerts-percent-d

mgr/alerts: interval can be None

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38930 from vbashkirtsev/master

client/fuse_ll.cc: use uint64_t for fuse_ll_forget() nlookup argument

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39132 from rzarzynski/wip-blk-ptr_node-for-aio

blk: avoid temporary bptrs on aio paths; use ptr_node instead.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39151 from batrick/radosstriper-refactor

libradosstriper: make default layout const and static

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39116 from tchaikov/wip-crimson-pgstat

crimson/osd: keep a local copy of pg_stat and cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39158 from tchaikov/wip-mgr-snap-stat-ann

pybind/mgr/{snap-schedule,status,iostat}: add type annotations

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39187 from tchaikov/wip-standalone-mon-misc.sh

qa/standalone/mon/misc: verify that len(monmap.features.persistent) == 9

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #39097 into master

* refs/pull/39097/head:
qa/workunits/cephadm/test_cephadm: fix argument
qa/workunits/cephadm/test_cephadm: show exporter output
qa/workunits/cephadm/test_cephadm: don't bother pulling octopus
qa/workunits/cephadm/test_cephadm: behave on terminal
qa/workunits/cepham/test_cephadm: pull only recent images, from quay.ceph.io
qa/workunits/cephadm/test_cephadm: don't respawn for python versions

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #39172 into master

* refs/pull/39172/head:
mgr/orchestrator: remove 'host' arg for 'orch ls'

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #39173 into master

* refs/pull/39173/head:
mgr/devicehealth: make CLI commands error when pool doesn't exist
mgr/devicehealth: only create pool when we have some osds

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/standalone/mon/misc: verify that len(monmap.features.persistent) == 9

in beb62c029abcf1d596ed624d58cbcb094ee6a292, FEATURE_QUINCY was added to
ceph::features::mon::get_persistent(), so update the test accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38892 from ceph/update_s3select_add_zero_segment_log

s3select: update s3select submodule to last commit ; add zero segment log

Merge pull request #38767 from myoungwon/wip-manifest-chunk-scrub

osd: add has_manifest_chunk to count chunks in snapshot

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #39039 into master

* refs/pull/39039/head:
src/test/cli/monmaptool: adjust for new monmap features
qa/tasks/cephadm: allow custom git_url for cephadm_branch pull
qa/suites/rados/upgrade: include pacific-x
qa/suites/upgrade/pacific-x/parallel
qa/suites: some clean up for quincy
mon: updates for quincy
mon: update for quincy ondisk features
script: add pacific
doc/dev/release-checklist: we tagged v17.0.0
ceph-volume: change to quincy
include/ceph_features: retire MON_SINGLE_PAXOS
include/ceph_features: define FEATURE_SERVER_QUINCY
mon/MgrMonitor: add always_on_modules for quincy
add feature/release name quincy
kickoff v17 quincy
doc/dev/release-checklists: uncheck everything!

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/workunits/cephadm/test_cephadm: fix argument

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #37144 from jmolmo/storage_devices_mgmt

doc: High level workflow for storage devices

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

src/test/cli/monmaptool: adjust for new monmap features

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: allow custom git_url for cephadm_branch pull

Normally the git_url is git://git.ceph.com/ceph-ci.git, which mirrors
upstream ceph-ci.git.  However, the release branches aren't present there.
Allow a custom git_url so we can pull these from the main ceph.git.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/devicehealth: make CLI commands error when pool doesn't exist

This is better than silently failing to do anything.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/devicehealth: only create pool when we have some osds

If we create the pool before we have OSDs, the PGs won't be active+clean
and we'll raise a health warning that we shouldn't.  This will annoy
a new users deploying a new cluster, and (more importantly?) make
qa tests fail while deploying initial clusters due to the health
warning.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39025 from yuvalif/wip-yuval-fix-48963

rgw: fix valgrind errors when protected_fixedsize_stack is used

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/orchestrator: remove 'host' arg for 'orch ls'

The host arg is unused, and also missing from the documentation.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39152 from neha-ojha/wip-46323

qa/*/thrash_cache_writeback_proxy_none.yaml: disable writeback overlay tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39118 from tchaikov/wip-cmake-rook-client

cmake: install rook-client-python using ExternalProject

Reviewed-by: Sebastian Wagner <swagner@suse.com>

pybind/mgr/snap-schedule: silence flake8 warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/snap-schedule: use use_fs if fs is not specified

the "vol_name" parameter of "fs subvolume getpath" command is not
optional, so we have to pass an fs name to it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/snap-schedule: close a string with single quote

if it starts with single quote, we should close it with a single quote
also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/snap-schedule: add typing annotations

please note, the behavior of `updates_schedule_db()` is changed so that
it now returns the return value of the decorated function. this change
makes it a real decorator which keeps the signature of the decorated
function unchanged. the reason why we need this change is that mypy does
not allow us to match the parameter pack of a Callable at the time of
writing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: flake8 cleanup

* remove unused import
* wrap long lines
* remove comments which do not start with "# "

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: always map 'mds_version' to a list of dict

more consistent this way, as json format is supposed to be consumed by a
parser not human being.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: construct defaultdict without key

it's simply wrong to construct defaultdict with the key which does not
have the mapped value, what we should do is to specify the
"default_factory" which construct the default value whose key does not
exist in the dict when being accessed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: cast return value to int

the return value of get_rate() is fed to mgr_util.format_dimless(),
which always cast the formatted value to int.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: correct a MgrModule annotation

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: use CLIReadCommand for defining commands

simpler this way. and will be more consistent with other mgr modules

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/iostat: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: silence flake8 warnings

silence following warnings from flake8

* line too long (107 > 100 characters) [E501]

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38686 from wornet-mwo/wip-doc-add-section-declarative-orch-cmd

doc: added section about retival of Service Specication

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38875 from pcuzner/new-host-to-maintenance

mgr/orchestrator: add maintenance mode for new hosts

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39083 from genesiscloud/orch-non-rotational-filter-fix

python-common/drivegroups: avoid dropping "rotational: 0" from Device Selection

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39007 from mgfritch/cephadm-podman-2.0

 cephadm: require podman >= 2.0.0

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39157 from xxhdx1985126/wip-crimson-bug-fix

crimson/osd: capture "intervals_included" by value in ReplicatedRecoveryBackend::submit_push_data

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39057 from aclamk/wip-mempool-shards-check-random

common/mempool: Improve mempool shard selection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crimson/osd: capture "intervals_included" by value in ReplicatedRecoveryBackend::submit_push_data

ReplicatedRecoveryBackend::submit_push_data()'s parameter "intervals_included" might be an
local varaiable in the outer method, so we should capture it by value to avoid a dangling
reference in the inner lambda

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>