mgr: add lookup_release_name(int) to mgr interface

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: verify container image version after we pull it

Apply the version checks to make sure this upgrade/downgrade is possible.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: only save version portion of version string

We don't want to 'ceph version ' prefix or sha1 suffix.  This matches us
up with 'orch upgrade start'

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: fix 'inspect' and 'pull'

With podman 2.2.1 I would get

ERRO[0000] Error printing inspect output: template: all inspect:1: function "json" not defined

Removing the 'json' portion of the query resolves it.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/upgrade: implement N-2 version checks on upgrade start

- Prevent major upgrades that span > 2 releases
- Prevent downgrades to an rc or dev release

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #38805 from kamoltat/wip-mgr-pg-autoscaler-improve-out-of-box-experience

pybind/mgr/pg_autoscaler: avoid scale-down until there is pressure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/pg_autoscaler: avoid scale-down until there is pressure

The autoscaler will start out with scaling each
pools to have a full complements of pgs from the start
and will only decrease it when pools need more due to
increased usage.

Introduced a unit test that tests only the
function get_final_pg_target_and_ratio() which
deals with the distrubtion of pgs amongst the
pools

Edited workunit script to reflect the change
of how pgs are calculated and distrubted.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #39200 from mitsu-ko/mypatch

doc/mgr/orchestrator: Unify the content of command and yaml

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39201 from tchaikov/wip-sphinx-openapi

doc: silence warnings from openapi sphinx extension

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39194 from tchaikov/wip-dashboard-enum

pybind/mgr/dashboard: do not install enum34

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

doc/mgr/orchestrator: Unify the content of command and yaml

Signed-off-by: Mitsumasa KONDO <kondo.mitsumasa@gmail.com>

doc: silence warnings from openapi sphinx extension

it prints out

skipping non-JSON example generation.

if an API does not contains example while ":example:" is specified for
openapi extension, and if this API is not a GET request, openapi
extention complains.

see also
https://github.com/sphinx-contrib/openapi/blob/9dbae9c9a65291c44b27041c63ded1b20611a2fc/sphinxcontrib/openapi/openapi30.py#L191

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39198 from tchaikov/wip-mypy.ini

mypy.ini: fix the unresolve conflict

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39196 from badone/wip-doc-python3-api

doc/rados/api: Update code samples to python3

Reviewed-by: Zac Dover <zac.dover@gmail.com>

mypy.ini: fix the unresolved conflict

it was introduced by a0930a63106e9f0d556e18bd70ffca6566388742, i failed
to resolve this conflict when resolving the merge confliction.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/api: Update code samples to python3

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #39106 from mgfritch/cephadm-apt-get

cephadm: use `apt-get` for package install/update

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39131 from sebastian-philipp/rook-disable-untyped

mgr/rook: disallow_untyped_defs = True

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

pybind/mgr/dashboard: do not install enum34

enum was introduced by python 3.4, but we require python3.6 and up.
so drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add rook_client._helper to mock_imports

This is due to `mgr/rook` now also includes `._helper`

Co-authored-by Kefu Chai <kchai@redhat.com>

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: disallow_untyped_defs = True

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Fix possibly infinite loop in _execute_blight_job

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39143 from adk3798/ha-rgw-data-dir

cephadm: fix get_data_dir calls for ha-rgw daemons

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge pull request #39081 from ifed01/wip-ifed-perpg-instrumentation

os/bluestore: be more verbose when fsck detects stray per-pg omaps

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39095 from tchaikov/wip-cephadm-fsid

cephadm: add fsid if --name is not specified

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39056 from sebastian-philipp/alerts-percent-d

mgr/alerts: interval can be None

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38930 from vbashkirtsev/master

client/fuse_ll.cc: use uint64_t for fuse_ll_forget() nlookup argument

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39132 from rzarzynski/wip-blk-ptr_node-for-aio

blk: avoid temporary bptrs on aio paths; use ptr_node instead.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39151 from batrick/radosstriper-refactor

libradosstriper: make default layout const and static

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39116 from tchaikov/wip-crimson-pgstat

crimson/osd: keep a local copy of pg_stat and cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39158 from tchaikov/wip-mgr-snap-stat-ann

pybind/mgr/{snap-schedule,status,iostat}: add type annotations

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39187 from tchaikov/wip-standalone-mon-misc.sh

qa/standalone/mon/misc: verify that len(monmap.features.persistent) == 9

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #39097 into master

* refs/pull/39097/head:
qa/workunits/cephadm/test_cephadm: fix argument
qa/workunits/cephadm/test_cephadm: show exporter output
qa/workunits/cephadm/test_cephadm: don't bother pulling octopus
qa/workunits/cephadm/test_cephadm: behave on terminal
qa/workunits/cepham/test_cephadm: pull only recent images, from quay.ceph.io
qa/workunits/cephadm/test_cephadm: don't respawn for python versions

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #39172 into master

* refs/pull/39172/head:
mgr/orchestrator: remove 'host' arg for 'orch ls'

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #39173 into master

* refs/pull/39173/head:
mgr/devicehealth: make CLI commands error when pool doesn't exist
mgr/devicehealth: only create pool when we have some osds

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/standalone/mon/misc: verify that len(monmap.features.persistent) == 9

in beb62c029abcf1d596ed624d58cbcb094ee6a292, FEATURE_QUINCY was added to
ceph::features::mon::get_persistent(), so update the test accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38892 from ceph/update_s3select_add_zero_segment_log

s3select: update s3select submodule to last commit ; add zero segment log

Merge pull request #38767 from myoungwon/wip-manifest-chunk-scrub

osd: add has_manifest_chunk to count chunks in snapshot

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #39039 into master

* refs/pull/39039/head:
src/test/cli/monmaptool: adjust for new monmap features
qa/tasks/cephadm: allow custom git_url for cephadm_branch pull
qa/suites/rados/upgrade: include pacific-x
qa/suites/upgrade/pacific-x/parallel
qa/suites: some clean up for quincy
mon: updates for quincy
mon: update for quincy ondisk features
script: add pacific
doc/dev/release-checklist: we tagged v17.0.0
ceph-volume: change to quincy
include/ceph_features: retire MON_SINGLE_PAXOS
include/ceph_features: define FEATURE_SERVER_QUINCY
mon/MgrMonitor: add always_on_modules for quincy
add feature/release name quincy
kickoff v17 quincy
doc/dev/release-checklists: uncheck everything!

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/workunits/cephadm/test_cephadm: fix argument

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #37144 from jmolmo/storage_devices_mgmt

doc: High level workflow for storage devices

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

src/test/cli/monmaptool: adjust for new monmap features

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: allow custom git_url for cephadm_branch pull

Normally the git_url is git://git.ceph.com/ceph-ci.git, which mirrors
upstream ceph-ci.git.  However, the release branches aren't present there.
Allow a custom git_url so we can pull these from the main ceph.git.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/devicehealth: make CLI commands error when pool doesn't exist

This is better than silently failing to do anything.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/devicehealth: only create pool when we have some osds

If we create the pool before we have OSDs, the PGs won't be active+clean
and we'll raise a health warning that we shouldn't.  This will annoy
a new users deploying a new cluster, and (more importantly?) make
qa tests fail while deploying initial clusters due to the health
warning.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39025 from yuvalif/wip-yuval-fix-48963

rgw: fix valgrind errors when protected_fixedsize_stack is used

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/orchestrator: remove 'host' arg for 'orch ls'

The host arg is unused, and also missing from the documentation.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39152 from neha-ojha/wip-46323

qa/*/thrash_cache_writeback_proxy_none.yaml: disable writeback overlay tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39118 from tchaikov/wip-cmake-rook-client

cmake: install rook-client-python using ExternalProject

Reviewed-by: Sebastian Wagner <swagner@suse.com>

pybind/mgr/snap-schedule: silence flake8 warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/snap-schedule: use use_fs if fs is not specified

the "vol_name" parameter of "fs subvolume getpath" command is not
optional, so we have to pass an fs name to it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/snap-schedule: close a string with single quote

if it starts with single quote, we should close it with a single quote
also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/snap-schedule: add typing annotations

please note, the behavior of `updates_schedule_db()` is changed so that
it now returns the return value of the decorated function. this change
makes it a real decorator which keeps the signature of the decorated
function unchanged. the reason why we need this change is that mypy does
not allow us to match the parameter pack of a Callable at the time of
writing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: flake8 cleanup

* remove unused import
* wrap long lines
* remove comments which do not start with "# "

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: always map 'mds_version' to a list of dict

more consistent this way, as json format is supposed to be consumed by a
parser not human being.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: construct defaultdict without key

it's simply wrong to construct defaultdict with the key which does not
have the mapped value, what we should do is to specify the
"default_factory" which construct the default value whose key does not
exist in the dict when being accessed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: cast return value to int

the return value of get_rate() is fed to mgr_util.format_dimless(),
which always cast the formatted value to int.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: correct a MgrModule annotation

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: use CLIReadCommand for defining commands

simpler this way. and will be more consistent with other mgr modules

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/iostat: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: silence flake8 warnings

silence following warnings from flake8

* line too long (107 > 100 characters) [E501]

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38686 from wornet-mwo/wip-doc-add-section-declarative-orch-cmd

doc: added section about retival of Service Specication

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38875 from pcuzner/new-host-to-maintenance

mgr/orchestrator: add maintenance mode for new hosts

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39083 from genesiscloud/orch-non-rotational-filter-fix

python-common/drivegroups: avoid dropping "rotational: 0" from Device Selection

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39007 from mgfritch/cephadm-podman-2.0

 cephadm: require podman >= 2.0.0

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39157 from xxhdx1985126/wip-crimson-bug-fix

crimson/osd: capture "intervals_included" by value in ReplicatedRecoveryBackend::submit_push_data

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39057 from aclamk/wip-mempool-shards-check-random

common/mempool: Improve mempool shard selection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crimson/osd: capture "intervals_included" by value in ReplicatedRecoveryBackend::submit_push_data

ReplicatedRecoveryBackend::submit_push_data()'s parameter "intervals_included" might be an
local varaiable in the outer method, so we should capture it by value to avoid a dangling
reference in the inner lambda

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

diagnose zero segment while s3select

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
using s3-test with the fix fix_test_file_permission_add_attrib

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
update s3select{1ea37ae459f957786b9d198510637dfa12f83350} ; add log tracing zero segments locations

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
s3select submodule update

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
fix comments

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
updating the s3select new features; covering the gap between latest documentation and latest faetures update

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
add some more examples; fixes

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
add between expression to doc; typo

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
typo

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
skipping empty segments

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
skip empty segments to the next segments

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

qa/*/thrash_cache_writeback_proxy_none.yaml: disable writeback overlay tests

thrash_cache_writeback_proxy_none tests have been failing consistently. Some investigation
shows that the writeback overlay tests are reponsible for it. Instead of removing these
cache tiering tests entirely, we'll disable them for now.

Related to: https://tracker.ceph.com/issues/46323
Signed-off-by: Neha Ojha <nojha@redhat.com>

libradosstriper: make default layout const and static

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39112 from pcuzner/fix-gather-facts

cephadm:add missing kernel_security property

qa/workunits/cephadm/test_cephadm: show exporter output

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/rados/upgrade: include pacific-x

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/upgrade/pacific-x/parallel

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites: some clean up for quincy

- remove upgrades from nautilus
- stubs for completing upgrade to quincy

Still missing the pacific-x upgrade tests.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: updates for quincy

Signed-off-by: Sage Weil <sage@newdream.net>

mon: update for quincy ondisk features

Signed-off-by: Sage Weil <sage@newdream.net>

script: add pacific

Signed-off-by: Sage Weil <sage@newdream.net>

doc/dev/release-checklist: we tagged v17.0.0

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-volume: change to quincy

Signed-off-by: Sage Weil <sage@newdream.net>

include/ceph_features: retire MON_SINGLE_PAXOS

X-3 = Q-3 = Nautilus

Signed-off-by: Sage Weil <sage@newdream.net>

include/ceph_features: define FEATURE_SERVER_QUINCY

Signed-off-by: Sage Weil <sage@newdream.net>

mon/MgrMonitor: add always_on_modules for quincy

Signed-off-by: Sage Weil <sage@newdream.net>

add feature/release name quincy

Signed-off-by: Sage Weil <sage@newdream.net>

kickoff v17 quincy

Signed-off-by: Sage Weil <sage@newdream.net>

doc/dev/release-checklists: uncheck everything!

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: fix get_data_dir calls for ha-rgw daemons

calls need to use the context's data dir rather than daemon type
puts these calls in line with the other daemons

Signed-off-by: Adam King <adking@redhat.com>

mgr/alerts: type checking

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/alerts: Avoid suspicious error message

... in case `interval` is not an integer.

Fixes: https://tracker.ceph.com/issues/48972
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/osd: use spaces for indent

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: pass function using rvalue reference

instead of passing a copy, pass a reference. so we can save the overhead
of copying the function object.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PeeringState: document prepare_stats_for_publish()

document its parameters

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd,crimson: pass optional<pg_stat_t> to prepare_stats_for_publish()

to simplify the interface. also save the overhead of passing an empty
pg_stat_t if the existing one is invalidated before calling
prepare_stats_for_publish().

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PeeringState: log epochs in pre_publish

does not make sense to print log the epochs in pg_stats_publish passed
in if it is invalid.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: keep a local copy of pg_stat

so we don't need to prepare it every time when sending pg stats to
mgr.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install rook-client-python using ExternalProject

so we don't need to rerun the generate_rook_ceph_client.sh script
everytime when building the script. cmake creates a stamp file for
tracking the dependencies and the time of modification of dependencies.

Signed-off-by: Kefu Chai <kchai@redhat.com>

blk: avoid temporary bptrs on aio paths; use ptr_node instead.

This is a slight optimization for the `HAVE_LIBAIO` paths
of the kernel-based `BlockDevice` implementation.
The overall idea is to squeeze temporary, short-living
instances of `ceph::bufferptr` as `ceph::bufferlist`
actually aggregates `ptr_node` (`bufferptr` with the extra
`next` pointer field to form a list). It can be created
directly and this commit switches to exactly this behavior.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>