Merge pull request #39116 from tchaikov/wip-crimson-pgstat

crimson/osd: keep a local copy of pg_stat and cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39158 from tchaikov/wip-mgr-snap-stat-ann

pybind/mgr/{snap-schedule,status,iostat}: add type annotations

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39187 from tchaikov/wip-standalone-mon-misc.sh

qa/standalone/mon/misc: verify that len(monmap.features.persistent) == 9

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #39097 into master

* refs/pull/39097/head:
qa/workunits/cephadm/test_cephadm: fix argument
qa/workunits/cephadm/test_cephadm: show exporter output
qa/workunits/cephadm/test_cephadm: don't bother pulling octopus
qa/workunits/cephadm/test_cephadm: behave on terminal
qa/workunits/cepham/test_cephadm: pull only recent images, from quay.ceph.io
qa/workunits/cephadm/test_cephadm: don't respawn for python versions

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #39172 into master

* refs/pull/39172/head:
mgr/orchestrator: remove 'host' arg for 'orch ls'

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #39173 into master

* refs/pull/39173/head:
mgr/devicehealth: make CLI commands error when pool doesn't exist
mgr/devicehealth: only create pool when we have some osds

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/standalone/mon/misc: verify that len(monmap.features.persistent) == 9

in beb62c029abcf1d596ed624d58cbcb094ee6a292, FEATURE_QUINCY was added to
ceph::features::mon::get_persistent(), so update the test accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38892 from ceph/update_s3select_add_zero_segment_log

s3select: update s3select submodule to last commit ; add zero segment log

Merge pull request #38767 from myoungwon/wip-manifest-chunk-scrub

osd: add has_manifest_chunk to count chunks in snapshot

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #39039 into master

* refs/pull/39039/head:
src/test/cli/monmaptool: adjust for new monmap features
qa/tasks/cephadm: allow custom git_url for cephadm_branch pull
qa/suites/rados/upgrade: include pacific-x
qa/suites/upgrade/pacific-x/parallel
qa/suites: some clean up for quincy
mon: updates for quincy
mon: update for quincy ondisk features
script: add pacific
doc/dev/release-checklist: we tagged v17.0.0
ceph-volume: change to quincy
include/ceph_features: retire MON_SINGLE_PAXOS
include/ceph_features: define FEATURE_SERVER_QUINCY
mon/MgrMonitor: add always_on_modules for quincy
add feature/release name quincy
kickoff v17 quincy
doc/dev/release-checklists: uncheck everything!

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/workunits/cephadm/test_cephadm: fix argument

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #37144 from jmolmo/storage_devices_mgmt

doc: High level workflow for storage devices

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

src/test/cli/monmaptool: adjust for new monmap features

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: allow custom git_url for cephadm_branch pull

Normally the git_url is git://git.ceph.com/ceph-ci.git, which mirrors
upstream ceph-ci.git.  However, the release branches aren't present there.
Allow a custom git_url so we can pull these from the main ceph.git.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/devicehealth: make CLI commands error when pool doesn't exist

This is better than silently failing to do anything.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/devicehealth: only create pool when we have some osds

If we create the pool before we have OSDs, the PGs won't be active+clean
and we'll raise a health warning that we shouldn't.  This will annoy
a new users deploying a new cluster, and (more importantly?) make
qa tests fail while deploying initial clusters due to the health
warning.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39025 from yuvalif/wip-yuval-fix-48963

rgw: fix valgrind errors when protected_fixedsize_stack is used

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/orchestrator: remove 'host' arg for 'orch ls'

The host arg is unused, and also missing from the documentation.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39152 from neha-ojha/wip-46323

qa/*/thrash_cache_writeback_proxy_none.yaml: disable writeback overlay tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39118 from tchaikov/wip-cmake-rook-client

cmake: install rook-client-python using ExternalProject

Reviewed-by: Sebastian Wagner <swagner@suse.com>

pybind/mgr/snap-schedule: silence flake8 warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/snap-schedule: use use_fs if fs is not specified

the "vol_name" parameter of "fs subvolume getpath" command is not
optional, so we have to pass an fs name to it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/snap-schedule: close a string with single quote

if it starts with single quote, we should close it with a single quote
also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/snap-schedule: add typing annotations

please note, the behavior of `updates_schedule_db()` is changed so that
it now returns the return value of the decorated function. this change
makes it a real decorator which keeps the signature of the decorated
function unchanged. the reason why we need this change is that mypy does
not allow us to match the parameter pack of a Callable at the time of
writing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: flake8 cleanup

* remove unused import
* wrap long lines
* remove comments which do not start with "# "

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: always map 'mds_version' to a list of dict

more consistent this way, as json format is supposed to be consumed by a
parser not human being.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: construct defaultdict without key

it's simply wrong to construct defaultdict with the key which does not
have the mapped value, what we should do is to specify the
"default_factory" which construct the default value whose key does not
exist in the dict when being accessed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: cast return value to int

the return value of get_rate() is fed to mgr_util.format_dimless(),
which always cast the formatted value to int.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: correct a MgrModule annotation

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: use CLIReadCommand for defining commands

simpler this way. and will be more consistent with other mgr modules

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/iostat: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: silence flake8 warnings

silence following warnings from flake8

* line too long (107 > 100 characters) [E501]

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38686 from wornet-mwo/wip-doc-add-section-declarative-orch-cmd

doc: added section about retival of Service Specication

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38875 from pcuzner/new-host-to-maintenance

mgr/orchestrator: add maintenance mode for new hosts

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39083 from genesiscloud/orch-non-rotational-filter-fix

python-common/drivegroups: avoid dropping "rotational: 0" from Device Selection

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39007 from mgfritch/cephadm-podman-2.0

 cephadm: require podman >= 2.0.0

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39157 from xxhdx1985126/wip-crimson-bug-fix

crimson/osd: capture "intervals_included" by value in ReplicatedRecoveryBackend::submit_push_data

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39057 from aclamk/wip-mempool-shards-check-random

common/mempool: Improve mempool shard selection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crimson/osd: capture "intervals_included" by value in ReplicatedRecoveryBackend::submit_push_data

ReplicatedRecoveryBackend::submit_push_data()'s parameter "intervals_included" might be an
local varaiable in the outer method, so we should capture it by value to avoid a dangling
reference in the inner lambda

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

diagnose zero segment while s3select

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
using s3-test with the fix fix_test_file_permission_add_attrib

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
update s3select{1ea37ae459f957786b9d198510637dfa12f83350} ; add log tracing zero segments locations

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
s3select submodule update

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
fix comments

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
updating the s3select new features; covering the gap between latest documentation and latest faetures update

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
add some more examples; fixes

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
add between expression to doc; typo

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
typo

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
skipping empty segments

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
skip empty segments to the next segments

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

qa/*/thrash_cache_writeback_proxy_none.yaml: disable writeback overlay tests

thrash_cache_writeback_proxy_none tests have been failing consistently. Some investigation
shows that the writeback overlay tests are reponsible for it. Instead of removing these
cache tiering tests entirely, we'll disable them for now.

Related to: https://tracker.ceph.com/issues/46323
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39112 from pcuzner/fix-gather-facts

cephadm:add missing kernel_security property

qa/workunits/cephadm/test_cephadm: show exporter output

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/rados/upgrade: include pacific-x

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/upgrade/pacific-x/parallel

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites: some clean up for quincy

- remove upgrades from nautilus
- stubs for completing upgrade to quincy

Still missing the pacific-x upgrade tests.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: updates for quincy

Signed-off-by: Sage Weil <sage@newdream.net>

mon: update for quincy ondisk features

Signed-off-by: Sage Weil <sage@newdream.net>

script: add pacific

Signed-off-by: Sage Weil <sage@newdream.net>

doc/dev/release-checklist: we tagged v17.0.0

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-volume: change to quincy

Signed-off-by: Sage Weil <sage@newdream.net>

include/ceph_features: retire MON_SINGLE_PAXOS

X-3 = Q-3 = Nautilus

Signed-off-by: Sage Weil <sage@newdream.net>

include/ceph_features: define FEATURE_SERVER_QUINCY

Signed-off-by: Sage Weil <sage@newdream.net>

mon/MgrMonitor: add always_on_modules for quincy

Signed-off-by: Sage Weil <sage@newdream.net>

add feature/release name quincy

Signed-off-by: Sage Weil <sage@newdream.net>

kickoff v17 quincy

Signed-off-by: Sage Weil <sage@newdream.net>

doc/dev/release-checklists: uncheck everything!

Signed-off-by: Sage Weil <sage@newdream.net>

crimson/osd: use spaces for indent

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: pass function using rvalue reference

instead of passing a copy, pass a reference. so we can save the overhead
of copying the function object.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PeeringState: document prepare_stats_for_publish()

document its parameters

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd,crimson: pass optional<pg_stat_t> to prepare_stats_for_publish()

to simplify the interface. also save the overhead of passing an empty
pg_stat_t if the existing one is invalidated before calling
prepare_stats_for_publish().

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PeeringState: log epochs in pre_publish

does not make sense to print log the epochs in pg_stats_publish passed
in if it is invalid.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: keep a local copy of pg_stat

so we don't need to prepare it every time when sending pg stats to
mgr.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install rook-client-python using ExternalProject

so we don't need to rerun the generate_rook_ceph_client.sh script
everytime when building the script. cmake creates a stamp file for
tracking the dependencies and the time of modification of dependencies.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39113 from liewegas/fix-datetime-test

python-common: fix test_datetime_to_str_2 on non-UTC hosts

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #39100 from xxhdx1985126/wip-crimson-bug-fix-2

crimson/osd: fix recovery related bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use do_with to capture MOSDPGPull::take_pulls()'s return

MOSDPGPull::take_pulls() returns a temporary object, so we need to use seastar::do_with
to create an object whose lives until seastar::do_with's inner continuation chain ends

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

rgw: fix valgrind errors when protected_fixedsize_stack is used

Fixes: https://tracker.ceph.com/issues/48963
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

crimson/osd: group stats related methods together

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: move PG::publish_stats_to_osd() to .cc

and move PG::get_stats() to .cc as well

for faster compilation

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: objects without omap header may not necessarily be a error

There are circumstances in which objects don't have omap headers, so this should
not be treated as an error

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: should capture seastar::do_with params by reference

The lambda within seastar::do_with need to use lvalue reference as its parameter,
otherwise it would be referencing a variable whose lifecycle ends when lambda finishes
runing

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39115 from wjwithagen/wjw-fix-QCOWFormat.cc

librbd: Fix variable declaration in capture list in lambda

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39108 from zdover23/wip-doc-leads-update-epuertat-28-Jan-2021

doc/dev: add Ernesto Puerta as dashboard lead

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39114 from liewegas/drop-rhel-8.0

qa/suites/rados/cephadm: drop rhel_8.0 tests

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #37949 from ideepika/wip-docs-update

doc/dev/developer_guide:  teuthology update developer guide

Reviewed-by: Neha Ojha <nojha@redhat.com>

librbd: Fix variable declaration in capture list in lambda

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa/suites/rados/cephadm/*/distro: drop rhel_8.0 targets

We still have centos 8.0.  We're missing an 8.0 image for the gibba.

Signed-off-by: Sage Weil <sage@newdream.net>

python-common: fix test_datetime_to_str_2 on non-UTC hosts

The old test parsed to a datetime without a tz, which was interpreted as
the local time zone when rendering back to a string.  Specify that it's a
UTC datetime so that behavior is consistent regardless of the test host
timezone.

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm:add missing kernel_security property

The propery decorator had gone missing which meant
the dump of host facts was missing the kernel security
(LSM) settings. This patch just adds the @property
decorator back

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #38861 from alimaredia/wip-rgw-create-bucket-dpp

rgw: Thread dpp through Create Bucket logging

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39105 from dillaman/wip-48989

librbd: tweaks to support Boost 1.75.0

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: Add DoutPrefixProvider to create_bucket logging

This commit hits all log messages for a create bucket
request and ensures each has a DoutPrefixProvider.

This commit gets dpp's all the way down to rgw/services
up way though to rgw_op.cc, rgw_rest.cc files,
rgw_admin.cc, etc.

This commit also removes store as dpp in rgw_sal.h

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #39068 from dillaman/wip-librbd-migration-qcow-2

librbd/migration: tweaks and unit tests for QCOW handler

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39107 from yuriw/wip-yuriw-crontab-master

qa/tests: reduced octopus, nautilus, master frequency

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39043 from tchaikov/wip-cephadm-call

cephadm: refactor call() using asyncio.asyncio.StreamReader

Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tests: changed priority - master 100, pacific 80

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/tests: reduced octopus, nautilus, master frequency

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #39015 from lixiaoy1/fix_lib

librbd/cmake: add library dependency for pwl_cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/dev: add Ernesto Puerta as dashboard lead

This commit removes Lenz Grimmer's name and github
handle from the list of component leads and replaces
it with Ernesto Puerta's name and github handle.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39101 from sebastian-philipp/cephadm-fix-ok-to-stop-return-value

mgr/cephadm: Fix wrong argument type to HandleCommandResult

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39098 from tchaikov/wip-zabbix-ann

mgr: add annotations to zabbix and devicehealth

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39065 from cbodley/wip-48988

cmake: boost>=1.74 adds BOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT to radosgw

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38786 from kotreshhr/evict-clients-of-deauthorized-authids

mgr/volumes: Evict clients based on auth-IDs and subvolume path

Reviewed-by: Victoria Martinez de la Cruz <victoria@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

librbd: tweaks to support Boost 1.75.0

Fixes: https://tracker.ceph.com/issues/48989
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: minor cleanup to QCOWFormat code

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37259 from soumyakoduri/head_req

rgw: Set CURLOPT_NOBODY for HEAD request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38982 from jmolmo/fix_iscsi_client_caps

cephadm: Fix iscsi client caps

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>