Merge PR #39172 into master

* refs/pull/39172/head:
mgr/orchestrator: remove 'host' arg for 'orch ls'

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #39173 into master

* refs/pull/39173/head:
mgr/devicehealth: make CLI commands error when pool doesn't exist
mgr/devicehealth: only create pool when we have some osds

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38892 from ceph/update_s3select_add_zero_segment_log

s3select: update s3select submodule to last commit ; add zero segment log

Merge pull request #38767 from myoungwon/wip-manifest-chunk-scrub

osd: add has_manifest_chunk to count chunks in snapshot

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #39039 into master

* refs/pull/39039/head:
src/test/cli/monmaptool: adjust for new monmap features
qa/tasks/cephadm: allow custom git_url for cephadm_branch pull
qa/suites/rados/upgrade: include pacific-x
qa/suites/upgrade/pacific-x/parallel
qa/suites: some clean up for quincy
mon: updates for quincy
mon: update for quincy ondisk features
script: add pacific
doc/dev/release-checklist: we tagged v17.0.0
ceph-volume: change to quincy
include/ceph_features: retire MON_SINGLE_PAXOS
include/ceph_features: define FEATURE_SERVER_QUINCY
mon/MgrMonitor: add always_on_modules for quincy
add feature/release name quincy
kickoff v17 quincy
doc/dev/release-checklists: uncheck everything!

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37144 from jmolmo/storage_devices_mgmt

doc: High level workflow for storage devices

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

src/test/cli/monmaptool: adjust for new monmap features

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: allow custom git_url for cephadm_branch pull

Normally the git_url is git://git.ceph.com/ceph-ci.git, which mirrors
upstream ceph-ci.git.  However, the release branches aren't present there.
Allow a custom git_url so we can pull these from the main ceph.git.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/devicehealth: make CLI commands error when pool doesn't exist

This is better than silently failing to do anything.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/devicehealth: only create pool when we have some osds

If we create the pool before we have OSDs, the PGs won't be active+clean
and we'll raise a health warning that we shouldn't.  This will annoy
a new users deploying a new cluster, and (more importantly?) make
qa tests fail while deploying initial clusters due to the health
warning.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39025 from yuvalif/wip-yuval-fix-48963

rgw: fix valgrind errors when protected_fixedsize_stack is used

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/orchestrator: remove 'host' arg for 'orch ls'

The host arg is unused, and also missing from the documentation.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39152 from neha-ojha/wip-46323

qa/*/thrash_cache_writeback_proxy_none.yaml: disable writeback overlay tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39118 from tchaikov/wip-cmake-rook-client

cmake: install rook-client-python using ExternalProject

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #38686 from wornet-mwo/wip-doc-add-section-declarative-orch-cmd

doc: added section about retival of Service Specication

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38875 from pcuzner/new-host-to-maintenance

mgr/orchestrator: add maintenance mode for new hosts

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39083 from genesiscloud/orch-non-rotational-filter-fix

python-common/drivegroups: avoid dropping "rotational: 0" from Device Selection

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39007 from mgfritch/cephadm-podman-2.0

 cephadm: require podman >= 2.0.0

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39157 from xxhdx1985126/wip-crimson-bug-fix

crimson/osd: capture "intervals_included" by value in ReplicatedRecoveryBackend::submit_push_data

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39057 from aclamk/wip-mempool-shards-check-random

common/mempool: Improve mempool shard selection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crimson/osd: capture "intervals_included" by value in ReplicatedRecoveryBackend::submit_push_data

ReplicatedRecoveryBackend::submit_push_data()'s parameter "intervals_included" might be an
local varaiable in the outer method, so we should capture it by value to avoid a dangling
reference in the inner lambda

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

diagnose zero segment while s3select

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
using s3-test with the fix fix_test_file_permission_add_attrib

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
update s3select{1ea37ae459f957786b9d198510637dfa12f83350} ; add log tracing zero segments locations

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
s3select submodule update

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
fix comments

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
updating the s3select new features; covering the gap between latest documentation and latest faetures update

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
add some more examples; fixes

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
add between expression to doc; typo

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
typo

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
skipping empty segments

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
skip empty segments to the next segments

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

qa/*/thrash_cache_writeback_proxy_none.yaml: disable writeback overlay tests

thrash_cache_writeback_proxy_none tests have been failing consistently. Some investigation
shows that the writeback overlay tests are reponsible for it. Instead of removing these
cache tiering tests entirely, we'll disable them for now.

Related to: https://tracker.ceph.com/issues/46323
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39112 from pcuzner/fix-gather-facts

cephadm:add missing kernel_security property

qa/suites/rados/upgrade: include pacific-x

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/upgrade/pacific-x/parallel

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites: some clean up for quincy

- remove upgrades from nautilus
- stubs for completing upgrade to quincy

Still missing the pacific-x upgrade tests.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: updates for quincy

Signed-off-by: Sage Weil <sage@newdream.net>

mon: update for quincy ondisk features

Signed-off-by: Sage Weil <sage@newdream.net>

script: add pacific

Signed-off-by: Sage Weil <sage@newdream.net>

doc/dev/release-checklist: we tagged v17.0.0

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-volume: change to quincy

Signed-off-by: Sage Weil <sage@newdream.net>

include/ceph_features: retire MON_SINGLE_PAXOS

X-3 = Q-3 = Nautilus

Signed-off-by: Sage Weil <sage@newdream.net>

include/ceph_features: define FEATURE_SERVER_QUINCY

Signed-off-by: Sage Weil <sage@newdream.net>

mon/MgrMonitor: add always_on_modules for quincy

Signed-off-by: Sage Weil <sage@newdream.net>

add feature/release name quincy

Signed-off-by: Sage Weil <sage@newdream.net>

kickoff v17 quincy

Signed-off-by: Sage Weil <sage@newdream.net>

doc/dev/release-checklists: uncheck everything!

Signed-off-by: Sage Weil <sage@newdream.net>

cmake: install rook-client-python using ExternalProject

so we don't need to rerun the generate_rook_ceph_client.sh script
everytime when building the script. cmake creates a stamp file for
tracking the dependencies and the time of modification of dependencies.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39113 from liewegas/fix-datetime-test

python-common: fix test_datetime_to_str_2 on non-UTC hosts

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #39100 from xxhdx1985126/wip-crimson-bug-fix-2

crimson/osd: fix recovery related bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use do_with to capture MOSDPGPull::take_pulls()'s return

MOSDPGPull::take_pulls() returns a temporary object, so we need to use seastar::do_with
to create an object whose lives until seastar::do_with's inner continuation chain ends

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

rgw: fix valgrind errors when protected_fixedsize_stack is used

Fixes: https://tracker.ceph.com/issues/48963
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

crimson/osd: objects without omap header may not necessarily be a error

There are circumstances in which objects don't have omap headers, so this should
not be treated as an error

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: should capture seastar::do_with params by reference

The lambda within seastar::do_with need to use lvalue reference as its parameter,
otherwise it would be referencing a variable whose lifecycle ends when lambda finishes
runing

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39115 from wjwithagen/wjw-fix-QCOWFormat.cc

librbd: Fix variable declaration in capture list in lambda

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39108 from zdover23/wip-doc-leads-update-epuertat-28-Jan-2021

doc/dev: add Ernesto Puerta as dashboard lead

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39114 from liewegas/drop-rhel-8.0

qa/suites/rados/cephadm: drop rhel_8.0 tests

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #37949 from ideepika/wip-docs-update

doc/dev/developer_guide:  teuthology update developer guide

Reviewed-by: Neha Ojha <nojha@redhat.com>

librbd: Fix variable declaration in capture list in lambda

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa/suites/rados/cephadm/*/distro: drop rhel_8.0 targets

We still have centos 8.0.  We're missing an 8.0 image for the gibba.

Signed-off-by: Sage Weil <sage@newdream.net>

python-common: fix test_datetime_to_str_2 on non-UTC hosts

The old test parsed to a datetime without a tz, which was interpreted as
the local time zone when rendering back to a string.  Specify that it's a
UTC datetime so that behavior is consistent regardless of the test host
timezone.

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm:add missing kernel_security property

The propery decorator had gone missing which meant
the dump of host facts was missing the kernel security
(LSM) settings. This patch just adds the @property
decorator back

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #38861 from alimaredia/wip-rgw-create-bucket-dpp

rgw: Thread dpp through Create Bucket logging

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39105 from dillaman/wip-48989

librbd: tweaks to support Boost 1.75.0

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: Add DoutPrefixProvider to create_bucket logging

This commit hits all log messages for a create bucket
request and ensures each has a DoutPrefixProvider.

This commit gets dpp's all the way down to rgw/services
up way though to rgw_op.cc, rgw_rest.cc files,
rgw_admin.cc, etc.

This commit also removes store as dpp in rgw_sal.h

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #39068 from dillaman/wip-librbd-migration-qcow-2

librbd/migration: tweaks and unit tests for QCOW handler

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39107 from yuriw/wip-yuriw-crontab-master

qa/tests: reduced octopus, nautilus, master frequency

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39043 from tchaikov/wip-cephadm-call

cephadm: refactor call() using asyncio.asyncio.StreamReader

Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tests: changed priority - master 100, pacific 80

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/tests: reduced octopus, nautilus, master frequency

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #39015 from lixiaoy1/fix_lib

librbd/cmake: add library dependency for pwl_cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/dev: add Ernesto Puerta as dashboard lead

This commit removes Lenz Grimmer's name and github
handle from the list of component leads and replaces
it with Ernesto Puerta's name and github handle.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39101 from sebastian-philipp/cephadm-fix-ok-to-stop-return-value

mgr/cephadm: Fix wrong argument type to HandleCommandResult

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39098 from tchaikov/wip-zabbix-ann

mgr: add annotations to zabbix and devicehealth

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39065 from cbodley/wip-48988

cmake: boost>=1.74 adds BOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT to radosgw

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38786 from kotreshhr/evict-clients-of-deauthorized-authids

mgr/volumes: Evict clients based on auth-IDs and subvolume path

Reviewed-by: Victoria Martinez de la Cruz <victoria@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

librbd: tweaks to support Boost 1.75.0

Fixes: https://tracker.ceph.com/issues/48989
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: minor cleanup to QCOWFormat code

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37259 from soumyakoduri/head_req

rgw: Set CURLOPT_NOBODY for HEAD request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38982 from jmolmo/fix_iscsi_client_caps

cephadm: Fix iscsi client caps

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39099 from ZhenLiu94/liuz

doc: Correct spelling mistakes.

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: Fix wrong argument type to HandleCommandResult

the argument must be a string

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38854 from adk3798/ots-monitoring

mgr/cephadm: force flag for ok-to-stop and ok-to-stop for monitoring stack

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38998 from sebastian-philipp/fix-serve-handle-json-error

mgr/cephadm: Properly handle JSON Decode error

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38764 from rhcs-dashboard/fix-48748-master

mgr/dashboard: trigger alert if some nodes have a MTU different than the median value

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/devicehealth: return -1 if prediction plugin is not available

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: remove unused variable

silences flake8 warning F841.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: remove handle_command()

as commands are collected by COMMANDS already.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: use CLICommand for command handling

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: do not catch exception with name

silences flake8 warning of F841:
https://www.flake8rules.com/rules/F841.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: cast port to int

to appease mypy

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: talk to zabbix_sender in text not bytes

as we expect read from / write to it in text mode.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: use "uint" for type of discovery_interval

we only support following types:

'uint', 'int', 'str', 'float', 'bool', 'addr', 'addrvec', 'uuid',
'size', 'secs',

see also str_to_type()

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: add annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: use Option for adding options

more type safety

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: add more annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: correct annotation

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Correct spelling mistakes.

Signed-off-by: ZhenLiu94 <zhenliu94@163.com>

Merge pull request #39093 from rzarzynski/wip-crimson-guard-op-lifetime

crimson: fix dangling ClientRequest::this pointer.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Applied text as suggested by @zdover23

Signed-off-by: Michael Wodniok <wodniok@wor.net>

Merge PR #39087 into master

* refs/pull/39087/head:
qa/suites/rados/cephadm: drop 20.04 podman
qa/suites/rados/cephadm: stop testing 18.04 + podman

Reviewed-by: Sebastian Wagner <swagner@suse.com>

crimson: fix dangling ClientRequest::this pointer.

`ShardServices::start_operation<T>()` spawns an operation,
calls `start()` on it and returns a future plus a smart
pointer controlling its life-time. Callers are responsible
to ensure the pointer doesn't go out-of-scope before
the entire execution is finished. This is error-prone.

`OSD::handle_osd_op()` forgets about its responsibility
which results in dangling `this` pointer of `ClientRequest`.
I believe the problem is much wider spread and the class
is just the tip of the iceberg.

In this commit `start_operation<T>()` is altered to extend
the life-time a bit. However, this isn't an ultimate solution
as callers are still able to e.g. put extra `this`-catching
lambdas on the returned future. A new `with_operation<T>()`-
like interface is expected as a follow-up.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39086 from orozery/qa-rbd-delete-passphrase-file

qa/tasks/rbd: cleanup passphrase file

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39089 from orozery/librbd-luks-cli-test

test/librbd: fix luks encryption cli test on unsupported runs

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39091 from tchaikov/wip-crimson-build

cmake: let alienstore link against zoned allocator

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cmake: let alienstore link against zoned allocator

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #39028 into master

* refs/pull/39028/head:
cephadm: retry for 30s instead of 10s
cephadm: try 'mgr stat' before 'mgr dump' to get the epoch
mon: add 'mgr stat' command

Reviewed-by: Sebastian Wagner <swagner@suse.com>

test/librbd: fix luks encryption cli test on unsupported runs

This commit changes the luks encryption cli test to return success instead of failure when not supported.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: initial unit tests for migration/QCOWFormat

Signed-off-by: Jason Dillaman <dillaman@redhat.com>