Merge pull request #38523 from tchaikov/wip-gcc-toolset-9

install-deps.sh: do not use gcc-toolset-9 from copr

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38157 from sebastian-philipp/labeler-orchestrator

github/labeler: Add some more to cephadm, add orchestrator

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38408 from trociny/wip-rbd-group-snap-create2

librbd: extend group snap create API to support flags

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38518 from trociny/wip-rbd-test-memleak

test/librbd: fix memory leak in TestLibRBD.ConcurentOperations

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38490 from sebastian-philipp/mypy-0.790

src,qa,dashboard: Upgrade to mypy 0.790

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38075 from joke-lee/s3-auth-date-fix

rgw: fix RequestTimeTooSkewed when Date contain RFC-822/ISO 8601 standard timezone

Merge pull request #37926 from BryceCao/wip-catch-non-int-exception

rgw : catch non int exception

Merge pull request #37909 from alexandrsushko/prometheus-pg-laggy-46142

mgr: Fix for dashboard/prometheus failure due to laggy pg state

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #38131 from linuxbox2/wip-lc-ix

rgwlc: correctly dimension lc shard index vector

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

github/labeler: Add some more to cephadm, add orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38524 from tchaikov/wip-crimson-qa-rbd

qa/suites: add crimson-rados/rbd test suites

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38470 from sebastian-philipp/cephadm-command-unit-verbose

cephadm: command_unit: call systemctl with verbose=True

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38009 from ideepika/wip-48058-cephadm

qa/tasks/cephadm: update_info_yam() only if ctx.archive is valid

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36551 from CongMinYin/immutable_object_cache_throttle

tools: add throttle mechanism to immutable object cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites: add crimson-rados/rbd test suites

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38435 from votdev/issue_48449_test_standby

mgr/dashboard: test_standby* (tasks.mgr.test_dashboard.TestDashboard) failed locally

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38441 from bk201/wip-48455

mgr/dashboard: adjust refresh intervals of Services and Daemons

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38212 from sebastian-philipp/mgr-cephadm-type-annotations-mandatory-module

mgr/cephadm: make type annotations for module.py mandatory

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

install-deps.sh: do not use gcc-toolset-9 from copr

since both CentOS and RHEL now include the updated
gcc-toolset-9-gcc-9.2.1-2.3.el8.

see also https://bugzilla.redhat.com/show_bug.cgi?id=1853900

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd: make "group snap create" support quiesce options

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: extend group snap create API to support flags

Signed-off-by: Mykola Golub <mgolub@suse.com>

test/librbd: fix memory leak in TestLibRBD.ConcurentOperations

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/prometheus/module.py: defaultdict for num_by_state

num_by_state[state] += count in get_pg_status method raises KeyError
if pg state is not in PG_STATES list. PG_STATES should be synced with
osd_types.cc:pg_state_string(). But sometimes it is not. After the
KeyError raise mgr metrics are not available at all.

Fixes: https://tracker.ceph.com/issues/46142
Signed-off-by: Alexander Sushko <alexandrsushko@gmail.com>

Merge pull request #38431 from tchaikov/wip-rtd-doc

pybind,doc: build stub using cython and build API doc with RTD

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@hotmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38329 from pcuzner/cephadm-exporter-docs

doc/dev/cephadm: document the cephadm exporter design

Merge branch 'master' into cephadm-exporter-docs

doc/dev/cephadm: update index links

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #38472 from ronen-fr/wip-ronenf-qa-scrub

test: cancelling both noscrub *and* nodeep-scrub

Merge PR #38407 into master

* refs/pull/38407/head:
qa: allow arbitrary mount options on kclient mounts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38266 into master

* refs/pull/38266/head:
client: add ceph.{cluster_fsid/client_id} vxattrs suppport

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38367 into master

* refs/pull/38367/head:
pybind/cephfs: fix missing terminating NULL char in readlink()'s C string

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36127 into master

* refs/pull/36127/head:
client: more reasonable statfs ffree

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

test: cancelling both noscrub *and* nodeep-scrub

as part of osd-scrub-test.sh.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #37971 from lixiaoy1/rwl_plugin

librbd/cache: make image persistent writeback cache as plugin

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38484 from rhcs-dashboard/update-ssl-error-1

mgr/dashboard: Updating the inbuilt ssl providers error

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38459 from ronen-fr/wip-ronenf-lua-include

rgw/lua: fixing incompatible declarations of CephContext

Merge pull request #37519 from pcuzner/docs-cephadm-config-drift

doc/dev/cephadm: high level design for a compliance check feature

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38498 from bk201/wip-48506

mgr/dashboard: remove pyOpenSSL version pinning

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38415 from insujang/master

msg/async: refine worker creation in NetworkStack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38457 from tchaikov/wip-mon-scrub-interval

mon: use TYPE_SEC for mon_scrub_interval

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mgr/cephadm: make type annotations for module.py mandatory

Fixing bogus json representation for registry-login

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

src,qa: Upgrade to mypy 0.790

dashboard/tools: Make sure {begin,end}_time are set.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: remove pyOpenSSL version pinning

Tests require a newer version of pyOpenSSL.

Fixes: https://tracker.ceph.com/issues/48506
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #38497 from tchaikov/wip-seastar

seastar: update seastar submodule

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38491 from agayev/fix-hmsmr-compilation

os/bluestore: Fix HMSMRDevice.cc compilation.

Reviewed-by: Kefu Chai <kchai@redhat.com>

seastar: update seastar submodule

to pick up recent fixes in file io

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38437 from ifed01/wip-ifed-alloc-replay-dump

os/bluestore: add more means to troubleshoot allocator(s)

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38454 from tchaikov/wip-compressor-cleanup

compressor: put Compressor into TOPNSPC namespace and use string_view

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38458 from tchaikov/wip-librados-cct

librados: move definition of cct_deleter to .cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38418 from anthonyeleven/anthonyeleven/clarify-op-priorities

doc: clarify osd recovery op priority and fix a couple of typos

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

os/bluestore: Fix HMSMRDevice.cc compilation.

Signed-off-by: Abutalib Aghayev <agayev@psu.edu>

Merge PR #38419 into master

* refs/pull/38419/head:
vstart_runner: print result line unless opt_rotate_log is set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Adding the alert bad certificate error to the ssl providers error

upstream tracked in https://github.com/cherrypy/cheroot/pull/348
Fixes: https://tracker.ceph.com/issues/48490
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Updating the inbuilt ssl providers error

Fixes: https://tracker.ceph.com/issues/48490
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37956 from dorindabassey/example-notific

examples/boto3/README: examples notification

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #38374 from rzarzynski/wip-crimson-backfill-remove

crimson: add support for backfill, part 2 -- MOSDPGBackfillRemove

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38392 from rzarzynski/wip-crimson-backfill-virtual_pgfacade

crimson: use explicit, abstract interfaces for backfill's facades.

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38208 from sebastian-philipp/mgr-cephadm-type-annotations-mandatory

mgr/cephadm: make type annotations mandatory for some modules

Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson: use explicit, abstract interfaces for backfill's facades.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37130 from pcuzner/cephadm-exporter

cephadm:Add a daemon mode for cephadm to provide a metadata endpoint

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #38483 from tchaikov/wip-crimson-cleanup

crimson: cleanups

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: make annotations for schedule.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for inventory.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for serve.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for upgrade.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38106 from varshar16/wip-fix-rook-orch-ps-img-id-name

mgr/rook: Fix container id and image id in 'orch ps'

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #38346 from varshar16/wip-remove-container-id-field-orch-ps

mgr/orchestrator: Remove 'CONTAINER ID' col from orch ps if unknown

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37952 from mgfritch/cephadm-json-decode

mgr/cephadm: handle JSONDecodeError

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38032 from mgfritch/cephadm-last-local-image-name

cephadm: reference the last local image by digest

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38064 from mgfritch/cephadm-rm-host-key-err

mgr/cephadm: verify the `host` key exists during refresh

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/test: make the backfill testing quieter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/test: make use of maybe_flush() in the backfill testing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for MOSDPGBackfillRemove at primary.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for MOSDPGBackfillRemove at replicas.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38130 from sebastian-philipp/cephadm-upgrade-fail-daemon

mgr/cephadm: upgrade: fail gracefully, if daemon redeploy fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38317 from sebastian-philipp/cephadm-reduce-log-dg-noise

mgr/cephadm: OSD service: reduce log noise

Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson/osd: return something in PG::do_delete_work()

this silences the warning of

crimson/osd/pg.cc:316:1: warning: no return statement in function returning non-void [-Wreturn-type]
  316 | }
      | ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: use a constexpr function for sanity test

since match_stage_t is an alias of uint_8. it is an unsigned type, so an
instance of this type should be always greater or equal to 0.

this silences warning like:

stage_types.h:37:50: warning: comparison is always true due to limited range of data type [-Wtype-limits]
   37 |     static_assert(STAGE >= STAGE_BOTTOM && STAGE <= STAGE_TOP);
      |                                            ~~~~~~^~~~~~~~~~~~

also, GCC complains if it is able to assure that the compare always
returns true. so a function helps to silence it.

this warning is meaningless per-se, as we are using static_assert() to
perform sanity test at *compile-time*.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd/cache: move image writeback cache to plugin

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #38460 from ronen-fr/wip-ronenf-lua-move

rgw: removing a pessimizing move

doc: clarify osd recovery op priority and fix a couple of typos

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

pybind/cephfs: fix missing terminating NULL char in readlink()'s C string

The error is in test_readlink() in src/test/pybind/test_cephfs.py:

...
test_cephfs.test_readlink ... FAIL

======================================================================
FAIL: test_cephfs.test_readlink
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/tmhoang/test_cephfs.py", line 99, in test_readlink
    assert_equal(d, b"/file-1")
AssertionError: b'/file-1\xe8' != b'/file-1'
...

Fixes: https://tracker.ceph.com/issues/48408
Signed-off-by: Tuan Hoang <tmhoang@linux.ibm.com>

cephadm: command_unit: call systemctl with verbose=True

Otherwise `cephadm unit ... status` won't print anything

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd/plugin: enable hook points for exclusive lock

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #36986 from cyx1231st/wip-seastar-onode-tree-upstream

crimson/seastore: add initial onode staged-fltree

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/test: fix test/tool make target names

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/staged-onode-tree: use LT, EQ, GT for comparison results

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add and fix comments

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38356 from icepic/patch-2

doc/radosgw: Fix misspelled word in index.rst

Reviewed-by: Zac Dover <zac.dover@gmail.com>

rgw: removing a pessimizing move

"warning: moving a temporary object prevents copy elision"

Co-authored-by: Kefu Chai <kefu@redhat.com>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38324 from tchaikov/wip-mon-cleanups

mon: drop QuorumService and cleanups.

Reviewed-by: Joao Eduardo Luis <joao@suse.com>

mon/ConfigKeyService: drop ConfigKeyService::dispatch()

this method is basically an alias of
ConfigKeyService::service_dispatch(), so get rid of it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: drop ConfigKeyService::get_name()

the only caller of this function is prefix(), so hardcode the service
name there instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: remove unused bits

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: store mon and paxos using reference

before this change, we always have a valid pointer as member variable.
so, a reference serve this purpose better.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: drop QuorumService

so far we only implemented ConfigKeyService, so move QuorumService into
the its only child class -- ConfigKeyService.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: move private member variable declaration to the end of class

to follow https://google.github.io/styleguide/cppguide.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: store a reference as member variable

before this change, we always have a valid pointer as member variable.
so, a reference serve this purpose better.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: use unique_ptr<> to manage lifecycle of Monitor::paxos

it is less error-prone, and has better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/lua: fixing incompatible declarations of CephContext

Existing forward declaration breaks OSD compilation (clang)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>