pybind/mgr/prometheus/module.py: defaultdict for num_by_state

num_by_state[state] += count in get_pg_status method raises KeyError
if pg state is not in PG_STATES list. PG_STATES should be synced with
osd_types.cc:pg_state_string(). But sometimes it is not. After the
KeyError raise mgr metrics are not available at all.

Fixes: https://tracker.ceph.com/issues/46142
Signed-off-by: Alexander Sushko <alexandrsushko@gmail.com>

Merge pull request #38431 from tchaikov/wip-rtd-doc

pybind,doc: build stub using cython and build API doc with RTD

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@hotmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38329 from pcuzner/cephadm-exporter-docs

doc/dev/cephadm: document the cephadm exporter design

Merge branch 'master' into cephadm-exporter-docs

doc/dev/cephadm: update index links

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #38472 from ronen-fr/wip-ronenf-qa-scrub

test: cancelling both noscrub *and* nodeep-scrub

Merge PR #38407 into master

* refs/pull/38407/head:
qa: allow arbitrary mount options on kclient mounts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38266 into master

* refs/pull/38266/head:
client: add ceph.{cluster_fsid/client_id} vxattrs suppport

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38367 into master

* refs/pull/38367/head:
pybind/cephfs: fix missing terminating NULL char in readlink()'s C string

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36127 into master

* refs/pull/36127/head:
client: more reasonable statfs ffree

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

test: cancelling both noscrub *and* nodeep-scrub

as part of osd-scrub-test.sh.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #37971 from lixiaoy1/rwl_plugin

librbd/cache: make image persistent writeback cache as plugin

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38484 from rhcs-dashboard/update-ssl-error-1

mgr/dashboard: Updating the inbuilt ssl providers error

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38459 from ronen-fr/wip-ronenf-lua-include

rgw/lua: fixing incompatible declarations of CephContext

Merge pull request #37519 from pcuzner/docs-cephadm-config-drift

doc/dev/cephadm: high level design for a compliance check feature

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38498 from bk201/wip-48506

mgr/dashboard: remove pyOpenSSL version pinning

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38415 from insujang/master

msg/async: refine worker creation in NetworkStack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38457 from tchaikov/wip-mon-scrub-interval

mon: use TYPE_SEC for mon_scrub_interval

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mgr/dashboard: remove pyOpenSSL version pinning

Tests require a newer version of pyOpenSSL.

Fixes: https://tracker.ceph.com/issues/48506
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #38497 from tchaikov/wip-seastar

seastar: update seastar submodule

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38491 from agayev/fix-hmsmr-compilation

os/bluestore: Fix HMSMRDevice.cc compilation.

Reviewed-by: Kefu Chai <kchai@redhat.com>

seastar: update seastar submodule

to pick up recent fixes in file io

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38437 from ifed01/wip-ifed-alloc-replay-dump

os/bluestore: add more means to troubleshoot allocator(s)

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38454 from tchaikov/wip-compressor-cleanup

compressor: put Compressor into TOPNSPC namespace and use string_view

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38458 from tchaikov/wip-librados-cct

librados: move definition of cct_deleter to .cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38418 from anthonyeleven/anthonyeleven/clarify-op-priorities

doc: clarify osd recovery op priority and fix a couple of typos

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

os/bluestore: Fix HMSMRDevice.cc compilation.

Signed-off-by: Abutalib Aghayev <agayev@psu.edu>

Merge PR #38419 into master

* refs/pull/38419/head:
vstart_runner: print result line unless opt_rotate_log is set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Adding the alert bad certificate error to the ssl providers error

upstream tracked in https://github.com/cherrypy/cheroot/pull/348
Fixes: https://tracker.ceph.com/issues/48490
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Updating the inbuilt ssl providers error

Fixes: https://tracker.ceph.com/issues/48490
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37956 from dorindabassey/example-notific

examples/boto3/README: examples notification

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #38374 from rzarzynski/wip-crimson-backfill-remove

crimson: add support for backfill, part 2 -- MOSDPGBackfillRemove

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38392 from rzarzynski/wip-crimson-backfill-virtual_pgfacade

crimson: use explicit, abstract interfaces for backfill's facades.

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38208 from sebastian-philipp/mgr-cephadm-type-annotations-mandatory

mgr/cephadm: make type annotations mandatory for some modules

Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson: use explicit, abstract interfaces for backfill's facades.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37130 from pcuzner/cephadm-exporter

cephadm:Add a daemon mode for cephadm to provide a metadata endpoint

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #38483 from tchaikov/wip-crimson-cleanup

crimson: cleanups

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: make annotations for schedule.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for inventory.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for serve.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for upgrade.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38106 from varshar16/wip-fix-rook-orch-ps-img-id-name

mgr/rook: Fix container id and image id in 'orch ps'

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #38346 from varshar16/wip-remove-container-id-field-orch-ps

mgr/orchestrator: Remove 'CONTAINER ID' col from orch ps if unknown

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37952 from mgfritch/cephadm-json-decode

mgr/cephadm: handle JSONDecodeError

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38032 from mgfritch/cephadm-last-local-image-name

cephadm: reference the last local image by digest

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38064 from mgfritch/cephadm-rm-host-key-err

mgr/cephadm: verify the `host` key exists during refresh

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/test: make the backfill testing quieter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/test: make use of maybe_flush() in the backfill testing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for MOSDPGBackfillRemove at primary.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for MOSDPGBackfillRemove at replicas.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38130 from sebastian-philipp/cephadm-upgrade-fail-daemon

mgr/cephadm: upgrade: fail gracefully, if daemon redeploy fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38317 from sebastian-philipp/cephadm-reduce-log-dg-noise

mgr/cephadm: OSD service: reduce log noise

Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson/osd: return something in PG::do_delete_work()

this silences the warning of

crimson/osd/pg.cc:316:1: warning: no return statement in function returning non-void [-Wreturn-type]
  316 | }
      | ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: use a constexpr function for sanity test

since match_stage_t is an alias of uint_8. it is an unsigned type, so an
instance of this type should be always greater or equal to 0.

this silences warning like:

stage_types.h:37:50: warning: comparison is always true due to limited range of data type [-Wtype-limits]
   37 |     static_assert(STAGE >= STAGE_BOTTOM && STAGE <= STAGE_TOP);
      |                                            ~~~~~~^~~~~~~~~~~~

also, GCC complains if it is able to assure that the compare always
returns true. so a function helps to silence it.

this warning is meaningless per-se, as we are using static_assert() to
perform sanity test at *compile-time*.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd/cache: move image writeback cache to plugin

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #38460 from ronen-fr/wip-ronenf-lua-move

rgw: removing a pessimizing move

doc: clarify osd recovery op priority and fix a couple of typos

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

pybind/cephfs: fix missing terminating NULL char in readlink()'s C string

The error is in test_readlink() in src/test/pybind/test_cephfs.py:

...
test_cephfs.test_readlink ... FAIL

======================================================================
FAIL: test_cephfs.test_readlink
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/tmhoang/test_cephfs.py", line 99, in test_readlink
    assert_equal(d, b"/file-1")
AssertionError: b'/file-1\xe8' != b'/file-1'
...

Fixes: https://tracker.ceph.com/issues/48408
Signed-off-by: Tuan Hoang <tmhoang@linux.ibm.com>

librbd/plugin: enable hook points for exclusive lock

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #36986 from cyx1231st/wip-seastar-onode-tree-upstream

crimson/seastore: add initial onode staged-fltree

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/test: fix test/tool make target names

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/staged-onode-tree: use LT, EQ, GT for comparison results

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add and fix comments

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38356 from icepic/patch-2

doc/radosgw: Fix misspelled word in index.rst

Reviewed-by: Zac Dover <zac.dover@gmail.com>

rgw: removing a pessimizing move

"warning: moving a temporary object prevents copy elision"

Co-authored-by: Kefu Chai <kefu@redhat.com>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38324 from tchaikov/wip-mon-cleanups

mon: drop QuorumService and cleanups.

Reviewed-by: Joao Eduardo Luis <joao@suse.com>

mon/ConfigKeyService: drop ConfigKeyService::dispatch()

this method is basically an alias of
ConfigKeyService::service_dispatch(), so get rid of it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: drop ConfigKeyService::get_name()

the only caller of this function is prefix(), so hardcode the service
name there instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: remove unused bits

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: store mon and paxos using reference

before this change, we always have a valid pointer as member variable.
so, a reference serve this purpose better.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: drop QuorumService

so far we only implemented ConfigKeyService, so move QuorumService into
the its only child class -- ConfigKeyService.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: move private member variable declaration to the end of class

to follow https://google.github.io/styleguide/cppguide.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: store a reference as member variable

before this change, we always have a valid pointer as member variable.
so, a reference serve this purpose better.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: use unique_ptr<> to manage lifecycle of Monitor::paxos

it is less error-prone, and has better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/lua: fixing incompatible declarations of CephContext

Existing forward declaration breaks OSD compilation (clang)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

librados: move definition of cct_deleter to .cc

so cct->get() and cct->put() are visually closer. this improves the
readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: use TYPE_SEC for mon_scrub_interval

* common/legacy_config_opts.h: drop mon_scrub_interval from legacy
  settings. as this option is not in critical path. so we can afford
  the overhead of look its value up in the map.
* common/options.cc: change mon_scrub_interval's type from TYPE_INT
  to TYPE_SECS, so we can use human friendly representations like
  "3 days" when specifying its value
* mon/Monitor.cc: adapt to the changes above.
* doc/rados: update the doc for "mon scrub interval" accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Timer: add SafeTimer::add_event_after() which accepts timespan

easier to use when we are moving from the number of seconds to a
C++ timespan.

Signed-off-by: Kefu Chai <kchai@redhat.com>

compressor: pass string_view instead of string

this allows us to find by string without creating a std::string
instance, even if this string is not nul terminated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

compressor: put Compressor into TOPNSPC namespace

we want to add the support for on-the-wire compression to msgr v2, and
this feature will be shared by classic osd and crimson. but
Compressor.cc is also used by bluestore which is linked against by
crimson also. Compressor depends on CephContext which has two different
implementation in two namespaces for classic osd and crison.

so to avoid violating ODR, we should also put Compressor into two
different namespaces so it can be shared by alienstore and crimson in
the same executable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38442 from badone/wip-bump-fmt-package-min-version

cmake: require fmt 6.0.0 and up

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38389 from trociny/wip-48412

librbd: fix regression in object map diff request

Reviewed-by: Jason Dillaman <dillaman@hotmail.com>

Merge pull request #35821 from Aran85/fix-ec-read

osd/ECBackend: optimize remaining read as readop contain multiple objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #37543 from ianwatsonrh/master

docs: With the addition of STS x-amz-security-token is now supported

Merge pull request #38401 from a16bitsysop/alpine-tests

test/lazy-omap-stats: Fix compilation on alpine linux

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: build api docs with Read the Docs

since we are able to build the python bindings using the stub
implementation of C binding, let's enable the API docs build.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: check for READTHEDOC env variable

for building stub C binding if building in Read the Docs environment,
see https://docs.readthedocs.io/en/stable/builds.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: do not build stub C binding anymore

since the python binding is able to build the stub by themselves, there
is no need to do this manually.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/cephfs: use stub implementations when building document

also define a stub for rados.Rados right in cephfs.pyx, so the cython
compiler needs it. but it's cubersome to copy rados.pxd from rados
python binding when building cephfs python binding. because, if we
install a local python package using requirement.txt, pip does not allow
us to specify the include-directory for Cython. and cephfs/setup.py is
located out of the source tree when it is being built by pip, and unlike
in our CMake based build, which specifies --cython-include-dirs with
absolute directory, we don't have access to the root directory of
project in Read the Docs environment, so it's impossible for us to
locate pybind/rados in setup.py.

we could pass "--global-option" as part of package specifier in
requirements.txt, like:

cephfs --global-option="build_ext" --global-option="--for-doc"

or

cephfs --global-option="build_doc"

but then we will have to override or create a command
for interpreting the command line options.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/cephfs: drop duplicated declaration

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/cephfs: extract non-shared interface definitions

* c_cephfs.pxd: for cephfs C declarations
* types.pxd: for types exposed by libc and kernel

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rbd: use stub implementations when building document

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rbd: extract non-shared interface definitions into c_rbd.pxd

so it'd be simpler if we want to parse and generate the stub
for these declarations when building document where where the
annotations are used.

ctime.pxd is extracted so the time related declarations can be shared by
c_rbd.pxd and rbd.pyx.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rgw: use stub implementations when building document

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rgw: s/bool/bint/

when compile a pure python cythonized binding, bool is not defined
because, i think, Cython does not include <stdbool.h> by default.
so use bint indead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rgw: extract non-shared interface definitions into c_rgw.pxd

so it'd be simpler if we want to parse and generate the stub
for these declarations when building document where where the
annotations are used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: use stub implementations when building document

* setup.py: inject "BUILD_DOC" into cython build context if "BUILD_DOC"
    env variable is set, this helps us to do conditional compilation.
    for instance, if we are building document, there is no need to
    use the header files or link against librados.so for building
    the python bindings.
* mock_rados.pxi: implement the mock functinons for their C counterparts
    in librados, so we can use them when building librados python
    binding for creating the document. Sphinx's autodoc extension
    reads annotation of the python package whose document is being
    built, after importing the python package into python runtime.
    but since we use Cython for creating our python binding,
    we have to provide the implementation of those referenced
    librados C API symbols. mock_rados.pxi implements them using Cython.
    the downside of this solution is that we need to mirror every
    function used by the Python binding by repeating them in both
    c_rados.pxd and mock_rados.pxi. the same will apply to other
    Ceph python bindings as long as they use C APIs not offered by
    libc or python runtime. so next step is to develop a simple
    parser which can be run at build time to create mock_*.pxi from
    c_*.pxd.
* rados.pyx: use mock_rados.pxi if BUILD_DOC, otherwise use
    c_rados.pyd. the latter requires a librados.so at runtime.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: check if BUILD_DOC in os.environ

no need to check if it is in keys()

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: remove unused import statement

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: extract non-shared interface definitions into c_rados.pxd

so it'd be simpler if we want to parse and generate the stub
for these declarations when building document where where the annotations
are used, but importable python bindgs of rados,ceph,rbd,cephfs are required.

also move the time.h declarations into ctime.pxd, are they can be shared
by the stub implementation and the non-stub one.

Signed-off-by: Kefu Chai <kchai@redhat.com>