osd: osdmap: do not assert target_v when encoding in stretch mode

Again, this was an overly-strict safety check. It passed testing against
(even old) userspace clients, but the kernel clients don't have all the
same feature bits and so when encoding for them, we see lower target_v values
and assert out erroneously.

As before, monitors and OSDs that actually need to interpret this data
are gated on cluster entry, so we don't need this safety check here.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: add a missing space in disallowed_leaders output

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: do not get pending crush to no purpose

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: osdmap: Do not increase compatv when using stretch mode

This is analagous to my monmap fix in 2e3643647bfbe955b54c62c8aaf114744dedb86e,
but for the OSDMap. I was previously trying to be safe by changing compat_v
when using stretch details. Unfortunately, older clients are trying to
interpret this data (even though its purpose is to be a cluster-only data
region! See https://tracker.ceph.com/issues/48489), and crashing as a result.

Because enabling stretch mode is blocked by all OSDs understanding it, and
OSDs which do not understand it are not allowed to boot once it's on,
we can simply remove the compat_v change.

This does leave the miniscule risk that somebody will enable stretch mode
and run with it, then turn it off, an old OSD will *then* connect to the
cluster, and maybe some bad peering choices will happen -- but I don't
see a lot of other choices and this scenario is so unlikely I don't
think it's worth worrying about.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: fix some erroneous safety asserts in OSDMonitor::try_enable_stretch_mode

This function asserts it is not committing in error cases, because it is supposed
to be run first in a trial mode, and then in a commit mode. But I wanted the
asserts to be more informative than that when they are printed, so I included
the failure condition as an OR clause in the assert.

Unfortunately, a few of them were either written wrong or became wrong due to
code movement. Fix these so they are valid and convey the desired information.

Reported-by: Ronen Friedman <rfriedma@redhat.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #38524 from tchaikov/wip-crimson-qa-rbd

qa/suites: add crimson-rados/rbd test suites

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38470 from sebastian-philipp/cephadm-command-unit-verbose

cephadm: command_unit: call systemctl with verbose=True

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38009 from ideepika/wip-48058-cephadm

qa/tasks/cephadm: update_info_yam() only if ctx.archive is valid

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36551 from CongMinYin/immutable_object_cache_throttle

tools: add throttle mechanism to immutable object cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites: add crimson-rados/rbd test suites

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38435 from votdev/issue_48449_test_standby

mgr/dashboard: test_standby* (tasks.mgr.test_dashboard.TestDashboard) failed locally

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38441 from bk201/wip-48455

mgr/dashboard: adjust refresh intervals of Services and Daemons

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38212 from sebastian-philipp/mgr-cephadm-type-annotations-mandatory-module

mgr/cephadm: make type annotations for module.py mandatory

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38431 from tchaikov/wip-rtd-doc

pybind,doc: build stub using cython and build API doc with RTD

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@hotmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38329 from pcuzner/cephadm-exporter-docs

doc/dev/cephadm: document the cephadm exporter design

Merge branch 'master' into cephadm-exporter-docs

doc/dev/cephadm: update index links

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #38472 from ronen-fr/wip-ronenf-qa-scrub

test: cancelling both noscrub *and* nodeep-scrub

Merge PR #38407 into master

* refs/pull/38407/head:
qa: allow arbitrary mount options on kclient mounts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38266 into master

* refs/pull/38266/head:
client: add ceph.{cluster_fsid/client_id} vxattrs suppport

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38367 into master

* refs/pull/38367/head:
pybind/cephfs: fix missing terminating NULL char in readlink()'s C string

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36127 into master

* refs/pull/36127/head:
client: more reasonable statfs ffree

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

test: cancelling both noscrub *and* nodeep-scrub

as part of osd-scrub-test.sh.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #37971 from lixiaoy1/rwl_plugin

librbd/cache: make image persistent writeback cache as plugin

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38484 from rhcs-dashboard/update-ssl-error-1

mgr/dashboard: Updating the inbuilt ssl providers error

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38459 from ronen-fr/wip-ronenf-lua-include

rgw/lua: fixing incompatible declarations of CephContext

Merge pull request #37519 from pcuzner/docs-cephadm-config-drift

doc/dev/cephadm: high level design for a compliance check feature

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38498 from bk201/wip-48506

mgr/dashboard: remove pyOpenSSL version pinning

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38415 from insujang/master

msg/async: refine worker creation in NetworkStack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38457 from tchaikov/wip-mon-scrub-interval

mon: use TYPE_SEC for mon_scrub_interval

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mgr/cephadm: make type annotations for module.py mandatory

Fixing bogus json representation for registry-login

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: remove pyOpenSSL version pinning

Tests require a newer version of pyOpenSSL.

Fixes: https://tracker.ceph.com/issues/48506
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #38497 from tchaikov/wip-seastar

seastar: update seastar submodule

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38491 from agayev/fix-hmsmr-compilation

os/bluestore: Fix HMSMRDevice.cc compilation.

Reviewed-by: Kefu Chai <kchai@redhat.com>

seastar: update seastar submodule

to pick up recent fixes in file io

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38437 from ifed01/wip-ifed-alloc-replay-dump

os/bluestore: add more means to troubleshoot allocator(s)

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38454 from tchaikov/wip-compressor-cleanup

compressor: put Compressor into TOPNSPC namespace and use string_view

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38458 from tchaikov/wip-librados-cct

librados: move definition of cct_deleter to .cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38418 from anthonyeleven/anthonyeleven/clarify-op-priorities

doc: clarify osd recovery op priority and fix a couple of typos

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

os/bluestore: Fix HMSMRDevice.cc compilation.

Signed-off-by: Abutalib Aghayev <agayev@psu.edu>

Merge PR #38419 into master

* refs/pull/38419/head:
vstart_runner: print result line unless opt_rotate_log is set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Adding the alert bad certificate error to the ssl providers error

upstream tracked in https://github.com/cherrypy/cheroot/pull/348
Fixes: https://tracker.ceph.com/issues/48490
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Updating the inbuilt ssl providers error

Fixes: https://tracker.ceph.com/issues/48490
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37956 from dorindabassey/example-notific

examples/boto3/README: examples notification

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #38374 from rzarzynski/wip-crimson-backfill-remove

crimson: add support for backfill, part 2 -- MOSDPGBackfillRemove

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38392 from rzarzynski/wip-crimson-backfill-virtual_pgfacade

crimson: use explicit, abstract interfaces for backfill's facades.

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38208 from sebastian-philipp/mgr-cephadm-type-annotations-mandatory

mgr/cephadm: make type annotations mandatory for some modules

Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson: use explicit, abstract interfaces for backfill's facades.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37130 from pcuzner/cephadm-exporter

cephadm:Add a daemon mode for cephadm to provide a metadata endpoint

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #38483 from tchaikov/wip-crimson-cleanup

crimson: cleanups

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: make annotations for schedule.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for inventory.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for serve.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make annotations for upgrade.py mandatory

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38106 from varshar16/wip-fix-rook-orch-ps-img-id-name

mgr/rook: Fix container id and image id in 'orch ps'

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #38346 from varshar16/wip-remove-container-id-field-orch-ps

mgr/orchestrator: Remove 'CONTAINER ID' col from orch ps if unknown

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37952 from mgfritch/cephadm-json-decode

mgr/cephadm: handle JSONDecodeError

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38032 from mgfritch/cephadm-last-local-image-name

cephadm: reference the last local image by digest

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38064 from mgfritch/cephadm-rm-host-key-err

mgr/cephadm: verify the `host` key exists during refresh

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/test: make the backfill testing quieter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/test: make use of maybe_flush() in the backfill testing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for MOSDPGBackfillRemove at primary.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for MOSDPGBackfillRemove at replicas.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38130 from sebastian-philipp/cephadm-upgrade-fail-daemon

mgr/cephadm: upgrade: fail gracefully, if daemon redeploy fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38317 from sebastian-philipp/cephadm-reduce-log-dg-noise

mgr/cephadm: OSD service: reduce log noise

Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson/osd: return something in PG::do_delete_work()

this silences the warning of

crimson/osd/pg.cc:316:1: warning: no return statement in function returning non-void [-Wreturn-type]
  316 | }
      | ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: use a constexpr function for sanity test

since match_stage_t is an alias of uint_8. it is an unsigned type, so an
instance of this type should be always greater or equal to 0.

this silences warning like:

stage_types.h:37:50: warning: comparison is always true due to limited range of data type [-Wtype-limits]
   37 |     static_assert(STAGE >= STAGE_BOTTOM && STAGE <= STAGE_TOP);
      |                                            ~~~~~~^~~~~~~~~~~~

also, GCC complains if it is able to assure that the compare always
returns true. so a function helps to silence it.

this warning is meaningless per-se, as we are using static_assert() to
perform sanity test at *compile-time*.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd/cache: move image writeback cache to plugin

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #38460 from ronen-fr/wip-ronenf-lua-move

rgw: removing a pessimizing move

doc: clarify osd recovery op priority and fix a couple of typos

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

pybind/cephfs: fix missing terminating NULL char in readlink()'s C string

The error is in test_readlink() in src/test/pybind/test_cephfs.py:

...
test_cephfs.test_readlink ... FAIL

======================================================================
FAIL: test_cephfs.test_readlink
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/tmhoang/test_cephfs.py", line 99, in test_readlink
    assert_equal(d, b"/file-1")
AssertionError: b'/file-1\xe8' != b'/file-1'
...

Fixes: https://tracker.ceph.com/issues/48408
Signed-off-by: Tuan Hoang <tmhoang@linux.ibm.com>

cephadm: command_unit: call systemctl with verbose=True

Otherwise `cephadm unit ... status` won't print anything

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd/plugin: enable hook points for exclusive lock

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #36986 from cyx1231st/wip-seastar-onode-tree-upstream

crimson/seastore: add initial onode staged-fltree

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/test: fix test/tool make target names

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/staged-onode-tree: use LT, EQ, GT for comparison results

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add and fix comments

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38356 from icepic/patch-2

doc/radosgw: Fix misspelled word in index.rst

Reviewed-by: Zac Dover <zac.dover@gmail.com>

rgw: removing a pessimizing move

"warning: moving a temporary object prevents copy elision"

Co-authored-by: Kefu Chai <kefu@redhat.com>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38324 from tchaikov/wip-mon-cleanups

mon: drop QuorumService and cleanups.

Reviewed-by: Joao Eduardo Luis <joao@suse.com>

mon/ConfigKeyService: drop ConfigKeyService::dispatch()

this method is basically an alias of
ConfigKeyService::service_dispatch(), so get rid of it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: drop ConfigKeyService::get_name()

the only caller of this function is prefix(), so hardcode the service
name there instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: remove unused bits

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: store mon and paxos using reference

before this change, we always have a valid pointer as member variable.
so, a reference serve this purpose better.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: drop QuorumService

so far we only implemented ConfigKeyService, so move QuorumService into
the its only child class -- ConfigKeyService.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigKeyService: move private member variable declaration to the end of class

to follow https://google.github.io/styleguide/cppguide.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: store a reference as member variable

before this change, we always have a valid pointer as member variable.
so, a reference serve this purpose better.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: use unique_ptr<> to manage lifecycle of Monitor::paxos

it is less error-prone, and has better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/lua: fixing incompatible declarations of CephContext

Existing forward declaration breaks OSD compilation (clang)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

librados: move definition of cct_deleter to .cc

so cct->get() and cct->put() are visually closer. this improves the
readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: use TYPE_SEC for mon_scrub_interval

* common/legacy_config_opts.h: drop mon_scrub_interval from legacy
  settings. as this option is not in critical path. so we can afford
  the overhead of look its value up in the map.
* common/options.cc: change mon_scrub_interval's type from TYPE_INT
  to TYPE_SECS, so we can use human friendly representations like
  "3 days" when specifying its value
* mon/Monitor.cc: adapt to the changes above.
* doc/rados: update the doc for "mon scrub interval" accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Timer: add SafeTimer::add_event_after() which accepts timespan

easier to use when we are moving from the number of seconds to a
C++ timespan.

Signed-off-by: Kefu Chai <kchai@redhat.com>

compressor: pass string_view instead of string

this allows us to find by string without creating a std::string
instance, even if this string is not nul terminated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

compressor: put Compressor into TOPNSPC namespace

we want to add the support for on-the-wire compression to msgr v2, and
this feature will be shared by classic osd and crimson. but
Compressor.cc is also used by bluestore which is linked against by
crimson also. Compressor depends on CephContext which has two different
implementation in two namespaces for classic osd and crison.

so to avoid violating ODR, we should also put Compressor into two
different namespaces so it can be shared by alienstore and crimson in
the same executable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38442 from badone/wip-bump-fmt-package-min-version

cmake: require fmt 6.0.0 and up

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38389 from trociny/wip-48412

librbd: fix regression in object map diff request

Reviewed-by: Jason Dillaman <dillaman@hotmail.com>

Merge pull request #35821 from Aran85/fix-ec-read

osd/ECBackend: optimize remaining read as readop contain multiple objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #37543 from ianwatsonrh/master

docs: With the addition of STS x-amz-security-token is now supported

Merge pull request #38401 from a16bitsysop/alpine-tests

test/lazy-omap-stats: Fix compilation on alpine linux

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: build api docs with Read the Docs

since we are able to build the python bindings using the stub
implementation of C binding, let's enable the API docs build.

Signed-off-by: Kefu Chai <kchai@redhat.com>