mgr/dashboard: use ListPaginator in rbd pagination

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: paginate services

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #48282 from ljflores/wip-telemetry-perf-fix

mgr/telemetry: handle daemons with complex ids

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

mgr/telemetry: handle daemons with complex ids

Treating daemons as `<daemon_type>.x` caused a crash
in the Telemetry module since the current method does not cover a case
where a daemon id is more complex, i.e. `<daemon_type>.x.y`.

When we parse the daemon type and daemon id, we should
split it into a maximum of two pieces rather than splitting
it by every `.` character. Specifying `1` in the Python
.split() function will limit the split to a maximum of two items.

Fixes: https://tracker.ceph.com/issues/57700
Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #45379 from tobias-urdin/bump-openstack-versions

qa/suites/rgw: fix and update tempest and barbican tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48267 from rhcs-dashboard/nfs-exports-fs-permission-error

mgr/dashboard: handle the cephfs permission issue in nfs exports

Reviewed-by: Pedro González Gómez <pegonzal@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #48114 from athanatos/sjust/wip-crimson-multicore-for-review

crimson: multicore support for pgs in crimson-osd

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: handle the cephfs permission issue in nfs exports

Fixes: https://tracker.ceph.com/issues/48686
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48256 from dang/wip-dang-zipper-admin-api

RGW - Re-do admin API split

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48092 from rosinL/fix-57537

os/bluestore: use direct write in BlueStore::_write_bdev_label

Reviewed-by: Igor Fedotov <ifedotov@croit.io>

Merge pull request #47876 from torchiaf/fix/43116-2

mgr/dashboard: Add text to empty life expectancy column

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

crimson/osd/main: call OSD::stop()

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/FuturizedCollection: use thread_safe_counter

This should work for now, long term we probably want to get
rid of FuturizedCollection entirely from the interface.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/heartbeat: drop ownership of messenger instances

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_shard_manager: add shard_id asserts

Signed-off-by: Samuel Just <sjust@redhat.com>

src/vstart.sh: add vstart --crimson-smp option

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/net: Message::conn needs to be a foreign_ptr

There are two main consequences of this:
1. Messages can't be default copy constructed in crimson.  MMonCommand
   seems to be the only user, and we simply add a copy constructor that
   duplicates data portions of the message.
2. We can't casually copy-construct the conn into other structures.
   The main user here is watch/notify.  We use copy() explicitely
   to populate the object_context structures and avoid passing
   ConnectionFRef by value.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operations/client_request: remove OSD reference

Capture ShardServices reference for use in put_historic directly.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: move send_incremental_map to OSDSingletonState

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/net/Connection: let send and keepalive be called from foreign cores

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/shard_services: check core on each PerShardState method

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operations/client_request: fix whitespace on do_process

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: update_heartbeat_peers can iterate pgs syncronously

OSDSingletonState has a local inventory of all pgids.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operations: use foreign_ptr for conn

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart: echo osd arguments

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operations/peering_event.cc: whitespace fix

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: remove compound_peering_request

The state shared between sub events creates problems for multicore.  The
only user is MOSDPGCreate2, so the optimization isn't really worth
salvaging.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/operation: release pipeline stages on the core they are on

Otherwise, tasks waiting on the stage will wake up on the wrong core.
Later, we may choose to statically enable this behavior only for stages
that can actually span cores, but this is ok for now.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: move perf counter ownership to OSDSingletonState

Later, we'll want to create a properly sharded implentation that avoids
barriers.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: shard PerShardState across cores

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: move hb_stamps to PerShardState

We may later need to syncronize this across cores, perhaps during tick.
This should work for now, however.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: replace global tid with unique core-local tids

We don't really want a global counter here if we can avoid it.  Instead,
assign tids with core-local counters prefixed with the core id.  We
continue to ensure that tids are unique within an osd boot, but lose
the property that sucessive tids on different cores are ordered.  I
don't see anything relying on that property, however, so this should be
fine.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: remove explicit pg counter

The main user of num_pgs was get_target_pg_log_entries(), but we
don't really want to be accessing a global counter there if we
can avoid it.  Instead, compute a core-local target and use the
core-local pg count.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: clean up PerShardState/OSDSingletonState ownership/access

- move heartbeat messenger ownership to OSD
- move OSDSingletonState/ShardServices initialization to
  PGShardManager::start() method
- add accessor OSD::get_shard_services(), remove OSD::shard_services
  reference
- accessor/forwarding cleanups within PGShardManager

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: move store responsiblity to PerShardState

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os: consolidate context handling in FuturizedStore

Context handling is pretty uniform accross all implementations,
may as well do it in the same place.

ShardedStoreProxy would need to handle it otherwise, since
callbacks need to be handled on the core do_transaction is
invoked on.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/futurized_store: create ShardedStoreProxy

For now, FuturizedStore implementations assume that methods are invoked
on core 0.  Later, we'll adapt each implementation to intelligently
support invocation on any pg core.  Until then, this wrapper converts
the existing implementations to a safe, if not particuarly performant,
proxy behavior.

AlienStore should be safe as is.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: adjust get_pg_stats() and for_each_pg() to return futures

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: refactor pg management for multicore

OSDSingletonState will now only be responsible for the spg_t->core
mapping for pgs, the individual PerShardState's will hold local
PGMap's.  PG management operations are now proxied from PGShardManager
to PerShardState.  Subsequent patches will shard PerShardState.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: modify start_pg_operation to return op_id

The op will actually be run generally on a different core, so we don't
want to be returning a reference to it.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_map: add PGShardMapping

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #48147 into main

* refs/pull/48147/head:
win32: install wget when building on ubuntu
win32: install deps with DEBIAN_FRONTEND=noninteractive

Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Reviewed-by: Justin Caratzas <jcaratza@redhat.com>

Merge PR #47153 into main

* refs/pull/47153/head:
build: fix atomic linking with LTO on s390x

Reviewed-by: Justin Caratzas <jcaratza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

RGW - Re-do admin API split

The mgr instantiation was re-added in the build re-org.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #48197 from phlogistonjohn/jjm-cephadm-cov-podman

cephadm: add test cases for container engine classes

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #47131 from aaryanporwal/vrt-doc-update

doc/dev: mgr/dashboard: Replace broken link

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa/tasks/barbican: dont copy policy files

In newer versions the policies is inside the code
so we don't need these files for default policy
anymore.

Signed-off-by: Tobias Urdin <tobias.urdin@binero.se>

Merge pull request #48217 from cyx1231st/wip-crimson-buffer

crimson: create buffer from temporary_buffer with foreign-ptr by default

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48235 from zdover23/wip-doc-2022-09-25-dev-guide-basic-workflow-fork-note-formatting

doc/dev: improve presentation of note (git remote)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: improve presentation of note (git remote)

This commit corrects the formatting of a note that previously
did not make console output and commands as clear as this commit
makes them.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #47767 from batrick/i57249

mds: correct storage of multiple damaged dentries in DamageTable

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #48148 from ktdreyer/win-remove-curl

win32: remove libcurl

Merge pull request #47080 from dparmar18/wip-dparmar-MDS-56522

mds/Server: Do not abort MDS on unknown messages

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/suites/rgw: blocklist object expiry tempest tests

Signed-off-by: Tobias Urdin <tobias.urdin@binero.se>

qa/tasks/barbican: sync barbican db and secret stores

Signed-off-by: Tobias Urdin <tobias.urdin@binero.se>

qa/tasks/barbican: copy barbican policy json or yaml

Signed-off-by: Tobias Urdin <tobias.urdin@binero.se>

qa/suites/rgw/tempest: bump openstack versions

Update the versions used for OpenStack Keystone,
Barbican and Tempest projects.

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

crimson: create buffer from temporary_buffer with foreign-ptr by default

temporary_buffer is internally shareable with a thread-unsafe
ref-counter, we need to make sure it is released in the same core where
it is constructed.

Users that need the extra efficiency can swap to create_local as needed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47265 from s0nea/wip-dashboard-redirect-fqdn

mgr/dashboard: add option to resolve ip addr

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

crimson/osd: rework OSDMap handling for multicore

OSDMaps can occupy significant space in memory.  Duplicating OSDMaps
across cores would multiply that memory usage as well as complicate
the internal structure sharing we do when applying OSDMap incrementals.

Because PeeringState and other interfaces expect efficient copying of
OSDMapRef objects we don't want to use foreign_ptr directly.

Instead, maintain a single cache and distribute local_shared_foreign_ptrs
to other cores.  ShardServices becomes the only OSDMapService.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common: introduce local_shared_foreign_ptr

Introduces a foreign_ptr wrapper to allow core-local refcounting.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48198 from mgfritch/cephadm-pytest-patch-logger

cephadm: patch the cephadm.logger class

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #48095 from ivancich/wip-unify-logic

rgw: fix bool/int logic error when calling get_obj_head_ioctx

Merge pull request #47756 from dparmar18/wip-dparmar-cephadm-after-revert

pybind/mgr/cephadm/upgrade: allow upgrades without reducing max_mds

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #48118 from s0nea/container-version-fix

cephadm: consider stdout to get container version

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #48120 from mgfritch/cephadm-haproxy-nonlocal-bind

cephadm: add `ip_nonlocal_bind` to haproxy deployment

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #48184 from SUSE/wip-fix-57609

python-common: Add 'KB' to supported suffixes in SizeMatcher

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #48200 from guits/fix-57627

ceph-volume: fix regression in activate

Merge pull request #48185 from zhscn/wip-dtype

crimson/os/seastore: introduce backend_type_t

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: make device according to device type

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: introduce get_backend_type to Device

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: make can_delay_allocation always return true

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: rename about device_type_t and journal_type_t

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: introduce backend_type_t

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

Merge pull request #48012 from Huber-ming/slave_requestPayment

rgw: set requestPayment in slave zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48021 from soumyakoduri/wip-multisite-crash

rgw: Drain async_processor request queue during shutdown

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

ceph-volume: fix regression in activate

bea9f4b643c introduced a regression that makes the activate process
take a very long time to complete.
`_get_bluestore_info()` which calls `ceph-bluestore-tool` binary via
subprocess is called in an exponential way while this is not needed.

Fixes: https://tracker.ceph.com/issues/57627
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

cephadm: add test cases for container engine classes

Fixes: https://tracker.ceph.com/issues/57621
Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: denote any mocked params as internal

prefix the mock patch params with a `_` char for consistancy with other
tests cases within the suite

Fixes: Fixes: https://tracker.ceph.com/issues/57621
Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: patch the `cephadm.logger` class

Patch the logger class instead of globally mocking the class from within
the loaded source file. This was inadvertently allowing for the entire
test run to succeed, while a single run of a test case would fail due to
the missing mock.

For example:
    `tox -e py3 tests/test_cephadm.py::TestShell::test_fsid`

Fixes: Fixes: https://tracker.ceph.com/issues/57621
Signed-off-by: Michael Fritch <mfritch@suse.com>

crimson/common/operation: generalized for multiple reactors

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: make mkfs static

Can't access pg_shard_manager state here without invoking
start().  Instead, let's just be explicit about meta_coll
etc.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/main: need to actually invoke ConfigProxy::start()

sharded_proxy()::start() is a sharded<> method.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/main: don't shard OSD

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: rename CoreState to OSDSingletonState

CoreState was a bad name -- "core" is going to be used
to refer to seastar reactors/cores as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/shard_services: formatting fixes

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common: introduce smp_helpers.h, crimson::submit_to

Introduce crimson::submit_to wrapper to transparently deal with
vanilla and errorated futures, adds helpers to errorator.h.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #47766 from cbodley/wip-qa-rgw-tempest-meta

qa/rgw/tempest: blocklist failing tests for removal of container metadata

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #48193 from rhcs-dashboard/fix-stoi

exporter: avoid stoi for empty pid_str

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #47900 from Matan-B/wip-matanb-librados-20

examples/librados: Compile with c++20 on Ubuntu

cephadm: add `ip_nonlocal_bind` to haproxy deployment

standalone haproxy deployments require the
`net.ipv4.ip_nonlocal_bind` sysctl setting

Fixes: https://tracker.ceph.com/issues/57563
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #48191 from phlogistonjohn/jjm-cephadm-tox-coverage

cephadm: add a tox env for generating a coverage report

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

mgr/dashboard: Add text to empty life expectancy column

- Add life_expectancy_enabled field to /api/osd/{svc_id}/devices
- Add new value 'n/a' for 'Life Expectancy' column

Fixes: https://tracker.ceph.com/issues/43116
Signed-off-by: Francesco Torchia <francesco.torchia@suse.com>

Merge pull request #47494 from rhcs-dashboard/pgs-repaired

mgr/prometheus: expose repaired pgs metrics

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48190 from ronen-fr/wip-rf-replica-init

osd/scrub: make on_replica_init() idempotent again

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

exporter: avoid stoi for empty pid_str

Fixes: https://tracker.ceph.com/issues/57619
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

crimson/.../client_request: use intrusive_ptr for instance_handle

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48121 from rzarzynski/wip-crimson-pgadvmap-critical-from

crimson/osd: initialize PGAdvanceMap::from inside critical section

Reviewed-by: Samuel Just <sjust@redhat.com>

cephadm: add a tox env for generating a coverage report

This "dormant" target is here for developers to run locally and generate
a HTML coverage report to review.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #48131 from adamemerson/wip-no-fno-new-ttp-matching

build: Remove -fno-new-ttp-matchingg flag

Reviewed-by: Kefu Chai <tchaikov@gmail.com>