node-proxy: RedfishClient class refactor

This implements BaseClient class and make RedfishClient inherit from it.
Same logic as BaseSystem / RedfishSystem given that any other backend could
need to implement a new client for collecting the data.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: fix mypy warning regarding Config.logging

Config's attributes are dynamically created so mypy complains.
using `__dict__['logging']` addresses that.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: rename server-v2.py

As the previous version has been removed, let's rename this file.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: drop old server.py

This version relies on flask.
At the end, we decided to migrate to cherrypy given that
we already use it quite a lot in ceph/ceph

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: create entrypoint main()

This creates a `main()` function in server.py that will be the
entrypoint of node-proxy.

This also implement arg parsing and add a `--config` parameter
to specify the configuration file.

Finally, this introduce a small refactor of class `Config` and class
`Logger` in util.py because there was a circular dependency between them.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: rename System to BaseSystem

In order to avoid confusion or redefinition issue with class System()
defined in server.py.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: add a timeout when posting data

if this call is stuck for any reason, the report will block
the whole daemon given that at this point it has acquired a lock.
We need to make sure this call won't block the daemon for a long time,
let's add a timeout.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: (Redfish_System) reuse the existing client when possible

Otherwise, the method start_client() recreates a new client.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: remove a redundant message

This message is not needed given that there's the same in
the RedFishClient class.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: add requirements.txt

This adds the requirements.txt file in order to manage the required
libraries.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: add a retry on redfish_client.get_path() calls

The idea is to retry multiple times before stating the endpoint is
definitely unreachable.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: add a decorator 'retry'

This decorator will be useful for calls that should do multiple
attempts before actually failing.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: add type annotation

This commit adds the type annotation in all files.
This was missing since the initial implementation, let's add
it before the project gets bigger.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: address some flake8 linting errors

This addresses some flake8 errors.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: implement config & logging management

This adds the classes 'Config' and 'Logger' in order to manage
the logging and the configuration within the node-proxy daemon.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: catch RequestException in reporter

This catches the requests.exceptions.RequestException
exception in the reporter agent so we can better handle the
case where it can't reach the endpoint when trying to send the
collected data.
Before this change, if for some reason the refreshed data couldn't be
sent to the endpoint, it wouldn't have retried because
`self.system.previous_data` was overwritten anyway.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: catch more error in redfish_client

This catches more potential exceptions in the redfish_client
class.
So if an error is caught we can log a more accurate and nicer message.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: add some logging in the reporter agent

This adds some calls to the logging module, mostly for
devel/debug purposes at the moment.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: fix a typo in redfish_system.get_status()

s/Status/status

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: redfish_system.get_system refactor

This method should return the 'unified structure' version of the
collected data instead of the huge json returned by redfish.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: add a lock mechanism

The loop in the reporter agent has to wait that the data are all
collected before checking and pushing them to the ceph-mgr (if needed).
The idea is to use the lock mechanism offered by the threading module
from python.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: migrate to cherrypy

cherrypy is already widely used in Ceph.
Let's not add new dependencies and use cherrypy instead of
python-flask

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: add method start_client() redfish_system class

This is going to be useful for a new endpoint '/start'

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: drop redfish_system._process_redfish_system method

This method isn't needed, let's drop it.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: display error messages when Exception is caught

This is mostly for development purposes.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: merge self._system with current values

Otherwise `self._system` gets reset in each iteration.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: add normalize_dict() function

this is to make sure all keys are converted into
lowercase.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: split RedfishSystem class

This class should be split because the logic will be different depending on the
hardware.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: implement storage endpoint

This adds the required logic for the endpoint '/system/storage'
to gather and return data about physical drives.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: implement network endpoint

This adds the required logic for the endpoint '/system/network'
to gather and return data about network interfaces.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: implement processors endpoint

This adds the required logic for the endpoint '/system/processors'
to gather and return data about CPUs.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: use `use_reloader=False`

In order to prevent the server from restarting in a loop
when an error shows up. Otherwise, it creates a bunch of new
redfish client session and make it quickly unavailable due to the
session limit.
Probably not intended to be kept.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: add a /shutdown endpoint

Add a '/shutdown' endpoint to force the client to logout and delete its current
session.
This is for devel puroposes and probably not intended to be kept.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: logout from redfish api on Exception

Otherwise it ends up recreating new session each time whereas the previous session
is left. After multiple failures, it reaches the limit and left sessions need to be
cleaned up manually.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: variabilize the system_endpoint

This makes it possible to define the value of the 'System endpoint'.
This can be different according to the hardware.

This probably means that the class `RedfishSystem` should be split itself.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: improve logging

this adds a new file `util.py` with a logger function in order
to improve a bit the logging.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: various unified interface changes

this slightly modifies the data structure of the unified interface.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit b853761836febe92f6460a13d554cd966ff2e529)

First hardware-monitoring draft version

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #55287 from ajarr/wip-64139

rbd-nbd: fix resize of images mapped using netlink

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55270 from afreen23/fix-cap-inconsistency-multisite

mgr/dashboard: Fix inconsistency in capitalisation of "Multi-site"

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: rosinL <NOT@FOUND>

Merge pull request #55182 from rkachach/fix_issue_64029

mgr/rook: adding some basic rook e2e testing

Merge pull request #55266 from athanatos/sjust/wip-63996

crimson: retain map references in OSDSingletonState::store_maps

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/shard_services: retain map references in OSDSingletonState::store_maps

Introduced: 3f11cd94
Fixes: https://tracker.ceph.com/issues/63996
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/shard_service.cc: convert to newer logging machinery

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd.cc: migrate logging to new style

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #55288 from athanatos/sjust/wip-64140

Revert "crimson/os/alienstore/alien_log: _flush concurrently"

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #54987 from batrick/i63822

pybind/mgr/devicehealth: skip legacy objects that cannot be loaded

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge pull request #54491 from jianwei1216/fix_osd_pg_stat_report_interval_max_cmain

fix: resolve inconsistent judgment of osd_pg_stat_report_interval_max

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #53250 from YiteGu/add-perfcount-for-allocator

os/bluestore: add perfcount for bluestore/bluefs allocator

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #52530 from amathuria/wip-amat-fix-59531

osd: Add memstore to unsupported objstores for QoS

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rbd-nbd: log errors during netlink_resize() using derr

When using rbd CLI to map the images to NBD devices via netlink,
any errors that arose during image resizing in netlink_resize()
were not logged. Switching the error logging from using cerr to
derr helps log the errors from netlink_resize().

Signed-off-by: Ramana Raja <rraja@redhat.com>

rbd_nbd: fix resize of images mapped using netlink

Include device identifier or cookie in the message sent to the kernel
to resize images mapped to NBD devices using netlink. Otherwise,
netlink_resize() fails and the size of the device isn't updated.

Fixes: https://tracker.ceph.com/issues/64139
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #49462 from rzarzynski/wip-bug-53789

osdc: fix the ENOCONN normalization in Objecter::_linger_reconnect()

Merge pull request #55219 from samarahu/rgw_asio_frontend_asserts

rgw/asio: Add asserts to rgw_asio_frontend.cc

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mgr/rook: increase minikube mem to 6GB to avoid stability issues

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #55192 from dparmar18/fix_docstrings_ceph_test_case

qa: typo fixes in ceph_test_case docstrings

Merge pull request #53320 from jzhu116-bloomberg/wip-62710

rgw/multisite: maintain endpoints connectable status and retry the requests to them when appropriate

Reviewed-by: Mark Kogan <mkogan@ibm.com>

Merge pull request #54941 from samsungceph/vstart_network_v2

vstart: Pick only CIDR-formatted routes when cephadm enabled

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53668 from mdw-at-linuxbox/wip-master-update-kmip-1

Update libkmip to pull in some portability changes.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/rook: adding some basic rook e2e testing
Fixes: https://tracker.ceph.com/issues/64029
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Revert "crimson/os/alienstore/alien_log: _flush concurrently"

While submitting the log line asyncronously is reasonable,
with this implementation the EntryVector &q parameter does
not necessarily outlive the submission continuation.

This reverts commit 511af83e2747361350b60ce0ce88e67a726d9343.

Fixes: https://tracker.ceph.com/issues/64140
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #55183 from galsalomon66/s3select_fixes_QE_bugs

rgw/s3select: bug fixes per QE recent defects

Update libkmip submodule to pull in some portability changes.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #55217 from ronen-fr/wip-rf-old-reserv

osd/scrub: check reservation replies for relevance

Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #55067 from yaarith/telemetry-pool-flags

mgr/telemetry: add pool flags

Merge pull request #55240 from rosinL/wip-fix-64032

install-deps: Force remove ceph-libboost* packages

Merge pull request #55278 from Himura2la/patch-2

Docs: Specify correct fs type for mkfs on volume creation

Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

bug fixes per QE recent defects
update for the engine_version message
s3select submodule

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

Merge pull request #55277 from yuvalif/wip-yuval-63578

rgw/lua: fix compilation issue when lua packages are disabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: specify correct fs type for mkfs

The default value is ext2, which is not supported (anymore?).

Signed-off-by: Vladislav Glagolev <vladislav.glagolev@devexpress.com>

rgw/lua: fix compilation issue when lua packages are disabled

Fixes: https://tracker.ceph.com/issues/63578#change-253102
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/dashboard: Fix inconsistency in capitalisation of "Multi-site"

fixes https://tracker.ceph.com/issues/64125

Across the dashboard, two instances are present: Multi-site and
Multi-Site.
Making it consistent all over by using Multi-site.

Signed-off-by: Afreen <afreen23.git@gmail.com>

osd/scrub: check reservation replies for relevance

Compare a token (nonce) carried in the reservation reply with the remembered
token of the reservation request.  If they don't match, the reply is
stale and should be ignored (and logged).

Fixes: https://tracker.ceph.com/issues/64052
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55107 from ronen-fr/wip-rf-rm-penaltyq

osd/scrub: remove the 'penalty queue' from the scrubber

Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #55269 from zdover23/wip-doc-2024-01-23-radosgw-admin-read-write-global-rate-limit-config

doc/radosgw: edit "read/write global rate limit" admin.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit "read/write global rate limit" admin.rst

Edit "Reading/Writing Global Rate Limit Configuration" in
doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55223 from athanatos/sjust/wip-64055

crimson: clear obc_registry on interval change

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rgw/multisite: add multisite test cases with some rgw instances down

Signed-off-by: Juan Zhu <jzhu4@dev-10-34-20-139.pw1.bcc.bloomberg.com>

Merge pull request #55070 from pdvian/wip-fix-progressevent

mon: initialize ProgressEvent::add_to_ceph_s

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #53154 from ifed01/wip-ifed-no-death-tests

test/store_test: get rid off assert_death.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #49415 from ljflores/wip-update-telemetry-upgrade

qa/workunits: update telemetry quincy workunits with `basic_pool_options_bluestore` collection

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

osd/scrub: update job's NB on failure

When a scrub job fails, update its NB to the current time plus a
fixed delay.  This prevents the job from being scheduled again
immediately.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: fix set_last_deep_scrub_stamp()

The call should update last_scrub_stamp, too, without
requiring an extra call to on_scrub_schedule_input_change()

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: introduce a 'not before' attribute for scrub jobs

The NB enables the OSD to delay the next attempt to schedule a specific
scrub job.  This is useful for jobs that have failed for whatever
reason, especially if the primary has failed to acquire the replicas.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: remove the 'penalized jobs' queue

The 'penalized jobs' queue was used to track scrub jobs that had failed
to acquire their replicas, and to prevent those jobs from being retried
too quickly.  This functionality will be replaced by a
simple 'not before' delay (see the next commits).

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55262 from Matan-B/wip-matanb-crimson-bluestore-submit

crimson/os/alienstore/alien_log: _flush concurrently

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/alienstore/alien_log: _flush concurrently

In continuation to c15e56e386251403a876454f6a4aa186284565e1

Authored-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55190 from zdover23/wip-doc-2024-01-16-radosgw-admin-enable-disable-bucket-rate-limit

doc/radosgw: edit "Enable/Disable Bucket Rate Limit"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53288 from rzarzynski/wip-crimson-dont-shadow-store-in-ecbackend

crimson: drop store from ECBackend to not shadow PGBackend::store

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54813 from amathuria/wip-crimson-amat-fix-config-set-cmd

src/crimson: Add support for the OSD to receive config changes

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55127 from idryomov/wip-63341

librbd: improve rbd_diff_iterate2() performance in fast-diff mode

Reviewed-by: Mykola Golub <mgolub@suse.com>

PendingReleaseNotes: add rbd_diff_iterate2 note

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: try to preserve object map for diff-iterate in fast-diff mode

As an optimization, try to ensure that the object map for the end
version is preloaded through the acquisition of exclusive lock and
as a consequence remains around until exclusive lock is released.
If it's not around, DiffRequest would (re)load it on each call.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/object_map: potentially use in-memory object map in DiffRequest

If the object map for the end version is around (already loaded in
memory, either due to the end version being a snapshot or due to
exclusive lock being held), use it to run diff-iterate against the
beginning of time.  Since it's the only object map needed in that
case, such calls would be satisfied locally.

Fixes: https://tracker.ceph.com/issues/63341
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/object_map: decouple object map processing in DiffRequest

In preparation for potentially using in-memory object map, decouple
object map processing from loading object maps and place the logic in
prepare_for_object_map() and process_object_map().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

common/bit_vector: fix iterator vs reference constness confusion

T (ConstIterator or Iterator) is confused with const T here:
IteratorImpl dereference operator is wrongly overloaded on const
and returns Reference instead of ConstReference for ConstIterator.
This then fails inside bufferlist bowels because Reference is
incompatible with bufferlist::const_iterator.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/object_map: make object map in handle_load_object_map() local

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/object_map: don't resize object map in handle_load_object_map()

Currently it's done in two cases:

- if the loaded object map is larger than expected based on byte size,
  it's truncated to expected number of objects
- in case of deep-copy, if the loaded object map is smaller than diff
  state, it's expanded to get "track the largest of all versions in the
  set" semantics

Both of these cases can be easily dealt with without modifying the
object map.  Being able to process a const object map is needed for
working on in-memory object map which is external to DiffRequest.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

common/bit_vector: fix IteratorImpl post-increment operator

It's totally broken: instead of returning the current position and
moving to the next position, it returns the next position and doesn't
move anywhere.  Luckily it hasn't been used until now.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: drop DiffIterate::diff_object_map() declaration

This is a leftover from commit 2b3a46801d39 ("librbd: switch
diff-iterate API to use new object-map diff helper").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>