client: check mds down status bofore getting mds_gid_t from mdsmap

Get mds_gid_t from up of MDSMap, will cause to core dump if target
mds_rank_t does not exist up map:

from: src/mds/MDSMap.h
  const auto& get_info(mds_rank_t m) const {
    return mds_info.at(up.at(m));
  }

reproduct:
1. ceph fs fail <fs_name>
2. curl -X GET "example.com:8080/api/cephfs/1/clients" ...
up.at(m) will cause to core dump.

Fixes: https://tracker.ceph.com/issues/61844
Signed-off-by: Yite Gu <yitegu0@gmail.com>

mgr/dashboard: allow sending back error status code fetching clients fails

Fixes: https://tracker.ceph.com/issues/64089
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit 564dba38c49864e71958ddcd970856b834e9c6b7)

Merge pull request #55330 from baum/centos9-install-build-deps

🧹build dependencies: centos9

Merge PR #55233 into main

* refs/pull/55233/head:
qa: use centos 9.stream for cephfs stock kernel testing

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53901 from cbodley/wip-qa-container-distros-s

qa/distros: bump container host distros from centos 8->9

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55413 from petrutlucian94/flipped_tid

msg: update MOSDOp() to use ceph_tid_t instead of long

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

msg: update MOSDOp() to use ceph_tid_t instead of long

The MOSDOp constructor receives the the transaction ID as a long
instead of ceph_tid_t.

The issue is that "long" uses 32b on Windows instead of 64 bits,
so it flips after about 2 billion requests. At that point, the OSD
replies are dropped because of transaction ID mismatches.

We'll solve the issue by using the correct type for the transaction
id, specifically ceph_tid_t.

Fixes: https://tracker.ceph.com/issues/64291
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #55373 from ivoalmeida/snapshot-schedule-form-directory-typeahead

Snapshot schedule form directory typeahead

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55393 from rkachach/fix_issue_64266

mgr/rook: adding deployment to ceph image built for rook e2e testing

mgr/rook: adding deployment to ceph image built for rook e2e testing
Fixes: https://tracker.ceph.com/issues/64266
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #47723 from mdw-at-linuxbox/wip-master-tempurl

rgw: swift: tempurl fixes for ceph

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55403 from cbodley/wip-doc-rgw-topic-attrs

doc/rgw: fix Attributes index in CreateTopic example

Merge pull request #54742 from guits/node-proxy

orch: implement hardware monitoring

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #55397 from jagombar/fixInvalidSnapId2

librbd: return ENOENT from Snapshot::get_timestamp for nonexistent snap_id

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rgw: fix Attributes index in CreateTopic example

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: added dir search to snap schdl form

Fixes: https://tracker.ceph.com/issues/64246
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #54781 from rhcs-dashboard/nvmeof-integration

mgr/dashboard: implement APIs for nvmeof management

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #54900 from ivoalmeida/snapshot-schedule-create

added snap schedule form

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55340 from ronen-fr/wip-rf-reserver2

osd/scrub: use an AsyncReserver to handle scrub reservations on the replica side

Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #55198 from JoshSalomon/balancer-perf

osd: Read balancer for devices with different sizes (performance)

doc: add node-proxy documentation

This commit adds some documentation about the
'hardware inventory / monitoring' feature (node-proxy agent).

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #55396 from zdover23/wip-doc-2024-01-31-arch-monitors

doc/architecture: improve some paragraphs

Merge pull request #55383 from ceph/cpp-redis-cxx-standard

cpp_redis: update to a version with the CMAKE_CXX_STANDARD fixed

Reviewed-by: Ali Maredia <amaredia@redhat.com>

librbd: return ENOENT from Snapshot::get_timestamp for nonexistent snap_id

Fixes: https://tracker.ceph.com/issues/47287
Signed-off-by: John Agombar <agombar@uk.ibm.com>

doc/architecture: improve some paragraphs

Improve paragraphs under the heading "The Ceph Storage Cluster". Remove
a sentence that was pleonastic in its context in the paragraph.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: added snap schedule form

Fixes: https://tracker.ceph.com/issues/63827
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #55305 from Matan-B/wip-matanb-crimson-do_delete_work

crimson/osd/pg: do_delete_work use claim_and_reset

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

cpp_redis: update to a version with the CMAKE_CXX_STANDARD fixed

This updates the cpp_redis submodule to a version
which contains https://github.com/ceph/cpp_redis/pull/5

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

test: fix test_scrub_sched following recent scrub changes

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: restore 'dump_scrub_reservations' functionality

As ScrubResources is no longer involved in remote reservations, some
of the data listed by 'dump_scrub_reservations' is now collected by
OsdScrub itself (prior to this change, OsdScrub just forwarded the
request to ScrubResources).

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: control reservation queueing using a config option

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: directly manage remote reservations in the FSM

The FSM now interacts with the scrub_reserver directly.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

vstart: add nvmeof_gw to the vstart script

so that it can be later used by the dashboard to configure the nvmeof
through UI

and create rbd pool in UI

Fixes: https://tracker.ceph.com/issues/64201
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55379 from ljflores/wip-revert-labler-changes

tools: add 'osd-size-aware' mode to osdmaptool

Sample command:
`osdmaptool om --read out.txt --read-pool cephfs.a.data --osd-size-aware`

Signed-off-by: Laura Flores <lflores@ibm.com>

mgr/balancer: add new keys to balancer mgr module

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #55372 from zdover23/wip-doc-2024-01-30-architecture-typo

doc/architecture.rst - fix typo

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

osd: unit tests for read balance osd size aware score

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>

osd: Add score for read balance osd size aware policy

This score works for pools in which the read_ratio
value is set.
Current limitations:
- This mechanism ignores osd read affinty
  - There is a plan adding support for read affinity 0
    in the next version.
- This mechanism works only when all PGs are full
- If read_ration is not set - the existing mechanism (named
  fair score) is used.

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>
Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>

osd: Unit tests for read balancer on OSDs with different sizes

Test cases for the read balancer which takes osd sizes into account.
Some balancing code refactoring and reorg for code that is used in
multiple tests

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>

osd: Read balancer for OSDs with different sizes

This commit adds calculation for desired primary distribution which
takes into account the osd size. This way smaller OSDs can take more
read operations (by adding more primaries) and the larger OSDs take less
primaries and the load of the cluater can increase. (This feature offset
a bit the weakest link in the chain effect under some conditions). In
order to calculate the loads correctly there is a need to know the
read/write ratio for the pool, and this commit assumes the read_ratio
parameter is available for the pool.

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>

osd: Add 'read_ratio' pool parameterr

This parameter is used for better read balancing with non identical
devices.
- This parameter is controlled using the commands 'ceph osd pool set/get'
- This parameter is applicable only for replicated pools
- Valid values are integers in the range [0..100] and represent the
  percentage of read IOs out of all IOs in the pool
  - Value of 0 unsets this parameter and the value will be the default
    value (this is the generic behavior of the command 'ceph osd pool
    set'
  - default value can be set by config parameter
    `osd_pool_default_read_ratio`

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>

.github/workflows: revert version bump to labler action

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #55364 from ljflores/wip-pr-triage

.github/workflows: disable sync-labels in pr-triage

agent/node-proxy: fix wrong host name used in data endpoint

data['cephx']['name'] will return something like:

node-proxy.hostname123

the prefix "node-proxy." has the be removed otherwise there will be
a mismatch between what is actually expected.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

mgr/cephadm: update node-proxy unit tests

The recent migration to a separate daemon implied
some changes which have broken these tests.
This commit fixes them.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

mgr/cephadm: add a new config option 'oob_default_addr'

So there's a default value (169.254.1.1) which is the default
address for the 'OS to iDrac pass-through' interface.
Given that node-proxy will reach the RedFish API through this interface,
we can make users avoid to pass that addr when providing the host spec
at bootstrap time.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

qa/distros: add supported-container-hosts for cephadm and upgrade suites

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53610 from Patty8122/documentation-update-fsck-flagdeep

doc/man/8: Add valid options in ceph-bluestore-tool.rst

Merge PR #54515 into main

* refs/pull/54515/head:
qa: get_global_id() returns a string (global_id)
qa: correct usage of DEBUGFS_META_DIR in dedent

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #55313 from rkachach/fix_issue_64183

mgr/rook: adding metrics monitoring e2e testing

doc/architecture.rst - fix typo

s/requies/requires

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55284 from rhcs-dashboard/cephfs-clone-create

mgr/dashboard: create cephfs snapshot clone

mgr/dashboard: adapt to new nvmeof APIs

Fixes: https://tracker.ceph.com/issues/64201
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: introduce APIs for NvmeOf management

Fixes: https://tracker.ceph.com/issues/64201
Signed-off-by: avanthakkar <avanjohn@gmail.com>

mgr/dashboard: gRPC client for nvmeof communication

Created a gRPC client by utilising the protobuf file available in nvmeof
repo

Copied the file to this repo and generated its output.

Fixes: https://tracker.ceph.com/issues/64201
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/cephadm: configure the dashboard gateways

cephadm configures the nvmeof gateways and add the gateways to a config
store which dashboard will later on fetch to make the grpc calls.

Fixes: https://tracker.ceph.com/issues/64201
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55351 from rosinL/wip-fix-64122

script/run-make: install lvm2 for make check cephadm test

Merge pull request #55228 from rhcs-dashboard/fix-64081-main

mgr/dashboard: Export rgw multisite status via the API

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

osd/scrub: remove remote reservation code from ScrubResources

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

common: allow AsyncReserver requests with no callbacks

As no callbacks are needed for request_reservation_or_fail(),
the synchronous request API.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: add a general data-carrying event type to the FSM

Based on Sam's Crimson FSM's 'value-event'.

Co-authored-by: Sam Just <sjust@redhat.com>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: define the scrub_reserver to track remote reservations

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #52652 into main

* refs/pull/52652/head:
PendingReleaseNotes: add note about new mdlog trimming configurations
mds: drive mdlog trimming via a separate thread
mds: allow runtime modification of mdlog trimming configuration
mds: remove a bunch of heuristics from MDLog::trim()
mds: add mdlog trimming threshold and decay counter

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #55184 into main

* refs/pull/55184/head:
qa: remove redundant test test_volumes.TestRenameCmd.test_rename_when_fs_is_online

Reviewed-by: Christopher Hoffman <choffman@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #55346 into main

* refs/pull/55346/head:
qa: remove error string checks and check w/ return value

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #55349 from afreen23/fix-64112

mgr/dashboard: Create subvol of same name in different group

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

PendingReleaseNotes: add note about new mdlog trimming configurations

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: drive mdlog trimming via a separate thread

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: allow runtime modification of mdlog trimming configuration

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: remove a bunch of heuristics from MDLog::trim()

These were probbaly introduced to workaround some sort of
resource overusage by the MDS during trimming, but now it
looks like they are not really neeeded, especially if we
introduce a dedicated thread for log trimming.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: add mdlog trimming threshold and decay counter

Fixes: http://tracker.ceph.com/issues/61908
Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa: remove error string checks and check w/ return value

I ran into this failure once #54972 was merged. The test is validating
the error string returned due to the failed mount. There aren't any
return value checks - which is a _more_ important check. Generic error
string checks will fail once a (error) string is changed (typo, etc..).

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge PR #54808 into main

* refs/pull/54808/head:
client: fix copying bufferlist to iovec structures in Client::_read
src/test: test sync call providing nullptr as ctx to async api

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Frank S. Filz <ffilzlnx@mindspring.com>

Merge PR #54031 into main

* refs/pull/54031/head:
qa: add test to mangle lost+found directory object and ensure safety
qa: run scrub before mounting client and validations

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge pull request #55241 from rhcs-dashboard/remove-bucket-policy

mgr/dashboard: Remove bucket policy

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Export rgw multisite status via the API

Fixes: https://tracker.ceph.com/issues/64081
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

.github/workflows: disable sync-labels in pr-triage

`sync-labels` is defined as "Whether or not to remove labels
when matching files are reverted or no longer changed by the PR".

We should disable this in case someone manually adds a label,
so the label doesn't get removed.

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #55326 from ceph/dependabot-github_actions-actions-labeler-5.0.0

.github: Bump actions/labeler from 4.0.2 to 5.0.0

mgr/dashboard: remove bucket policy
add a button on the bucket form to clear the bucket policy

Fixes: https://tracker.ceph.com/issues/64096
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #55062 from rhcs-dashboard/set-bucket-policy

mgr/dashboard: set bucket policies

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54853 from ivoalmeida/snapshot-schedule-module

mgr/dashboard: snap schedule module detect/enable

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55255 from phlogistonjohn/jjm-mypy-updates

Various mypy related updates

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #55329 from cbodley/wip-64189

rgw/rest: fix url decode of post params for iam/sts/sns

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

mgr/dashboard: set bucket policies

- Adds support to set bucket policies through the Dashboard.
- Rename rgw bucket policy from 'policy' to 'bucket policy' and tab 'Permissions' to 'Policies'
- Fix: hide Tags when none are present on bucket list details and sets bucket form dirty after deleting a tag
- Added service to manage the formatting of a textArea that works with json

Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Fixes: https://tracker.ceph.com/issues/63942

Merge PR #54446 into main

* refs/pull/54446/head:
qa: bump up scrub status command timeout

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #54683 into main

* refs/pull/54683/head:
qa/tasks/cephfs/test_misc: switch duration to timeout
qa/tasks/cephfs/test_misc: fix the incorrect comments

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #53578 into main

* refs/pull/53578/head:
mds: adhere to mds_health_summarize_threshold config when reporting laggy clients

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #53177 into main

* refs/pull/53177/head:
qa: test nfs export apply -i failure returns pseudo path in output JSON
mgr/nfs: include pseudo in JSON output when nfs export apply -i fails

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge PR #53734 into main

* refs/pull/53734/head:
qa: refactor client upgrade yamls and other minor touchups
qa/upgrade/nofs: upgrade pacific->reef
qa/upgrade/upgraded_client: upgrade nautilus->pacific and pacific->reef

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #55309 into main

* refs/pull/55309/head:
qa: `fs volume rename` requires `fs fail` and `refuse_client_session` set

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

mgr/rook: adding metrics monitoring e2e testing
Fixes: https://tracker.ceph.com/issues/64183
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge PR #54451 into main

* refs/pull/54451/head:
common/admin_socket: improvements to the RaiseHook

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/osd/pg_map: add assertion

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg_map: increase verbosity

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

script/run-make: install lvm2 for make check cephadm test

The make check test `run-tox-cephdam` reports error:
```
        if errors:
>           raise Error('\nERROR: '.join(errors))
E           cephadmlib.exceptions.Error: lvcreate binary does not appear to be installed

cephadm.py:4434: Error
```
So let's installing lvm2 for make check cephadm test.

Fixes: https://tracker.ceph.com/issues/64122
Signed-off-by: luo rixin <luorixin@huawei.com>

mgr/dashboard: snap schedule module detect/eneable

Fixes: https://tracker.ceph.com/issues/63768
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

osd: improve MOSDScrubReserver log formatting

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

common: add some missing include files to AsyncReserver.h

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55131 from ronen-fr/wip-rf-reserver

osd/scrub: partial implementation of scrub reserver
Reviewed-by: Samuel Just <sjust@redhat.com>-