mgr/dashboard: Migrate from promtail to grafana alloy

Since promtail is now deprecated, we need to start using grafana alloy for centralized logging setup

Fixes: https://tracker.ceph.com/issues/71072
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge PR #64324 into main

* refs/pull/64324/head:
qa/cephfs: fix test_subvolume_group_charmap_inheritance test
doc: add name mangling documentation for subvolume group creation
qa: add tests for name mangling in subvolume group creation
pybind/mgr: add name mangling options to subvolume group creation

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #64992 from rhcs-dashboard/tiering-enhancement

mgr/dashboard: Tiering form -  Placement Target in Advanced Section

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: pujashahu <pshahu@redhat.com>

Merge pull request #65115 from rhcs-dashboard/72628-fix-inputfield-css-carbon

mgr/dashboard : fix css for carbon input fields

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #64683 from tobias-urdin/rgwadmin-max-entries-fix

radosgw-admin: Pass max_entries for bucket list

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #65321 from guits/support-extra-dmcrypt-options

ceph-volume: support additional dmcrypt params

Merge pull request #64855 from rhcs-dashboard/72431-mirrored-images-have-incorrect-usage-info

mgr/dashboard : Fixed mirrored image usage info bar

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #65312 from rhcs-dashboard/72788-group-similar-alerts

mgr/dashboard: Group similar alerts

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #64190 from SundownRises/carbon-toast

mgr/dasboard: Carbonised Toast Notification

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

qa/cephfs: fix test_subvolume_group_charmap_inheritance test

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #65158 from jzhu116-bloomberg/wip-72667

rgw: discard olh_ attributes when copying object from a versioning-suspended bucket to a versioning-disabled bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #65393 from MaxKellermann/rgw_includes

rgw: include cleanup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #64800 from BBoozmen/wip-72375-bulk_delete_olh

RGW: multi object delete op; skip olh update for all deletes but the last one

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #65470 from rhcs-dashboard/add-user-agent

monitoring: add user-agent headers to the urllib

Merge pull request #65428 from rkachach/fix_rgw_docs_certmgr

doc: update RGW HTTPS configuration to use certmgr and new fields

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #65406 from rkachach/fix_grafana_docs

doc: update Grafana certificate configuration to use certmgr

Reviewed-by: Adam King <adking@redhat.com>

monitoring: add user-agent headers to the urllib

The documentation started raising 403 suddenly. Adding User-Agent
headers to the request

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge PR #65320 into main

* refs/pull/65320/head:

Reviewed-by: Anoop C S <anoopcs@cryptolab.net>

Merge pull request #64463 from ljflores/wip-qa-summary-script

script: add script to help format QA review summaries

Merge pull request #58926 from TRYTOBE8TME/wip-shard-id-option

src/rgw: Adding "sync error trim" option

mgr/dashboard: Tiering form - Placement Target in Advanced Section

Fixes: https://tracker.ceph.com/issues/72545
Signed-off-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

mgr/dashboard: Carbonised Toast Notification
Used carbon toast component to carbonise toast notifications

Dashboard: Toast Notification carbonised
This PR replaces the existing ngx-toastr implementation with Carbon Design System toast notifications to maintain UI consistency across the Ceph dashboard application.

Fixes:https://tracker.ceph.com/issues/71735
Signed-off-by: Anikait Sehwag <anikaitsehwag.amg@gmail.com>

libcephfs_proxy: fix userperm pointer decoding for older protocols

The random data used to decode pointers coming from the old protocol was
taken from the client instead of using the global_random data, which is
the correct one.

Fixes: https://tracker.ceph.com/issues/72800
Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: remove unnecessary protocol references in daemon

With the new protocol structure definitions, it's not necessary to
explicitly access each field inside its version substructure (v0, for
example). Now all fields of the latest version are declared inside an
anonymous substructure that can be accessed without a prefix.

Fixes: https://tracker.ceph.com/issues/72800
Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: remove unnecessary protocol references in client

With the new protocol structure definitions, it's not necessary to
explicitly access each field inside its version substructure (v0, for
example). Now all fields of the latest version are declared inside an
anonymous substructure that can be accessed without a prefix.

Fixes: https://tracker.ceph.com/issues/72800
Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: fix protocol structures for backward compatibility

The structures used for transferring data between the proxy client and
the proxy daemon had been reworked in a recent change to be able to
expand the protocol. This caused an inconsistency in the size of the
data transferred when communication with a peer using the older version.
The result was that the peer receiving the data with an unexpected size
was closing the connection, causing unexpected errors.

The discrepancy in size is the result of how compilers pad structures
combined with the change in the structure layout introduced when
extending the protocol. With these changes, the computation of the size
of each version of the structures was not done correctly.

This change makes the layout equal to the older version, so that
computing the size of the structures becomes easier and doesn't depend
on unexpected paddings.

Fixes: https://tracker.ceph.com/issues/72800
Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

Merge pull request #63895 from Kushal-deb/rgw-qat-compression

cephadm: improve hw qat experience with cephadm

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #65387 from yuvalif/wip-yuval-72542

rgw/logging: allow committing empty objects

Merge pull request #64844 from ljflores/wip-tracker-72312

qa/tasks/thrashosds-health: fine tune ignorelist for degraded and undersized pgs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Group similar alerts
fixes : https://tracker.ceph.com/issues/72788

Signed-off-by: Abhishek Desai <abhishek.desai1@ibm.com>

Merge pull request #64375 from SundownRises/header-component

mgr/dashboard: Carbonised Notification Header

Reviewed-by: Afreen Misbah <afreen@ibm.com>

doc: update RGW HTTPS configuration to use certmgr and new fields

With the introduction of certmgr, RGW services now support three
certificate sources: cephadm-signed (default), inline, and reference.
Docs have been updated to:

- Show how to provide inline certificates using the new ssl_cert/ssl_key
  fields instead of the deprecated rgw_frontend_ssl_certificate.
- Explain how to register and reference user-provided certs/keys
- Clarify that cephadm-signed certificates remain the default, with
  optional wildcard SANs support.

The usage of rgw_frontend_ssl_certificate is still supported for
backward compatibility, but is now documented as deprecated.

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #59515 from kamoltat/wip-ksirivad-fix-67801

mon [stretch mode]: restrict changing mon election strategy post stretch mode

Merge PR #63636 into main

* refs/pull/63636/head:

Reviewed-by: Christopher Hoffman <choffman@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #65015 from connorfawcett/pg-autoscale-threshold-cmd

mgr/pg_autoscaler: Add 'osd pool get threshold' command which returns the current threshold value

Merge pull request #64788 from rhcs-dashboard/acl-mapping

mgr/dashboard:RGW- Storage Class ACL Mapping

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge PR #64958 into main

* refs/pull/64958/head:

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #64967 into main

* refs/pull/64967/head:

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #65436 from gbregman/main

mgr/cephadm/nvmeof: Add fields for prometheus frequency to NVMEOF spec file

test/libcephfs: validate asynchronous write and fsync executing concurrently

This synthetic reproducer does three things:

- setup a client mount with a configuration to delay write operations and
  initiates a write operation via a thread.
- a thread that invokes asynchronous fsync
- a thread that invokes setxattr for the client to track early replies

Without the fix[0], the test reproduces the following crash:

```
/home/vshankar/ceph/src/client/Client.cc: In function 'void Client::put_request(MetaRequest*)' thread 7f7210ff9640 time 2025-06-03T09:34:45.634974+0000
/home/vshankar/ceph/src/client/Client.cc: 2290: FAILED ceph_assert(request->ref >= 1)
 ceph version 20.3.0-673-gdd152807f7e (dd152807f7e7f7a82df6cfc0159f5fc65f60ecd5) tentacle (dev - Debug)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x119) [0x7f72222ebb98]
 2: (ceph::__ceph_assert_fail(ceph::assert_data const&)+0x17) [0x7f72222ebedc]
 3: /home/vshankar/ceph/build/lib/libcephfs.so.2(+0x6a075) [0x7f7222e6a075]
 4: /home/vshankar/ceph/build/lib/libcephfs.so.2(+0xb8289) [0x7f7222eb8289]
 5: /home/vshankar/ceph/build/lib/libcephfs.so.2(+0xee951) [0x7f7222eee951]
 6: /home/vshankar/ceph/build/lib/libcephfs.so.2(+0xf167c) [0x7f7222ef167c]
 7: (Context::complete(int)+0x9) [0x7f7222e5949d]
 8: /home/vshankar/ceph/build/lib/libcephfs.so.2(+0x16a853) [0x7f7222f6a853]
 9: /home/vshankar/ceph/build/lib/libcephfs.so.2(+0xa7cc5) [0x7f7222ea7cc5]
 10: /home/vshankar/ceph/build/lib/libcephfs.so.2(+0xf128d) [0x7f7222ef128d]
 11: /home/vshankar/ceph/build/lib/libcephfs.so.2(+0x16e09d) [0x7f7222f6e09d]
 12: (Context::complete(int)+0x9) [0x7f7222e5949d]
 13: /home/vshankar/ceph/build/lib/libcephfs.so.2(+0x6d108) [0x7f7222e6d108]
 14: (Context::complete(int)+0x9) [0x7f7222e5949d]
 15: (Finisher::finisher_thread_entry()+0x665) [0x7f722226fdc1]
 16: (Finisher::FinisherThread::entry()+0xd) [0x7f7222270ddf]
 17: (Thread::entry_wrapper()+0x2f) [0x7f72222b88f5]
 18: (Thread::_entry_func(void*)+0x9) [0x7f72222b8907]
 19: /lib64/libc.so.6(+0x89e92) [0x7f7221089e92]
 20: /lib64/libc.so.6(+0x10ef20) [0x7f722110ef20]
[1]    2162689 IOT instruction (core dumped)  ./bin/ceph_test_libcephfs --gtest_filter=LibCephFS.ConcurrentWriteAndFsync
```

[0]: https://github.com/ceph/ceph/pull/63619

Fixes: http://tracker.ceph.com/issues/71515
Signed-off-by: Venky Shankar <vshankar@redhat.com>

client: catch buggy reference count drop for MetaRequest

With the prior commit that introduces a synthetic delay in write
operation so as to write a test reproducer which would interleave
asynchronous fsync and an operation that makes the MDS send a early
reply to the client (therefore, having the client track the early
replied response for an inode in Inode::unsafe_ops). Now, this is
enough to trick the client into the code path that causes a buggy
reference drop for the request (MetaRequest), but, hitting the
_exact_ crash backtrace requires the request to be a in various
[x]list's.

This last bit is tricky to synthetically massage in the test. So,
in order to catch the buggy reference drop, it would suffice to
assert on the reference count dropping to less than zero (0).

Signed-off-by: Venky Shankar <vshankar@redhat.com>

client: synthetically delay write operation

To allow the client to hold Fb caps for an extended period of
time, to allow an asynchronous fsync to intervene and block, so
as to hunt [0].

[0]: https://tracker.ceph.com/issues/71510

Signed-off-by: Venky Shankar <vshankar@redhat.com>

client: log unsafe operation count (for debugging)

Signed-off-by: Venky Shankar <vshankar@redhat.com>

libcephfs/client: asynchronous fsync interface

Mostly for writing test for hunting [0].

[0]: https://tracker.ceph.com/issues/71510

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #64999 from rishabh-d-dave/fs-pyx-chown

cephfs.pyx: handle when UID/GID passed to chown() is -1

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #64927 from NitzanMordhai/wip-nitzan-suites-ignore-still-running-cephadm-osds-suites

suites/rados/cephadm: typo in ignotr list for still running message

Merge pull request #65431 from afreen23/doc-release-notes

Update dashboard Pending release notes

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #65419 from ljflores/wip-tracker-72897

doc/rados/operations: add kernel client procedure to read balancer documentation

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #64837 from rzarzynski/wip-bug-72412

osd: stop scrub_purged_snaps() from ignoring osd_beacon_report_interval

mgr/cephadm/nvmeof: Add fields for prometheus frequency to NVMEOF spec file.

Fixes: https://tracker.ceph.com/issues/72805
Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #65384 from rhcs-dashboard/fix-72868-main

mgr/dashboard: fix RGW Bucket Notification Dashboard units

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

doc: wq!Update dashboard Pending release notes

- added for tentacle
- moved the inccorect notes added in tentacke to umbrella

Signed-off-by: Afreen Misbah <afreen@ibm.com>

doc/rados/operations: add kernel client procedure to read balancer documentation

As of now, the kernel client does not support `pg-upmap-primary`. I have
added some troubleshooting steps to help users who are unable to
mount images and filesystems with the kernel client while using `pg-upmap-primary`.

Once the feature is supported by the kernel client, users will be able
to perform mounts along with `pg-upmap-primary`.

Fixes: https://tracker.ceph.com/issues/72897
Signed-off-by: Laura Flores <lflores@ibm.com>

rgw/logging: allow committing empty objects

Fixes: https://tracker.ceph.com/issues/72542
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #62747 from bill-scales/issue70844

test: add replica pool support to ceph_test_rados_io_sequence

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Connor Fawcett <connorfa@uk.ibm.com>

Merge pull request #65281 from nbalacha/wip-nbalacha-72740

rgw/logging: fixes data loss during rollover

Merge pull request #65231 from anthonyeleven/improve-osd-dot-cc

src/osd: Improve message in OSD.cc

rgw/logging: fixes data loss during rollover

Multiple threads attempting to roll over the same log object can result
in the creation of numerous orphan tail objects, each with a single record.
This occurs when a NULL RGWObjVersionTracker is used during the creation of
a new logging object. These records are inaccessible, leading to data loss,
which is particularly critical in Journal mode.
Furthermore, valid log tail objects may be added to the Garbage Collection (GC)
list, exacerbating data loss.

Fixes: https://tracker.ceph.com/issues/72740
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>

Merge pull request #65421 from ronen-fr/wip-rf-ec72898

osd/scrub: clear m_ec_digest_map between objects

Reviewed-by: Jon Bailey <jonathan.bailey1@ibm.com>

Merge pull request #62106 from rkachach/fix_certmgr_v2

Add cephadm-signed certificate support for all services

Reviewed-by: John Mulligan <jmulligan@redhat.com>

test: add replica pool support to ceph_test_rados_io_sequence

Make 'ceph_test_rados_io_sequenece --pool rbd' work, replica
pools don't have an erausre code profile and do not have the
ec_allow_overwrites or ec_allow_optimizations flags

Fixes: https://tracker.ceph.com/issues/70844
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

common: Added values to json::OSDPoolGetReply

OSDPoolGetReply actually returns a lot more values than what is currently supplied. These have been added in as optionals (as they can not be give as well) so its possible to query them to find out if they exist and use them if they do.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

osd/scrub: clear m_ec_digest_map between objects

Fixing a bug introduced by commit 4c61079e931
("caluculate EC digest map size only once").

Fixes: https://tracker.ceph.com/issues/72897
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #65150 from rkachach/fix_issue_nvmeof_prometheus

mgr/cepahdm: fixing nvmeof scraping Prometheus config generation

Reviewed-by: Kushal Deb <Kushal.Deb@ibm.com>

Merge pull request #65338 from ronen-fr/wip-rf-be-st3

osd/scrub: modify OMAP stats collection

Reviewed-by: Jon Bailey <jonathan.bailey1@ibm.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #65193 from tchaikov/wip-osd-scrub-fix-buffer-overflow

osd/scrub: fix heap-buffer-overflow when checking digest emptiness

Reviewed-by: Jon Bailey <jonathan.bailey1@ibm.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

mgr/cepahdm: fixing nvmeof scraping Prometheus config generation

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

doc/cephadm: updating certmgr docs to reflects new changes

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: Adding RGW migration for the new certmgr certs format

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: removing call to previous method to storing all certs

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: fixing nvmeof section in cert_mgr UT + new UT

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adapting UT after all the changes

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: using 5 years for service-discovery internal certs

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding self-signed certifiactes support for nvmeof svc

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adapting node-proxy service to use the new cert mgmt

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adapting agent service to use the new cert mgmt

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adapting node-exporter service to use the new cert mgmt

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adapting Prometheus service to use the new cert mgmt

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adapting alertmanager service to use the new cert mgmt

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adapting oauth2-proxy service to use the new cert mgmt

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adapting Grafana service to use the new cert mgmt

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adapting mgmt-gateway service to use the new cert mgmt

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adapting iscsi service to use the new cert mgmt

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adapting ingress svc to use the new cert mgmt approach

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding automated certificates mgmt for cephadmservice

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adding new filtring options and certificate ref checks

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding support to prevent editing readonly certs

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding enahanced support for self-signed certs

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: add support for custom duration when generating certs

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adding support to extract ips and fqdns from cert

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adapting python-common UT to the spec changes

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding new spec generic fields fo ssl/certifiactes

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

osd/scrub: reinstate one-warning-per-chunk behaviour

Modify collect_omap_stats() to guarantee that only
one 'large omap entry' warning message is logged
per chunk, thus maintaining the existing behaviour.

Unlike the existing behaviour - all 'large omap'
entries are counted.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: modify OMAP stats collection

Replace the separate all-chunk loop with object-by-object
handling. Use the selected authoritative version of the object
as the info source.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #64859 from ronen-fr/wip-rf-72426movedref

osd/scrub: avoid using moved-from auth_n_errs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #64963 from edwinzrodriguez/ceph-wip-72452

test/rgw: Refactor CORS configuration handling for S3

Merge pull request #64801 from JoshuaGabriel/wip-jblanch/rgw-reindex

rgw: auto-create missing bucket index shards during reindex operations

Merge pull request #65414 from ceph/fix-api-tests

pybind/mgr/dashboard: Use teuthology's actual requirements

pybind/mgr/dashboard: Use teuthology's actual requirements

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #65011 from irq0/pr/fix-barbican-deprecated-call

rgw: Replace deprecated Barbican payload endpoint with alternative