mgr: Modify unit tests to reflect changes
Modify calc_final_pg_target test cases to reflect optimization algorithm
Add test cases that ensure cost is respected to ensure pools get rounded
closely to target

Fixes: https://tracker.ceph.com/issues/73418
Signed-off-by: Eric Zhang <emzhang@ibm.com>

mgr: Fix autoscaler not distributing PGs evenly + wrong pg_left update
Add optimization algorithm to distribute PGs among pools
to minimize cost (absolute difference between target PG and rounded PG).
Fix autoscaler still allocating pools when autoscale is off but bulk is on

Fixes: https://tracker.ceph.com/issues/73418
       https://tracker.ceph.com/issues/75424
Signed-off-by: Eric Zhang <emzhang@ibm.com>

Merge pull request #67665 from tchaikov/wip-test-rgw-fix-warning

test/rgw: fix -Wstring-plus-char warning in test_d4n_policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #67189 from cbodley/wip-74231

rgw: fix 'bucket stats' when bucket index doesn't exist

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #67958 from cbodley/wip-qa-rgw-compress-encrypted

qa/rgw: crypt subsuite tests compress-encrypted

Reviewed-by: Matthew N. Heler <matthew.heler@hotmail.com>

Merge pull request #68393 from Hezko/traddr-compat

mgr/dashboard: introduce traddr for backward compatibility

qa: Add direct_io test for fscrypt

Fixes: https://tracker.ceph.com/issues/73347
Signed-off-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #67841 from rhcs-dashboard/fix-pools-app-list-names

mgr/dashboard : Fix application names in pools form

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

Merge pull request #68011 from mheler/wip-75650

rgw: always decompress in CopyObject when source is compressed

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #68324 from mheler/wip-multipart-cloud-transition

rgw: fix cloud tier multipart resume using part number 0

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #67560 from mheler/wip-cloud-restore-etag-fix

rgw/cloud-restore: strip quotes from ETag on cloud tier fetch

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #67966 from mheler/wip-75649

rgw: fix fixup_range cascade for compressed+encrypted objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw/gc: mark rgw_gc_max_objs as startup-only

Raising rgw_gc_max_objs at runtime crashes the GC thread.

Reject runtime changes and size the IOManager vectors from the
stored max_objs.

Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>

Merge pull request #68466 from bluikko/wip-doc-fix-tentacle-releases

doc/releases: fix Tentacle data in releases.yml

qa/rgw: crypt subsuite tests compress-encrypted

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: rgw task can enable/disable zone features

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: don't duplicate 'user list' commands for default zone

some commands during setup expect the zone to exist already, so run
'radosgw-admin user list' to make sure a default zone/zonegroup are
created. avoid duplicating this in several subtasks by moving this to
its own subtask that runs when a realm is not configured

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson/tests: fix test_remap_pin_concurrent

The test failed in the following sequence:

  1. 32 coroutines start in parallel, all calling try_get_pin(t, offset)
     with the same offset
  2. One coroutine succeeds in submit_transaction, which calls
     test_mappings.consume(delta) and modifies the shared test_mappings
  3. Other coroutines that haven't reached try_get_pin yet now hit the
     assertion because offset no longer exists in test_mappings

The fix is to add a guard before try_get_pin to check if test_mappings still
contains the offset. If it doesn't, we can safely exit early since that
means another coroutine already succeeded and modified the shared state.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

qa/suites/rados/thrash-old-clients: ignore "Reduced data availability" warning

This type of warning is typical during tests where we are thrashing the
OSDs.

Fixes: https://tracker.ceph.com/issues/73147
Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #67863 from mheler/wip-bug-75569

rgw: stop crashing when an admin tries to assume a nonexistent role

rgw: fix cloud tier multipart resume starting at part number 0

When resuming a cloud tier multipart upload, the part-size
calculation was inside the fresh-init block and never executed.
cur_part, num_parts, and part_size stayed at 0, causing the
remote endpoint to reject part number 0 as invalid.

Move the part-size calculation out of the init block so it
runs for both fresh and resumed uploads.

Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>

rgw/cloud-restore: strip quotes from ETag on cloud tier fetch

When objects are restored from a cloud endpoint, the ETag value read
from the HTTP response includes surrounding double-quotes per RFC 7232.
RGW stores ETags unquoted internally, and dump_etag() adds its own
quotes when serving responses. The mismatch results in double-quoted
ETags like ""abc123-6"" on restored objects.

Strip the quotes from both the etag output parameter and the
RGW_ATTR_ETAG attribute after fetching from the cloud endpoint,
matching the unquoted format RGW uses everywhere else.

Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>

Merge pull request #68204 from abhishek593/bug-73437

rgw: fix uninitalized fields in pubsub topic creation

test/rgw/kafka: cleanup kafka tgz file

Fixes: https://tracker.ceph.com/issues/76127
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #67548 from mertsunacoglu/wip-rgw-lua-postauth

rgw/lua: Add new Lua postAuth hook that is being run after authorization

Merge pull request #68097 from cheese-cakee/wip-72088-sync-checkpoint-stdout

radosgw-admin: output bucket sync checkpoint to stdout

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #68279 from wuhongsong/fix/rgw-vault-content-type

rgw: add Content-Type header to Vault KMS requests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #67479 from nbalacha/wip-nbalacha-75127

rgw: Lua Package management fixes

mgr/dashboard : Select replicated rule by default in pools form
fixes : https://tracker.ceph.com/issues/75632
Signed-off-by: Abhishek Desai <abhishek.desai1@ibm.com>

Merge pull request #66778 from Kushal-deb/d3n-cephadm-automation

mgr/cephadm: Implement D3N L1 persistent datacache support for RGW

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

mgr/dashboard : Fix application names in pools form
Signed-off-by: Abhishek Desai <abhishek.desai1@ibm.com>

Merge pull request #66194 from ShwetaBhosale1/fix_issue_73774_nfs_tls_add_xprtsec

mgr/nfs: Cephadm support for NFS-Ganesha TLS configuration adding new option xprtsec

Reviewed-by: Adam King adking@redhat.com

Merge pull request #66763 from Shubhaj1810/fix-issue-2359181

mgr/orchestrator: sort host labels in ceph orch host ls output

Reviewed-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
Reviewed-by: Kushal Deb <Kushal.Deb@ibm.com>
Reviewed-by: Ujjawal Anand <Ujjawal.Anand@ibm.com>

mgr/dashboard: show rados ns in 'ceph nvmeof top io'

In nvmeof_top_cli.py, show RADOS namespace name
in "RBD Image" column.

Example:
Before: “rbd/image10”
After: “rbd/ns2/image10"

Fixes: https://tracker.ceph.com/issues/76116
Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #67828 from linuxbox2/wip-75540

rgw_file: always assign string_view of current common_prefix

Merge pull request #67722 from kchheda3/wip-add-transid-notification

rgw/notification: notification Event is not printing the correct  x_amz_request_id.

Merge pull request #67656 from kchheda3/wip-oidc-thumbprint-optional

rgw/oidc: make the thumprint field optional

Merge pull request #67616 from smanjara/wip-bucket-sync-run

rgw/multisite: 'bucket sync run' command during full sync does not take a bucket-wide lock

Merge pull request #60387 from thotz/rgw-placement-store-standard-storageclass

rgw/placement: display storage class in user info dump

Merge pull request #67219 from mertsunacoglu/wip-lua-bucket-metadata

RGW: Add Bucket metadata attribute for Lua scripts

Merge pull request #67365 from Kushal-deb/fix-test-nvmeof-sysctl

cephadm: reapply hugepages for nvmeof at service start

Reviewed-by: Adam King adking@redhat.com

Merge pull request #67999 from Shubhaj1810/nfs-ganesha-servicemap-fix

mgr/cephadm: align nodeid and add register_service for NFS Ganesha service visibility

Reviewed-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>

Merge pull request #68454 from idryomov/wip-76101

qa: fix setting rbd_sparse_read_threshold_bytes in test_migration_clone()

Reviewed-by: Miki Patel <miki.patel132@gmail.com>

mgr/cephadm: preserve in-flight D3N allocations during fresh deploy

Avoid pruning per-(service, host) D3N device assignments when
daemon details are not yet visible in the mgr cache.

This keeps initial daemon-to-device mappings stable on fresh
deployment so 1:1 allocation is preserved when free devices exist.

Signed-off-by: Kushal Deb <Kushal.Deb@ibm.com>

Merge pull request #67735 from bachmanity1/fix/cephadm-host-rm-keyerror

mgr/cephadm: fix KeyError when host is removed during serve loop

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

doc/releases: fix Tentacle data in releases.yml

Fixes the Ceph Releases doc that is now completely missing 20.2.0
and renders the Tentacle gantt bar starting at the time of 20.2.1.

Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>

Merge pull request #67836 from rhcs-dashboard/74243-add-bottom-padding-to-dashboard

mgr/dashboard : Add bottom padding for dashboard screens

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Devika Babrekar <devika.babrekar@ibm.com>

Merge pull request #67952 from rhcs-dashboard/stretched-cluster-pool-size

mgr/dashboard : add stretch cluster validation for pools form

Reviewed-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: introduce traddr together with existing server_address for backward compatibilty

Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

mgr/nfs: Cephadm support for NFS-Ganesha TLS configuration adding new option xprtsec

Fixes: https://tracker.ceph.com/issues/73774
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>

Merge pull request #65641 from rkachach/fix_issue_69325

mgr/cephadm: Fix alertmanager TLS and global security handling

Reviewed-by: Adam King adking@redhat.com

Merge pull request #66121 from ShwetaBhosale1/fix_issue_73712_fix_nfs_client_commands_hung_issue

mgr/cephadm: increase default backend health check interval for NFS

Reviewed-by: Adam King adking@redhat.com
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #67598 from Kushal-deb/fix-zg-modify

Avoid calling zonegroup modify if realm/zonegroup/zone are not present

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #68061 from JoshuaGabriel/75744-cephadm-zap

mgr/cephadm: Fixes infinite loop when OSD zap fails during removal

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

mgr/dashboard: tearsheet-layout-fix

Fixes: https://tracker.ceph.com/issues/76084
Signed-off-by: Sagar Gopale <sagar.gopale@ibm.com>

Fix test_remote_executables

update ast.ButOr -> ast.BitOr

Signed-off-by: Kushal Deb <Kushal.Deb@ibm.com>

mgr/cephadm: Implement D3N L1 persistent datacache support for RGW

Add RGW D3N L1 persistent datacache support backed by host block devices.
Select devices deterministically per (service, host) with intra-service
sharing, forbid cross-service reuse, prepare/mount devices, and
bind-mount per-daemon cache directories into the container.

Signed-off-by: Kushal Deb <Kushal.Deb@ibm.com>

qa/cephfs: exclude ceph-osd-{classic,crimson} in fs:upgrade:featureful_client:upgraded_client

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/cephfs: make test_journal_migration immune to stdout changes

Fixes: http://tracker.ceph.com/issues/75083
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard : add stretch cluster validation for pools form
fixes : https://tracker.ceph.com/issues/75667
Signed-off-by: Abhishek Desai <abhishek.desai1@ibm.com>

Merge pull request #67685 from mheler/wip-cloud-restore-days-zero

rgw: require RestoreObject Days >= 1

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

mgr/dashboard : Unable to remove gateway node from gateway group

Fixes: https://tracker.ceph.com/issues/75864
Signed-off-by: pujaoshahu <pshahu@redhat.com>

Merge pull request #68456 from gbregman/main

mgr/dashboard: Allow upper case encryption algorithm values in NVMeoF CLI

mgr/dashboard: Allow upper case encryption algorithm values in NVMeoF CLI
Fixes: https://tracker.ceph.com/issues/76106
Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

dashboard: align notifications footer to bottom when body has fewer items

- Fix footer alignment using sticky positioning
- Ensure footer stays at bottom for short notification lists
- Resolve ExpressionChangedAfterItHasBeenCheckedError
- Remove manual change detection causing test instability

Signed-off-by: Shashiranjan singh <singhshashiranjan34@gmail.com>

Merge pull request #68400 from MaxKellermann/mon_includes2

mon: include cleanup

Reviewed-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #68423 from ceph/caroav-patch-1

doc/rbd: Update NVMe-oF deployment instructions with group name

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rbd: Add group name to nvmeof-target-configure.rst

Signed-off-by: Aviv Caro <133020857+caroav@users.noreply.github.com>

qa: Fix checksum calculation on empty directories

Fixes: https://tracker.ceph.com/issues/75804
Signed-off-by: Kotresh HR <khiremat@redhat.com>

qa: Add mirror test for snapshot with only dir

Fixes: https://tracker.ceph.com/issues/75804
Signed-off-by: Kotresh HR <khiremat@redhat.com>

tools/cephfs_mirror: Fix sync hang

The snapshot mirror sync can hang if all of the following
are true.
 1. The snapshot being synced contain only directories and
    no files.
 2. The crawler finishes syncing dirs and completes crawling
    before datasync threads picks it up from syncm queue.

The above scenario can be achieved as below.

 1. Configure say /d0 and /d1 for mirroring.
 2. Create around 10k files in /d0
 3. Create a single dir say /d1/dir0
 4. snapshot /d0 and wait for status to change 'syncing'
 5. Now, snapshot /d1.

The /d1 snapshot will be stuck in syncing, as datasync
thread's has_pending_work logic would never pick it up
as there are no files to be synced (i.e. dataq is empty
and crawl is finished which is essentially job done) and
never notifies crawler thread to proceed with taking
snapshot.

The fix is to pick up the syncm if the crawler is finished
and dataq is empty to avoid missing notification to the
crawler thread.

Fixes: https://tracker.ceph.com/issues/75804
Signed-off-by: Kotresh HR <khiremat@redhat.com>

qa: fix setting rbd_sparse_read_threshold_bytes in test_migration_clone()

Currently it's set on the intermediary clone instead of the parent.
As a result the setting is effective only for reads that terminate at
the intermediary clone -- reads that go all the way to the parent may
end up being handled as not sparse depending on their size.

Fixes: https://tracker.ceph.com/issues/76101
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rgw: require RestoreObject Days >= 1

AWS S3 requires Days to be a positive non-zero integer. Parse Days as
a signed integer and validate in get_params() before any restore state
is modified, returning InvalidArgument for values less than 1.

Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>

qa: Valgrind suppression additions to pattern match inline functions stemming from CPython in mgr

Fixes: https://tracker.ceph.com/issues/72938
Signed-off-by: szuraski898 <steven.zuraski@ibm.com>
Conflicts:
  qa/valgrind.supp

The qa/valgrind.supp file had a conflict with an extra curly brace at the EOF as a result of changes adding CPython suppressions in the unittest-mgr branch, as well as additions coming form main.
Resolved by keeping both: the CPython suppressions for the mgr unit tests and all the rocky10 suppressions from main (rocky10_gcc14_mismatch_delete_map_erase
and rocky10 still reachable rocksdb leak), and removing the extra curly brace after merging.

test/mgr: clang-format changes to test_ttlcache

Fixes: https://tracker.ceph.com/issues/72938
Signed-off-by: szuraski898 <steven.zuraski@ibm.com>

test/mgr: Add unit tests to CMake

Fixes: https://tracker.ceph.com/issues/72938
Signed-off-by: szuraski898 <steven.zuraski@ibm.com>

Merge pull request #67601 from soumyakoduri/wip-restore-storage-class

rgw/cloud-restore: Restoring to non-existent storage-class should fail

Reviewed-by: Matthew N. Heler <matthew.heler@hotmail.com>

rgw/cloud-restore: Correct the restore svc shutdown order

Stop restore worker threads prior to obj_expirer and other services
(svc) shutdown which they rely on.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #68438 from ShwetaBhosale1/fix_76079_fix_test_scheduling

mgr/cephadm: Fixed test_scheduling.py

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #67749 from jamiepryde/update-isal-2.32.0

isa: Update ISA-L from 2.30.0 to 2.32.0

Merge pull request #67188 from cbodley/wip-74736

rgw/iam: match value of Null condition

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mgr/cephadm: fix NFS ganesha service registration via nodeid and register_service

The NFS Ganesha service was not consistently visible in `ceph -s`,
especially in multi-daemon deployments. This was due to missing or
incorrect service registration with the Ceph manager.

This change updates the ganesha.conf template to explicitly set:

  CEPH {
      register_service = true;
      nodeid = "{{ namespace }}.{{ nodeid }}";
  }

Key points:
- Enables proper service registration in mgr via register_service
- Ensures unique nodeid per daemon using namespace + nodeid
- Fixes visibility of NFS daemons in `ceph -s`
- Works correctly for both single and multi-node deployments

Validation:
- Verified with single NFS daemon → visible in `ceph -s`
- Verified with 3 NFS daemons → all correctly aggregated and visible
- Confirmed export creation activates service visibility
- Tested using Ceph 9.1 (Ganesha 9.7)

No regressions observed.

Fixes: https://tracker.ceph.com/issues/75709
Signed-off-by: Shubha Jain <SHUBHA.JAIN1@ibm.com>

mgr/cephadm: add nodeid and register_service for NFS Ganesha service visibility

- Add 'name' to template context in nfs.py
- Use consistent nodeid in RADOS_KV and CEPH blocks
- Enable register_service in CEPH block for service map visibility

Note: CEPH block is ignored in Ganesha 5.9 (seen as 'Unknown block (CEPH)' in logs),
so ceph -s visibility cannot be validated upstream. This change is forward-compatible
with newer Ganesha versions.

Fixes: https://tracker.ceph.com/issues/75709
Signed-off-by: Shubha Jain <SHUBHA.JAIN1@ibm.com>

Merge pull request #67585 from tchaikov/librbd-silence-asan

librbd: use try_lexical_convert instead of throwing lexical_cast

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

python-common/hostspec: normalize hostnames for case-insensitive matching

Normalize HostSpec and related placement parsing to ensure hostnames
are treated case-insensitively when supplied via service specs.

This prevents service placement failures caused by mixed-case hostnames.

Fixes: https://tracker.ceph.com/issues/74299
Signed-off-by: Shubha Jain <SHUBHA.JAIN1@ibm.com>

Merge pull request #68202 from idryomov/wip-73831

librbd: avoid losing sparseness in read_parent()

Reviewed-by: Mykola Golub <mykola.golub@clyso.com>

mgr/orchestrator: sort host labels in ceph orch host ls output

Sort host labels alphabetically to improve readability and
consistency across hosts. This matches the behavior of
structured formats such as yaml.

Fixes: https://tracker.ceph.com/issues/74290
Signed-off-by: Shubha Jain <SHUBHA.JAIN1@ibm.com>

mgr/cephadm: increase default backend health check interval for NFS

Fixes: https://tracker.ceph.com/issues/73712
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>

mgr/cephadm: Fixed test_scheduling.py
Fixes: https://tracker.ceph.com/issues/76079
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>

mgr/cephadm: Added retry logic for execute command if command fails with connection error

Fixes: https://tracker.ceph.com/issues/74179
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>

Merge pull request #68392 from rkachach/fix_issue_cepahdm_qa_task

qa: fix misleading "in cluster log" failures during cluster log scan

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Zack Cerza <zack@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

mgr/dashboard: sync policy created for a bucket in Object >> Multi-site >> Sync-policy, is not reflecting under bucket's replication

Fixes: https://tracker.ceph.com/issues/75581
Signed-off-by: Naman Munet <naman.munet@ibm.com>

Merge pull request #68297 from tchaikov/wip-feedback-without-tracker

mgr/feedback: fix flaky test_issue_tracker_create_with_invalid_key

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge PR #68245 into main

* refs/pull/68245/head:
mon/MonClient: check stopping for auth request handling

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mgr: Properly set description in labeled get_perf_schema_python

Fixes: https://tracker.ceph.com/issues/76048
Signed-off-by: stzuraski898 <steven.zuraski@ibm.com>

test/mgr: Add negative unit tests to MgrCap

Fixes: https://tracker.ceph.com/issues/72938
Signed-off-by: szuraski898 <steven.zuraski@ibm.com>

test/mgr: Add ServiceMap unit tests

Fixes: https://tracker.ceph.com/issues/72938
Signed-off-by: szuraski898 <steven.zuraski@ibm.com>

test/mgr: Add PyUtil unit tests

Fixes: https://tracker.ceph.com/issues/72938
Signed-off-by: szuraski898 <steven.zuraski@ibm.com>

test/mgr: Add PyFormatter unit tests

Fixes: https://tracker.ceph.com/issues/72938
Signed-off-by: szuraski898 <steven.zuraski@ibm.com>

test/mgr: Add DaemonState unit tests

Fixes: https://tracker.ceph.com/issues/72938
Signed-off-by: szuraski898 <steven.zuraski@ibm.com>