mgr: Control PG autoscaler during upgrades with pg_autoscale_during_upgrade

Fixes: https://tracker.ceph.com/issues/69477
Signed-off-by: Ashwin M. Joshi <ashjosh1@in.ibm.com>
 Conflicts:
src/pybind/mgr/cephadm/tests/test_upgrade.py
src/pybind/mgr/cephadm/upgrade.py

Merge pull request #67030 from indirasawant/wip-isawant-volumes-info-log

mgr/volumes: reduce noisy health check logs

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: reduce noisy health check logs

Previously, the manager logged connection cleanup messages at info level
even when no idle connections existed, adding unnecessary noise to logs.

This change logs cleanup actions at info level only when idle connections
are found, and moves the 'no idle connections' message to debug level.

Fixes: https://tracker.ceph.com/issues/73635
Signed-off-by: Indira Sawant <indira.sawant@ibm.com>

Merge PR #68128 into main

* refs/pull/68128/head:
qa: Fix checksum calculation on empty directories
qa: Add mirror test for snapshot with only dir
tools/cephfs_mirror: Fix sync hang

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #68389 into main

* refs/pull/68389/head:
qa: Handle TypeError in test_filelock

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #68141 into main

* refs/pull/68141/head:
client: Use correct size for fscrypt dummy key

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #68446 into main

* refs/pull/68446/head:
mds: remove duplicate context completion calls
mds: add retry request to MDSRank wait queue rather via finisher
mds: adjust scan_stray_dir after fixing up MDSContext class
Revert "mds: move MDSContext completion handling to finish method"

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #68748 from guits/update-cv-doc

doc: warn against default cephadm shell for ceph-volume

Merge pull request #68052 from rhcs-dashboard/enable-overview-page

mgr/dashbaord: Enable overview landing page

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #68084 from tchaikov/osd-odr

osd: fix ASAN ODR violations in denc-mod-osd

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #66820 from Shubhaj1810/fix-hostname-case-insensitive-v2

python-common/hostspec: normalize hostnames for case-insensitive matc…

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #67707 from Shubhaj1810/fix-upgrade-order-validation

mgr/cephadm: Fix upgrade order validation when using daemon_types with hosts

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ashwin M. Joshi <ashjosh1@in.ibm.com>

Merge pull request #66189 from timqn22/mon-public-network-updating

mgr/cephadm: mon public network updating

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #68398 from ashjosh1git/ceph-tracker-75603-ok-to-upgrade-bucket-params

mgr: Bucket scoped OSD upgrades using ok-to-upgrade

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #68484 from kginonredhat/issue-75967-ceph-orch-daemon-incorrectly-sets-container_image-to-force

Correct: ceph orch daemon incorrectly setting container image to force

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

mgr/dashboard: Updates to empty state component

- added state for no storage in empty state component
- extended the icon component to take into account the scenario of button with icon
- fix unit tests

Signed-off-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: Update cypress dashboard e2e tests

- removed dashboard v3 tests
-fixed login, navigation, mirroring, language, osd, page header e2e tests

Signed-off-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: Allow checks for prometheus disablement

- dont fire promethues queries if promethues is disabled

Signed-off-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: Fix cephadm e2e tests

- these tests failing due to new onboarding page changes

Fixes https://tracker.ceph.com/issues/75697

Signed-off-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: Fix creat-cluster welcome tests

Fixes https://tracker.ceph.com/issues/75697

Signed-off-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: Fix overview a11y tests

Fixes https://tracker.ceph.com/issues/75696

Signed-off-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: Fix a11y tests of navigation

Signed-off-by: Afreen Misbah <afreen@ibm.com>

mgr/dashbaord: Enable overview landing page

- removes feature toggle
- removed unused dashboard component, dashboard v3 component, and helper pipes and components

Fixes https://tracker.ceph.com/issues/75749

Signed-off-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #68333 from lumir-sliva/crimson/fix-stat-enoent

crimson/os/seastore: handle enoent in SeaStore::Shard::stat

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #68132 from myoungwon/wip-fastpath-logmanager

crimson/os/seastore: make the common write case the fast path in logmanager

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #68630 from xxhdx1985126/wip-76268

crimson/os/seastore: destroy Transaction only when no other reference exists

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #68839 from tchaikov/crimson-get-segment-manager-cleanups

crimson/os/seastore: SegmentManager::get_segment_manager() cleanups

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #68534 from xxhdx1985126/wip-76197

crimson/os/seastore/cache: conconrrent read of EXIST_CLEAN extents can

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #68544 from myoungwon/wip-coroutine-cjs

crimson/os/seastore/journal: switch CircularJournalSpace to coroutines

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #68340 from myoungwon/wip-avoid-continuation-delta-overwrite

crimson/os/seastore: remove an extra continuation in delta-overwrite path

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #68085 from tchaikov/mgr-python-cleanup

mgr: replace deprecated PyImport_ImportModuleNoBlock with PyImport_ImportModule

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #68806 from rhcs-dashboard/db-sso-oauth2-fixes

mgr/dashboard: add oauth2 sso prerequisites and fixes missing claims and expired token

Reviewed-by: Afreen Misbah <afreen@ibm.com>

crimson: consolidate the return paths of get_segment_manager()

before this change, two branches both return `BlockSegmentManager`,
which is redundant. in this change, consolidate them so that the
`HAVE_ZNS` path becomes an early return. this improves readability.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

crimson: abort on ioctl(BLKGETNRZONES) failure

previously, we did not check the return value of ioctl(BLKGETNRZONES).

we query the number of zones of the storage device to determine which
seastore backend to use. the only possible error from this ioctl is
-EFAULT (invalid user pointer), which indicates a programming error
and should never happen in practice. use ceph_assert() to catch this.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

crimson: use uint32_t when calling ioctl(BLKGETNRZONES)

before this change, we pass a pointer to a `size_t` to
ioctl(BLKGETNRZONES), but in the Linux kernel,
include/uapi/linux/blkzoned.h:

```c
 #define BLKGETNRZONES _IOR(0x12, 133, __u32)
```
this API reads 32 bits of data into the pointer. on 64-bit
architectures, size_t is 64 bits. fortunately, we initialize
nr_zones with 0, so the upper 32 bits remain zero. this works
on little-endian systems, but not on big-endian systems. it is
also semantically wrong. we should pass a pointer to a 32-bit
value when calling ioctl(BLKGETNRZONES).

in this change, we change the type of nr_zones from size_t to
uint32_t to match what the Linux kernel expects.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

crimson: coroutinize SegmentManager::get_segment_manager()

this change was inspired by following warning:

```
[1/3] Building CXX object src/crimson/os/seastore/CMakeFiles/crimson-seastore.dir/segment_manager.cc.o
/home/kefu/dev/ceph/src/crimson/os/seastore/segment_manager.cc:45:15: warning: lambda capture 'FNAME' is not used [-Wunused-lambda-capture]
   45 |       ).then([FNAME,
      |               ^
```

but we went further by coroutinize the whole method. because the return
value of ioctl() is not checked before this change, and clang correctly
flagged this with a warning, we marker it with `[[maybe_unused]]`, we
will fix it in a separate change.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #68131 from rzarzynski/wip-ec-asserted-isa-prepare

ec: validate tcache retrievals in ErasureCodeIsaDefault::prepare()

Merge pull request #68609 from aainscow/attr_rollback_fix

osd: Fix incorrect rollback logic for partial write OI

Merge pull request #67292 from JonBailey1993/stats_fix_part_2

osd: Reduce pg_stats invalidations occurring in fast ec

Merge pull request #66559 from timqn22/crash-dir-permission-setting

src/cephadm: updated crash dir creation

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #68848 from rkachach/fix_issue_76511

qa/cephadm: start upgrade tests from tentacle instead of reef on main

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #68365 from kamoltat/wip-ksirivad-fix-75418

qa/suites/upgrade: ignore PG_DAMAGED

Reviewed-by: Laura Flores <lflores@ibm.com>

Merge pull request #67915 from falconlee236/fix-osd-df-sorting-main

mon/PGMap: sort 'osd df' and 'osd perf' outputs by OSD ID

Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #68326 from ljflores/wip-tracker-75763

qa/suites/rados/encoder: remove rocky from supported distros

Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #67715 from NitzanMordhai/wip-nitzan-is_pg_clean-hang-after-teardown

test/ceph-helpers: add timeout to ceph pg query

Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #68847 from ceph/wip-doc-SageMcTSecurityCSC

docs/security: added workinggroup.rst and securitylead.rst

Merge pull request #68401 from phlogistonjohn/jjm-pypkg

build: Update python packaging for src/python-common

Reviewed-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #68747 from Kushal-deb/fix-nvmeof-apply-path

mgr/cephadm: allow nvmeof group assignment for NVMe-oF services

Reviewed-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>

docs/security: added workinggroup.rst and securitylead.rst
Signed-off-by: Sage McTaggart <sagemct@ibm.com>

Merge pull request #67344 from dermalikmann/fix-mgmt-gateway-use-vip

mgr/cephadm: mgmt-gateway bind to virtual_ip

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

mgr: add prerequisites check before enabling dashboard oauth2 sso

Assisted-by: Claude:claude-4.6-sonnet
Fixes: https://tracker.ceph.com/issues/76476
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@ibm.com>

qa/cephadm: start upgrade tests from tentacle instead of reef on main

Since main is what will become umbrella at this point

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit a137bccbf4330a88f17588d465218165ec03b009)

mgr/dashboard: fix missing claims on oauth2 sso

Fixes: https://tracker.ceph.com/issues/76479
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@ibm.com>

mgr/dashboard: raise exception on oauth2 sso expired token

Fixes: https://tracker.ceph.com/issues/76478
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@ibm.com>

Merge pull request #67422 from MattyWilliams22/cls-fifo

cls/test: Stop the cls_fifo.get_info test from Segmentation Faulting

Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>
Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>

Merge pull request #68386 from tchaikov/wip-crimson-throttling

crimson/osd: acquire throttle when scanning replica/primary for backfill

Reviewed-by: Mohit Agrawal <moagrawa@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #68834 from tchaikov/rgw-d4n-boost-1.91

rgw/d4n: fix deprecated async_run overload in RedisPool

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #68676 from JoshuaGabriel/msgr-activecon-perfcounter

msg/async: make msgr_active_connections counter a gauge to prevent underflow

https://tracker.ceph.com/issues/76440

Merge pull request #53891 from Hu-Yuxuan/fix-bug-#63137

osd: Improved pg-upmap computing speed

https://tracker.ceph.com/issues/76440

Merge pull request #68489 from ronen-fr/wip-rf-pintest-crimson

crimson/tests: fix test_remap_pin_concurrent

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #68812 from tchaikov/wip-doc-lower-require-min-compat

doc/rados: warn against lowering require_min_compat_client in read-balancer

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

doc/rados: warn against lowering require_min_compat_client in read-balancer

Add a note that the value should not be lowered after being set, to
avoid accidentally breaking features that depend on a newer release.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #68750 from tchaikov/wip-crimson-merge-coll

crimson/os: implement OP_MERGE_COLLECTION

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #67671 from bluikko/wip-doc-start-bg-improve

doc/start: Improve beginners-guide.rst

Merge pull request #68833 from tchaikov/wip-rgw-lua-5.5

rgw: fix build with Lua 5.5

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw/d4n: fix deprecated async_run overload in RedisPool

The async_run overload taking a logger argument is deprecated since
Boost 1.89. Use the 2-arg async_run(config, token) overload when
building with Boost >= 1.89, and fall back to the 3-arg overload
for Boost 1.87-1.88.

See https://www.boost.org/doc/libs/1_89_0/libs/redis/doc/html/redis/reference/boost/redis/basic_connection/async_run-04.html

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #68628 from MaxKellermann/msg__includes2

msg: include cleanup

Reviewed-by: Kefu Chai <k.chai@proxmox.com>

rgw: fix build with Lua 5.5

Lua 5.5 added a `seed` parameter to `lua_newstate()`. Use a
preprocessor conditional to pass the extra argument when building
against Lua >= 5.5.

Since we require Lua >= 5.3 in src/CMakeLists.txt, we need to be
compatible with Lua 5.5 as well.

See https://www.lua.org/manual/5.5/manual.html#lua_newstate

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #68702 from nhoad/compiler-warning

rgw: Move declaration inline to solve compiler warning about an unused variable

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #68522 from phlogistonjohn/jjm-smb-rm-all

smb: wildcard deletion of smb shares and clusters

Reviewed-by: Xavi Hernandez <xhernandez@gmail.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #68711 from jamiepryde/isal-arm-build-typo

cmake: Fix ISA-L build on arm

Merge pull request #68422 from tchaikov/wip-speedy-backport

script/ceph-backport: skip fetch if merge commit already exists locally

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

msg/async/Protocol: include cleanup

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>

msg/async/AsyncConnection: include cleanup

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>

msg/Connection: include cleanup

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>

mon: add missing includes

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>

mgr: Accept only osd daemon type for bucket params to adhere to upgrade sequence

Fixes: https://tracker.ceph.com/issues/75603
Signed-off-by: Ashwin M. Joshi <ashjosh1@in.ibm.com>

Merge pull request #68790 from MaxKellermann/crimson__missing_includes

crimson: add missing includes

Reviewed-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #68151 from bluikko/wip-doc-man-ceph-improvements1

doc/man: improve ceph.rst

Merge pull request #68156 from bluikko/wip-doc-man-spelling1

doc/man: fix spelling etc errors (1 of 2)

qa/suites/upgrade: ignore PG_DAMAGED

we can simply ignore this warning that pops up temporary on the logs,
since do check for active+clean in ceph.healthy

Fixes: https://tracker.ceph.com/issues/72424
Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

Merge pull request #68637 from aainscow/do_read_assert

osd: Avoid assertion on empty object read when reading multiple objects

Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>

Merge pull request #68797 from JonBailey1993/fix-design-definition

doc: Clarification of text in ec stretch cluster design

Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>

Merge pull request #68364 from linuxbox2/wip-complete-mpu-etag

rgw: return an etag header for all successful complete-multipart

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #68788 from cloudbehl/fix-Application-overview

monitoring: Fix application overview to show Raw used

Reviewed-by: Afreen Misbah <afreen@ibm.com>

doc: Clarification of text in ec stretch cluster design

Information regarding min_size in the EC Cluster Design doc was unclear in regards to the intention of what we want to develop. This commit is to clarify this so it is clear to readers.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

Merge pull request #68735 from ronen-fr/wip-rf-scrubNtrim-crimson

crimson/osd: defer snap trimming while scrubbing

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #68614 from tchaikov/wip-test-osd-fix-leaks

test/osd: fix Message and Connection refcount leaks

Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>

crimson/osd: acquire throttle when scanning replica/primary for backfill

The backfill state machine called budget_available() before deciding to
scan, but request_primary_scan() and request_replica_scan() never
actually acquired the throttle slot.  This meant scans could proceed
without any resource reservation, defeating the QoS intent of the
throttler introduced in 791772f1c0.

In this change, we fix this by acquiring the throttle before initiating
each scan.

Fixes: https://tracker.ceph.com/issues/70808
Signed-off-by: Kefu Chai <k.chai@proxmox.com>

crimson: add missing includes

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>

monitoring: Fix application overview to show Raw used

- Updated capacity used to show Raw capacity
- Pool table shows Raw capacity
- Total used capacity graph shows raw capacity

fixes: https://tracker.ceph.com/issues/76456

Signed-off-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #66430 from NitzanMordhai/wip-nitzan-deadlock-eio

aio_cxx: Fix mutual deadlock and resolve test unreliability

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Laura Flores <lflores@ibm.com>

Merge PR #68634 into main

* refs/pull/68634/head:
mgr: change cleanup and scanning cephfs connection logs to be less noisy

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: anuragbandhu <anuragbandhu007@gmail.com>

doc: document new options for smb share rm and cluster rm

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add test cases for cluster rm w/ wildcard/recursive

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add --wildcard and --recursive to smb cluster rm

Add new --wildcard and --recursive flags to the smb cluster rm
subcommands. These allow deleting clusters in bulk.  The --wildcard
option works like the same option for share rm in that it allows the use
of globbing for the cluster IDs, this includes '*' to delete all
clusters. The --recursive option tells the command to also delete all
child resources (shares) when deleting a cluster.

This was previously doable by streaming the output of `ceph smb show
...` through (sed or) jq and flipping the intent to removed and piping
that to `ceph smb apply` - but this is clearly not obvious nor easy to
document versus  these new options.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add unit tests to verify deleting shares by wildcard

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add a --wildcard option to the smb share rm subcommand

The new wildcard option will enable matching multiple shares to delete
or even all shares in a cluster using '*'.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add a new error type to reflect non-matching inputs

It's not that the input was bad, it just reflects that nothing
matching the input value was found.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add glob style wildcard support to matcher object

Add glob/wildcard support to the matcher type in the handler.py file.
This will be used in future changes to make matching shares and/or
clusters easier by supporting glob style wildcards on some commands.

Signed-off-by: John Mulligan <jmulligan@redhat.com>