Merge pull request #60737 from zdover23/wip-doc-2024-11-15-backport-60703-to-squid

squid: doc/rados: correct "full ratio" note

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #60809 from zdover23/wip-doc-2024-11-24-backport-60804-to-squid-test-2

squid: doc/cephadm: Clarify "Deploying a new Cluster"

doc/cephadm: Clarify "Deploying a new Cluster"

Change the title of the section "Deploying a new Ceph cluster" to "Using
cephadm to Deploy a New Ceph Cluster". This is part of the initiative to
separate package-related documentation from container-based
documenation.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 8cf77d03a1f3ecb73931ddf96d13ef098aa61c1a)

Merge pull request #60751 from idryomov/wip-68743-squid

squid: Make mon addrs consistent with mon info

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #60718 from ljflores/wip-68910-squid

squid: mgr/balancer: optimize 'balancer status detail'

Merge pull request #58914 from ljflores/wip-67245-squid

squid: mon: Remove any pg_upmap_primary mapping during remove a pool

Merge pull request #60766 from zdover23/wip-doc-2024-11-19-backport-60765-to-squid

squid: doc/start: update os-recommendations.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: update os-recommendations.rst

Remove information about the operating systems that support Ceph's
official container images from the "Platforms" table in
doc/start/os-recommendations.rst and add that information to the (new)
table that shows the operating systems that support Ceph's official
container images.

Credit for this change should go to Enrico Bocchi, who noticed a
discrepancy that motivated it.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit c2f1fac3b94e946b253bc64e579429660cbc2d59)

Merge pull request #60164 from cbodley/wip-68425-squid

squid: cls/user: reset stats only returns marker when truncated

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59302 from yuvalif/wip-67574-squid

squid: rgw/notifications: fixing radosgw-admin notification json

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59120 from k0ste/wip-64499-squid

squid: rgw: add s3select usage to log usage

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59931 from adk3798/wip-68136-squid

squid: mgr/cephadm: add command to expose systemd units of all daemons

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60734 from guits/wip-68961-squid

squid: ceph-volume: fix dmcrypt activation regression

Merge pull request #60130 from cbodley/wip-68399-squid

squid: rgw: decrypt multipart get part when encrypted

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59754 from yuvalif/wip-68047-squid

squid: rgw/kafka: refactor topic creation to avoid rd_kafka_topic_name()

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59678 from jzhu116-bloomberg/wip-67959-squid

squid: rgw/multipart: use cls_version to avoid racing between part upload and multipart complete

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59123 from k0ste/wip-67099-squid

squid: [RGW] Fix the handling of HEAD requests that do not comply with RFC standards

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Make mon addrs consistent with mon info

For legacy librbd clients, sanitize_mons will panic if addrs are not
consistent with mon_info_t.

Signed-off-by: shenjiatong <yshxxsjt715@163.com>
fixes: https://tracker.ceph.com/issues/68743
(cherry picked from commit 14b09519739f3b2dc715647fd0f04dd22c17aeb3)

doc/rados: correct "full ratio" note

Correct a note that directed users not to add an OSD after the cluster
has reached its "full ratio". The note now says "Do not let your cluster
reach its full ratio before adding an OSD."

Hat tip: Oskar Berggren

Fixes: https://tracker.ceph.com/issues/68900
Co-authored-by: Oskar Berggren <oskar.berggren@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f1a2637c79a15c26a769661dd72ca68d766b2f0d)

Merge pull request #60671 from zdover23/wip-doc-2024-11-08-backport-56771-to-squid

squid: doc: Document which options are disabled by mClock.

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

ceph-volume: fix dmcrypt activation regression

`ceph.with_tpm` tag is actually a `str`, which means that
`bool('0')` will return `True`.

Fixes: https://tracker.ceph.com/issues/68944
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit 3cf00cde820ec838e4fbda1799d2ce336c9e2d86)

Merge pull request #59948 from aaSharma14/wip-68203-squid

squid: mgr/cephadm: use host address while updating rgw zone endpoints

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59945 from aaSharma14/wip-68201-squid

squid: mgr/rgwam: use realm/zonegroup/zone method arguments for period update

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59935 from adk3798/squid-cephadm-no-exception-cert-store-get

squid: mgr/cephadm: add --no-exception-when-missing flag to cert-store cert/key get

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59934 from adk3798/squid-cephadm-iscsi-teuth-extension

squid: qa/cephadm: extend iscsi teuth test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59933 from adk3798/wip-68159-squid

squid: cephadm: Support Docker Live Restore

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59930 from adk3798/wip-68114-squid

squid: qa/task: update alertmanager endpoints version

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59929 from adk3798/wip-68019-squid

squid: cephadm: emit warning if daemon's image is not to be used

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59710 from adk3798/cephadm-squid-ssh-keepalive-configurable

squid: mgr/cephadm: make ssh keepalive settings configurable

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59472 from adk3798/wip-67462-squid

squid: qa/upgrade: fix checks to make sure upgrade is still in progress

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58711 from rhcs-dashboard/bump-monitoring-stack-squid

squid: mgr/cephadm: bump monitoring stacks version

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58515 from adk3798/squid-cephadm-ingress-fix-ipv6-binding

squid: mgr/cephadm/services/ingress Fix HAProxy to listen on IPv4 and IPv6

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58381 from adk3798/squid-cephadm-haproxy-bugged-config

squid: cephadm/services/ingress: fixed keepalived config bug

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60466 from rhcs-dashboard/wip-68686-squid

squid: mgr/dashboard: fix bucket get for s3 account owned bucket

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60620 from YiteGu/wip-67094-squid

squid: mgr/dashboard: add cephfs rename REST API

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60632 from rhcs-dashboard/wip-68844-squid

squid: mgr/dashboard: remove cherrypy_backports.py

doc: document balancer configurable "update_pg_upmap_activity"

Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit c4274c0c3a608eeb0b1183c8a96ef3a314bf797c)

mgr/balancer: tie `update_pg_upmap_activity` to a configurable

This addition gives users the option of enabling/disabling this feature.

They can do so by running:
ceph config set mgr mgr/balancer/update_pg_upmap_activity <True|False>

The feature is off by default.

Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit 62c9f152e31affe3e19eb50e917cbeb10624f8fd)

mgr/balancer: optimize 'balancer status detail'

Before, we were updating the balancer status by
iterating through all pg upmap entires. This was
affecting the loading time of other mgr modules
on clusters with a large number of pgs (600+).

This can be optimized by simply pulling from
the incremental.

Fixes: https://tracker.ceph.com/issues/68657
Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit 31c498da96fa1acc1700515edcd544dead86dab9)

Merge pull request #60707 from zdover23/wip-doc-2024-11-12-backport-60705-to-squid

squid: doc/radosgw: s/Poliicy/Policy/

Merge pull request #60708 from zdover23/wip-doc-2024-11-12-backport-60704-to-squid

squid: doc/start: fix "are are" typo

doc/start: fix "are are" typo

Fix typo reading "They are are single-threaded".
s/are are/are/

Fixes: https://tracker.ceph.com/issues/68901
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 22010719d7edf2a6cf7de7b115ed797e7942ec47)

doc/radosgw: s/Poliicy/Policy/

Correct "AttachGroupPoliicy" to "AttachGroupPolicy".

Fixes: https://tracker.ceph.com/issues/68902
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 961aef1bb4e59912ac1c8827a392c18b912c71f0)

Merge pull request #60698 from zdover23/wip-doc-2024-11-12-backport-60529-to-squid

squid: doc/start: separate package chart from container chart

doc/start: separate package chart from container chart

Separate the packages-and-containers chart into two charts:
   (1) a chart that shows which OSes Ceph builds packages for
   (2) a chart that shows which OSes support Ceph's containers

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit e5ff1916e3d5e2cc3f47b4cea2762730dfd335c0)

Merge pull request #60679 from zdover23/wip-doc-2024-11-09-backport-60605-to-squid

squid: doc/cephadm: clean "Adv. OSD Service Specs"

doc/cephadm: clean "Adv. OSD Service Specs"

Clean the English grammar and the .RST in the section "Advanced OSD
Service Specifications" in the file doc/cephadm/services/osd.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f6b56f9ee5d9f0b54dcaf1832d7a977895eb9c94)

Merge pull request #59035 from sseshasa/wip-67355-squid

squid: osd/scheduler: add mclock queue length perfcounter

Merge pull request #58785 from sseshasa/wip-67108-squid

squid: qa/suites/rados/verify/validater: increase heartbeat grace timeout

doc: Document which options are disabled by mClock.

Not only in the mClock docs, but also in the reference
of the options that are disabled.

Otherwise users are bound to miss it, and surprised
why their options are ignored or reset.

Signed-off-by: Niklas Hambüchen <mail@nh2.me>
(cherry picked from commit 790afd277730f24801a3b59c08d2dce0f2a9659f)

qa/task: update alertmanager endpoints version

the v1 endpoint is deprecated and v2 is suggested to use.

Fixes: https://tracker.ceph.com/issues/67183
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit ac998b58487f6f9176b7f934ed61ae57e59c2ee5)

mgr/dashboard: adapt dashboard e2e for grafana

Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit a4374ab53806f353e49f3f818421b2630aad6f0b)

mgr/cephadm: bump monitoring stacks versions

Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit c719d8ac2d7ca70b6c33d9f5893b99360029c947)

Merge pull request #60644 from zdover23/wip-doc-2024-11-07-backport-60643-to-squid

squid: doc/cephadm: link to "host pattern" matching sect

doc/cephadm: link to "host pattern" matching sect

Link to the "Placement by Pattern Matching" section in
doc/cephadm/services/index.rst from the "Advanced OSD Service
Specifications" section in doc/cephadm/services/osd.rst.

Fixes: https://tracker.ceph.com/issues/68812
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit cdfa299a2511ee35dfe0a857702af6d3ba257efd)

Merge pull request #60637 from zdover23/wip-doc-2024-11-06-backport-56599-to-squid

squid: doc/cephadm: Update operations.rst

doc/cephadm: Update operations.rst

Fixed typo 'no' to 'not' in sentence "If this test fails, cephadm will
no be able to manage services on that host."

Signed-off-by: rhkelson <119636079+rhkelson@users.noreply.github.com>
(cherry picked from commit 34bc35a73443fa56908395a83ea2dd404a8fc4a6)

mgr/dashboard: remove cherrypy_backports.py

since its mostly used only for older cherrypy versions which we don't
support anymore in any of our recent upstream releases, we could remove
it completely

Fixes: https://tracker.ceph.com/issues/68802
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 7ae56f4f561999898a7411840f94e742981ca6bc)

Merge pull request #59217 from ifed01/wip-ifed-bs-fix-unused-squid

squid: os/bluestore: Mute warnings

Reviewed-by: Md Mahamudur Rahaman Sajib <mahamudur.sajib@croit.io>

Merge pull request #58740 from ifed01/wip-ifed-async-compact-squ

squid: os/bluestore: enable async manual compactions

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #58676 from ifed01/wip-ifed-many-many-extents-read-squi

squid: blk/aio: fix long batch (64+K entries) submission

Reviewed-by: Md Mahamudur Rahaman Sajib <mahamudur.sajib@croit.io>

Merge pull request #57819 from NitzanMordhai/wip-66319-squid

squid: objclass: deprecate cls_cxx_gather

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #59974 from ronen-fr/wip-rf-59944-squid

squid: qa/standalone/scrub: increase status updates frequency

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: add cephfs rename REST API

Introduce rename for the cephfs REST API controller, we can
rename the existing file or directory by it.

Fixes: https://tracker.ceph.com/issues/66797
Signed-off-by: Yite Gu <yitegu0@gmail.com>
(cherry picked from commit bac2689fdd0221a9b77833f20b0c512115f9d9e5)

Merge pull request #60609 from zdover23/wip-doc-2024-11-05-backport-60608-to-squid

doc/rgw/notification: add missing admin commands

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw/notification: add missing admin commands

also remove obsolete admin commands

Fixes: https://tracker.ceph.com/issues/68818
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit ec272797c4caa5cf62319cdec104cf11575727b1)

Merge pull request #59398 from lxbsz/wip-67432

squid: qa/cephfs: fix TestRenameCommand and unmount the clinet before failin…

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59406 from joscollin/wip-67668-squid

squid: cephfs_mirror: update peer status for invalid metadata in remote snapshot

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59487 from vshankar/wip-67775-squid

squid: client: calls to _ll_fh_exists() should hold client_lock

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59517 from batrick/wip-67745-squid

squid: mds: encode quiesce payload on demand

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59521 from batrick/wip-67806-squid

squid: qa: load all dirfrags before testing altname recovery

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59402 from Matan-B/wip-67664-squid

squid: mon/OSDMonitor: Add force-remove-snap mon command

Merge pull request #59284 from k0ste/wip-65042-squid

squid: librados: use CEPH_OSD_FLAG_FULL_FORCE for IoCtxImpl::remove

Merge pull request #59156 from badone/wip-tracker-59380-admin-socket-injectfull-squid

squid: qa/suites/rados: Cancel injectfull to allow cleanup

Merge pull request #59065 from YiteGu/wip-blk-discard-squid

squid: a series of optimizations for kerneldevice discard

mgr/dashboard: fix bucket get for s3 account owned bucket

**Issue 1:**

When a bucket is created with a user that is owner by the account user,
it fails to fetch the info for the bucket because the owner of that
bucket is considered as owned by the account rather than the user.

dashboard api try to fetch some info for the bucket on the basis of the
bucket owner which in this case fails since its not a real user.

To keep the existing behavior as it is and fix the current issue, I am
doing a check to make sure the user id exists or not. if it doesn't
exist, we goes on to fetch using the dashboard user.

**Issue 2**
Editing the bucket owner by the account is broken. So disabling the
ownership change for the bucket owned by account until we have user
account management for rgw in the dashboard

Fixes: https://tracker.ceph.com/issues/68622
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit dd4549238e525d371414538bb43e8333af19cb96)

 Conflicts:
src/pybind/mgr/dashboard/controllers/rgw.py
 - remove imports for multisite-automation
src/pybind/mgr/dashboard/services/service.py
 - this file is not part of squid. hence the user parameter is added to
   rgw_client.py instead where it originally was

Merge pull request #58587 from YiteGu/wip-64606-squid

squid: client: check mds down status before getting mds_gid_t from mdsmap

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59097 from batrick/wip-67438-squid

squid: mds: only authpin on wrlock when not a locallock

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59306 from batrick/wip-67613-squid

squid: mon/MonClient: handle ms_handle_fast_authentication return

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60367 from rhcs-dashboard/squid-i18n

squid: mgr/dashboard: update translations for squid

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60593 from zdover23/wip-doc-2024-11-02-backport-60584-to-squid

squid: doc: s/Whereas,/Although/

doc: s/Whereas,/Although/

Change the subordinating conjunction "Whereas" followed by a comma to
the less grammatically-incorrect "Although". I've been meaning to do
this since 22 Mar 2023.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ea1a470fb4809d312f854e72fc5f8081cbcb2c49)

Merge pull request #60545 from yuvalif/wip-yuval-fix-notif-test

squid: test/rgw/notification: fixing backport issues in the tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60557 from zdover23/wip-doc-2024-10-30-backport-60520-to-squid

squid: doc/start: add supported Squid distros

test/rgw/notification: replace calls to get_ip_http with get_ip

this fix is complements the fix done
by: 4d74fe5e508ebdadf2d033b8b4c43d6b18986569
that was causing python errors when running the bucket notifications
http tests.
this is also adding a missing function from:
970113d4092567bae9c6fab060bc32d241f6c581
the above commit cannot be cherry-picked as-is, since it has other
dependencies to the product code (notification list json format).

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

doc/start: add supported Squid distros

Add the distros that support Squid to the chart on
start/os-recommendations.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 2368dcb402e7b4425c1617ca12c923c2a2ad14bf)

Merge pull request #60537 from zdover23/wip-doc-2024-10-29-backport-60508-to-squid

squid: doc/rados: add blaum_roth coding guidance

Merge pull request #60525 from zdover23/wip-doc-2024-10-29-backport-60470-to-squid

squid: doc: explain the consequence of enabling mirroring through monitor co…

doc/rados: add blaum_roth coding guidance

Direct Ceph administrators using blaum_roth coding for erasure-coded
pools to change the default value of w=7 to a different value in order
to ensure that w+1 is prime.

This information was provided to the Ceph upstream by Benjamin Mare in
September of 2024.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6648d94aed5ff5643f50fcd4f3f07d97b846e885)

Merge pull request #60486 from guits/wip-68599-squid

squid: orch: disk replacement enhancement

Reviewed-by: Adam King <adking@redhat.com>

doc: explain the consequence of enabling mirroring through monitor commands

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit ed3ce72fbc0b097131945cbb39caa0831401b94a)

Merge pull request #59671 from yuvalif/wip-67952-squid

squid: rgw/notifications: free completion pointer using unique_ptr

Reviewed-By: Yuri Weinstein <yweinste@ibm.com>, Daniel Gryniewicz <dang1@ibm.com>, Shilpa Jagannath <shilpa.jagannath@ibm.com>

Merge pull request #59885 from ronen-fr/wip-rf-59793-to-squid

squid: osd/scrub: reduce osd_requested_scrub_priority default value

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #60500 from zdover23/wip-doc-2024-10-26-backport-60444-to-squid

squid: doc/rados: standardize markup of "clean"

Merge pull request #60503 from zdover23/wip-doc-2024-10-26-backport-60434-to-squid

squid: doc/glossary: add "PLP" to glossary

doc/glossary: add "PLP" to glossary

Add "PLP" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 599bdbf0a240ee718ed6da4104141b0f23330fbe)

doc/rados: standardize markup of "clean"

Standardize the markup around the status "clean" in the documentation so
that readers don't mistakenly get the idea that inconsistent
presentation of the word "clean" implies a never-stated difference
between one instance and the other.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit c8b4155d2d27228917d4be1f11514504100c55c7)

Merge pull request #60496 from zdover23/wip-doc-2024-10-26-backport-58647-to-squid

squid: doc/cephfs: add doc for disabling mgr/volumes plugin

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: add doc for disabling mgr/volumes plugin

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 373910de34fc9e1d98218aed0361cd9bf3badffa)

orch: disk replacement enhancement

This introduces a new `ceph orch device replace` command in order to
improve the user experience when it comes to replacing the underlying
device of an OSD.

Fixes: https://tracker.ceph.com/issues/68456
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit 212c8740831a7650b5be86c27d14f8c0b6eacbef)