rgw/multisite: in order to sleep between mdlog polling events, we check if the mdlog_marker is not modified by comparing
mdlog_marker and max_marker. but max_marker is exposed to changes from RGWReadMDLogEntriesCR, and if there is a race
coming from mdlog trimming which could render max_marker empty, then its comparison with mdlog polling can be incorrect.
To fix this, we now save the previous mdlog marker and compare with the updated mdlog marker.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit bae10f22e8b008088f0861db2785d8c0ba7ee506)

Merge pull request #60766 from zdover23/wip-doc-2024-11-19-backport-60765-to-squid

squid: doc/start: update os-recommendations.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: update os-recommendations.rst

Remove information about the operating systems that support Ceph's
official container images from the "Platforms" table in
doc/start/os-recommendations.rst and add that information to the (new)
table that shows the operating systems that support Ceph's official
container images.

Credit for this change should go to Enrico Bocchi, who noticed a
discrepancy that motivated it.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit c2f1fac3b94e946b253bc64e579429660cbc2d59)

Merge pull request #60164 from cbodley/wip-68425-squid

squid: cls/user: reset stats only returns marker when truncated

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59302 from yuvalif/wip-67574-squid

squid: rgw/notifications: fixing radosgw-admin notification json

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59120 from k0ste/wip-64499-squid

squid: rgw: add s3select usage to log usage

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59931 from adk3798/wip-68136-squid

squid: mgr/cephadm: add command to expose systemd units of all daemons

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60734 from guits/wip-68961-squid

squid: ceph-volume: fix dmcrypt activation regression

Merge pull request #60130 from cbodley/wip-68399-squid

squid: rgw: decrypt multipart get part when encrypted

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59754 from yuvalif/wip-68047-squid

squid: rgw/kafka: refactor topic creation to avoid rd_kafka_topic_name()

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59678 from jzhu116-bloomberg/wip-67959-squid

squid: rgw/multipart: use cls_version to avoid racing between part upload and multipart complete

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59123 from k0ste/wip-67099-squid

squid: [RGW] Fix the handling of HEAD requests that do not comply with RFC standards

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #60671 from zdover23/wip-doc-2024-11-08-backport-56771-to-squid

squid: doc: Document which options are disabled by mClock.

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

ceph-volume: fix dmcrypt activation regression

`ceph.with_tpm` tag is actually a `str`, which means that
`bool('0')` will return `True`.

Fixes: https://tracker.ceph.com/issues/68944
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit 3cf00cde820ec838e4fbda1799d2ce336c9e2d86)

Merge pull request #59948 from aaSharma14/wip-68203-squid

squid: mgr/cephadm: use host address while updating rgw zone endpoints

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59945 from aaSharma14/wip-68201-squid

squid: mgr/rgwam: use realm/zonegroup/zone method arguments for period update

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59935 from adk3798/squid-cephadm-no-exception-cert-store-get

squid: mgr/cephadm: add --no-exception-when-missing flag to cert-store cert/key get

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59934 from adk3798/squid-cephadm-iscsi-teuth-extension

squid: qa/cephadm: extend iscsi teuth test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59933 from adk3798/wip-68159-squid

squid: cephadm: Support Docker Live Restore

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59930 from adk3798/wip-68114-squid

squid: qa/task: update alertmanager endpoints version

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59929 from adk3798/wip-68019-squid

squid: cephadm: emit warning if daemon's image is not to be used

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59710 from adk3798/cephadm-squid-ssh-keepalive-configurable

squid: mgr/cephadm: make ssh keepalive settings configurable

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59472 from adk3798/wip-67462-squid

squid: qa/upgrade: fix checks to make sure upgrade is still in progress

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58711 from rhcs-dashboard/bump-monitoring-stack-squid

squid: mgr/cephadm: bump monitoring stacks version

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58515 from adk3798/squid-cephadm-ingress-fix-ipv6-binding

squid: mgr/cephadm/services/ingress Fix HAProxy to listen on IPv4 and IPv6

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58381 from adk3798/squid-cephadm-haproxy-bugged-config

squid: cephadm/services/ingress: fixed keepalived config bug

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60466 from rhcs-dashboard/wip-68686-squid

squid: mgr/dashboard: fix bucket get for s3 account owned bucket

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60620 from YiteGu/wip-67094-squid

squid: mgr/dashboard: add cephfs rename REST API

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60632 from rhcs-dashboard/wip-68844-squid

squid: mgr/dashboard: remove cherrypy_backports.py

Merge pull request #60707 from zdover23/wip-doc-2024-11-12-backport-60705-to-squid

squid: doc/radosgw: s/Poliicy/Policy/

Merge pull request #60708 from zdover23/wip-doc-2024-11-12-backport-60704-to-squid

squid: doc/start: fix "are are" typo

doc/start: fix "are are" typo

Fix typo reading "They are are single-threaded".
s/are are/are/

Fixes: https://tracker.ceph.com/issues/68901
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 22010719d7edf2a6cf7de7b115ed797e7942ec47)

doc/radosgw: s/Poliicy/Policy/

Correct "AttachGroupPoliicy" to "AttachGroupPolicy".

Fixes: https://tracker.ceph.com/issues/68902
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 961aef1bb4e59912ac1c8827a392c18b912c71f0)

Merge pull request #60698 from zdover23/wip-doc-2024-11-12-backport-60529-to-squid

squid: doc/start: separate package chart from container chart

doc/start: separate package chart from container chart

Separate the packages-and-containers chart into two charts:
   (1) a chart that shows which OSes Ceph builds packages for
   (2) a chart that shows which OSes support Ceph's containers

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit e5ff1916e3d5e2cc3f47b4cea2762730dfd335c0)

Merge pull request #60679 from zdover23/wip-doc-2024-11-09-backport-60605-to-squid

squid: doc/cephadm: clean "Adv. OSD Service Specs"

doc/cephadm: clean "Adv. OSD Service Specs"

Clean the English grammar and the .RST in the section "Advanced OSD
Service Specifications" in the file doc/cephadm/services/osd.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f6b56f9ee5d9f0b54dcaf1832d7a977895eb9c94)

Merge pull request #59035 from sseshasa/wip-67355-squid

squid: osd/scheduler: add mclock queue length perfcounter

Merge pull request #58785 from sseshasa/wip-67108-squid

squid: qa/suites/rados/verify/validater: increase heartbeat grace timeout

doc: Document which options are disabled by mClock.

Not only in the mClock docs, but also in the reference
of the options that are disabled.

Otherwise users are bound to miss it, and surprised
why their options are ignored or reset.

Signed-off-by: Niklas Hambüchen <mail@nh2.me>
(cherry picked from commit 790afd277730f24801a3b59c08d2dce0f2a9659f)

qa/task: update alertmanager endpoints version

the v1 endpoint is deprecated and v2 is suggested to use.

Fixes: https://tracker.ceph.com/issues/67183
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit ac998b58487f6f9176b7f934ed61ae57e59c2ee5)

mgr/dashboard: adapt dashboard e2e for grafana

Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit a4374ab53806f353e49f3f818421b2630aad6f0b)

mgr/cephadm: bump monitoring stacks versions

Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit c719d8ac2d7ca70b6c33d9f5893b99360029c947)

Merge pull request #60644 from zdover23/wip-doc-2024-11-07-backport-60643-to-squid

squid: doc/cephadm: link to "host pattern" matching sect

doc/cephadm: link to "host pattern" matching sect

Link to the "Placement by Pattern Matching" section in
doc/cephadm/services/index.rst from the "Advanced OSD Service
Specifications" section in doc/cephadm/services/osd.rst.

Fixes: https://tracker.ceph.com/issues/68812
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit cdfa299a2511ee35dfe0a857702af6d3ba257efd)

Merge pull request #60637 from zdover23/wip-doc-2024-11-06-backport-56599-to-squid

squid: doc/cephadm: Update operations.rst

doc/cephadm: Update operations.rst

Fixed typo 'no' to 'not' in sentence "If this test fails, cephadm will
no be able to manage services on that host."

Signed-off-by: rhkelson <119636079+rhkelson@users.noreply.github.com>
(cherry picked from commit 34bc35a73443fa56908395a83ea2dd404a8fc4a6)

mgr/dashboard: remove cherrypy_backports.py

since its mostly used only for older cherrypy versions which we don't
support anymore in any of our recent upstream releases, we could remove
it completely

Fixes: https://tracker.ceph.com/issues/68802
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 7ae56f4f561999898a7411840f94e742981ca6bc)

Merge pull request #59217 from ifed01/wip-ifed-bs-fix-unused-squid

squid: os/bluestore: Mute warnings

Reviewed-by: Md Mahamudur Rahaman Sajib <mahamudur.sajib@croit.io>

Merge pull request #58740 from ifed01/wip-ifed-async-compact-squ

squid: os/bluestore: enable async manual compactions

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #58676 from ifed01/wip-ifed-many-many-extents-read-squi

squid: blk/aio: fix long batch (64+K entries) submission

Reviewed-by: Md Mahamudur Rahaman Sajib <mahamudur.sajib@croit.io>

Merge pull request #57819 from NitzanMordhai/wip-66319-squid

squid: objclass: deprecate cls_cxx_gather

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #59974 from ronen-fr/wip-rf-59944-squid

squid: qa/standalone/scrub: increase status updates frequency

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: add cephfs rename REST API

Introduce rename for the cephfs REST API controller, we can
rename the existing file or directory by it.

Fixes: https://tracker.ceph.com/issues/66797
Signed-off-by: Yite Gu <yitegu0@gmail.com>
(cherry picked from commit bac2689fdd0221a9b77833f20b0c512115f9d9e5)

Merge pull request #60609 from zdover23/wip-doc-2024-11-05-backport-60608-to-squid

doc/rgw/notification: add missing admin commands

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw/notification: add missing admin commands

also remove obsolete admin commands

Fixes: https://tracker.ceph.com/issues/68818
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit ec272797c4caa5cf62319cdec104cf11575727b1)

Merge pull request #59398 from lxbsz/wip-67432

squid: qa/cephfs: fix TestRenameCommand and unmount the clinet before failin…

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59406 from joscollin/wip-67668-squid

squid: cephfs_mirror: update peer status for invalid metadata in remote snapshot

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59487 from vshankar/wip-67775-squid

squid: client: calls to _ll_fh_exists() should hold client_lock

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59517 from batrick/wip-67745-squid

squid: mds: encode quiesce payload on demand

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59521 from batrick/wip-67806-squid

squid: qa: load all dirfrags before testing altname recovery

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59402 from Matan-B/wip-67664-squid

squid: mon/OSDMonitor: Add force-remove-snap mon command

Merge pull request #59284 from k0ste/wip-65042-squid

squid: librados: use CEPH_OSD_FLAG_FULL_FORCE for IoCtxImpl::remove

Merge pull request #59156 from badone/wip-tracker-59380-admin-socket-injectfull-squid

squid: qa/suites/rados: Cancel injectfull to allow cleanup

Merge pull request #59065 from YiteGu/wip-blk-discard-squid

squid: a series of optimizations for kerneldevice discard

mgr/dashboard: fix bucket get for s3 account owned bucket

**Issue 1:**

When a bucket is created with a user that is owner by the account user,
it fails to fetch the info for the bucket because the owner of that
bucket is considered as owned by the account rather than the user.

dashboard api try to fetch some info for the bucket on the basis of the
bucket owner which in this case fails since its not a real user.

To keep the existing behavior as it is and fix the current issue, I am
doing a check to make sure the user id exists or not. if it doesn't
exist, we goes on to fetch using the dashboard user.

**Issue 2**
Editing the bucket owner by the account is broken. So disabling the
ownership change for the bucket owned by account until we have user
account management for rgw in the dashboard

Fixes: https://tracker.ceph.com/issues/68622
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit dd4549238e525d371414538bb43e8333af19cb96)

 Conflicts:
src/pybind/mgr/dashboard/controllers/rgw.py
 - remove imports for multisite-automation
src/pybind/mgr/dashboard/services/service.py
 - this file is not part of squid. hence the user parameter is added to
   rgw_client.py instead where it originally was

Merge pull request #58587 from YiteGu/wip-64606-squid

squid: client: check mds down status before getting mds_gid_t from mdsmap

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59097 from batrick/wip-67438-squid

squid: mds: only authpin on wrlock when not a locallock

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59306 from batrick/wip-67613-squid

squid: mon/MonClient: handle ms_handle_fast_authentication return

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60367 from rhcs-dashboard/squid-i18n

squid: mgr/dashboard: update translations for squid

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60593 from zdover23/wip-doc-2024-11-02-backport-60584-to-squid

squid: doc: s/Whereas,/Although/

doc: s/Whereas,/Although/

Change the subordinating conjunction "Whereas" followed by a comma to
the less grammatically-incorrect "Although". I've been meaning to do
this since 22 Mar 2023.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ea1a470fb4809d312f854e72fc5f8081cbcb2c49)

Merge pull request #60545 from yuvalif/wip-yuval-fix-notif-test

squid: test/rgw/notification: fixing backport issues in the tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60557 from zdover23/wip-doc-2024-10-30-backport-60520-to-squid

squid: doc/start: add supported Squid distros

test/rgw/notification: replace calls to get_ip_http with get_ip

this fix is complements the fix done
by: 4d74fe5e508ebdadf2d033b8b4c43d6b18986569
that was causing python errors when running the bucket notifications
http tests.
this is also adding a missing function from:
970113d4092567bae9c6fab060bc32d241f6c581
the above commit cannot be cherry-picked as-is, since it has other
dependencies to the product code (notification list json format).

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

doc/start: add supported Squid distros

Add the distros that support Squid to the chart on
start/os-recommendations.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 2368dcb402e7b4425c1617ca12c923c2a2ad14bf)

Merge pull request #60537 from zdover23/wip-doc-2024-10-29-backport-60508-to-squid

squid: doc/rados: add blaum_roth coding guidance

Merge pull request #60525 from zdover23/wip-doc-2024-10-29-backport-60470-to-squid

squid: doc: explain the consequence of enabling mirroring through monitor co…

doc/rados: add blaum_roth coding guidance

Direct Ceph administrators using blaum_roth coding for erasure-coded
pools to change the default value of w=7 to a different value in order
to ensure that w+1 is prime.

This information was provided to the Ceph upstream by Benjamin Mare in
September of 2024.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6648d94aed5ff5643f50fcd4f3f07d97b846e885)

Merge pull request #60486 from guits/wip-68599-squid

squid: orch: disk replacement enhancement

Reviewed-by: Adam King <adking@redhat.com>

doc: explain the consequence of enabling mirroring through monitor commands

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit ed3ce72fbc0b097131945cbb39caa0831401b94a)

Merge pull request #59671 from yuvalif/wip-67952-squid

squid: rgw/notifications: free completion pointer using unique_ptr

Reviewed-By: Yuri Weinstein <yweinste@ibm.com>, Daniel Gryniewicz <dang1@ibm.com>, Shilpa Jagannath <shilpa.jagannath@ibm.com>

Merge pull request #59885 from ronen-fr/wip-rf-59793-to-squid

squid: osd/scrub: reduce osd_requested_scrub_priority default value

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #60500 from zdover23/wip-doc-2024-10-26-backport-60444-to-squid

squid: doc/rados: standardize markup of "clean"

Merge pull request #60503 from zdover23/wip-doc-2024-10-26-backport-60434-to-squid

squid: doc/glossary: add "PLP" to glossary

doc/glossary: add "PLP" to glossary

Add "PLP" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 599bdbf0a240ee718ed6da4104141b0f23330fbe)

doc/rados: standardize markup of "clean"

Standardize the markup around the status "clean" in the documentation so
that readers don't mistakenly get the idea that inconsistent
presentation of the word "clean" implies a never-stated difference
between one instance and the other.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit c8b4155d2d27228917d4be1f11514504100c55c7)

Merge pull request #60496 from zdover23/wip-doc-2024-10-26-backport-58647-to-squid

squid: doc/cephfs: add doc for disabling mgr/volumes plugin

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: add doc for disabling mgr/volumes plugin

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 373910de34fc9e1d98218aed0361cd9bf3badffa)

orch: disk replacement enhancement

This introduces a new `ceph orch device replace` command in order to
improve the user experience when it comes to replacing the underlying
device of an OSD.

Fixes: https://tracker.ceph.com/issues/68456
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit 212c8740831a7650b5be86c27d14f8c0b6eacbef)

ceph-volume: address mypy errors in disk.py

typical error:

```
ceph_volume/util/disk.py:1374: error: Incompatible types in assignment (expression has type "Optional[str]", variable has type "str")  [assignment]
```

This commits addresses it.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit b5e7008d28a5acd63ea9cd0c6b27f400dad409af)

ceph-volume: address test_activate_dmcrypt_tpm

This mocks the call to `luks_close()`, otherwise this test
fails when run on a system where `cryptsetup` isn't available.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit dd2a150f40fb11abe6bd1ee51bca03419aaa7d7f)

Merge pull request #60482 from zdover23/wip-doc-2024-10-25-backport-60480-to-squid

squid: doc/cephfs: use 'p' flag to set layouts or quotas

doc/cephfs: use 'p' flag to set layouts or quotas

Remind users that clients must use the 'p' flag when writing file
layouts.

This PR is made in response to
https://www.reddit.com/r/ceph/comments/1gbad51/comment/ltkgd5v/

Co-authored-by: Zac Dover <zac.dover@proton.me>
Signed-off-by: TruongSinh Tran-Nguyen <i@truongsinh.pro>
(cherry picked from commit 13c7c9d99881c7b4491ba1a693283c047161ea8a)

Merge pull request #59603 from cbodley/wip-67882-squid

squid: qa/rgw/multisite: specify realm/zonegroup/zone args for 'account create'

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #60244 from cbodley/wip-67957-squid

squid: rgw/beast: optimize for accept when meeting error in listenning

Merge pull request #60065 from cbodley/wip-68330-squid

squid: rgw/rados: don't rely on IoCtx::get_last_version() for async ops

Merge pull request #59917 from yuvalif/wip-67648-squid

squid: test/rgw/notifications: don't check for full queue if topics expired

Merge pull request #59670 from mkogan1/wip-67922-squid

squid: rgw: decrement qlen/qactive perf counters on error

Merge pull request #59439 from cbodley/wip-67712-squid

squid: rgw/http: finish_request() after logging errors