Merge pull request #57819 from NitzanMordhai/wip-66319-squid

squid: objclass: deprecate cls_cxx_gather

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #59974 from ronen-fr/wip-rf-59944-squid

squid: qa/standalone/scrub: increase status updates frequency

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #60609 from zdover23/wip-doc-2024-11-05-backport-60608-to-squid

doc/rgw/notification: add missing admin commands

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw/notification: add missing admin commands

also remove obsolete admin commands

Fixes: https://tracker.ceph.com/issues/68818
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit ec272797c4caa5cf62319cdec104cf11575727b1)

Merge pull request #59398 from lxbsz/wip-67432

squid: qa/cephfs: fix TestRenameCommand and unmount the clinet before failin…

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59406 from joscollin/wip-67668-squid

squid: cephfs_mirror: update peer status for invalid metadata in remote snapshot

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59487 from vshankar/wip-67775-squid

squid: client: calls to _ll_fh_exists() should hold client_lock

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59517 from batrick/wip-67745-squid

squid: mds: encode quiesce payload on demand

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59521 from batrick/wip-67806-squid

squid: qa: load all dirfrags before testing altname recovery

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59402 from Matan-B/wip-67664-squid

squid: mon/OSDMonitor: Add force-remove-snap mon command

Merge pull request #59284 from k0ste/wip-65042-squid

squid: librados: use CEPH_OSD_FLAG_FULL_FORCE for IoCtxImpl::remove

Merge pull request #59156 from badone/wip-tracker-59380-admin-socket-injectfull-squid

squid: qa/suites/rados: Cancel injectfull to allow cleanup

Merge pull request #59065 from YiteGu/wip-blk-discard-squid

squid: a series of optimizations for kerneldevice discard

Merge pull request #58587 from YiteGu/wip-64606-squid

squid: client: check mds down status before getting mds_gid_t from mdsmap

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59097 from batrick/wip-67438-squid

squid: mds: only authpin on wrlock when not a locallock

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59306 from batrick/wip-67613-squid

squid: mon/MonClient: handle ms_handle_fast_authentication return

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60367 from rhcs-dashboard/squid-i18n

squid: mgr/dashboard: update translations for squid

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60593 from zdover23/wip-doc-2024-11-02-backport-60584-to-squid

squid: doc: s/Whereas,/Although/

doc: s/Whereas,/Although/

Change the subordinating conjunction "Whereas" followed by a comma to
the less grammatically-incorrect "Although". I've been meaning to do
this since 22 Mar 2023.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ea1a470fb4809d312f854e72fc5f8081cbcb2c49)

Merge pull request #60545 from yuvalif/wip-yuval-fix-notif-test

squid: test/rgw/notification: fixing backport issues in the tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60557 from zdover23/wip-doc-2024-10-30-backport-60520-to-squid

squid: doc/start: add supported Squid distros

test/rgw/notification: replace calls to get_ip_http with get_ip

this fix is complements the fix done
by: 4d74fe5e508ebdadf2d033b8b4c43d6b18986569
that was causing python errors when running the bucket notifications
http tests.
this is also adding a missing function from:
970113d4092567bae9c6fab060bc32d241f6c581
the above commit cannot be cherry-picked as-is, since it has other
dependencies to the product code (notification list json format).

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

doc/start: add supported Squid distros

Add the distros that support Squid to the chart on
start/os-recommendations.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 2368dcb402e7b4425c1617ca12c923c2a2ad14bf)

Merge pull request #60537 from zdover23/wip-doc-2024-10-29-backport-60508-to-squid

squid: doc/rados: add blaum_roth coding guidance

Merge pull request #60525 from zdover23/wip-doc-2024-10-29-backport-60470-to-squid

squid: doc: explain the consequence of enabling mirroring through monitor co…

doc/rados: add blaum_roth coding guidance

Direct Ceph administrators using blaum_roth coding for erasure-coded
pools to change the default value of w=7 to a different value in order
to ensure that w+1 is prime.

This information was provided to the Ceph upstream by Benjamin Mare in
September of 2024.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6648d94aed5ff5643f50fcd4f3f07d97b846e885)

Merge pull request #60486 from guits/wip-68599-squid

squid: orch: disk replacement enhancement

Reviewed-by: Adam King <adking@redhat.com>

doc: explain the consequence of enabling mirroring through monitor commands

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit ed3ce72fbc0b097131945cbb39caa0831401b94a)

Merge pull request #59671 from yuvalif/wip-67952-squid

squid: rgw/notifications: free completion pointer using unique_ptr

Reviewed-By: Yuri Weinstein <yweinste@ibm.com>, Daniel Gryniewicz <dang1@ibm.com>, Shilpa Jagannath <shilpa.jagannath@ibm.com>

Merge pull request #59885 from ronen-fr/wip-rf-59793-to-squid

squid: osd/scrub: reduce osd_requested_scrub_priority default value

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #60500 from zdover23/wip-doc-2024-10-26-backport-60444-to-squid

squid: doc/rados: standardize markup of "clean"

Merge pull request #60503 from zdover23/wip-doc-2024-10-26-backport-60434-to-squid

squid: doc/glossary: add "PLP" to glossary

doc/glossary: add "PLP" to glossary

Add "PLP" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 599bdbf0a240ee718ed6da4104141b0f23330fbe)

doc/rados: standardize markup of "clean"

Standardize the markup around the status "clean" in the documentation so
that readers don't mistakenly get the idea that inconsistent
presentation of the word "clean" implies a never-stated difference
between one instance and the other.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit c8b4155d2d27228917d4be1f11514504100c55c7)

Merge pull request #60496 from zdover23/wip-doc-2024-10-26-backport-58647-to-squid

squid: doc/cephfs: add doc for disabling mgr/volumes plugin

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: add doc for disabling mgr/volumes plugin

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 373910de34fc9e1d98218aed0361cd9bf3badffa)

orch: disk replacement enhancement

This introduces a new `ceph orch device replace` command in order to
improve the user experience when it comes to replacing the underlying
device of an OSD.

Fixes: https://tracker.ceph.com/issues/68456
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit 212c8740831a7650b5be86c27d14f8c0b6eacbef)

ceph-volume: address mypy errors in disk.py

typical error:

```
ceph_volume/util/disk.py:1374: error: Incompatible types in assignment (expression has type "Optional[str]", variable has type "str")  [assignment]
```

This commits addresses it.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit b5e7008d28a5acd63ea9cd0c6b27f400dad409af)

ceph-volume: address test_activate_dmcrypt_tpm

This mocks the call to `luks_close()`, otherwise this test
fails when run on a system where `cryptsetup` isn't available.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit dd2a150f40fb11abe6bd1ee51bca03419aaa7d7f)

Merge pull request #60482 from zdover23/wip-doc-2024-10-25-backport-60480-to-squid

squid: doc/cephfs: use 'p' flag to set layouts or quotas

doc/cephfs: use 'p' flag to set layouts or quotas

Remind users that clients must use the 'p' flag when writing file
layouts.

This PR is made in response to
https://www.reddit.com/r/ceph/comments/1gbad51/comment/ltkgd5v/

Co-authored-by: Zac Dover <zac.dover@proton.me>
Signed-off-by: TruongSinh Tran-Nguyen <i@truongsinh.pro>
(cherry picked from commit 13c7c9d99881c7b4491ba1a693283c047161ea8a)

Merge pull request #59603 from cbodley/wip-67882-squid

squid: qa/rgw/multisite: specify realm/zonegroup/zone args for 'account create'

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #60244 from cbodley/wip-67957-squid

squid: rgw/beast: optimize for accept when meeting error in listenning

Merge pull request #60065 from cbodley/wip-68330-squid

squid: rgw/rados: don't rely on IoCtx::get_last_version() for async ops

Merge pull request #59917 from yuvalif/wip-67648-squid

squid: test/rgw/notifications: don't check for full queue if topics expired

Merge pull request #59670 from mkogan1/wip-67922-squid

squid: rgw: decrement qlen/qactive perf counters on error

Merge pull request #59439 from cbodley/wip-67712-squid

squid: rgw/http: finish_request() after logging errors

Merge pull request #59413 from cbodley/wip-67686-squid

squid: rgw: load copy source bucket attrs in putobj

Merge pull request #59303 from yuvalif/wip-67609-squid

squid: test/rgw/notification: use real ip address instead of localhost

Merge pull request #59226 from cbodley/wip-67558-squid

squid: rgw/notify: visit() returns copy of owner string

Merge pull request #59223 from cbodley/wip-67557-squid

squid: rgwlc: fix typo in getlc (ObjectSizeGreaterThan)

Merge pull request #60461 from zdover23/wip-doc-2024-10-24-backport-60460-to-squid

squid: doc/start: add vstart install guide

doc/start: add vstart install guide

Add "vstart Cluster Installation and Configuration Procedure" to the
Beginner's Guide.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit a42be085bd7b9159f4924853ac2b2d92c37884b0)

Merge pull request #60354 from guits/wip-68556-squid

squid: cephadm: pull container images from quay.io

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60353 from guits/wip-68566-squid

squid: ceph-volume: drop unnecessary call to `get_single_lv()`

Merge pull request #60091 from guits/wip-68343-squid

squid: ceph-volume: add new class UdevData

Merge pull request #60435 from zdover23/wip-doc-2024-10-23-backport-60433-to-squid

squid: doc/cephfs: rearrange subvolume group information

doc/cephfs: rearrange subvolume group information

Rearranging the documentation to align it with CephFS encapsulation design.

Signed-off-by: Indira Sawant <indira.sawant@ibm.com>
(cherry picked from commit f6cf7ae5281f5740ac78bfa5e2a22913353b8167)

Merge pull request #60432 from zdover23/wip-doc-2024-10-23-backport-60423-to-squid

squid: doc/cephadm: correct "ceph orch apply" command

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

cephadm: pull container images from quay.io

Now that all required images are hosted and/or mirrored on quay.io
we can move away from docker.io

Fixes: https://tracker.ceph.com/issues/68323
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit fd895dde9d69ae7be7a78d8db37a2f94ded27080)

doc/cephadm: correct "ceph orch apply" command

s/orch apply/ceph orch apply/ in a command where "ceph" was missing.

This was reported by Eugen Block to Zac Dover on 22 Oct 2024 through
Slack.

Co-authored-by: Eugen Block <eblock@nde.ag>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit c42066766428c72884ed6f5dd6fa88244f436a1f)

Merge pull request #60127 from adamemerson/wip-40692-squid

squid: global: Call getnam_r with a 64KiB buffer on the heap

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #59661 from idryomov/wip-67845-squid

squid: librbd/migration: prune snapshot extents in RawFormat::list_snaps()

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58542 from k0ste/wip-66530-squid

squid: cephadm: fix apparmor profiles with spaces in the names

Reviewed-by: Adam King adking@redhat.com

Merge pull request #60171 from idryomov/wip-63184-squid

squid: librbd/crypto/LoadRequest: clone format for migration source image

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #59771 from nbalacha/wip-67764-squid

squid: rbd-mirror: use correct ioctx for namespace

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #59502 from idryomov/wip-67698-squid

squid: rbd: "rbd bench" always writes the same byte

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #59145 from idryomov/wip-53674-squid

squid: librbd/crypto: fix issue when live-migrating from encrypted export

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #60335 from aclamk/wip-68560-squid

squid: os/bluestore: Fix ceph-bluestore-tool allocmap command

Merge pull request #59969 from aclamk/wip-68222-squid

squid: os/bluestore: Fix BlueFS allocating bdev label reserved location

Merge pull request #60336 from aclamk/wip-68558-squid

squid: os/bluestore: Fix repair of multilabel when collides with BlueFS

Merge pull request #60338 from zdover23/wip-doc-2024-10-16-backport-60302-to-squid

squid: doc,mailmap: update my email / association to ibm

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #60003 from kamoltat/wip-68280-squid

squid: src/mon/ConnectionTracker.cc: Fix dump function
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #60378 from rhcs-dashboard/wip-68597-squid

squid: mgr/dashboard: fix lifecycle issues

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60279 from mchangir/wip-log-save-thread-name-in-log-entries-squid

squid: log: save/fetch thread name infra

Reviewed-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

mgr/dashboard: fix lifecycle issues

Conflicts:
src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-details/rgw-bucket-details.component.html

Fixes: https://tracker.ceph.com/issues/68434
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
(cherry picked from commit 87612f499f86c9864c3bf6371cdd46954176e5ab)

Merge pull request #60356 from rhcs-dashboard/wip-68565-squid

Squid: mgr/dashboard: The subvolumes are missing from the dropdown menu on the "Create NFS export" page

mgr/dashboard: update translations for squid

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #60346 from rhcs-dashboard/wip-68554-squid

Squid: mgr/dashboard: sync policy's in Object >> Multi-site >> Sync-policy, does not show the zonegroup to which policy belongs to

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60348 from afreen23/wip-68559-squid

squid: mgr/dashboard: fix group name bugs in the nvmeof API

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60332 from cbodley/wip-doc-rgw-multisite-secondary-commit-squid

squid: doc/radosgw/multisite: fix Configuring Secondary Zones -> Updating the Period

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58991 from cbodley/wip-67307-squid

squid: msg: insert PriorityDispatchers in sorted position

Merge pull request #60343 from zdover23/wip-doc-2024-10-16-backport-60256-to-squid

squid: doc: update Key Idea in cephfs-mirroring.rst

Squid: mgr/dashboard: The subvolumes are missing from the dropdown menu on the "Create NFS export" page

Fixes: https://tracker.ceph.com/issues/68519
Signed-off-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>
(cherry picked from commit e036fde7e4b03241e617cbb3ef8f19a703aae716)

ceph-volume: drop unnecessary call to `get_single_lv()`

`Zap.zap_lv()` currently makes a call to `get_single_lv()`:

```
        lv = api.get_single_lv(filters={'lv_name': device.lv_name,
                                        'vg_name': device.vg_name})
```

this isn't needed and redundant as zap_lv() takes an instance of `Device()`
as argument which has already a `lv_api` attribute:

class Device in device.py:

```
            else:
                vgname, lvname = self.path.split('/')
                filters = {'lv_name': lvname, 'vg_name': vgname}
            lv = lvm.get_single_lv(filters=filters)         # <---- same call

        if lv:
            self.lv_api = lv
```

This implies a duplicate call to `subprocess.Popen()` unnecessarily.

Fixes: https://tracker.ceph.com/issues/68312
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit c0e05bf36067294420631f33c5e43c32077eeb82)

os/bluestore: Fix repair of multilabel when collides with BlueFS

The problem was that BDEV_FIRST_LABEL_POSITION was removed from
bdev_label_valid_locations set.
Now, if label at BDEV_FIRST_LABEL_POSITION is valid, it is in the set.

Fixes: https://tracker.ceph.com/issues/68558
Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
(cherry picked from commit 7343be720870d4a5f82b55beee4685457a003067)

mgr/dashboard: fix group name bugs in the nvmeof API

there are 2 issues

1. in cephadm, i was always using the first daemon to populate the group
   in all the services for the dashboard config.

2. in the API, if there are more than 1 gateways listed in the config,
   rather than chosing a random gateway from the group, raise an
   exception and warn user to specify the gw_group parameter in the api
   request

Fixes: https://tracker.ceph.com/issues/68463
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit f9b50b2e88ae5d9ac4f2cab986527a0a12317da9)

 Conflicts:
src/pybind/mgr/cephadm/services/nvmeof.py

Merge pull request #60311 from rhcs-dashboard/wip-68410-squid

squid: mgr/dashboard: ignore exceptions raised when no cert/key found

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60321 from aaSharma14/wip-68546-squid

squid: mgr/cephadm: Update multi-site configs before deploying  daemons on rgw service create

Reviewed-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: sync policy's in Object >> Multi-site >> Sync-policy, does not show the zonegroup to which policy belongs to

Fixes: https://tracker.ceph.com/issues/68355
Fixes Includes: Added default zonegroup name with the sync policy details

Signed-off-by: Naman Munet <namanmunet@li-ff83bccc-26af-11b2-a85c-a4b04bfb1003.ibm.com>
(cherry picked from commit 517ab013e2a9bd23b482134121bcb85c5a32f028)

doc: update Key Idea in cephfs-mirroring.rst

Updates the snapdiff feature and it's url.

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 629922bf6a0905cc87707f5e2d027f6320aafd99)

mailmap: add my ibm email

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit e4177406f9734f1c8af91f8292aa972d01fb77f9)

doc/governance: update my CSC email

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit 2f61b2847d92b5156408dbcfa5b6e09e2de404c1)

os/bluestore: Fix ceph-bluestore-tool allocmap command

BlueStore::read_allocation_from_drive_for_bluestore_tool was
not informed that multiple bdev labels can exist and reserve space.
Comparison of real alloc vs recovered alloc was failing.

Fixes: https://tracker.ceph.com/issues/68560
Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
(cherry picked from commit 358f33a148c9a65478e33648f16e8c8af73c98f2)

doc/radosgw/multisite: fix Configuring Secondary Zones -> Updating the Period

this was copy/pasted from Configuring a Master Zone -> Update the Period
but still referred to the 'master zone'

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit cfe254758b1ca9647c4dcfb13b6a3310558b88d2)

mgr/cephadm: RGW service deployment defaults to 'default' realm/zonegroup/zone despite non-default spec in service

When we create an RGW service using the ceph orch apply command, the service is always deployed in the default realm, zonegroup, and zone, even if we specify a different realm, zonegroup, or zone in the service spec. This happens because certain configuration values, like rgw_realm, rgw_zonegroup, and rgw_zone, need to be set for the RGW instances before the daemons are deployed. Currently, these configurations are being applied after the RGW daemons are deployed, which requires a service restart to reflect the correct realm, zonegroup, and zone. Ideally, these configurations should be applied before the RGW daemons are deployed, so they are correctly placed in the desired realm, zonegroup, and zone from the start.

Fixes: https://tracker.ceph.com/issues/68461
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit ad147f2e8d820ff251e1499c1e4c3fe57d1a2082)

Merge pull request #59140 from zdover23/wip-doc-2024-08-10-backport-58951-to-squid

squid: mgr/dashboard: fix ceph-users api doc

Reviewed-by: Afreen <afreen23.git@gmail.com>

log: thread name save/fetch infra

* pthread name is saved in a thread_local storage
* the thread_local name is copied into Entry object's ctor
* Log::dump_recent() reads the thread name from the Entry
  object's data member when dumping logs

Fixes: https://tracker.ceph.com/issues/50743
Signed-off-by: Milind Changire <mchangir@redhat.com>
(cherry picked from commit 0be8d01c9ddde0d7d24edd34dc75f6cfc861b5ba)

Merge pull request #60287 from ronen-fr/wip-rf-60198-squid

squid: qa/standalone/scrub: remove TEST_recovery_scrub_2

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #60293 from rhcs-dashboard/squid-sync-policy-pipe-edit-fixes

Squid: mgr/dashboard: unable to edit pipe config for bucket level policy of bucket

Reviewed-by: Afreen Misbah <afreen@ibm.com>