cephadm: add python-common/ceph to cephadm zipapp

We want to use this location for files to be shared between
the cephadm binary and cephadm mgr module so it must be included
as part of the zipapp

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #60541 from gbregman/main

mgr/cephadm: Change NVME image version to 1.3.3.

Merge pull request #60546 from phlogistonjohn/jjm-python-common-codeowners

CODEOWNERS: let orchestration cover all of python-common

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #60520 from zdover23/wip-doc-2024-10-29-start-os-recs-add-supported-squid-distros

doc/start: add supported Squid distros

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge branch 'ceph:main' into main

Merge pull request #60422 from rhcs-dashboard/modal-carbonized

mgr/dashboard: Replacing bootstrap modal service with cds modal service

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #50105 from zhsgao/mds_export_state

mds: add an asok command to dump export states

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59072 from lxbsz/wip-66991

qa: increase the http.maxRequestBuffer to 100MB and enable the git debug logs

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge branch 'ceph:main' into main

CODEOWNERS: let orchestration cover all of python-common

While it's not perfectly correct to say the orchestration team owns
src/python-common, it's closer to the truth than not. It also ensures
that someone with domain specific knowledge is looking at the rest of
python-common outside just the deployment section (short of creating
some sort of new python-sig type group). This is relevant due to the
changes in PR #60377 as python-common's ceph package will be included
in cephadm.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57299 from rhcs-dashboard/fix-47066-main

mgr/{restful,zabbix}: remove deprecated module

Reviewed-by: @tchaikov
Reviewed-by: @ktdreyer
Reviewed-by: @jmolmo

mgr/dashboard: Replacing bootstrap modal service with cds modal service

Fixes: https://tracker.ceph.com/issues/68650
Signed-off-by: Dnyaneshwari talwekar <dtalweka@redhat.com>

Merge PR #60518 into main

* refs/pull/60518/head:
doc/governance: updates based on 2024q4 election

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #60532 from joscollin/wip-mirroring-create-snapshot

doc: mention how to initiate snapshot mirroring

mgr/cephadm: Change NVME image version to 1.3.3.
Fixes https://tracker.ceph.com/issues/68755

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge PR #60010 into main

* refs/pull/60010/head:
script/ceph-debug-docker: add debuginfod for ubuntu
script/ceph-debug-docker: drop /ceph mount

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #60508 from zdover23/wip-doc-2024-10-27-rados-operations-erasure-code-jerasure

doc/rados: add blaum_roth coding guidance

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #60510 from Matan-B/wip-matanb-backfill-crimson-docs

doc/dev/crimson: Add BackfillMachine state graph

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

doc: mention how to initiate snapshot mirroring

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #60054 from rhcs-dashboard/realm-popup

mgr/dashboard: Carbon Design - Create realm popup

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60393 from rhcs-dashboard/nvmeofServiceUnitTest

Add API tests for nvmeof service.ts file

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60476 from rhcs-dashboard/usermanagment-roles-defect

mgr/dashboard:Roles section is not aligned with column name

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60318 from ronen-fr/wip-rf-schedlogl

osd/scrub: improve scrub information conveyed in standard

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60470 from joscollin/wip-doc-cephfs-mirroring-improve-note

doc: explain the consequence of enabling mirroring through monitor commands

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #60400 into main

* refs/pull/60400/head:
mon,cephfs: fix indentation level of a code block

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

doc/dev/crimson: update index

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

doc/dev/crimson: Add initial BackfillMachine docs

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60190 from avanthakkar/teuthology-smb-cluster-public-addrs

qa/suites/orch: add a tests for testing public_addr for smb cluster create cmd

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

doc/start: add supported Squid distros

Add the distros that support Squid to the chart on
start/os-recommendations.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/governance: updates based on 2024q4 election

https://vote.heliosvoting.org/helios/elections/e03494ce-e04c-41d0-bb05-ec5ccc632ce4/view

Question #1
    Update election requirements for Ceph Executive Council Elections?
        Remove "ranked-choice" requirement 13
        Keep "ranked-choice" requirement (no change) 16

Question #2
    Require periodic elections in governance charter?
        No (no change) 8
        Annual 15
        Semi-annual 3
        Quarterly 2

Question #3
    Update the Ceph Executive Council term length?
        Change to 3 years 14
        Keep 2 years (no change) 14

Question #4
    Amend governance document to require a supermajority of votes for amendments to the governance model? The current requirement is a simple majority.
        Require a supermajority 20
        Require a simple majority (no change) 9

Question #5
    Clarify "supermajority" and "majority" election requirements?
        Of members voting on a given question (abstaining does not bias the vote) 18
        Of members voting on the election (abstaining is an implicit "no") 6
        Of members in the CSC 3

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

mgr/{restful,zabbix}: document removal

Fixes: https://tracker.ceph.com/issues/47066
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/zabbix: remove deprecated module

This (already deprecated) module is removed as a side-effect of the
deprecation and removal of the `restful` module.

Fixes: https://tracker.ceph.com/issues/47066
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/restful: remove deprecated module

Detailed changes:
* Remove `restful` mgr module dir,
* Remove Python depedencies (`pecan`, `werkzeug`) from ceph.spec and
  debian control,
* Remove docs,
* Remove associated QA tests,
* Update vstart.

Fixes: https://tracker.ceph.com/issues/47066
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #59465 from galsalomon66/limit_mem_usage_on_parquet_flow

rgw/s3select: limit memory usage on Parquet flow

Merge pull request #60454 from batrick/mds-warn-cleanup

mds: remove obsolete comments

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/rados: add blaum_roth coding guidance

Direct Ceph administrators using blaum_roth coding for erasure-coded
pools to change the default value of w=7 to a different value in order
to ensure that w+1 is prime.

This information was provided to the Ceph upstream by Benjamin Mare in
September of 2024.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60357 from ronen-fr/wip-rf-transitive

osd/scrub: make sched-targets comparator transitive

Reviewed-by: Samuel Just <sjust@redhat.com>

osd/scrub: improve scrub information conveyed in standard
PG log line references

When "mentioning" a PG in a log message, we include a set of data
items, including some scrub related information. This
commit improves the scrubber information conveyed, following
changes to the scrub scheduler.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #60506 from zdover23/2024-10-26-fix-grammar-check_license_name-main

.github/check-license: correct grammar in "license check"

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

.github/check-license: correct grammar in "license check"

Change "Check Incompatible Licenses" to "Check for Incompatible
Licenses".

This commit is made as part of a project to separate the changes in
https://github.com/ceph/ceph/pull/60448 into individual pull requests.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60434 from zdover23/wip-doc-2024-10-23-glossary-plp

doc/glossary: add "PLP" to glossary

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #60444 from zdover23/wip-doc-2024-10-23-rados-ops-health-checks

doc/rados: standardize markup of "clean"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #60471 from Jayaprakash-ibm/patch-2

Update .githubmap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #60386 into main

* refs/pull/60386/head:
qa/cephfs: override testing kernel with -k option

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/suites/orch: add a tests for testing public_addr for smb cluster create cmd

Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #60117 from rishabh-d-dave/mgr-vol-base-init

mgr/vol: better to call base class __init__() at beginning

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60153 from rishabh-d-dave/mgr-vol-cv-waiting

mgr/vol: remove unused variable

Reviewed-by: Jos Collin <jcollin@redhat.com>

mgr/dashboard: Carbon Design - Create realm popup

Fixes: https://tracker.ceph.com/issues/68341
Signed-off-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

Merge pull request #60395 from guits/replace-osd-enhancement

ceph-volume: support zapping by osd-id for RAW OSDs

ceph-volume: support zapping by osd-id for RAW OSDs

Currently it is only possible to zap by osd-id for LVM-based OSDs.

This commit introduces the required changes in order to support zapping
by osd-id for RAW-based OSDs.

Fixes: https://tracker.ceph.com/issues/68576
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #59847 from rishabh-d-dave/mgr-vol-spawn-threads

mgr/vol: reuse code to spawn threads

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Add API tests for nvmeof service.ts file

Fixes: https://tracker.ceph.com/issues/68551
Signed-off-by: Achint Kaur <ackaur@redhat.com>

Merge pull request #60480 from truongsinh/patch-2

doc: Note that to set layouts or quotas, clients require the `p` flag

doc/cephfs: use 'p' flag to set layouts or quotas

Remind users that clients must use the 'p' flag when writing file
layouts.

This PR is made in response to
https://www.reddit.com/r/ceph/comments/1gbad51/comment/ltkgd5v/

Co-authored-by: Zac Dover <zac.dover@proton.me>
Signed-off-by: TruongSinh Tran-Nguyen <i@truongsinh.pro>

mgr/dashboard:Roles section is not aligned with column name

Fixes:https://tracker.ceph.com/issues/68706

Signed-off-by:Puja Shahu pshahu@redhat.com
Signed-off-by: pujaoshahu <pshahu@redhat.com>

doc: explain the consequence of enabling mirroring through monitor commands

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #59982 from rkachach/fix_issue_mgmt_gw_high_availability

Adding HA support for mgmt-gateway and oauth2-proxy services

Reviewed-by: Adam king <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #58898 from neesingh-rh/wip-fix-strict-iec-cast

src/common : proper handling of units in `strict_iec_cast`

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

mgr/vol: add helpers to spawn all threads and more threads

Also add log messages for in these helper methods to allow tracking when
and why more threads were spawned.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

mgr/vol: reuse code for spawning threads

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #60473 into main

* refs/pull/60473/head:
mailmap, githubmap, organisationmap: Add Anuradha Gadge

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #60457 into main

* refs/pull/60457/head:
.github/check-license: add friendly name

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #59002 from neesingh-rh/wip-65820

qa/tasks/fwd_scrub: remove unnecessary traceback

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #60119 from rishabh-d-dave/mgr-vol-comments

mgr/vol: add comments to explain queuing data structures

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #60042 from sunilangadi2/migration_with_namespace_test

qa/workunits/rbd: cover native migration between namespaces

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mailmap, githubmap, organisationmap: Add Anuradha Gadge

Signed-off-by: Anuradha Gadge <anuradha.gadge@ibm.com>

Merge pull request #60420 from nbalacha/wip-nbalacha-namespace-doc

doc/rbd: namespace mirroring updates

Reviewed-by: Sunil Angadi <Sunil.Angadi@ibm.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Update .githubmap

Signed-off-by: Jayaprakash-ibm <jayaprakash@ibm.com>

Merge pull request #60355 from rhcs-dashboard/carbonize-hosts-form

mgr/dashboard: carbonize Cluster > Hosts form

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60465 from zdover23/wip-doc-2024-10-24-cephfs-troubleshooting

doc/cephfs: edit "Disabling Volumes Plugin"

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/cephfs: edit "Disabling Volumes Plugin"

Improve the English language in the section 'Disabling Volumes Plugin"
(which here becomes "Disabling the Volumes Plugin") in the
doc/cephfs/troubleshooting.rst file.

The change here is similar to a change introduced in
https://github.com/ceph/ceph/pull/60448/. That PR seems to be unable to
pass the "make check", and this PR is part of a test to see if breaking
the changes in that PR into several smaller changes results in mergeable
PRs.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59027 from hit1943/65971_fix

client: fixed a bug that read operation hung

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/rbd : add namespace mirroring updates

Updates the rbd docs with the new remote-namespace
mirroring configuration option.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #60276 from ceph/bypass_sr

test/crimson: disable stall-detector on aarch64 to fix asan failures

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60460 from zdover23/wip-doc-2024-10-24-start-beginners-guide

doc/start: add vstart install guide

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: add vstart install guide

Add "vstart Cluster Installation and Configuration Procedure" to the
Beginner's Guide.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60413 from gbregman/main

 Add namespace netmask parameters to nvmeof configuration

.github/check-license: add friendly name

In the absence of `name`, GH uses the `job` ID of the action to select which checks are required, and `pull_request` is rather confusing.

Signed-off-by: Ernesto Puerta <37327689+epuertat@users.noreply.github.com>

mgr/dashboard: carbonize Cluster > Hosts form

Fixes: https://tracker.ceph.com/issues/68264
Signed-off-by: Dnyaneshwari Talwekar <dtalweka@redbat.com>

mds: remove obsolete comments

An mdr is a smart pointer now. The reference is now codified.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #60399 from rhcs-dashboard/rgw-bucket-get-fix-for-account

mgr/dashboard: fix bucket get for s3 account owned bucket

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60388 from ceph/fix-check_license_typo-main

ci: fix typo in check-license

added tests migration with namespace

Signed-off-by: Sunil Angadi <Sunil.Angadi@ibm.com>

Merge pull request #58647 from rishabh-d-dave/mgr-vol-mod-disable

mgr: allow disabling always-on modules

Merge pull request #60316 from VallariAg/wip-nvmeof-teuthology-mtls-test

qa/suites/nvmeof: add mtls test

Merge pull request #60157 from soumyakoduri/wip-skoduri-lc-nullinstance

rgw/lc: Fix issues with non-current objects with instance empty

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60258 from aclamk/wip-aclamk-cbt-improve-show-label

os/bluestore/ceph-bluestore-tool: Modify show-label for many devs

doc/glossary: add "PLP" to glossary

Add "PLP" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: standardize markup of "clean"

Standardize the markup around the status "clean" in the documentation so
that readers don't mistakenly get the idea that inconsistent
presentation of the word "clean" implies a never-stated difference
between one instance and the other.

Signed-off-by: Zac Dover <zac.dover@proton.me>

test/crimson: disable stall-detector on aarch64 to fix asan failures

Ceph crimson uses seastar on debug mode which would enable ASAN, while stall-detector uses glibc backtrace function which would cause ASAN failures on aarch64.
Reason see scylladb/scylladb#15090 (comment)

Because arm ci servers in lab are "elderly", causing stall happened often, this PR is to disable stall-detector until seastar upstream migrated to libunwind, see scylladb/seastar#1878
Seastar does not provide a function to disable stall-detector, at present, increase stall-detector time to avoid stall-detector timeout on arm platform.

fixes: https://tracker.ceph.com/issues/65635

Signed-off-by: cailianchun <arm7star@qq.com>

Merge pull request #60433 from indirasawant/doc_fix_1

doc/cephfs: rearrange subvolume group information

doc/cephfs: rearrange subvolume group information

Rearranging the documentation to align it with CephFS encapsulation design.

Signed-off-by: Indira Sawant <indira.sawant@ibm.com>

Merge pull request #60423 from zdover23/wip-doc-2024-10-22-cephadm-services-smb

doc/cephadm: correct "ceph orch apply" command

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

mgr/dashboard: fix bucket get for s3 account owned bucket

**Issue 1:**

When a bucket is created with a user that is owner by the account user,
it fails to fetch the info for the bucket because the owner of that
bucket is considered as owned by the account rather than the user.

dashboard api try to fetch some info for the bucket on the basis of the
bucket owner which in this case fails since its not a real user.

To keep the existing behavior as it is and fix the current issue, I am
doing a check to make sure the user id exists or not. if it doesn't
exist, we goes on to fetch using the dashboard user.

**Issue 2**
Editing the bucket owner by the account is broken. So disabling the
ownership change for the bucket owned by account until we have user
account management for rgw in the dashboard

Fixes: https://tracker.ceph.com/issues/68622
Signed-off-by: Nizamudeen A <nia@redhat.com>

rgw/lc: Fix issues with non-current objects with instance empty

When the bucket versioning is enabled, old plain object entry is converted
to versioned by updating its instance as "null" in its raw head/old object.
However its instance remains empty in the bi list entry. Same is the case for
the entries created after versioning is suspended and re-enabled.

So to access such objects which are non-current, we need to set rgw_obj_key.instance as
1) "null" to read the actual raw obj and
2) empty while accessing/updating their bi entry.

Fixes: https://tracker.ceph.com/issues/68402
Signed-off-by: Soumya Koduri <skoduri@redhat.com>

doc/cephadm: correct "ceph orch apply" command

s/orch apply/ceph orch apply/ in a command where "ceph" was missing.

This was reported by Eugen Block to Zac Dover on 22 Oct 2024 through
Slack.

Co-authored-by: Eugen Block <eblock@nde.ag>
Signed-off-by: Zac Dover <zac.dover@proton.me>

osd/scrub: remove the 'only_deadlined' flag

as we no longer allow overdue targets to ignore time/load
constraints.

Also: fix the documentation for the relevant configuration
options.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: make sched-targets comparator transitive

The comparator used to sort the scrub targets wasn't transitive, which
can lead to undefined behavior (and likely - to a crash of the sort
algorithm in some implementations). That - due to the clause comparing
the two targets of the same PG.

This commit fixes the comparator. The sorted queue is no longer
guaranteed to have a ripe deep target before the corresponding
shallow one, which is less than optimal - but it won't crash.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #60106 into main

* refs/pull/60106/head:
msg/async/ProtocolV2: pass `desc` as `std::string_view` to write()

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #60174 into main

* refs/pull/60174/head:
common/Finisher: pass name as std::string_view to ctor
common/Finisher: add method get_thread_name()
mgr/ActivePyModule: build thread name with fmt
mgr/ActivePyModule: return std::string_view instead of std::string copy
common/Finisher: use fmt to build strings
common/Finisher: un-inline ctor and dtor
common/Finisher: add `const` to several fields
common/Finisher: merge duplicate field initializers
common/Finisher: call notify_one() instead of notify_all()
common/Finisher: wake up after pushing to the queue
common/Finisher: do not wake up the thread if already running
common/Finisher: call logger without holding the lock
common/Finisher: use `std::lock_guard` instead of `std::unique_lock`
common/Finisher: merge all queue() container methods into one template

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>