osd/scheduler/mClockScheduler: Use same profile and client ids for all clients

Currently, without the distributed mClock (dmClock) feature, all external
clients classified under op_scheduler_class::client must share the
allocated reservation and limit as set within default_external_client_info.
This is realized only if all the external clients are assigned the same
id. This was not true because the client_id field within the
client_profile_id_t was set to item.get_owner() which uniquely identified
each client. As a result, an increase in the client count resulted in a
proportional increase in the consumption of the allocated limit which is
not the desired outcome in the current implementation.

The fix is to use the same client_id and profile_id within the
client_profile_id_t structure for all clients. These fields are set to '0'.
This ensures that all such clients get the same QoS controls which means
reservation and limit is shared between them. In other words, regardless
of the number of clients active, the reservation and limit consumption
remains constant as per the mClock profile setting and is shared amongst
them.

NOTE: When dmClock feature (client vs client QoS) is implemented, the
above fields would change with client_id set to the global id (client.####)
and the profile_id set to the id generated by the client's QoS profile.

Fixes: https://tracker.ceph.com/issues/62293
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #52635 from rhcs-dashboard/list-subvolume-groups

mgr/dashboard: List subvolume groups

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #52702 from zdover23/wip-doc-2023-07-31-documenting-ceph-linking-customs

doc/start: update linking conventions

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #52715 from zdover23/wip-doc-2023-08-01-radosgw-admin-rate-limit-management

doc/radosgw: update rate limit management

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #52225 from yuvalif/wip-yuval-58412

rgw/lua: give tables fully qualified names

Merge pull request #52864 from adk3798/cephadm-debian-recommends

debian/control: add docker-ce as recommends for cephadm package

Reviewed-by: John Mulligan <jmulligan@redhat.com>

mgr/dashboard: list subvolume groups

Fixes: https://tracker.ceph.com/issues/62168
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

rgw/lua: allow metatable fields to be cached

this is by making metatable names fully qualified names
that contain the entire "path" for reaching them and not
just the name of the object they point to.

with the fix, the code would either create a new metatable,
as in this case:

local o1 = Request.Object
-- new metatable is created to represent the Object in Request.Object
local o2 = Request.CopyFrom.Object
-- new metatable (with different upvalues) is created to represent Request.CopyFrom.Object
print(o1.Name)
print(o2.Name)

or, will reuse an existing metatable, as in this case:

local o1 = Request.Object
-- new metatable is created to represent the Object in Request.Object
local o2 = Request.Object
-- reuse the same metatable
print(o1.Name)
print(o2.Name)

Fixes: https://tracker.ceph.com/issues/58412
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw/lua: expose correct value in Request.Bucket.User

a change in zipper caused bucket->get_info().owner to return an empty
string. so the lua value now expose: bucket->get_owner()->get_id()

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #52273 from yuvalif/wip-yuval-lua-mem-limit

rgw/lua: add configurable memory limit for the lua state

Merge pull request #52154 from rhcs-dashboard/hosts-pagination

mgr/dashboard: host pagination

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52745 from zdover23/wip-doc-2023-08-02-rados-operations-add-or-rm-mons-1-of-x

doc/rados: edit operations/add-or-rm-mons (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52884 from phlogistonjohn/jjm-doc-typo-fix

doc/cephadm: fix typo in cephadm initial crush location section

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephadm: fix typo in cephadm initial crush location section

A minor typo fix found while skimming through the cephadm docs:
"will no remove" -> "will not remove".

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc/rados: edit operations/add-or-rm-mons (1 of x)

Edit the first part of doc/rados/operations/add-or-rm-mons.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52543 from aclamk/wip-aclamk-fix-valgrind-bluefs-test

os/bluestore: Fix unittest_bluefs

mgr/dashboard: paginate hosts

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>
Fixes: https://tracker.ceph.com/issues/56513

Merge pull request #52792 from leonid-s-usov/bulk-data-pool

mgr/volumes: create bulk data pool for new volumes

Merge pull request #50427 from ifed01/wip-ifed-encrypted-ceph-volume

ceph_volume: support encrypted volumes for lvm new-db/new-wal/migrate  commands

Merge pull request #52870 from rhcs-dashboard/gh-add-to-project-link-broken

.github: pr-triage broken link fix

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

.github: pr-triage broken link fix

The link I put was broken so replacing it with the correct one:

Fixes: https://github.com/ceph/ceph/pull/52455
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52786 from rhcs-dashboard/fs-subvolumes-create

mgr/dashboard: cephfs subvolume creation form

Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #52455 from ceph/gh-add-to-project-fix

.github: pr-triage fix

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge PR #52788 into main

* refs/pull/52788/head:
mds: remove unused C_MDC_OpenSnapRealms

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #52838 from zztaki/fix-backport-resolve-issue

script/backport-resolve-issue: fix script to handle RC versions

Merge pull request #52490 from zdover23/wip-doc-2023-07-18-add-reef-18-2-0-release-notes

doc: Add Reef 18.2.0 release notes

doc: Add Reef 18.2.0 release notes

Add release notes for the first stable release of Ceph Reef (18.2.0).

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>
Co-authored-by: Laura Flores <lflores@redhat.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52736 from ljflores/wip-update-stale

.github/workflows: pin the stale bot to the latest version

Merge pull request #52494 from alimaredia/wip-rgw-crypt-keystone-fix

qa: fix keystone in rgw/crypt/barbican.yaml

Reviewed-by: Ali Maredia <amaredia@redhat.com>

debian/control: add docker-ce as recommends for cephadm package

The issue this is addressing is that if somebody has
docker-ce installed but not docker.io with the
current recommends it will install podman. If
the user has docker-ce we don't want it to do that.

Signed-off-by: Adam King <adking@redhat.com>

.github: pr-triage fix

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52860 from Matan-B/wip-matanb-monstore-osd_snap-docs

tools/ceph_monstore_tool: add docs

doc/man/8/ceph-monstore-tool: add documentation

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52720 from debjyoti-pandit/api-docs-routes

mgr/dashboard: exposed the open api documentations to details card

Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: cephfs subvolume creation form

Fixes: https://tracker.ceph.com/issues/62345
Signed-off-by: Nizamudeen A <nia@redhat.com>

ceph-volume: make raw prepare use encryption_units.prepare_dmcrypt

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

mgr/dashboard: create/generalize checked table scope_permission table

Fixes: https://tracker.ceph.com/issues/62345
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: exposed the open api documentations to details card

Fixes: https://tracker.ceph.com/issues/62296
Signed-off-by: dpandit <dpandit@li-6f946d4c-287f-11b2-a85c-b0cb07ed936d.ibm.com>

Merge pull request #52395 from rhcs-dashboard/upgrade-ui-enhancements

mgr/dashboard: add logs and daemon version to upgrade UI

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #52647 from rhcs-dashboard/rbd-typing

mgr/dashboard: rbd type status

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52703 from zdover23/wip-doc-2023-07-31-documenting-ceph-linking-customs-2

doc/start: update linking conventions

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #52827 from zdover23/wip-doc-2023-08-05-troubleshooting-troubleshooting-mon-2-of-x

doc/rados: edit troubleshooting-mon.rst (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw/lua: add configurable memory limit for the lua state

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

script/backport-resolve-issue: fix script to handle RC versions

Fixes: https://tracker.ceph.com/issues/61746
Signed-off-by: Zhelong Zhao <zztaki@outlook.com>
Signed-off-by: Zhelong Zhao <50295684+zztaki@users.noreply.github.com>

Merge pull request #52785 from Matan-B/wip-matanb-crimson-add-asan-req

ceph.spec.in: add libasan to seastar requires

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/rados: edit troubleshooting-mon.rst (2 of x)

Edit the second part of
doc/rados/troubleshooting/troubleshooting-mon.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52824 from bluikko/bluikko-doc-radosgw-config-language

doc/radosgw: Improve language and formatting in config-ref.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/radosgw: Improve language and formatting in config-ref.rst

The "Topic persistency settings" section includes convoluted
and incomplete sentences so attempt to improve the
language. The section lacks context so add a link to the
relevant documentation about Notifications.

The "Garbage Collection settings" note includes a sentence
that looks incomplete so try to improve the language.

Don't capitalize "Ops" in "RGW Ops" especially when the
same sentence talks about "op queue" with lower case "o".

Also improve formatting: indentation of a note and wrap
several very long lines.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

ceph.spec.in: add sanitizers to seastar requires

```
CMake Error at src/seastar/CMakeLists.txt:782 (message):
  Sanitizers not found!
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52784 from Matan-B/wip-matanb-crimson-only-centos-9

qa/suites/crimson-rados: add centos9 to supported distros

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52778 from zdover23/wip-doc-2023-08-03-rados-operations-add-or-rm-mons-2-of-x

doc/rados: edit operations/add-or-rm-mons (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit operations/add-or-rm-mons (2 of x)

Edit the second part of doc/rados/operations/add-or-rm-mons.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52802 from dvanders/support_the_foundation

doc: Add a Ceph Foundation footer to docs.ceph.com

Merge pull request #52814 from cbodley/wip-doc-rgw-zonegroup-feature-default

doc/rgw: correct statement about default zone features

Merge pull request #52787 from rhcs-dashboard/fix-rgw-average-object-size

mgr/dashboard: fix average object size calculation in rgw overview dashboard

Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: update the upgrade layout by introducing new items

Shows Cluster Health
Shows the count of mgr daemons
Shows a table with all the daemon versions
Display the cluster logs

Fixes: https://tracker.ceph.com/issues/62016
Signed-off-by: Nizamudeen A <nia@redhat.com>

doc/rgw: correct statement about default zone features

this sentence wasn't updated when we added the Default column to the
table below

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: pin rgw/crypt suite temporarily to ubuntu_latest

barbican tests are only passing on ubunutu latest,
this commit allows us to have a green crypt suite.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

qa: change admin_host to admin_url in barbican.py

Signed-off-by: Ali Maredia <amaredia@redhat.com>

qa: bump keystone version in barbican.yaml

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #52248 from cbodley/wip-46062

rgw: fetch_remote_obj() preserves original part lengths for BlockDecrypt

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

mgr/dashboard: fix average object size calculation in rgw overview
dashboard

Fixes: https://tracker.ceph.com/issues/62318
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #52274 from cbodley/wip-61862

qa/rgw: specify cluster name in 'radosgw-admin gc process'

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #52800 from cbodley/wip-62312

doc/rgw: several response headers are supported

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc: Add a Ceph Foundation footer to docs.ceph.com

Add an info box to the footer of every docs page inviting visitors
to support the Ceph Foundation.

Signed-off-by: Dan van der Ster <dan.vanderster@clyso.com>

doc/rgw: several response headers are supported

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ReleaseNotes: document recovery of encrypted multipart objects

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add command to resync encrypted multipart objects

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRados::set_attrs() takes optional mtime

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: expose part id in RGWObjManifest iterator

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #51786 from cbodley/wip-61473

rgw/crypt: apply rgw_crypt_default_encryption_key by default

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

mon/MDSMonitor: print a note when adding data pool to a file system that doesn't have 'bulk' flag set

The note will only be printed if the pool has pg_autoscale_mode set to ON and the bulk flag is missing

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mgr/volumes: set the 'bulk' flag for data pools created automatically for a new volume

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Fixes: https://tracker.ceph.com/issues/61595

mgr/dashboard: make the logs component configurable and reusable

This will make the logs component easier to reuse for now in upgrade
component and later, wherever we want

Fixes: https://tracker.ceph.com/issues/62016
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52447 from liumiaomiaoIntel/qatzip_update

common/compressor: update the interfaces and data structures

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52634 from AliMasarweh/wip-alimasa-persistant-q-enhance

RGW: control the persistency of the notification

Merge pull request #52405 from rhcs-dashboard/rgw-overview-dashboard-graphs

mgr/dashboard: add graphs to rgw overview dashboard

Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mds: remove unused C_MDC_OpenSnapRealms

This was just left and forgot to be removed.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #52711 from cbodley/wip-62250

rgw: retry metadata cache notifications with INVALIDATE_OBJ

Reviewed-by: Adam Emerson <aemerson@redhat.com>

rgw: preserve RGW_ATTR_CRYPT_PARTS of already-replicated objects

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fetch_remote_obj() preserves original part lengths for BlockDecrypt

because multisite replicates multipart objects as a single part, we lose
information about the part sizes from the original manifest that is
necessary to correctly decrypt across those part boundaries

on replication, parse the part lengths out of the source object's
manifest, and store them in a separate RGW_ATTR_CRYPT_PARTS for use on
decryption

Fixes: https://tracker.ceph.com/issues/46062
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BlockDecrypt filter parses manifest parts before construction

users now call a static read_manifest_parts() function, and pass the
resulting vector into the BlockDecrypt constructor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52657 from rhcs-dashboard/fs-subvolumes

mgr/dashboard: cephfs subvolume list

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: cephfs subvolume list

Fixes: https://tracker.ceph.com/issues/62182
Signed-off-by: Nizamudeen A <nia@redhat.com>

qa/suites/crimson-rados: use crimson-supported-all-distro

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/distros: add crimson-supported-all-distro

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

add graphs to rgw overview dashboard

Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit a63026348cb4fcb287046c6dc5432d5709d5dff8)

Merge pull request #52757 from aisakaki/wip-retire-cp

crimson/os/seastore: retire_extent_addr clean up

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #52775 from dvanders/kplus1

 doc/rados/operations: Correct EC min_size recommendation to K+1 in erasure-code.rst

Merge pull request #52719 from cbodley/wip-rgw-get_ws_listing_op

rgw/swift: remove redundant moves of object name

Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>

Merge pull request #52714 from cbodley/wip-ec-clay-unit

ec: initialize lost_chunk in ErasureCodeClay

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: for EC we recommend K+1

Update the doc to match the reality in the code. I don't know where
the recommendation to have min_size = k+2 came from, but for awhile
now we've defaulted to K+1. See PR #8008.

Signed-off-by: Dan van der Ster <dan.vanderster@clyso.com>

Merge pull request #52007 from tcoldrick-bb/fix-invalid-json

rgw/rgw_admin.cc: Handle pg_ver and source_zone explicitly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #48732 into main

* refs/pull/48732/head:
doc: add MDS treatise on segments
pybind/mgr/dashboard: bump teuthology version
qa/tasks/vstart_runner: stop overriding _run_python
qa: stop overriding ceph_w prefix in vstart_runner
qa/tasks/vstart_runner: update teuthology helper tool paths
qa/tasks/vstart_runner: allow writing to command's stdin
vstart.sh: always add CEPH_CONF to vstart_environment.sh
qa: use stdin-killer for python3 command
qa: add killpoint testing for mds shutdown
qa: fix background exit condition
qa: add filesystem helper for setting transient config
qa: add helper for waiting for a rank to fail
mds: add incompat feature for minor log segments
mds: introduce ELid event to create/close log
mds: change EResetJournal to major segment boundary
mds: add killpoints for MDS shutdown
qa: add numerous subtree test
mds: track larger log events in perf dump
mds: add minor LogSegment boundaries
mds: obviate MDLog::start_entry
mds: retype to properly sized unsigned ints
mds: use unsigned type for event count
mds: use base Context class for generalization
mds: optimize segment lookup
mds: add stream dump for LogSegment
mds: handle conf changes in mdlog
mds: remove redundant comment
mds: remove unused method
mds: set a reasonable minimum number of segments
mds: sort configs

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #52754 into main

* refs/pull/52754/head:
test: explicitly link to ceph-common for some libcephfs tests

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #52761 from bluikko/patch-18

doc/rados/configuration: Avoid repeating "support" in msgr2.rst

Merge pull request #52709 from rishabh-d-dave/cephfs-test_snapshots

qa/cephfs: fix test_disallow_monitor_managed_snaps_for_fs_pools

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52762 from yuvalif/wip-yuval-62264

rgw/amqp: skip idleness tests since it needs to sleep longer than 30s

Reviewed-by: Casey Bodley <cbodley@redhat.com>

RGW: control the persistency of the notification
via adding expiry by number of retries and time
and controling the frequency of retries by sleep duration (in the options of vstart)

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

Merge pull request #52559 from adamemerson/wip-62097

build: Remove ceph-libboost* packages in install-deps

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>