doc/architecture: edit "Calculating PG IDs"

Edit the section "Calcluating PG IDs" in doc/architecture.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53580 from AliMasarweh/wip-alimasa-persistent-q-migration

RGW | Bucket Notification: migrating old entries to support persistency control

Reviewed-by: yuvalif <ylifshit@redhat.com>

Merge pull request #46692 from pdvian/wip-unreachable-osd

osd: Report health error if OSD public address is not within subnet

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52509 from ericqzhao/main

mgr: add throttle policy for DaemonServer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #53740 from zdover23/wip-doc-2023-09-30-architecture-15-of-x

doc/architecture: edit several sections

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53618 from adk3798/cephadm-refactor-more-net-utils

cephadm: move more funcs into net_utils.py

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

doc/architecture: edit several sections

Edit the following sections in doc/architecture.rst:

 1. Dynamic Cluster Management
 2. About Pools
 3. Mapping PGs to OSDs

The tone of "Dynamic Cluster Management" remains a bit too close to the
tone of marketing material, in my opinion, but I will return to firm it
up when I have finished a once-over of architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53702 from adk3798/cephadm-refactor-registry-login

cephadm: move registry_login to container_engines.py

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53617 from adk3798/cephadm-refactor-more-data-utils

cephadm: move more funcs into data_utils.py

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #53612 from adk3798/fix-refreshed-column

mgr/cephadm: fix REFRESHED column of orch ps being unpopulated

Reviewed-by: Michael Fritch <mfritch@suse.com>

cephadm: re-format black data_utils.py

Signed-off-by: Adam King <adking@redhat.com>

cephadm: move more funcs into data_utils.py

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #53682 from phlogistonjohn/jjm-cephadm-move-pathify

cephadm: move pathify & get_file_timestamp to file_utils

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53683 from phlogistonjohn/jjm-cephadm-move-getters

cephadm: move context getters

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53726 from zdover23/wip-doc-2023-09-29-architecture-14-of-x

doc/architecture: edit "Replication"

Reviewed-by: Cole Mitchell <cole.mitchell.ceph.@gmail.com>

Merge pull request #53605 from sayantani11/osd-1

script/backport-resolve-issue: Update script with latest versions

Merge pull request #53723 from zdover23/wip-doc-2023-09-29-architecture-13-of-x

doc/architecture: edit "Data Scrubbing"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "Replication"

Edit "Replication" in the "Smart Daemons Enable Hyperscale" section of
doc/architecture.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/architecture: edit "Data Scrubbing"

Edit the "Data Scrubbing" listitem in the list of benefits conferred by
the use by OSDs of the aggregate power of the cluster, in the section
"Smart Daemons Enable Hyperscale" in doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #53594 into main

* refs/pull/53594/head:
qa: fix "no orch backend set" in nfs suite

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53707 from zdover23/wip-doc-2023-09-28-architecture-12-of-x

doc/architecture: edit "OSD Membership and Status"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "OSD Membership and Status"

Edit "OSD Membership and Status" in the "Smart Daemons Enable
Hyperscale" section of doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53608 from cbodley/wip-62536

qa/rgw/sts: keycloak task installs java manually

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

RGW | Bucket Notification: migrating old entries to support persistency control

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

Merge pull request #53432 from liu-chunmei/seastore_device_size

crimson/vstart: default seastore_device_size will be out of space f…

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

cephadm: re-format black container_engines.py

Signed-off-by: Adam King <adking@redhat.com>

cephadm: move registry_login to container_engines.py

Signed-off-by: Adam King <adking@redhat.com>

cephadm: move logging from registry_login to command_registry_login

So that registry_login can be moved to container_engines.py
without creating a dependency on logging there

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #53607 from cfsnyder/wip-cfsnyder-bucket-check-bug

rgw: fix radosgw-admin bucket check stat calculation bug

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52393 from yuvalif/wip-yuval-enable-trace-lua

common/tracer: remove is_enabled check in add_span methods

reviewed-by: cbodley, ljflores

Merge pull request #53523 from rhcs-dashboard/add-ceph-daemon-label-labeled-counters

exporter: add ceph_daemon labels to labeled counters as well

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52828 from rhcs-dashboard/angular-15

mgr/dashboard: upgrade to angular v15

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #53596 from rhcs-dashboard/fix-validations-in-cephfs-forms

mgr/dashboard: fix cephfs forms validations

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge PR #53529 into main

* refs/pull/53529/head:
doc: update colorama, packaging
cephfs-shell: drop LooseVersion for version.parse

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53685 from zdover23/wip-doc-2023-09-27-architecture-11-of-x

doc/architecture: edit "OSDs service clients directly"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "OSDs service clients directly"

Edit "OSDs service clients directly" in the list in
"Smart Daemons Enable Hyperscale" in doc/architecure.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

cephadm: remove (doc)string

Remove a, now irrelevant (IMO), docstring that might have been
associated with the recently moved `cached_stdin` global. It's not
really clear how helpful it is in light of the new "compiled"
cephadm, so I am opting to remove it rather than move it.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: black format context_getters.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: rename fetch_tcp_ports to fetch_endpoints

Rename fetch_tcp_ports to fetch_endpoints to more closely match what
the function is doing.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move context based getters to context_getters.py

Move functions that exist mainly to pull information out of the
CephadmContext in various ways to a new context_getters.py module.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move a logging line closer to where the data is used

Move a logging line closer to where the data being logging is
used. This avoids having a dependency on logging in a fairly
simple function and should make moving the function in a future
commit easier.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move pathify & get_file_timestamp to file_utils

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53615 from ronen-fr/wip-rf-great-move

tools, rgw/test: add std:: qualifiers to 'move'

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #53048 from YiteGu/add-slow-count-committed-kv

os/bluestore: add some slow count for bluestore

Merge pull request #52930 from leonid-s-usov/msg_wait

msg/AsyncMessenger: re-evaluate the stop condition when woken up in 'wait()'

Merge pull request #52906 from sseshasa/wip-fix-config-dump-pretty-print

mon/ConfigMonitor: Show localized name in "config dump --format json" output

Merge pull request #52200 from jinmyeonglee/61819

client: append to buffer list to save all result from wildcard command

Merge pull request #50585 from ssdohammer-sl/fix-metadata-obj-promote

osd: chunked type manifest object not to be promoted

mgr/dashboard: upgrade to cypress 12

Looks like chrome 117 will need cypress >=12.15.0
https://github.com/cypress-io/cypress-documentation/issues/5479

Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: adapt and refactor jest test files

Use the `configureTestBed` as the placeholder for adding the
declarations, imports... that is required for the unit tests to run

Fixes: https://tracker.ceph.com/issues/62844
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51923 from kamoltat/wip-ksirivad-fix-61570

pybind/mgr/pg_autoscaler: fix warn when not too few pgs
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #53324 from aclamk/wip-aclamk-osd-op-scraper-fixes

Set of scraper improvements

exporter: add ceph_daemon labels to labeled counters as well

Exporter missed adding the `ceph_daemon` or `instance_id`
labels(in case if rgw metrics) to the new labeled performance counters.

Fixes: https://tracker.ceph.com/issues/62874
Signed-off-by: avanthakkar <avanjohn@gmail.com>

Merge pull request #53284 from xxhdx1985126/wip-seastore-remap-intermediate_key

crimson/os/seastore/transaction_manager: move intermediate_key by "remap_offset" when remapping the "back" half of the original pin

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #53455 from xxhdx1985126/wip-seastore-no-touch-pending

crimson/os/seastore/cache: don't add EXIST_CLEAN extents to lru

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #53657 from zdover23/wip-doc-2023-09-26-architecture-10-of-x

doc/architecture: edit "SDEH"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "SDEH"

Edit the front matter of the "Smart Daemons Enable Hyperscale" section
of doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/cache: replace is_clean by is_stable_clean wherever
possible

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: move intermediate_key by
"remap_offset" when remapping the "back" half of the original pin

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #52442 from kamoltat/wip-ksirivad-fix-tracker-61922

pybind/mgr/pg_autoscaler: noautoscale flag retains individual pool configs
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

mgr/cephadm: add unit test for _process_ls_output

This is a weird function to make a unit test for
since it's essentially just moving data from a
list of dicts into a list of DaemonDescriptions,
but wanted to have some coverage to lower the
chance of breaking something again.

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: fix REFRESHED column of orch ps being unpopulated

The way the daemon ls data was processed was changed in
https://github.com/ceph/ceph/commit/1fd4132c7c03602719f29230732b12c8afa04779
and it seems that commit removed a line that set the
last_refresh field. This commit just adds it back
in the new location after the change.

Without this in "ceph orch ps" the REFRESHED column
for every daemon just reports "-"

Fixes: https://tracker.ceph.com/issues/62954
Signed-off-by: Adam King <adking@redhat.com>

cephadm: re-format black net_utils.py

There was a conflict here between what black
and flake8 were okay with. After running
format-black flake8 would report

cephadmlib/net_utils.py:211:29: E203 whitespace before ':'
cephadmlib/net_utils.py:259:25: E203 whitespace before ':'
cephadmlib/net_utils.py:272:27: E203 whitespace before ':'

but removing the whitespace before the ":" would
cause black to complain. For parse_mon_ip and
parse_mon_addrv, it was doing array slicing with
a start of "0" so I believe we can just remove the
start point without affecting anything (since "0" is
just the beginning of the string anyway). For
get_ipv6_address it had to actually be altered in
a way that had the potential to be done incorrectly,
so I added a unit test for it in a previous commit
in order to make sure we maintain the behavior.

Signed-off-by: Adam King <adking@redhat.com>

cephadm: add unit test for get_ipv6_address

I wanted to modify this function slightly
to try to make both black and flake8 happy
with it, so adding a unit test to make sure
I don't break it.

Signed-off-by: Adam King <adking@redhat.com>

cephadm: move more funcs into net_utils.py

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #51926 from adk3798/host-drain-zap

mgr/cephadm: add ability to zap OSDs' devices while draining host

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #53589 from adk3798/cephadm-refactor-host-facts

cephadm: create host_facts.py in cephadmlib

Reviewed-by: John Mulligan <jmulligan@redhat.com>

cephadm: format black host_facts.py

Signed-off-by: Adam King <adking@redhat.com>

cephadm: create host_facts.py in cephadmlib

For storing classes/functions related to gathering
information about the hosts such as disk enclosures
and networks

Signed-off-by: Adam King <adking@redhat.com>

qa/workunits: modified tests for noautoscale flag change

modified:

`qa/workunits/mon/test_noautoscale_flag.sh`
`qa/workunits/cephtool/test.sh`

adding test coverage to files mentioned above

Fixes: https://tracker.ceph.com/issues/61922
Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #53482 from adk3798/cephadm-refactor-decorators

cephadm: start decorators.py in cephadmlib

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53479 from rhcs-dashboard/upgrade-ui-popover-fix

mgr/dashboard: remove empty popover when there are no health warns

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

rgw: fix output formatting of bucket index check admin api

The bucket index check admin API was previously returning invalid
JSON.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>

Merge pull request #53512 from rhcs-dashboard/promqls-enhancements

mgr/dashboard: fix prometheus queries subscriptions

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53631 from zdover23/wip-doc-2023-09-25-architecture-9-of-x

doc/architecture: edit "HA Authentication"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "HA Authentication"

Edit "High Availability Authentication" in doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53575 from rkachach/fix_issue_62838

mgr/cephadm: removing double quotes from the generated nvmeof config

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

cephadm: black format initial decorators.py

Signed-off-by: Adam King <adking@redhat.com>

cephadm: start decorators.py in cephadmlib

Originally, wanted to move all the decorators into
their own files. Unfortunately, that isn't possible
at this time as most of them depend on things that
are still within cephadm.py This includes

list_daemons
_rm_cluster
is_fsid
termcolor
ContainerInfo
Ceph

and I'm sure I'm missing some others. We'll have to
revisit this again later when more of these things
have moved, or they can be slowly moved as their
dependencies are.

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #53468 from phlogistonjohn/jjm-cephadm-move-container-types

cephadm: move container types

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53613 from zdover23/wip-doc-2023-09-23-architecture-8-of-x

doc/architecture: "Edit HA Auth"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: "Edit HA Auth"

Rewrite the explanation of how a client authenticates against a monitor.
This is a rewrite of a single paragraph, and has been set apart in its
own PR so that it can receive the maximum amount of scrutiny that the
upstream Ceph community can muster.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53097 from mkogan1/rgw-wip-t62141-valg-supp

valgrind: UninitCondition under __run_exit_handlers suppression

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49405 from tchaikov/clang-16

run-make-check.sh: use clang-17 if available

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #51890 from lxbsz/wip-61394

test: unmount the mountpoint just before exiting

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

run-make-check.sh: use clang-17 if available

now that clang-17 has been released, let's use it if available.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

rgw/test: add std:: qualifiers to 'move'

to silence compiler warnings.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

tools: add std:: qualifiers to 'move'

to silence compiler warnings.
e.g. (ceph_dedup_tool.cc:1104:32: warning: unqualified call to
'std::move' [-Wunqualified-std-cast-call]
     estimate_threads.push_back(move(ptr));
                               ^
                               std::

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #53546 from mcv21/deb_package_fixes

debian: update to dh compat 12, fix more serious packaging errors, correct copyright syntax

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #50611 from cbodley/wip-rgw-sal-list-buckets

rgw/sal: list_buckets() returns RGWBucketEnts

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

qa: fix "no orch backend set" in nfs suite

Fixes: https://tracker.ceph.com/issues/62870
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

qa/rgw/sts: keycloak task installs java manually

java had already been installed automatically before centos 9. add an
override to install the jdk-17 packages manually

Fixes: https://tracker.ceph.com/issues/62536
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix radosgw-admin bucket check stat calculation bug

Fixes a regression with radosgw-admin bucket check stat
calculation and bucket reshard stat calculation when
there are objects that have transitioned from unversioned
to versioned. The bug was introduced in
152aadb71b61c53a4832a1c8cf82fce3d64b68d1.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>

rgw: add test case to reproduce bucket check stats bug for versioned bucket

Reproduces a regression where radosgw-admin bucket check incorrectly counts
objects that started as unversioned and later transitioned to versioned.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>

script/backport-resolve-issue: Update script with latest versions
Signed-off-by: Sayantani Saha <ii.sayantani.ii@gmail.com>

rgw/sal: StoreBucket no longer wraps RGWBucketEnt

`sal::Bucket` no longer needs to wrap `RGWBucketEnt` to support user
bucket listings, so can be represented by `RGWBucketInfo` alone. the
bucket stats interfaces that relied on RGWBucketEnt internally now
return their result as either `RGWBucketEnt` or `RGWStorageStats`

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: list_buckets() returns RGWBucketEnts

`sal::User::list_buckets()` no longer returns a map of `sal::Bucket`
handles. it now uses `std::span<RGWBucketEnt>` for input and output.
`RGWBucketEnt` contains all of the information we need to satisfy
ListBuckets requests, and also stores the `rgw_bucket` key for use with
`Driver::get_bucket()` where a `sal::Bucket` handle is necessary

`sal::BucketList` contains the span of results and the `next_marker`.
the `is_truncated` flag was removed in favor of `!next_marker.empty()`

the checks for `user->get_max_buckets()` on bucket creation now use a
paginated `check_user_max_buckets()` helper function that limits the
number of allocated entries to `rgw_list_buckets_max_chunk`

Signed-off-by: Casey Bodley <cbodley@redhat.com>

debian: call dh_python3 for ceph-{base,common,fuse,volume}

In the cases of ceph-base, ceph-common, and ceph-fuse, this picks up
that these packages contain python scripts and adds a necessary
python3 dependency. In the case of ceph-volume it additionally parses
the requirements.txt file.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian: radosgw.init to installinit, remove auto_build override

Installation of init scripts properly belongs with dh_installinit, so
move the installation there.

That means we no longer need the override of dh_auto_build, which
simplifies the rules file.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian: specify a dependency on python3 for cephadm

cephadm is a compressed zipapp, and dh3_python3 doesn't understand
this sort of binary file, so fails to produce the required python3
dependency. So specify this explicitly in debian/control

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>