cephadm: re-format black data_utils.py

Signed-off-by: Adam King <adking@redhat.com>

cephadm: move more funcs into data_utils.py

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #53682 from phlogistonjohn/jjm-cephadm-move-pathify

cephadm: move pathify & get_file_timestamp to file_utils

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53683 from phlogistonjohn/jjm-cephadm-move-getters

cephadm: move context getters

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53726 from zdover23/wip-doc-2023-09-29-architecture-14-of-x

doc/architecture: edit "Replication"

Reviewed-by: Cole Mitchell <cole.mitchell.ceph.@gmail.com>

Merge pull request #53605 from sayantani11/osd-1

script/backport-resolve-issue: Update script with latest versions

Merge pull request #53723 from zdover23/wip-doc-2023-09-29-architecture-13-of-x

doc/architecture: edit "Data Scrubbing"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "Replication"

Edit "Replication" in the "Smart Daemons Enable Hyperscale" section of
doc/architecture.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/architecture: edit "Data Scrubbing"

Edit the "Data Scrubbing" listitem in the list of benefits conferred by
the use by OSDs of the aggregate power of the cluster, in the section
"Smart Daemons Enable Hyperscale" in doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #53594 into main

* refs/pull/53594/head:
qa: fix "no orch backend set" in nfs suite

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53707 from zdover23/wip-doc-2023-09-28-architecture-12-of-x

doc/architecture: edit "OSD Membership and Status"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "OSD Membership and Status"

Edit "OSD Membership and Status" in the "Smart Daemons Enable
Hyperscale" section of doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53608 from cbodley/wip-62536

qa/rgw/sts: keycloak task installs java manually

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #53432 from liu-chunmei/seastore_device_size

crimson/vstart: default seastore_device_size will be out of space f…

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #53607 from cfsnyder/wip-cfsnyder-bucket-check-bug

rgw: fix radosgw-admin bucket check stat calculation bug

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52393 from yuvalif/wip-yuval-enable-trace-lua

common/tracer: remove is_enabled check in add_span methods

reviewed-by: cbodley, ljflores

Merge pull request #53523 from rhcs-dashboard/add-ceph-daemon-label-labeled-counters

exporter: add ceph_daemon labels to labeled counters as well

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52828 from rhcs-dashboard/angular-15

mgr/dashboard: upgrade to angular v15

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #53596 from rhcs-dashboard/fix-validations-in-cephfs-forms

mgr/dashboard: fix cephfs forms validations

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge PR #53529 into main

* refs/pull/53529/head:
doc: update colorama, packaging
cephfs-shell: drop LooseVersion for version.parse

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53685 from zdover23/wip-doc-2023-09-27-architecture-11-of-x

doc/architecture: edit "OSDs service clients directly"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "OSDs service clients directly"

Edit "OSDs service clients directly" in the list in
"Smart Daemons Enable Hyperscale" in doc/architecure.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

cephadm: remove (doc)string

Remove a, now irrelevant (IMO), docstring that might have been
associated with the recently moved `cached_stdin` global. It's not
really clear how helpful it is in light of the new "compiled"
cephadm, so I am opting to remove it rather than move it.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: black format context_getters.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: rename fetch_tcp_ports to fetch_endpoints

Rename fetch_tcp_ports to fetch_endpoints to more closely match what
the function is doing.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move context based getters to context_getters.py

Move functions that exist mainly to pull information out of the
CephadmContext in various ways to a new context_getters.py module.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move a logging line closer to where the data is used

Move a logging line closer to where the data being logging is
used. This avoids having a dependency on logging in a fairly
simple function and should make moving the function in a future
commit easier.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move pathify & get_file_timestamp to file_utils

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53615 from ronen-fr/wip-rf-great-move

tools, rgw/test: add std:: qualifiers to 'move'

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #53048 from YiteGu/add-slow-count-committed-kv

os/bluestore: add some slow count for bluestore

Merge pull request #52930 from leonid-s-usov/msg_wait

msg/AsyncMessenger: re-evaluate the stop condition when woken up in 'wait()'

Merge pull request #52906 from sseshasa/wip-fix-config-dump-pretty-print

mon/ConfigMonitor: Show localized name in "config dump --format json" output

Merge pull request #52200 from jinmyeonglee/61819

client: append to buffer list to save all result from wildcard command

Merge pull request #50585 from ssdohammer-sl/fix-metadata-obj-promote

osd: chunked type manifest object not to be promoted

mgr/dashboard: upgrade to cypress 12

Looks like chrome 117 will need cypress >=12.15.0
https://github.com/cypress-io/cypress-documentation/issues/5479

Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: adapt and refactor jest test files

Use the `configureTestBed` as the placeholder for adding the
declarations, imports... that is required for the unit tests to run

Fixes: https://tracker.ceph.com/issues/62844
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51923 from kamoltat/wip-ksirivad-fix-61570

pybind/mgr/pg_autoscaler: fix warn when not too few pgs
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #53324 from aclamk/wip-aclamk-osd-op-scraper-fixes

Set of scraper improvements

exporter: add ceph_daemon labels to labeled counters as well

Exporter missed adding the `ceph_daemon` or `instance_id`
labels(in case if rgw metrics) to the new labeled performance counters.

Fixes: https://tracker.ceph.com/issues/62874
Signed-off-by: avanthakkar <avanjohn@gmail.com>

Merge pull request #53284 from xxhdx1985126/wip-seastore-remap-intermediate_key

crimson/os/seastore/transaction_manager: move intermediate_key by "remap_offset" when remapping the "back" half of the original pin

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #53455 from xxhdx1985126/wip-seastore-no-touch-pending

crimson/os/seastore/cache: don't add EXIST_CLEAN extents to lru

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #53657 from zdover23/wip-doc-2023-09-26-architecture-10-of-x

doc/architecture: edit "SDEH"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "SDEH"

Edit the front matter of the "Smart Daemons Enable Hyperscale" section
of doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/cache: replace is_clean by is_stable_clean wherever
possible

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: move intermediate_key by
"remap_offset" when remapping the "back" half of the original pin

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #52442 from kamoltat/wip-ksirivad-fix-tracker-61922

pybind/mgr/pg_autoscaler: noautoscale flag retains individual pool configs
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #51926 from adk3798/host-drain-zap

mgr/cephadm: add ability to zap OSDs' devices while draining host

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #53589 from adk3798/cephadm-refactor-host-facts

cephadm: create host_facts.py in cephadmlib

Reviewed-by: John Mulligan <jmulligan@redhat.com>

cephadm: format black host_facts.py

Signed-off-by: Adam King <adking@redhat.com>

cephadm: create host_facts.py in cephadmlib

For storing classes/functions related to gathering
information about the hosts such as disk enclosures
and networks

Signed-off-by: Adam King <adking@redhat.com>

qa/workunits: modified tests for noautoscale flag change

modified:

`qa/workunits/mon/test_noautoscale_flag.sh`
`qa/workunits/cephtool/test.sh`

adding test coverage to files mentioned above

Fixes: https://tracker.ceph.com/issues/61922
Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #53482 from adk3798/cephadm-refactor-decorators

cephadm: start decorators.py in cephadmlib

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53479 from rhcs-dashboard/upgrade-ui-popover-fix

mgr/dashboard: remove empty popover when there are no health warns

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

rgw: fix output formatting of bucket index check admin api

The bucket index check admin API was previously returning invalid
JSON.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>

Merge pull request #53512 from rhcs-dashboard/promqls-enhancements

mgr/dashboard: fix prometheus queries subscriptions

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53631 from zdover23/wip-doc-2023-09-25-architecture-9-of-x

doc/architecture: edit "HA Authentication"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "HA Authentication"

Edit "High Availability Authentication" in doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53575 from rkachach/fix_issue_62838

mgr/cephadm: removing double quotes from the generated nvmeof config

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

cephadm: black format initial decorators.py

Signed-off-by: Adam King <adking@redhat.com>

cephadm: start decorators.py in cephadmlib

Originally, wanted to move all the decorators into
their own files. Unfortunately, that isn't possible
at this time as most of them depend on things that
are still within cephadm.py This includes

list_daemons
_rm_cluster
is_fsid
termcolor
ContainerInfo
Ceph

and I'm sure I'm missing some others. We'll have to
revisit this again later when more of these things
have moved, or they can be slowly moved as their
dependencies are.

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #53468 from phlogistonjohn/jjm-cephadm-move-container-types

cephadm: move container types

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53613 from zdover23/wip-doc-2023-09-23-architecture-8-of-x

doc/architecture: "Edit HA Auth"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: "Edit HA Auth"

Rewrite the explanation of how a client authenticates against a monitor.
This is a rewrite of a single paragraph, and has been set apart in its
own PR so that it can receive the maximum amount of scrutiny that the
upstream Ceph community can muster.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53097 from mkogan1/rgw-wip-t62141-valg-supp

valgrind: UninitCondition under __run_exit_handlers suppression

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49405 from tchaikov/clang-16

run-make-check.sh: use clang-17 if available

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #51890 from lxbsz/wip-61394

test: unmount the mountpoint just before exiting

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

run-make-check.sh: use clang-17 if available

now that clang-17 has been released, let's use it if available.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

rgw/test: add std:: qualifiers to 'move'

to silence compiler warnings.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

tools: add std:: qualifiers to 'move'

to silence compiler warnings.
e.g. (ceph_dedup_tool.cc:1104:32: warning: unqualified call to
'std::move' [-Wunqualified-std-cast-call]
     estimate_threads.push_back(move(ptr));
                               ^
                               std::

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #53546 from mcv21/deb_package_fixes

debian: update to dh compat 12, fix more serious packaging errors, correct copyright syntax

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #50611 from cbodley/wip-rgw-sal-list-buckets

rgw/sal: list_buckets() returns RGWBucketEnts

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

qa: fix "no orch backend set" in nfs suite

Fixes: https://tracker.ceph.com/issues/62870
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

qa/rgw/sts: keycloak task installs java manually

java had already been installed automatically before centos 9. add an
override to install the jdk-17 packages manually

Fixes: https://tracker.ceph.com/issues/62536
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix radosgw-admin bucket check stat calculation bug

Fixes a regression with radosgw-admin bucket check stat
calculation and bucket reshard stat calculation when
there are objects that have transitioned from unversioned
to versioned. The bug was introduced in
152aadb71b61c53a4832a1c8cf82fce3d64b68d1.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>

rgw: add test case to reproduce bucket check stats bug for versioned bucket

Reproduces a regression where radosgw-admin bucket check incorrectly counts
objects that started as unversioned and later transitioned to versioned.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>

script/backport-resolve-issue: Update script with latest versions
Signed-off-by: Sayantani Saha <ii.sayantani.ii@gmail.com>

rgw/sal: StoreBucket no longer wraps RGWBucketEnt

`sal::Bucket` no longer needs to wrap `RGWBucketEnt` to support user
bucket listings, so can be represented by `RGWBucketInfo` alone. the
bucket stats interfaces that relied on RGWBucketEnt internally now
return their result as either `RGWBucketEnt` or `RGWStorageStats`

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: list_buckets() returns RGWBucketEnts

`sal::User::list_buckets()` no longer returns a map of `sal::Bucket`
handles. it now uses `std::span<RGWBucketEnt>` for input and output.
`RGWBucketEnt` contains all of the information we need to satisfy
ListBuckets requests, and also stores the `rgw_bucket` key for use with
`Driver::get_bucket()` where a `sal::Bucket` handle is necessary

`sal::BucketList` contains the span of results and the `next_marker`.
the `is_truncated` flag was removed in favor of `!next_marker.empty()`

the checks for `user->get_max_buckets()` on bucket creation now use a
paginated `check_user_max_buckets()` helper function that limits the
number of allocated entries to `rgw_list_buckets_max_chunk`

Signed-off-by: Casey Bodley <cbodley@redhat.com>

debian: call dh_python3 for ceph-{base,common,fuse,volume}

In the cases of ceph-base, ceph-common, and ceph-fuse, this picks up
that these packages contain python scripts and adds a necessary
python3 dependency. In the case of ceph-volume it additionally parses
the requirements.txt file.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian: radosgw.init to installinit, remove auto_build override

Installation of init scripts properly belongs with dh_installinit, so
move the installation there.

That means we no longer need the override of dh_auto_build, which
simplifies the rules file.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian: specify a dependency on python3 for cephadm

cephadm is a compressed zipapp, and dh3_python3 doesn't understand
this sort of binary file, so fails to produce the required python3
dependency. So specify this explicitly in debian/control

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian: remove obsolete ceph-base.docs, restore dh_installdocs

debian/ceph-base.docs only referred to a README that doesn't exist, so
remove it. Because dpkg-source doesn't reflect deletions from debian/
cf the orig.tar.gz, also remove the file in dh_auto_clean.

Then do away with the removal of the empty override of dh_installdocs;
the main benefit of which here is that debian/copyright gets installed
in all of the built packages, which otherwise lack a copyright
file.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian: correct maintainer address

This means that debian/control matches changelog entries, and that the
Maintainer address is up to date.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian: dh compat to 12, necessary init/systemd adjustments

Bring the dh compat level to 12, the most recent supported by the
oldest supported Ubuntu LTS release, 20.04. This necessitates changes
to how initscripts & systemd packaging are done.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian/copyright: update syntax, maintainer, add license stanzas

Update the header paragraph to link to the canonical URL for the
format, and point to dev@ceph.io as the Contact.

Also add License: stanzas to reflect the licences in use (and refer to
fuller versions in /usr/share/common-licenses/ as appropriate).

This means that packages containing this copyright file are better in
compliance with the licences concerned.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian: add missing item separators in debian/control

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian: remove {Build-,}Depends on essential packages

Unless there's a version requirement (which there isn't here),
packages should not declare a Build-Depends: or Depends: relationship
on essential packages. Policy link:

https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian: specify interpreters for ceph-mon and ceph-osd postinsts

These were previously missing. The requirement for interpreters is in
Debian policy section 10.4:
https://www.debian.org/doc/debian-policy/ch-files.html#s-scripts

Debian's packaging already adds the #! to these two postinsts. In
practice, a text executible without a #! line will likely be executed
by the calling shell, so a lot of the time we'd get away with it
unless the administrator is using an incompatible shell like tcsh.

This behaviour of shells is documented in POSIX section 1(e)(i)(b)
here:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian: Build-Depend on g++ 11 or greater

Rely on the packaging system to provide a suitable g++ of version 11
or greater, and removing the corresponding hard-coding from
debian/rules, since cmake will then find a suitable version. This
seems better than trying to hard-code a particular version in
debian/rules, and Debian package building tools like e.g. sbuild will
then do the right thing.

This enables Reef (v18.2.0) to build on Debian bookworm in a clean
chroot.

Fixes: https://tracker.ceph.com/issues/61845
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

Merge pull request #52524 from dparmar18/wip-62072

tools/cephfs_mirror: only perform actions if init succeed

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #53407 into main

* refs/pull/53407/head:
Client: Fix nonblocking-io zero by read

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #53279 into main

* refs/pull/53279/head:
qa: descrease pgbench scale factor to 32 for postgresql database test

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #52747 into main

* refs/pull/52747/head:
cephfs-mirror: restart failed/blocklisted replayer instances
cephfs-mirror: do not run concurrent C_RestartMirroring context

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #53418 into main

* refs/pull/53418/head:
qa: use exisitng ignorelist override list for fs:mirror[-ha]

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #48027 from lxbsz/wip-57154

mds/client: check the cephx mds auth access in client side

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52576 from cfsnyder/wip-62075

rgw: add radosgw-admin bucket check olh/unlinked commands

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52042 from rishabh-d-dave/better-errmsg-for-perm

MDSAuthCaps: print better error message for perm flag in MDS caps

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #47533 from rishabh-d-dave/ceph-fuse-cmd

qa/cephfs: cleanup code for ceph-fuse command

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #52259 from cbodley/wip-61859

qa/rgw: disable 'rgw lc debug interval' in most suites

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #53531 from ronen-fr/wip-rf-squeue2

osd/scrub: extract scrub initiation code out of the OSD

Reviewed-by: Samuel Just <sjust@redhat.com>