Merge pull request #47366 from adk3798/quoted-networks

cephadm: support quotes around public/cluster network in config passed to bootstrap

Reviewed-by: Francesco Pantano <fpantano@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #47346 from rhcs-dashboard/remove-grafana-loki-dependency

mgr/dashboard: add flag to automatically deploy loki/promtail service at bootstrap

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47489 from cyx1231st/wip-seastore-trimmer

crimson/os/seastore: introduce JournalTrimmer with refactors

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Zhang Song <zhangsong325@gmail.com>

crimson/os/seastore: introduce JournalTrimmer

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: decouple prepare_record() from segment_provider

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: be explicit about detecting out-dated delta using segment info

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: decouple segment release from transaction manager

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: construct TransactionManager classes after device mount

To construct TransactionManager after all the devices are discoverred.

Also, it makes the following cleanups possible:
* Cleanup SeaStore and TransactionManager factory methods.
* Decouple TransactionManager from SegmentManagerGroup.
* Drop the unnecessary tm_make_config_t.
* Drop the unnecessary add_device() methods.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/epm: configure prefer_ool through the primary_device

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47548 from cbodley/wip-57073

install-deps: fix centos 8's use of gcc-toolset-11

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47544 from ceph/pmem-c9

ceph.spec.in: disable system_pmdk on aarch64

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #47543 from cbodley/wip-cmake-cxx20-compiler-versions

cmake: enforce minimum compiler versions for c++20

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47534 from tchaikov/wip-cmake-motr

cmake: find motr libraries and header before using them

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: unit test for parsing networks from config

Signed-off-by: Adam King <adking@redhat.com>

cephadm: support quotes around public/cluster network in config passed to bootstrap

Fixes: https://tracker.ceph.com/issues/56973
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #47421 from adk3798/conf-osd-reconfig

mgr/cephadm: recreate osd config when redeploy/reconfiguring

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #47464 from michaelalang/master

mgr/cephadm: provide an additional hint when running into I/O closed exception

Reviewed-by: Adam King <adking@redhat.com>

ceph.spec.in: install gcc-toolset-11-libatomic-devel in x86_64 also

otherwise after enabling gcc-toolset-11, cmake fails with:

- Performing Test HAVE_LIBATOMIC - Failed
CMake Error at cmake/modules/CheckCxxAtomic.cmake:66 (message):
  Host compiler /opt/rh/gcc-toolset-11/root/usr/bin/g++ requires libatomic,
  but it is not found

Signed-off-by: Casey Bodley <cbodley@redhat.com>

install-deps: bump gcc-toolset version to 11

match the toolset versions installed from ceph.spec.in

Signed-off-by: Casey Bodley <cbodley@redhat.com>

install-deps: update ensure_decent_gcc_on_rh for gcc-toolset

Signed-off-by: Casey Bodley <cbodley@redhat.com>

install-deps: centos8 doesn't have or need centos-release-scl

Fixes: https://tracker.ceph.com/issues/57073
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46378 from pritha-srivastava/wip-rgw-sts-modify-session-duration

rgw/sts: modify max_session_duration using update role REST API/ radosgw-admin command.

ceph.spec.in: disable system_pmdk on aarch64

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #47538 from tchaikov/wip-enable-gts

ceph.spec.in: %enable_devtoolset11 only if the macro is defined

cmake: enforce minimum compiler versions for c++20

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47541 from ceph/1723

doc: Add missing index links for 17.2.3

doc: Add missing index links for 17.2.3

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph.spec.in: always use stock compiler on el9

as RHEL/CentOS 9, we have the access to GCC-11, which is good enough for
compiling main HEAD even with WITH_SEASTAR=ON.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #46400 from rkachach/fix_issue_55733

mgr/cephadm: adding dynamic prometheus configuration based on http_sd_config

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

ceph.spec.in: define %gts_prefix

less repeatings this way, also  allow maintainer to use other version of GTS.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

ceph.spec.in: %enable_devtoolset11 only if the macro is defined

there is chance that we are using `yum-builddep` to prepare the
build dependencies. in that case, gcc-toolset-11-build is not
installed. it's like a chicken-egg dilemma, but the point is
`yum-builddep` is able to pull in the gcc-toolset-11-build. once
gcc-toolset-11-build is installed, we will have the %enable_devtoolset11
rpm macro.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43155 from ljflores/wip-install-deps-fix

install-deps.sh: ensure that pip ugrades to most recent version within virtualenv

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47278 from rzarzynski/wip-cephx-verify_auth_cleanups

monc, cephx: improve debugs and slightly clean invalidate_ticket() up

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #47092 from dparmar18/wip-dparmar-cephadm-simple-1

pybind/mgr/cephadm/upgrade: allow upgrades without reducing max_mds

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

mgr/cephadm: setting version field to a 17.2.3

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

mgr/cephadm: adding dynamic prometheus config based on http_sd

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

cmake: find motr libraries and header before using them

cmake should fail when generating the building system, if the necessary
development libraries or the header files are missing.

in this change, Findmotr.cmake is added to detect the motr library
and header so that we can use it to find the motr package before using
it.

See-also: https://tracker.ceph.com/issues/57079
Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47479 from ceph/wip-backportbot_script-main

.github/workflows: add create-backport action

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

.github/workflows: add create-backport action

Currently there's a cron job in a teuthology VM running a script to find all trackers in needs-backports and to create their corresponding backport trackers. This is done through the [Backport Bot](https://tracker.ceph.com/users/12172) Redmine account.

This PR intends to run this cron job task as a periodic Github Action.

Signed-off-by: Ernesto Puerta <37327689+epuertat@users.noreply.github.com>
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #47514 from guits/c-v-thread-inventory-2

ceph-volume: system.get_mounts() refactor

Merge pull request #47198 from ceph/wip-yuriw-release-15.2.17-main

doc: 15.2.17 Release Notes

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases/octopus.rst: add note for CVE-2022-0670

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases/octopus.rst: add note for rbd perf image iostat/iotop

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases/octopus.rst: add release note for rados

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

doc: 15.2.17 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #47179 from rhcs-dashboard/service-id-info

mgr/dashboard: validate mds service_id and helper text for service_id

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #47517 from tchaikov/wip-cortx-motr-jammy

install-deps: install cortx-motr from chacra on jammy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47488 from cbodley/wip-rgw-lc-shuffle

rgw/lc: random_sequence() uses default_random_engine

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #47504 from cbodley/wip-57050

rgw/cmake: stop building all of radosgw a shared lib

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #47503 from ceph/infosec

.gitleaks.toml: Allowlist cephadm test

qa/suites/fs/upgrade: added upgrade_without_reducing_max_mds for testing the fail_fs option

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

qa/tasks/fs: add cases in post_upgrade_checks() for fail_fs

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

doc/cephadm/upgrade: Add doc for mds upgrade without reducing mds_mds to 1.

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

PendingReleaseNotes: added note related to new mds upgrade option using cephadm

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

pybind/mgr/cephadm/upgrade: allow upgrades without reducing max_mds

Fixes: https://tracker.ceph.com/issues/55715
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #46101 from myoungwon/wip-dedup-tool-update

tool: modify object-dedup command to run on manifest object

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #46242 from kamoltat/wip-ksirivad-fix-overlapping-message

pybind/mgr/pg_autoscaler: change overlapping roots to warning

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #47408 from ronen-fr/wip-rf-snprefix

osd: a faster version of SnapMapper::get_prefix()

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #46573 from aclamk/wip-one-cmdparse

common/cmdparse: Unsplit crimson/classic

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #47379 from NitzanMordhai/wip-nitzan-valgrind-handle-recovery-delete

osd: remove invalid put on message

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #47500 from tchaikov/wip-no-more-distutils

pybind/mgr/dashboard: do not use distutils.version.StrictVersion

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

install-deps: install cortx-motr from chacra

this is an intermediate solution before cotx-motr upstream builds the
packages for jammy and upload them to github.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

install-deps: extract install_cortx_motr_on_ubuntu out

for better readability and maintainability

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47496 from tchaikov/wip-atomic-fc36

ceph.spec.in: add libatomic to BuildRequires on fedora

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

pybind/mgr/dashboard: do not use distutils.version.StrictVersion

replace `distutils.version.StrictVersion` with
`pkg_resources.parse_version()`

as the former is deprecated, see https://peps.python.org/pep-0632/.
let's use `pkg_resources` instead. this change also addresses
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1010894.
we have this issue when testing with an ubuntu jammy test node.
see https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1967139

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47505 from ceph/dot

doc: Install graphviz

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

ceph-volume: system.get_mounts() refactor

When a network mount is present in `/proc/mounts` but for any reason
the corresponding server is down, this function hangs forever.
In a cluster deployed with cephadm, the consequence is that
it triggers `ceph-volume inventory` commands that hang and stay in D
state.

The idea here is to use a thread with a timeout to abort the call if the
timeout is reached.
`get_mounts()` is now a method of a class so we can exclude a path
altogether during the whole `inventory` execution (otherwise,
ceph-volume would try to access it as many devices there is on the
host which could slow down the inventory execution)

Fixes: https://tracker.ceph.com/issues/57070
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47476 from rosinL/fix-enoent

crimson/os/seastore: return future when no element error tolerated

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47425 from cyx1231st/wip-seastore-backref-cleanup

crimson/os/seastore: backref related cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

doc: Install graphviz

Fixes `WARNING: dot command 'dot' cannot be run (needed for graphviz output), check the graphviz_dot setting`

Signed-off-by: David Galloway <dgallowa@redhat.com>

- mgr/cephadm: provide an additional hint when running into I/O closed exception from execnet seen as
               Error EINVAL: Can't communicate with remote host `127.0.0.1`, possibly because python3 is not installed there: cannot send (already closed?)

Signed-off-by: Michaela Lang <milang@redhat.com>

rgw/cmake: stop building all of radosgw a shared lib

Fixes: https://tracker.ceph.com/issues/57050
Signed-off-by: Casey Bodley <cbodley@redhat.com>

.gitleaks.toml: Allowlist cephadm test

I signed up for Red Hat's gitleaks scanning service.  The self-signed key is just for testing purposes.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #47475 from Huber-ming/stale_delete

rgw: correct help message of 'reshard stale-instances delete'

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

ceph.spec.in: add libatomic to BuildRequires on fedora

otherwise we'd have failures like

/opt/compiler-explorer/gcc-trunk-20220808/bin/../lib/gcc/x86_64-linux-gnu/13.0.0/../../../../x86_64-linux-gnu/bin/ld:
/tmp/ccVlMbVh.o: in function `std::atomic<tagged_ptr>::store(tagged_ptr,
std::memory_order)':
/opt/compiler-explorer/gcc-trunk-20220808/include/c++/13.0.0/atomic:273:
undefined reference to `__atomic_store_16'

when generating the building system using CMake on fedora 36.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47486 from tchaikov/wip-dashboard-newer-teuth

mgr/dashboard: bump up teuthology

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #46972 from nSedrickm/button-name-accessibility-update

mgr/dashboard: Improve level A accessibility for buttons

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #46927 from nmshelke/fix-55762

mgr/volumes: filter internal directories in 'subvolumegroup ls' command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #47436 from SUSE/wip-mib-snmp-dir

ceph.spec.in: ceph-mib: require snmp-mibs on SUSE distros

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Justin Caratzas <jcaratza@redhat.com>

mgr/dashboard: add flag to automatically deploy loki/promtail service at bootstrap

Fixes: https://tracker.ceph.com/issues/56964
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #47423 from tchaikov/wip-crimson-cmp

crimson/os: rewrite ordering using std::strong_ordering

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

rgw/lc: random_sequence() uses default_random_engine

we used random_device to seed a default_random_engine, but didn't use
the engine in our call to shuffle()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson, test: s/std::random_shuffle/std::shuffle/

to silence warnings like:

../src/test/crimson/seastore/onode_tree/test_staged_fltree.cc:371:10: warning: 'random_shuffle<__gnu_cxx::__normal_iterator<std::pair<ghobject_t, crimson::os::seastore::onode::test_item_t> *, std::vector<std::pair<ghobject_t, crimson::os::seastore::onode::test_item_t>>>>' is deprecated: use 'std::shuffle' instead [-Wdeprecated-declarations]
    std::random_shuffle(kvs.begin(), kvs.end());
         ^

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: rewrite ordering using std::strong_ordering

the goals are

1. to use std::strong_ordering mechinary for better readability
   and maintainbility
2. to remove unnecessary abstraction
3. use concept for better error message and readability.

changes:

* replace MatchKindCMP with std::strong_ordering
* replace compare_to() with operator<=>
* introduce a concept `IsFullKey` so we can use it when developing
  generic facilities to operate on both materialized key or view.
* use `IsFullKey` in place of `KeyT` when appropriate

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47485 from tchaikov/crimson-os-cleanups

crimson/os: cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #47487 from snosratiershad/examples/add-allow-unordered-to-list-objects-v2-extention-support

examples/boto3: add ListObjectsV2Request shape to boto3 sdk extras with AllowUnordered as member

examples/boto3: add ListObjectsV2Request shape to boto3 sdk extras with AllowUnordered as member

feat: add ListObjectsV2Request shape to boto3 sdk extras with AllowUnordered as member

feat: add list-objects-v2 unordered supports to doc

Signed-off-by: Salar Nosrati-Ershad <snosratiershad@gmail.com>

Merge pull request #44772 from ybwang0211/Bucket-Operations-doc

doc: Make the SYNTAX code writing of some interfaces of the api document consistent with others

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #46989 from petrutlucian94/boost_shared_mutex

common: use boost::shared_mutex on Windows

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47216 from sseshasa/wip-fix-high-cpu-util-in-process

osd: Handle oncommits and wait for future work items from mClock queue

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: bump up teuthology

to include the fix of e7c5d67e10fe29da22180f9e09b8973ae166c8fc,
see https://github.com/ceph/teuthology/pull/1746.
to address the test failure on ubuntu jammy. where we have python3.10

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/osd: trade macro for type alias

for better readablity.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: drop unused function

Btree::compare(const ghobject_t &l, const ghobject_t &r) is never used,
so let's drop it.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47309 from Thingee/update-foundation-mems-202207

doc: Updating Ceph Foundation members for July

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #47478 from tchaikov/wip-cmake-cython

cmake: remove spaces in macro used for compiling cython code

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #47481 from ljflores/wip-github-actions

.github/workflows: increase operations-per-run to 100 in stale bot

.github/workflows: increase operations-per-run to 100 in stale bot

The stale bot's `operations-per-run`
(https://github.com/actions/stale#operations-per-run) corresponds to the max
number of API calls it is allowed to make per hour. Currently,
`operations-per-run` is set to 30, which means that the stale bot
can make up to 30 API calls per hour.

With this limit in place, the stale bot is only able to process 400 PRs at a time.
Since there are 900+ PRs in the Ceph repository, we should increase the number of
operations to cover them all. This needs to be done with care though, since GitHub
has a rate limit
(https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting)
depending on business plan.

According to GitHub's documentation on GitHub action requests
(https://docs.github.com/en/rest/overview/resources-in-the-rest-api#requests-from-github-actions),
the rate limit is 1,000 requests per hour per repository when using `GITHUB_TOKEN` (which we are).
For enterprise accounts, GitHub Enterprise Cloud's rate limit applies, and the limit is 15,000
requests per hour per repository.

Based on this information, we should be fine to increase the max `operations-per-run`
to 100. This would cover a little over 1000 PRs, which should be enough
to process the 900-some-odd PRs in the Ceph repository.

Signed-off-by: Laura Flores <lflores@redhat.com>

test/test_snap_mapper: add tests for key format stability

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #47459 from tchaikov/wip-cmake-debian-python

cmake: install pure python module to deb_system path

Merge pull request #47397 from kalebskeithley/luamgr

rgw: lua, add support for lua packages to the lua manager