Merge pull request #55411 from rishabh-d-dave/qa-cephfs-caps_helper

qa/cephfs: declare f-string to get values to substitute variable names

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #55493 into main

* refs/pull/55493/head:
client: remove unused/undefined fake_write_size()

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #55499 from cbodley/wip-squid-deprecate-blkin

doc: deprecate blkin tracing

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #55332 from athanatos/sjust/wip-crush-multi-choose

crush: add multistep retry rules

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #54624 from cbodley/wip-cmake-qat

cmake: add ExternalProjects to enable QAT by default

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55490 from eroldan/patch-2

doc/start: fix typo in hw-recs

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc:start.rst fix typo in hw-recs

s/power less/power loss/

Signed-off-by: Eduardo Roldan <trazor@gmail.com>

doc: deprecate blkin tracing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55267 from athos-ribeiro/cmake-empty-submodules

cmake: quote variable to allow empty values

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55398 from ifed01/wip-ifed-fix-ceph-vol-migrate-wal

ceph-volume: Fix migration from WAL to data with no DB

client: remove unused/undefined fake_write_size()

there are no usages of this function in entire codebase

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #55416 from ivoalmeida/snapshot-schedule-edit

mgr/dashboard: added edit functionality

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55465 from neesingh-rh/wip-63991

doc: add description of metric fields for cephfs-top

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #55479 from ronen-fr/wip-rf-fifolegacy

rgw: mark maybe_unused variables

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55475 from zdover23/wip-doc-2024-02-07-radosgw-admin-confval

doc/radosgw: add confval directives

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55149 from rkachach/fix_issue_63992

mgr/prometheus: fix orch check to prevent Prometheus from crashing

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

cmake: WITH_QATLIB/ZIP disabled on aarch64

Signed-off-by: Casey Bodley <cbodley@redhat.com>

compressor: move QatAccel out of common

move the QatAccel instance out of the Compressor base class and into
the zlib and lz4 compressors that can use it

this avoids having to link QAT into the ceph-common library, and only
the plugins where it's necessary

had to add LZ4Compressor.cc to store the new static variable

Signed-off-by: Casey Bodley <cbodley@redhat.com>

build/win32: disable qat options

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rpm: add system_qat option, default on for centos9

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: enable WITH_QATLIB and WITH_QATZIP by default

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: WITH_QATZIP requires WITH_QATLIB or WITH_QATDRV

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: build from qatzip submodule

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: build qatzip from source when WITH_SYSTEM_QATZIP=OFF

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: build from qatlib submodule

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: build qat from source when WITH_SYSTEM_QATLIB=OFF

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54623 from alimaredia/wip-rgw-op-metrics-section-rename

rgw: change section name of rgw_op counters

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/prometheus: fix orch check to prevent Prometheus from crashing
Fixes: https://tracker.ceph.com/issues/63992
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

cmake: rename qatzip->QATzip and support pkg-config

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: add WITH_QATLIB to build against qatlib

FindQAT.cmake adds library targets QAT::qat and QAT::usdm for qatlib,
and FindQatDrv.cmake adds alias targets to the same

src/compressor and src/crypto/qat link against these common QAT targets

src/crypto/qat/qcccrypto.h uses new HAVE_QATDRV define to determine
which headers to include

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: rename WITH_QAT to WITH_QATDRV

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crypto/qat: remove unused variable qcc_alloc_mutex

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crypto/qat: remove unused captures of 'this'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crypto/qat: add missing include use_future.hpp

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crypto/qat: qualify spawn::yield_context

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55476 from zdover23/wip-doc-2024-02-07-cephadm-tshooting

doc/cephadm - edit t11ing.

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: add confval directives

Add confval directives to the documentation of "quota cache" options.
This addresses a request made by Antony D'Atri in https://github.com/ceph/ceph/pull/55075/files#r1444006246.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw: mark maybe_unused variables

Specifically, variables that are only used in debug builds.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54596 from cbodley/wip-cmake-dencoder-modules

cmake: ceph-dencoder depends on ceph-dencoder-modules

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55376 from guits/guits-cv-64248

ceph-volume: fix zap_partitions() in devices.lvm.zap

mgr/dashboard: snapshot schedule edit form

Fixes: https://tracker.ceph.com/issues/64331
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

doc/cephadm - edit t11ing.

Edit "Running Repeated Debugging Sessions" in
doc/cephadm/troubleshooting.rst. Improve grammar. Fix a misspelling.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc: add description of metric fields for cephfs-top

Fixes: https://tracker.ceph.com/issues/63991
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #55444 from Matan-B/wip-crimson-seastar-upgrade

Crimson: bump up seastar version

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54280 from xxhdx1985126/wip-rbm-scattered-allocation

crimson/os/seastore/random_block_manager: allow allocating multiple extents for a single disk space allocation request

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

ceph.spec.in: crimson add protobuf

Following https://github.com/scylladb/seastar/commit/eea4102091bb4d689e0b5a769512966b716ee8a2

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/crush/CrushLocation: fix compilation errors

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

CMakeLists.txt: fix Seastar expected dirs

```
  # create the directory so cmake won't complain when looking at the imported
  # target: Seastar exports this directory created at build-time
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

seastar: bump up seastar submodule version

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55054 from pereman2/zns-remove

os/bluestore: remove zoned namespace support
It has never been finished and now its in the way of future improvements.

Merge pull request #55275 from qiuxinyidian/rgw-noti-dev

rgw: add topic owner user check when creating

reviewed-by: cbodley, kchheda3

Merge pull request #55447 from zdover23/wip-doc-2024-02-05-rados-operations-pools

doc/rados: update PG guidance

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: modify topic owner check when creating

add tests to cover topic policies
as well as behavior when no policies are defined

Fixes: https://tracker.ceph.com/issues/64124
Signed-off-by: Zhipeng Li <qiuxinyidian@gmail.com>

Merge pull request #54311 from gabriel-samfira/do-not-escape-slash

RGW: fix cloud-sync not being able to sync folders

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55252 from jzhu116-bloomberg/wip-21800

rgw/multisite: avoid writing multipart parts to the bucket index log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55293 from jzhu116-bloomberg/wip-63178

rgw/lc: pass in flag as an argument when calling function complete(..)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54661 from lightmelodies/rgw-gc

rgw: optimize gc chain size calculation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55452 from nbalacha/wip-typo-fix

rbd-mirror: fix a typo in NotifyAckPayload::dump()

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55406 from cbodley/wip-64255

rgw: SiteConfig::load() falls back to local zonegroup

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #55448 from guits/subclass-thread

node-proxy: refactor entrypoint

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #55280 from ronen-fr/wip-rf-extra-lg

osd: remove superfluous '>' in spg_t formatter

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55414 from yuvalif/wip_yuval_url_decode_sns

rgw/rest: fix url decode of post params passed as attributes

reviewed-by: cbodley

rbd-mirror: fixed a typo in instance_watcher

Fixes a typo in NotifyAckPayload::dump().

Signed-off-by: N Balachandran <nibalach@redhat.com>

node-proxy: refactor entrypoint

This commit introduces a major refactor of the main
entrypoint.

- subclass threading.Thread:
  - Introduce a new class `BaseThread()` that is a
    `threading.Thread()` abstraction class in order
    to monitor the different threads.
  - `BaseSystem()` inherits from `BaseThread()`.
  - Handle `SIGTERM` signal in order to gracefully shutdown
    node-proxy (make threads exit gracefully, log out from RedFish API, etc.)

Additionally, this:
  - drops the class `Logger()` from util.py which
was not adding value. It is now replaced with a simple `get_logger()`
function.
  - changes the node-proxy API port from 8080 to 9456
    (8080 being widely used for frontend apps...)
  - changes the container entrypoint in order to use the
    `ceph-node-proxy` binary from the packaging

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

doc/rados: update PG guidance

Update the "Creating a Pool" section of doc/rados/operations/pools.rst
so that the documentation no longer insists that the user change the
values of "osd_pool_default_pg_num" and "osd_pool_default_pgp_num".

See also: https://github.com/ceph/ceph/pull/55419

Tracker: https://tracker.ceph.com/issues/64259

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/transaction_manager: allow allocating multiple
extents when rewriting data extents

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/object_data_handler: allow alloc multiple extents
for a single request

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: fix errorator mismatch

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

test/crimson/seastore: add unittest cases for scattered allocation

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

test/crimson/seastore: test avlallocator's scattered allocation

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: TransactionManager::alloc_extent(s) may return enospc

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: add new interfaces to alloc multiple extents for a
single request

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #55089 from cbodley/wip-boost-mirrors

make-dist: update boost mirrors

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55434 from anthonyeleven/rgwyamlin

src/common/options: Correct typo in rgw.yaml.in

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55276 from Matan-B/wip-matanb-crimson-alienstore-cores

crimson: rework CPU pinning

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

tasks/.../test_erasure_code_profile: assertSubset in test_create_plugin

Newly added profile options may break this test otherwise.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/cli/osdmaptool/crush.t: adjust --import-crush size output

Signed-off-by: Samuel Just <sjust@redhat.com>

test/cli/crushtool/choose-args.t: add msr related json output

Signed-off-by: Samuel Just <sjust@redhat.com>

qa/erasure-code: modify jerasure 4/2 ec test case to use msr

Signed-off-by: Samuel Just <sjust@redhat.com>

doc/rados/operations: add CRUSH MSR documentation

Signed-off-by: Samuel Just <sjust@redhat.com>

erasure-code: add support for multiple osds in a single failure domain

Adds support for crush-osds-per-failure-domain and
crush-num-failure-domains via MSR rules.

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart.sh: add --require-osd-and-client-version and --use-crush-tunables flags

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart.sh: add --osds-per-host

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crush/crush.cc: add tests specifically for MSR

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crush/crush.cc: add test variants for firstn rules

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crush/crush.cc: convert indep test cases to test MSR as well

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crush/crush.cc: s/NULL/nullptr/g

Signed-off-by: Samuel Just <sjust@redhat.com>

mon/OSDMonitor: generalize rule type check for pools

Add rule_valid_for_pool_type to CrushWrapper to generalize
rule type <-> pool type mapping to include the new MSR
types.

Signed-off-by: Samuel Just <sjust@redhat.com>

doc/dev/crush-msr.rst: add developer summary of crush msr

Signed-off-by: Samuel Just <sjust@redhat.com>

crush/mapper: add support for MSR types

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #55419 from zdover23/wip-doc-2024-02-02-rados-config-pool-pg-config-ref

doc/rados: update config for autoscaler

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

src/common/options: Correct typo in rgw.yaml.in

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

doc/rados: update config for autoscaler

Update doc/rados/configuration/pool-pg-config-ref.rst to account for the
behavior of autoscaler.

Previously, this file was last meaningfully altered in 2013, prior to
the invention of autoscaler. A recent confusion was brought to my
attention on the Ceph Slack whereby a user attempted to alter the
default values of a Quincy cluster, as suggested in this documentation.
That alteration caused Ceph to throw the error "Error ERANGE: 'pgp_num'
must be greater than 0 and lower or equal than 'pg_num', which in this
case is one" and a related "rgw_init_ioctx ERROR" reading in part
"Numerical result out of range". The user removed the
"osd_pool_default_pgp_num" configuration line from ceph.conf and the
cluster worked as expected. I presume that this is because the removal
of this configuration line allowed autoscaler to work as intended.

Fixes: https://tracker.ceph.com/issues/64259
Co-authored-by: David Orman <ormandj@corenode.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw: SiteConfig::load() falls back to local zonegroup

allow radosgw-admin commands like 'user create' to operate on a new zone
that hasn't been committed to the period yet. this follows similar logic
in RGWSI_Zone::do_start()

Fixes: https://tracker.ceph.com/issues/64255
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54312 from dvanders/dvanders_mcltf_true

qa/tasks/cephadm: enable mon_cluster_log_to_file

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54837 from imtzw/urlencode

rgw: fix list obj v1 nextmarker

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54867 from BBoozmen/oozmen_lc_subsystem

rgw/common: add rgw lifecycle specific debug log subsystem

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>

Merge pull request #55162 from cfsnyder/wip-64014-cfsnyder-pacific

rgw: fix issue with concurrent versioned deletes leaving behind olh entries

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #55229 from tobias-urdin/http-error-keystone-issue-admin-token-request

rgw: fix http error check in keystone issue_admin_token_request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55330 from baum/centos9-install-build-deps

🧹build dependencies: centos9