qa: add qa/tasks/nvmeof.py and rbd/nvmeof_basic_task and fio workunits

This is v2 of the rbd/nvmeof test: It deploys 1 gateway and 1 initiator.
Then does basic verification on nvme commands and runs fio.

This commit creates:
1. qa/tasks/nvmeof.py: adds a new 'Nvmeof' task which deploys
    the gateway and shares config with the initiator hosts.
    Sharing config was previously done by 'nvmeof_gateway_cfg' task
    in qa/tasks/cephadm.py (that task is removed in this commit).
2. qa/workunits/rbd/nvmeof_basic_tests.sh:
    Runs nvme commands (discovery, connect, connect-all, disconnect-all,
    and list-subsys) and does basic verification of the output.
3. qa/workunits/rbd/nvmeof_fio_test.sh:
    Runs fio command. Also runs iostat in parallel if IOSTAT_INTERVAL
    variable is set. This variable configures the delay between each iostat
    print.

nvmeof-cli upgrade from v0.0.6 to v0.0.7 introduced major changes
to all nvmeof commands. This commit changes v0.0.6 commands to
v0.0.7 in qa/workunits/rbd/nvmeof_initiator.sh

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #55467 from rhcs-dashboard/rgw-s3-analytics-grafana-dashboard

mgr/dashboard: Add RGW per user/bucket panels in  grafana

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55494 from ivoalmeida/snapshot-schedule-delete-activate

mgr/dashboard: added delete and activation actions

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55529 from tchaikov/cmake-cap

cmake: find_package(cap) before linking against it

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: added delete and activation actions

Fixes: https://tracker.ceph.com/issues/64355
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #55528 from tchaikov/cmake-boost-debug

cmake: build boost debug variant when CMAKE_BUILD_TYPE is Debug

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55199 from igomon-bloomberg/wip-publish-commit-async

rgw/s3-notifications: async commit to persistent notification queue

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #55371 from pritha-srivastava/wip-rgw-correct-err-msg

rgw/auth: correcting the error message to indicate

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55250 from cbodley/wip-64084

rgw/auth/s3: validate x-amz-content-sha256 for empty payloads

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #55478 from ronen-fr/wip-rf-update-times

osd: restoring timely collection of PG stats

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

cmake: find_package(cap) before linking against it

before this change, we link against libcap without finding it. this
works fine as long as libcap-devel or libcap-dev is installed in the
system. but if it is not, the source would fail to build due to missing
`sys/capability.h`. this is not a great developer experience.

in this change, a `Findcap.cmake` is added to find the capability
library. which would fail the build at the configure phase.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cmake: build boost debug variant when CMAKE_BUILD_TYPE is Debug

boost has some different predefined  build variants. they are quite
like CMake's CMAKE_BUILD_TYPE. in which, "debug" enables some
features related features. so it would be nice if we can have it
enabled for the Debug build, if the boost is built from source.
see also
https://www.boost.org/build/doc/html/bbv2/overview/builtins/features.html

before this change, we always build the "release" variant.  in this
change, "debug" variant is built if Ceph's is built with
CMAKE_BUILD_TYPE=Debug. please note, this change does not change
the way how boost is built when packaging Ceph, as our debian/rpm
receipts do not define CMAKE_BUILD_TYPE and respect the distros'
settings, in that case, the "release" variant is still built.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55525 from zdover23/wip-doc-2024-02-11-radosgw-55524-rollback

doc/radosgw: undo 55524

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/radosgw: undo 55524

Roll back the docs changes made in
https://github.com/ceph/ceph/pull/55524, in accordance with Casey
Bodley's instructions to me here:
https://github.com/ceph/ceph/pull/55524#issuecomment-1937020543.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55524 from zdover23/wip-doc-2024-02-10-radosgw-lua-scripting-options

doc/radosgw: remove invalid LUA context options

Merge pull request #55507 from ljflores/wip-tracker-64343

doc/radosgw: remove invalid LUA context options

Remove "background", "getdata", and "putdata" from the list of LUA
context options. Passing these options throws the following error:
"ERROR: invalid script context: background. must be one of: preRequest,
postRequest".

Fixes: https://tracker.ceph.com/issues/64327
Signed-off-by: Zac Dover <zac.dover@proton.me>

qa/suites/orch: whitelist warnings that are expected in test environments

Fixes: https://tracker.ceph.com/issues/64343
Signed-off-by: Laura Flores <lflores@ibm.com>

mgr/dashboard: Add RGW per user/bucket panels in  grafana

Fixes: https://tracker.ceph.com/issues/64359
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #53521 from xxhdx1985126/wip-62857

crimson/osd/replicated_backend: don't put pg's metadata updates in repops

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54650 from pereman2/buffer_map-noptr

os/bluestore: use buffer values instead of pointers in buffer_map

osd: no 'legacy' form for two configuration options

also - fixing review comments not addressed in the original PR.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd: restoring timely collection of PG stats

500 seconds is way too long, e.g. when compared to the 5s
default configuration option for the manager collection of the OSD data.

Fixes tracker issue 53342 note 5 (a specific scenario leading
to 'not all pgs scrubbed')

Fixes: https://tracker.ceph.com/issues/53342 - partial fix
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55462 from afreen23/fix-64270

mgr/dashboard: fix error while accessing roles tab when policy attached

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55365 from kotreshhr/ceph-mds-replica-debug-logs

mds: Add debug messages in replica handling

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #55411 from rishabh-d-dave/qa-cephfs-caps_helper

qa/cephfs: declare f-string to get values to substitute variable names

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #55493 into main

* refs/pull/55493/head:
client: remove unused/undefined fake_write_size()

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #55499 from cbodley/wip-squid-deprecate-blkin

doc: deprecate blkin tracing

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #55332 from athanatos/sjust/wip-crush-multi-choose

crush: add multistep retry rules

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #54624 from cbodley/wip-cmake-qat

cmake: add ExternalProjects to enable QAT by default

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55490 from eroldan/patch-2

doc/start: fix typo in hw-recs

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc:start.rst fix typo in hw-recs

s/power less/power loss/

Signed-off-by: Eduardo Roldan <trazor@gmail.com>

doc: deprecate blkin tracing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55267 from athos-ribeiro/cmake-empty-submodules

cmake: quote variable to allow empty values

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55398 from ifed01/wip-ifed-fix-ceph-vol-migrate-wal

ceph-volume: Fix migration from WAL to data with no DB

client: remove unused/undefined fake_write_size()

there are no usages of this function in entire codebase

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

mgr/dashboard: fix error while accessing roles tab when policy attached

Fixes https://tracker.ceph.com/issues/64270

Issue:
======
Accessing Object->Users-Roles tab causing 500 internal servor error.
This is due to the "PermissionPolicies" which are attached to role and
backend was not handling this field for rgw roles.

Fix:
====
Added "PermissionPolicies" as the valid field in backend and updated
frontend to render the attached policy in formatted JSON

Signed-off-by: Afreen <afreen23.git@gmail.com>

Merge pull request #55416 from ivoalmeida/snapshot-schedule-edit

mgr/dashboard: added edit functionality

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55465 from neesingh-rh/wip-63991

doc: add description of metric fields for cephfs-top

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #55479 from ronen-fr/wip-rf-fifolegacy

rgw: mark maybe_unused variables

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55475 from zdover23/wip-doc-2024-02-07-radosgw-admin-confval

doc/radosgw: add confval directives

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55149 from rkachach/fix_issue_63992

mgr/prometheus: fix orch check to prevent Prometheus from crashing

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

cmake: WITH_QATLIB/ZIP disabled on aarch64

Signed-off-by: Casey Bodley <cbodley@redhat.com>

compressor: move QatAccel out of common

move the QatAccel instance out of the Compressor base class and into
the zlib and lz4 compressors that can use it

this avoids having to link QAT into the ceph-common library, and only
the plugins where it's necessary

had to add LZ4Compressor.cc to store the new static variable

Signed-off-by: Casey Bodley <cbodley@redhat.com>

build/win32: disable qat options

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rpm: add system_qat option, default on for centos9

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: enable WITH_QATLIB and WITH_QATZIP by default

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: WITH_QATZIP requires WITH_QATLIB or WITH_QATDRV

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: build from qatzip submodule

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: build qatzip from source when WITH_SYSTEM_QATZIP=OFF

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: build from qatlib submodule

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: build qat from source when WITH_SYSTEM_QATLIB=OFF

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54623 from alimaredia/wip-rgw-op-metrics-section-rename

rgw: change section name of rgw_op counters

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/prometheus: fix orch check to prevent Prometheus from crashing
Fixes: https://tracker.ceph.com/issues/63992
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

cmake: rename qatzip->QATzip and support pkg-config

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: add WITH_QATLIB to build against qatlib

FindQAT.cmake adds library targets QAT::qat and QAT::usdm for qatlib,
and FindQatDrv.cmake adds alias targets to the same

src/compressor and src/crypto/qat link against these common QAT targets

src/crypto/qat/qcccrypto.h uses new HAVE_QATDRV define to determine
which headers to include

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: rename WITH_QAT to WITH_QATDRV

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crypto/qat: remove unused variable qcc_alloc_mutex

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crypto/qat: remove unused captures of 'this'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crypto/qat: add missing include use_future.hpp

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crypto/qat: qualify spawn::yield_context

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55476 from zdover23/wip-doc-2024-02-07-cephadm-tshooting

doc/cephadm - edit t11ing.

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: add confval directives

Add confval directives to the documentation of "quota cache" options.
This addresses a request made by Antony D'Atri in https://github.com/ceph/ceph/pull/55075/files#r1444006246.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw: mark maybe_unused variables

Specifically, variables that are only used in debug builds.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54596 from cbodley/wip-cmake-dencoder-modules

cmake: ceph-dencoder depends on ceph-dencoder-modules

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55376 from guits/guits-cv-64248

ceph-volume: fix zap_partitions() in devices.lvm.zap

mgr/dashboard: snapshot schedule edit form

Fixes: https://tracker.ceph.com/issues/64331
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

doc/cephadm - edit t11ing.

Edit "Running Repeated Debugging Sessions" in
doc/cephadm/troubleshooting.rst. Improve grammar. Fix a misspelling.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc: add description of metric fields for cephfs-top

Fixes: https://tracker.ceph.com/issues/63991
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #55444 from Matan-B/wip-crimson-seastar-upgrade

Crimson: bump up seastar version

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54280 from xxhdx1985126/wip-rbm-scattered-allocation

crimson/os/seastore/random_block_manager: allow allocating multiple extents for a single disk space allocation request

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

ceph.spec.in: crimson add protobuf

Following https://github.com/scylladb/seastar/commit/eea4102091bb4d689e0b5a769512966b716ee8a2

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/crush/CrushLocation: fix compilation errors

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

CMakeLists.txt: fix Seastar expected dirs

```
  # create the directory so cmake won't complain when looking at the imported
  # target: Seastar exports this directory created at build-time
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

seastar: bump up seastar submodule version

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55054 from pereman2/zns-remove

os/bluestore: remove zoned namespace support
It has never been finished and now its in the way of future improvements.

Merge pull request #55275 from qiuxinyidian/rgw-noti-dev

rgw: add topic owner user check when creating

reviewed-by: cbodley, kchheda3

Merge pull request #55447 from zdover23/wip-doc-2024-02-05-rados-operations-pools

doc/rados: update PG guidance

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: modify topic owner check when creating

add tests to cover topic policies
as well as behavior when no policies are defined

Fixes: https://tracker.ceph.com/issues/64124
Signed-off-by: Zhipeng Li <qiuxinyidian@gmail.com>

Merge pull request #54311 from gabriel-samfira/do-not-escape-slash

RGW: fix cloud-sync not being able to sync folders

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55252 from jzhu116-bloomberg/wip-21800

rgw/multisite: avoid writing multipart parts to the bucket index log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55293 from jzhu116-bloomberg/wip-63178

rgw/lc: pass in flag as an argument when calling function complete(..)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54661 from lightmelodies/rgw-gc

rgw: optimize gc chain size calculation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55452 from nbalacha/wip-typo-fix

rbd-mirror: fix a typo in NotifyAckPayload::dump()

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55406 from cbodley/wip-64255

rgw: SiteConfig::load() falls back to local zonegroup

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #55448 from guits/subclass-thread

node-proxy: refactor entrypoint

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #55280 from ronen-fr/wip-rf-extra-lg

osd: remove superfluous '>' in spg_t formatter

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55414 from yuvalif/wip_yuval_url_decode_sns

rgw/rest: fix url decode of post params passed as attributes

reviewed-by: cbodley

rgw/auth: correcting the error message to indicate
clearly that a pre-signed URL has expired.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rbd-mirror: fixed a typo in instance_watcher

Fixes a typo in NotifyAckPayload::dump().

Signed-off-by: N Balachandran <nibalach@redhat.com>

node-proxy: refactor entrypoint

This commit introduces a major refactor of the main
entrypoint.

- subclass threading.Thread:
  - Introduce a new class `BaseThread()` that is a
    `threading.Thread()` abstraction class in order
    to monitor the different threads.
  - `BaseSystem()` inherits from `BaseThread()`.
  - Handle `SIGTERM` signal in order to gracefully shutdown
    node-proxy (make threads exit gracefully, log out from RedFish API, etc.)

Additionally, this:
  - drops the class `Logger()` from util.py which
was not adding value. It is now replaced with a simple `get_logger()`
function.
  - changes the node-proxy API port from 8080 to 9456
    (8080 being widely used for frontend apps...)
  - changes the container entrypoint in order to use the
    `ceph-node-proxy` binary from the packaging

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

doc/rados: update PG guidance

Update the "Creating a Pool" section of doc/rados/operations/pools.rst
so that the documentation no longer insists that the user change the
values of "osd_pool_default_pg_num" and "osd_pool_default_pgp_num".

See also: https://github.com/ceph/ceph/pull/55419

Tracker: https://tracker.ceph.com/issues/64259

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/transaction_manager: allow allocating multiple
extents when rewriting data extents

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/object_data_handler: allow alloc multiple extents
for a single request

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: fix errorator mismatch

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

test/crimson/seastore: add unittest cases for scattered allocation

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

test/crimson/seastore: test avlallocator's scattered allocation

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: TransactionManager::alloc_extent(s) may return enospc

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: add new interfaces to alloc multiple extents for a
single request

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #55089 from cbodley/wip-boost-mirrors

make-dist: update boost mirrors

Reviewed-by: Kefu Chai <tchaikov@gmail.com>