qa/tests: added squid option

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #55566 from zdover23/wip-doc-2024-02-14-cephadm-services-nfs

doc/cephadm: correct nfs config pool name

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55539 from zdover23/wip-doc-2024-02-12-glossary-osd-terms

doc/glossary: improve OSD definitions

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55537 from guits/node-proxy-handle-fetch-oob-error

node-proxy: improve http error handling in fetch_oob_details

Merge pull request #55297 from VallariAg/fix-gtest-xml-path

qa/workunits/rados/test.sh: fix GTEST_OUTPUT path

doc/glossary: improve OSD definitions

Remove expletive construction from three OSD-id-related terms.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55585 from zdover23/wip-doc-2024-02-15-rados-operations-placement-groups-sage-blog-post

doc/rados: add link to pg blog post

Merge pull request #55584 from batrick/backport-create-squid

script: add squid release to backport-create-issue

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55533 from ronen-fr/wip-rf-sdocs-1

doc/osd: documentation changes related to scrub

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

script: add squid to backports

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rados: add link to pg blog post

Add a link in the "Placement Groups" section that links to Sage Weil's
April 5, 2019 blog post "New in Nautilus: PG merging and autotuning".

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55551 from ivoalmeida/cephfs-mount-details

mgr/dashboard: added cephfs mount details

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #55577 from cityofships/typo

doc/start: fix wording & syntax

doc/osd: fix 'scrub a PG' commands in the documentation

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

doc/start: fix wording & syntax

Signed-off-by: Piotr Parczewski <piotr@stackhpc.com>

mgr/dashboard: added cephfs mount details

Fixes: https://tracker.ceph.com/issues/64405
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

mgr/cephadm: add fullreport in ceph orch CLI (node-proxy)

This adds the `fullreport` category to the `ceph orch hardware status` CLI.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

mgr/cephadm: conditionally deploy node-proxy

Only attempt to deploy the node-proxy agent when oob details
are provided for a given host.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: call time.sleep() after each reporter iteration

This makes sure time.sleep() is always call in the reporter loop.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

node-proxy: improve http error handling in fetch_oob_details

This makes the daemon handle the error nicer in fetch_oob_details()

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #54295 from rhcs-dashboard/edit-bucket-acl-tags

mgr/dashboard: add ACLs

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #54690 into main

* refs/pull/54690/head:
client: handle callback completion if the async I/O failed
client: make sure the callback is finished when returning ENOTCONN
client: do not accept zero byte write request
client: check for negative value of iovcnt
src/test: test zero bytes async i/o
src/test: test async I/O with negative iov structures count
src/test: test async I/O if the client is not mounted
src/test: test async I/O with read only file
src/test: test async I/O with a file created with O_PATH

Reviewed-by: Frank S. Filz <ffilzlnx@mindspring.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

doc/cephadm: correct nfs config pool name

s/nfs-ganesha/.nfs/ in doc/cephadm/services/nfs, in accordance with
information given to me by John Mulligan over Slack.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55521 from ivoalmeida/snapshot-schedule-subvolume-copies

mgr/dashboard: snapshot schedule added subvolume and translations

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: add ACLs

Fixes: https://tracker.ceph.com/issues/63547
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #54959 from VallariAg/wip-nvmeof-test-v2

qa: add qa/tasks/nvmeof.py

Merge pull request #55314 from cloudbehl/rgw-dashboard-json

mgr/dashboard: Fixing RGW graph panels

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #55546 from zdover23/wip-doc-2024-02-13-glossary-crush

doc/glossary: Add link to CRUSH paper

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53527 from rhcs-dashboard/add-rgw-perf-counter-cache

cephadm/rgw: make rgw perf counters cache and it's size configurable

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #55355 from adk3798/cephadm-nvmeof-version-bug

cephadm: fix get_version for nvmeof

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #50422 from rzarzynski/wip-bug-58915

osd: don't send stale hb msgr's addresses in MOSDBoot

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #55156 from alimaredia/wip-rgw-add-cap-user-without-keys

rgw: add new cap user-info-without-keys

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55487 from cbodley/wip-qa-rgw-hadoop-distro

qa/rgw: hadoop-s3a uses supported-random-distro$

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #55360 from shreyanshjain7174/fix_2252048

RGW: Getting an RGW service Segfault when assigning an attribute to an IAM role.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55486 from jzhu116-bloomberg/wip-64340

rgw: do not copy olh attributes in versioning suspended bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55458 from tobias-urdin/cors-signature

rgw/auth: ignoring signatures for HTTP OPTIONS calls

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: added subvolume and translations

This commit adds support for subvolume snap scheduling and translations for snap schedule repeat schedules and retention policies

Fixes: https://tracker.ceph.com/issues/64372
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #55347 from lightmelodies/mon

mon: fix health store size growing infinitely

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #55251 from rzarzynski/wip-bug-64104

tools/cot: respect generic, Ceph-wide argvs

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

doc/glossary: Add link to CRUSH paper

Add a link in the CRUSH definition to the original CRUSH paper.

Signed-off-by: Zac Dover <zac.dover@proton.me>

qa: add qa/tasks/nvmeof.py and rbd/nvmeof_basic_task and fio workunits

This is v2 of the rbd/nvmeof test: It deploys 1 gateway and 1 initiator.
Then does basic verification on nvme commands and runs fio.

This commit creates:
1. qa/tasks/nvmeof.py: adds a new 'Nvmeof' task which deploys
    the gateway and shares config with the initiator hosts.
    Sharing config was previously done by 'nvmeof_gateway_cfg' task
    in qa/tasks/cephadm.py (that task is removed in this commit).
2. qa/workunits/rbd/nvmeof_basic_tests.sh:
    Runs nvme commands (discovery, connect, connect-all, disconnect-all,
    and list-subsys) and does basic verification of the output.
3. qa/workunits/rbd/nvmeof_fio_test.sh:
    Runs fio command. Also runs iostat in parallel if IOSTAT_INTERVAL
    variable is set. This variable configures the delay between each iostat
    print.

nvmeof-cli upgrade from v0.0.6 to v0.0.7 introduced major changes
to all nvmeof commands. This commit changes v0.0.6 commands to
v0.0.7 in qa/workunits/rbd/nvmeof_initiator.sh

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #55467 from rhcs-dashboard/rgw-s3-analytics-grafana-dashboard

mgr/dashboard: Add RGW per user/bucket panels in  grafana

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55494 from ivoalmeida/snapshot-schedule-delete-activate

mgr/dashboard: added delete and activation actions

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55529 from tchaikov/cmake-cap

cmake: find_package(cap) before linking against it

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: added delete and activation actions

Fixes: https://tracker.ceph.com/issues/64355
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

doc/osd: update the documentation of several scrub-related options

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55528 from tchaikov/cmake-boost-debug

cmake: build boost debug variant when CMAKE_BUILD_TYPE is Debug

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55199 from igomon-bloomberg/wip-publish-commit-async

rgw/s3-notifications: async commit to persistent notification queue

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #55371 from pritha-srivastava/wip-rgw-correct-err-msg

rgw/auth: correcting the error message to indicate

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55250 from cbodley/wip-64084

rgw/auth/s3: validate x-amz-content-sha256 for empty payloads

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #55478 from ronen-fr/wip-rf-update-times

osd: restoring timely collection of PG stats

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

cmake: find_package(cap) before linking against it

before this change, we link against libcap without finding it. this
works fine as long as libcap-devel or libcap-dev is installed in the
system. but if it is not, the source would fail to build due to missing
`sys/capability.h`. this is not a great developer experience.

in this change, a `Findcap.cmake` is added to find the capability
library. which would fail the build at the configure phase.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cmake: build boost debug variant when CMAKE_BUILD_TYPE is Debug

boost has some different predefined  build variants. they are quite
like CMake's CMAKE_BUILD_TYPE. in which, "debug" enables some
features related features. so it would be nice if we can have it
enabled for the Debug build, if the boost is built from source.
see also
https://www.boost.org/build/doc/html/bbv2/overview/builtins/features.html

before this change, we always build the "release" variant.  in this
change, "debug" variant is built if Ceph's is built with
CMAKE_BUILD_TYPE=Debug. please note, this change does not change
the way how boost is built when packaging Ceph, as our debian/rpm
receipts do not define CMAKE_BUILD_TYPE and respect the distros'
settings, in that case, the "release" variant is still built.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55525 from zdover23/wip-doc-2024-02-11-radosgw-55524-rollback

doc/radosgw: undo 55524

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/radosgw: undo 55524

Roll back the docs changes made in
https://github.com/ceph/ceph/pull/55524, in accordance with Casey
Bodley's instructions to me here:
https://github.com/ceph/ceph/pull/55524#issuecomment-1937020543.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55524 from zdover23/wip-doc-2024-02-10-radosgw-lua-scripting-options

doc/radosgw: remove invalid LUA context options

Merge pull request #55507 from ljflores/wip-tracker-64343

doc/radosgw: remove invalid LUA context options

Remove "background", "getdata", and "putdata" from the list of LUA
context options. Passing these options throws the following error:
"ERROR: invalid script context: background. must be one of: preRequest,
postRequest".

Fixes: https://tracker.ceph.com/issues/64327
Signed-off-by: Zac Dover <zac.dover@proton.me>

qa/suites/orch: whitelist warnings that are expected in test environments

Fixes: https://tracker.ceph.com/issues/64343
Signed-off-by: Laura Flores <lflores@ibm.com>

mgr/dashboard: Add RGW per user/bucket panels in  grafana

Fixes: https://tracker.ceph.com/issues/64359
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #53521 from xxhdx1985126/wip-62857

crimson/osd/replicated_backend: don't put pg's metadata updates in repops

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54650 from pereman2/buffer_map-noptr

os/bluestore: use buffer values instead of pointers in buffer_map

osd: no 'legacy' form for two configuration options

also - fixing review comments not addressed in the original PR.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd: restoring timely collection of PG stats

500 seconds is way too long, e.g. when compared to the 5s
default configuration option for the manager collection of the OSD data.

Fixes tracker issue 53342 note 5 (a specific scenario leading
to 'not all pgs scrubbed')

Fixes: https://tracker.ceph.com/issues/53342 - partial fix
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55462 from afreen23/fix-64270

mgr/dashboard: fix error while accessing roles tab when policy attached

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55365 from kotreshhr/ceph-mds-replica-debug-logs

mds: Add debug messages in replica handling

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #55411 from rishabh-d-dave/qa-cephfs-caps_helper

qa/cephfs: declare f-string to get values to substitute variable names

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #55493 into main

* refs/pull/55493/head:
client: remove unused/undefined fake_write_size()

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #55499 from cbodley/wip-squid-deprecate-blkin

doc: deprecate blkin tracing

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #55332 from athanatos/sjust/wip-crush-multi-choose

crush: add multistep retry rules

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #54624 from cbodley/wip-cmake-qat

cmake: add ExternalProjects to enable QAT by default

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55490 from eroldan/patch-2

doc/start: fix typo in hw-recs

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc:start.rst fix typo in hw-recs

s/power less/power loss/

Signed-off-by: Eduardo Roldan <trazor@gmail.com>

doc: deprecate blkin tracing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55267 from athos-ribeiro/cmake-empty-submodules

cmake: quote variable to allow empty values

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55398 from ifed01/wip-ifed-fix-ceph-vol-migrate-wal

ceph-volume: Fix migration from WAL to data with no DB

client: remove unused/undefined fake_write_size()

there are no usages of this function in entire codebase

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

mgr/dashboard: fix error while accessing roles tab when policy attached

Fixes https://tracker.ceph.com/issues/64270

Issue:
======
Accessing Object->Users-Roles tab causing 500 internal servor error.
This is due to the "PermissionPolicies" which are attached to role and
backend was not handling this field for rgw roles.

Fix:
====
Added "PermissionPolicies" as the valid field in backend and updated
frontend to render the attached policy in formatted JSON

Signed-off-by: Afreen <afreen23.git@gmail.com>

Merge pull request #55416 from ivoalmeida/snapshot-schedule-edit

mgr/dashboard: added edit functionality

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55465 from neesingh-rh/wip-63991

doc: add description of metric fields for cephfs-top

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

qa/rgw: hadoop-s3a uses supported-random-distro$

without this, teuthology-suite tries to schedule against centos8 which
is not supported for squid

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55479 from ronen-fr/wip-rf-fifolegacy

rgw: mark maybe_unused variables

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: do not copy olh attributes in versioning suspended bucket

Signed-off-by: Juan Zhu <jzhu4@dev-10-34-20-139.pw1.bcc.bloomberg.com>

Merge pull request #55475 from zdover23/wip-doc-2024-02-07-radosgw-admin-confval

doc/radosgw: add confval directives

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55149 from rkachach/fix_issue_63992

mgr/prometheus: fix orch check to prevent Prometheus from crashing

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

cmake: WITH_QATLIB/ZIP disabled on aarch64

Signed-off-by: Casey Bodley <cbodley@redhat.com>

compressor: move QatAccel out of common

move the QatAccel instance out of the Compressor base class and into
the zlib and lz4 compressors that can use it

this avoids having to link QAT into the ceph-common library, and only
the plugins where it's necessary

had to add LZ4Compressor.cc to store the new static variable

Signed-off-by: Casey Bodley <cbodley@redhat.com>

build/win32: disable qat options

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rpm: add system_qat option, default on for centos9

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: enable WITH_QATLIB and WITH_QATZIP by default

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: WITH_QATZIP requires WITH_QATLIB or WITH_QATDRV

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: build from qatzip submodule

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: build qatzip from source when WITH_SYSTEM_QATZIP=OFF

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: build from qatlib submodule

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: build qat from source when WITH_SYSTEM_QATLIB=OFF

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54623 from alimaredia/wip-rgw-op-metrics-section-rename

rgw: change section name of rgw_op counters

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/prometheus: fix orch check to prevent Prometheus from crashing
Fixes: https://tracker.ceph.com/issues/63992
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

cmake: rename qatzip->QATzip and support pkg-config

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: add WITH_QATLIB to build against qatlib

FindQAT.cmake adds library targets QAT::qat and QAT::usdm for qatlib,
and FindQatDrv.cmake adds alias targets to the same

src/compressor and src/crypto/qat link against these common QAT targets

src/crypto/qat/qcccrypto.h uses new HAVE_QATDRV define to determine
which headers to include

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: rename WITH_QAT to WITH_QATDRV

Signed-off-by: Casey Bodley <cbodley@redhat.com>