Do not duplicate query-string in ops-log

Fixes: https://tracker.ceph.com/issues/59059
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #50339 from cbodley/wip-qa-rgw-reef-upgrade

qa/rgw: add rgw/upgrade suite

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #50409 from rhcs-dashboard/formly

mgr/dashboard: replace ajsf with formly

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #45960 from rishabh-d-dave/xfstests-dev-run-more-tests

qa/cephfs: upgrade xfstests_dev.py to run more tests

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50390 from Svelar/vstart-doc

doc/crimson: add SeaStore when using vstart.

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #48411 from liangmingyuanneo/wip-rgw-reshard-collision

rgw/reshard: fix hold reshard lock collisions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48482 from cbodley/wip-rgw-site-config

rgw: add SiteConfig to load/create zonegroup configuration

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #50347 from cbodley/wip-rgw-sal-aio

rgw/aio: remove RGWSI_RADOS from generic Aio::get()

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #50343 from pkalever/fix-errormsg

librbd: report better errors when failing to enable mirroring on an image

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50500 from jdurgin/wip-release-checklist

doc/dev/release-checklist: add another step for ceph-build

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #50494 from rhcs-dashboard/fix-58942-main

mgr/dashboard: force TLS 1.3

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: replace ajsf with formly

ajsf json schema library for angular doesn't seem to be actively
maintained. Instead, fromly is a well maintained replacement with extra
stuff like validators builtin, support for json schemas, custom
components, etc...

Textareas weren't supported on ajsf, therefore, it made sense to move to
this dep instead.

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #50147 from cbodley/wip-debian-mgr-pytest

debian/mgr: remove pytest from debian/ceph-mgr-dashboard.requires

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #50282 from mhmdawnallah/fix-access-a-destructed-temporary-variable-in-rgw_sync_checkpoint

src/rgw/rgw_sync_checkpoint.cc: fix a destructed temporary variable access

reviewed-by: adamemerson, yuvalif

Merge pull request #50482 from neha-ojha/wip-milestore

.github/milestone.yml: add reef

doc/dev/release-checklist: add another step for ceph-build

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #50493 into main

* refs/pull/50493/head:
qa: fixup quota values exceeding 64 bit signed int value
qa, cleanup: no need to specify human readable quota values in comment

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #48382 into main

* refs/pull/48382/head:
test: add test to validate attrs for snapdir after snap operations
client: update {m,c}time and change attr for snapdir from last modification time of a snap realm
mds: update last modification time and change attr for a snap realm node
mds: include last modification time and change attr of a snap realm node in snap trace reply
mds: use change_attr from closest ancestor when initializing a new snaprealm
mds: initialize last_modified to current timestamp for root inode snaprealm
mds: track last modification time and change attr for a snap realm node (srnode)

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge PR #49970 into main

* refs/pull/49970/head:
mds: force replay sessionmap version

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #50049 into main

* refs/pull/50049/head:
qa: add cephfs_mirror test case to check root dir modes
cephfs_mirror: sync snap dir root mode with remote dir

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

rbd: improve log msgs when failing to enable mirroring on image

Attempting to enable mirroring on an an image within a namespace bails
out without any useful msg.

$ rbd mirror image enable mi_pool/ns_0/big snapshot
2021-11-17T08:33:21.321+0000 7f4b36f9c2c0 -1 librbd::api::Mirror:
  image_enable: cannot enable mirroring in the current pool mirroring mode

Fixes: https://tracker.ceph.com/issues/58895
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

mgr/dashboard: force TLS 1.3

Fixes: https://tracker.ceph.com/issues/58942
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

qa: fixup quota values exceeding 64 bit signed int value

The test does the following:

        setfattr -n ceph.quota.max_bytes -v 8388608Ti .

This failes since 8388608Ti exceeds 64 bit signed integer value.
Change that to the approprivate value and expect faliure when
quota is set for this exceeded value.

Fixes: http://tracker.ceph.com/issues/55940
Introduced-by: 733065148250e892fca941cdeaad3e3a8193b0d8
Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa, cleanup: no need to specify human readable quota values in comment

... since the values themselves are self-explanatory.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/cephfs: run all generic tests from xfstests-dev

Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc/crimson: add SeaStore when using vstart.

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

rgw/rados: copy_obj_data() can use yielding throttle

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRealmReloader reloads rgw::SiteConfig

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/main: init_storage() creates a ConfigStore and loads SiteConfig

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add SiteConfig to load/create zonegroup configuration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/main: null-initialize pointer members

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50383 from cbodley/wip-rgw-admin-formatter-v2

rgw/admin: fix formatter leaks again

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

test/rgw: ceph_test_rgw_throttle no longer needs rados

run it as a unittest instead of a workunit

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/aio: update Aio callers in rados and d3n

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/aio: remove RGWSI_RADOS from generic Aio::get()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/aio: librados_op() requires librados::IoCtx

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50385 from cbodley/wip-cmake-asan

cmake: some fixes and suppressions for WITH_ASAN under vstart.sh

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50473 from guits/cv-fix-cv-tests

ceph-volume: update the OS before deploying Ceph

Merge pull request #50483 from cbodley/wip-rgw-beast-server-name

rgw/beast: add extra_response_headers for Server name

Merge pull request #50168 from cbodley/wip-23838

rgw: store torrent file for GetObjectTorrent in object attrs instead of omap

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #50346 from vedanshbhartia/rm-57938

rgw: use unique_ptr for flat_map emplace in BucketTrimWatche

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50429 from cbodley/wip-58932

rgw/admin: 'bucket stats' displays non-empty mtime

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #50211 from bbc/add_quota_containers_rgw_rest_header

rgw: add containers quota to swift rest headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49466 from linuxbox2/wip-lc-size

rgw/notifications: fetch object state to get size, in rgw_lc.cc

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #47040 from hualongfeng/qat_batch_mode

Crypto: Add QAT batch mode

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: remove rgw_thread_pool_size from frontend options

this is a config variable, not a frontend option. and the description is
not accurate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/beast: add extra_response_headers for Server name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

.github/milestone.yml: add reef

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #50470 from zdover23/wip-doc-2023-03-10-landing-page-uniquely

doc/index: remove "uniquely" from landing page

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49963 from ifed01/wip-ifed-fix-prefixed-kv-iterator

kv/RocksDBStore: don't use real wholespace iterator for prefixed access

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Cory Snyder <csnyder@iland.com>

Merge pull request #49958 from ifed01/wip-ifed-report-min-alloc-size

os/bluestore: report min_alloc_size through "ceph osd metadata"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49768 from rzarzynski/wip-crimson-fmt9-compile-guard

crush: fix compilation guard around specialization for libfmt => 9

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #49524 from rzarzynski/wip-osd-snaptrimmer-awaitasyncwork-better-ret

osd: slightly improve readability in handling of snap trim errors

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #48948 from ljflores/wip-telemetry-show-all

mgr/telemetry: make sure histograms are formatted in `all` commands

Merge pull request #50348 from ivancich/wip-restore-bucket-index

rgw: experimental support for restoring a lost bucket index

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50465 from ceph/wip-yuri-branch_fix-main

qa/tests: added reef branch

Reviewed-by: Laura Flores <lflores@redhat.com>

ceph-volume: add bluefs_buffered_io: false in functional tests

This is a workaround to make dmcrypt scenarios pass the CI.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #49807 from DHB-liuhong/lh-bucket-notification-cache

rgw: add bucket notification cache

reviewed-by: cbodley, yuvalif

ceph-volume: update the OS before deploying Ceph

ceph-volume tests are failing, OSDs never get up and running.
For some reason, updating the OS early in the testing workflow
addresses that issue in the CI.

-- to be continued ... --

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

rgw/admin: hide bencoded torrent data from 'object stat'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/cephfs: add helper methods to run tests from xfstests-dev

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #50013 into main

* refs/pull/50013/head:
mgr/volumes: avoid returning -ESHUTDOWN back to cli

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge PR #48280 into main

* refs/pull/48280/head:
qa: test new client with old cluster

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #49102 into main

* refs/pull/49102/head:
qa: add test for schedule auto deactivation
doc/cephfs: add note about path usage in snap-schedule
mgr/snap_schedule: add debug log for missing or wrong path

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

doc/index: remove "uniquely" from landing page

Change "Ceph uniquely delivers[...]" to "Ceph delivers[...]"

Signed-off-by: Zac Dover <zac.dover@proton.me>

test/rgw: Add optional_yield in rgw_crypto

Signed-off-by: Feng Hualong <hualong.feng@intel.com>

Crypto: Add QAT batch mode

Now, the current code does not allow qat to exert
its acceleration advantage, but leads to poor performance
of qat. The QAT batch mode is implemented here, so that
the QAT performance can be displayed.

When we scale up the number of concurrent requests and worry about
QAT instance will be bottleneck, we can fall back to CPU.

And there add a parameter to set the times of the number of QAT
instance to wait for free instance, which can avoid QAT to be
not busy and make sure QAT full of utilization as much as possible.

max_queue_size is up to max_requests

Add optional_yield in RGWPutObj_BlockEncrypt and
RGWGetObj_BlockDecrypt. Make it with coroutine and non-coroutine mode.

Signed-off-by: Feng Hualong <hualong.feng@intel.com>

qa: test new client with old cluster

Fixes: https://tracker.ceph.com/issues/53573
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

rgw: add bucket notification cache

Bucket notification supports reading the rule configuration from the cache first, rather than directly from the bottom layer

Signed-off-by: liuhong <liuhong_yewu@cmss.chinamobile.com>

Merge PR #45224 into main

* refs/pull/45224/head:
mds: add support for snapshots alternate_name

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #50107 into main

* refs/pull/50107/head:
client: fix CEPH_CAP_FILE_WR caps reference leakage in _write()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #50127 into main

* refs/pull/50127/head:
client: correct quota check in Client::_rename()

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

qa/tests: added reef branch

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #49870 from ifed01/wip-ifed-rm-range-threshold-on-the-fly

kv/rocksdbstore: apply rocksdb_delete_range_threshold on the fly

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #49628 from Rethan/feat-osd-op-thread-timeout

osd: allow osd_op_thread_timeout and suicide_timeout to be adjusted o…

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #50199 from athanatos/sjust/wip-scrub-event-helpers-50088

scrub: use a generic interface for scheduling timer based events

Merge pull request #50283 from ronen-fr/wip-rf-partial-unpub

osd/scrub: only call publish_stats_to_osd() where allowed and needed

Merge pull request #50065 from ajarr/fix-58691

mgr: store names of modules that register RADOS clients in the MgrMap

Merge pull request #49546 from Matan-B/wip-matanb-pull-push-naming

osd/ReplicatedBackend: Rename Push/Pull Info

rgw: add rgw-restore-bucket-index

Adds an experimental script that allows a bucket index of a
non-versioned bucket to be restored by applying `radosgw-admin object
reindex ...` to all objects in the specified bucket. The objects in
the bucket are determined by scanning the data pool for head objects
containing the bucket's marker.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add an "object reindex" subcommand to radosgw-admin

This subcommand will take a bucket and an object (or a list of objects
in a file) and add those objects to the bucket's index. It does so by
preparing the bucket index entry update and then allowing the
so-called "dir suggest" mechansim, which is triggered by listing the
bucket, to completing the bucket index entry. This mechanism is
triggered by listing the bucket.  Using this mechanism has the
advantage of doing this lazily that both allows the reindex operation
to run more quickly and distributes the workload over time.

This subcommand can help restore lost bucket index entries.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #50195 into main

* refs/pull/50195/head:
cephfs-top: Handle `METRIC_TYPE_NONE` fields for sorting

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50456 from rhcs-dashboard/custom-image-kcli-bootstrap

mgr/dashboard: custom image for kcli bootstrap script

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: custom image for kcli bootstrap script

the stable branches like quincy pulls from the quay.io/ceph/ceph:v17 to
bootstrap the ceph cluster in test environments. This will cause issues
because the branches are changing constantly but the image is not. So
using the quay.ceph.io repo to bring the cluster in test environment.

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge PR #50177 into main

* refs/pull/50177/head:
cephfs-top: navigate to home screen when no fs

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #49953 from rhcs-dashboard/dashboard-rgw-multisite

mgr/dashboard: Dashboard RGW multisite configuration

Reviewed-by: aashish1412 <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: Dashboard RGW multisite configuration

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #50439 from anthonyeleven/anthonyeleven/caps

doc/rados/operations: Improve wording, capitalization, formatting

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/rados/operations: Improve wording, capitalizatiopn, formatting

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #50149 from fmount/fs_cleanup

Remove the filestore section from ceph-volume

Merge pull request #50082 from adk3798/move-repos

mgr/cephadm: make upgrade respect use_repo_digest

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #50318 from Svelar/extra_conf

cephadm: eliminate duplication of sections

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #50413 from adk3798/prometheus-cephadm-dep

mgr/prometheus: remove dependency on cephadm module

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #50402 from ronen-fr/wip-rf-operator-log

osd: ECTransactions.cc: remove __func__ used inside a lambda

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50437 from Svelar/rados-doc

doc/rados: fix grammar mistake

doc/rados: fix grammar mistake

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Merge PR #49008 into main

* refs/pull/49008/head:
doc/quota: accept human readable quota value documented
qa/workunits/fs/quota.sh: use human readable format for ceph.quota.max_bytes
qa: added three testcases
mds: accept human readable values for quotas

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

Merge pull request #50430 from zdover23/wip-doc-2023-03-08-install-minor-grammar-fix

doc/install: update index.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge PR #46357 into main

* refs/pull/46357/head:
PendingReleaseNotes: add reference to the new mdsmap max_xattr_size field
qa/tasks/mgr/dashboard/test_health: Add 'max_xattr_size' to the mdsmap schema
mds: prevent exceeding xattrs limits on initial set
mds: prevent clients from exceeding the xattrs key/value limits
mon: add new configuration to limit filesystem xattrs size

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #49766 into main

* refs/pull/49766/head:
mds: record and dump last tid for trimming completed requests (or flushes)

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>