Merge pull request #62744 from rhcs-dashboard/accounts-e2e

mgr/dashboard: rgw accounts UI e2e tests

Merge pull request #61992 from dparmar18/ib61357

qa: add parallelized cleanup step in _rebuild_metadata()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #62259 into main

* refs/pull/62259/head:
qa/tasks/cephfs/mount: introduce _default_gateway method
qa/tasks/cephfs/mount: use 'ip r' instead 'route'

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #62553 into main

* refs/pull/62553/head:
mds: fix issues with use-after-free in C_Flush_Journal
mds: fix issues with use-after-free in C_Flush_Journal

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #62764 from rhcs-dashboard/bucket-ratelimit-fix

mgr/dashboard: bucket ratelimit API failing while changing owner with ratelimit config

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #59533 from ifed01/wip-ifed-no-async-compact-in-splillover-tests

test/store_test: enforce sync compaction for spillover test cases.

Reviewed-by: Md Mahamudur Rahaman Sajib mahamudur.sajib@croit.io

Merge pull request #62807 from ceph/wip-yuriw-release.pocess-main

doc: changed refs from quincy to squid

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61786 from Hezko/ns-create-size-fix

mgr/dashboard: Ns create size fix

Merge pull request #62799 from anthonyeleven/fix-formatting

doc/cephadm/services: Fix formatting in osd.rst

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61610 from rzarzynski/wip-test-bl-lastp-on

test/bufferlist: turn the old last_p fix verification on

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #56922 from ifed01/wip-ifed-store-test-no-sleeps

test/store_test: get rid off sleep() call wherever possible

Reviewed-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #62804 from Matan-B/wip-matanb-rgw-spec.in

ceph.spec.in: package man/rgw-gap-list

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #62783 into main

* refs/pull/62783/head:
libcephfs_proxy: fully initialize async I/O structure

Reviewed-by: Sachin Prabhu <sp@spui.uk>
Reviewed-by: Shwetha Acharya <sacharya@redhat.com>

doc: changed refs from quincy to squid

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #62692 from ronen-fr/wip-rf-watch-ratiob

osd/scrub: additional configuration parameters to trigger scrub reschedule

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

libcephfs_proxy: fully initialize async I/O structure

The ceph_ll_io_info structure has recently been extended to support
zerocopy operations. The proxy was initializing just the known members,
so, after the zerocopy support, it was passing garbage in some fields,
causing failures.

This patch completely clears the whole structure to be sure that
everything is initialized to its default value.

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

Merge pull request #62761 from rhcs-dashboard/tiering-fix

mgr/dashboard: RGW - Reatin Head Object Fix

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: fix bucket rate limit API on owner change

Fixes: https://tracker.ceph.com/issues/70874
PR covers & fixes below scenarios:
Whenever we change the owner of bucket from non-tenanted to tenanted and
vice-versa with the rate-limit changes, there was issue in sending bucket name
Scenario 1: Changing the bucket owner from tenanted to non-tenanted
Scenario 2: Changing the bucket owner from non-tenanted to tenanted
Scenario 3: Keeping the owner(tenanted) same and changing only rate limit

Signed-off-by: Naman Munet <naman.munet@ibm.com>

Merge pull request #61273 from rhcs-dashboard/topicmanagment-ui

mgr/dashboard: Add RGW topic listing in dashboard

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Naman Munet <nmunet@redhat.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

ceph.spec.in: add man/rgw-gap-list

Introduced in https://github.com/ceph/ceph/pull/62749
Centos builds fail with:
```
error: Installed (but unpackaged) file(s) found:
   /usr/share/man/man8/rgw-gap-list.8.gz

RPM build errors:
    Installed (but unpackaged) file(s) found:
   /usr/share/man/man8/rgw-gap-list.8.gz
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62776 from rhcs-dashboard/tiering-secretKey-fix

mgr/dashboard: RGW - Tiering form - secret key fix

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

doc/cephadm/services: Fix formatting in osd.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #62784 from anthonyeleven/osd-restart-warn

doc/cephadm: Add admonition re restarting an OSD service

Reviewed-by: Zac Dover <zac.dover@proton.me>

osd/scrub: additional configuration params to trigger scrub reschedule

Adding the following parameters to the (small) set of configuration
options that, if changed, trigger re-computation of the next scrub
schedule:
- osd_scrub_interval_randomize_ratio,
- osd_deep_scrub_interval_cv, and
- osd_deep_scrub_interval (which was missing in the list of
  parameters watched by the OSD).

Fixes: https://tracker.ceph.com/issues/70806
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #62732 from Hezko/nvmeof-cli-gw-addr

mgr/dashboard: allow selecting target gw in nvmeof cli/api endpoints

Merge pull request #62786 from ceph/wip-yuriw-release.pocess-main

doc: minor additions based on the latest mods

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #62490 from bill-scales/ec_select_primary

osd: Restrict choice of primary shard for ec_optimizations pools

Merge pull request #62757 from rhcs-dashboard/fix-70866-main

mgr/dashboard: add description for default Pool Application select options

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

doc: minor additions based on the latest mods and updated example to squid

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #62704 from sseshasa/wip-fix-recovery-counters-calc

osd/scheduler/OpSchedulerItem: Fix calculation of recovery latency counters

Merge pull request #62749 from ivancich/wip-document-gap-list

rgw/doc: add man documentation for the rgw-gap-list tool

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw: add man documentation for the rgw-gap-list tool

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #62780 from aainscow/sv_notify_fix

rgw: Fix compilation issue with clang on centos

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/cephadm: Add admonition re restarting an OSD service

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #62771 from idryomov/wip-drop-remote-namespace-cls-err

cls/rbd: drop overzealous CLS_ERR message in mirror_remote_namespace_get()

Reviewed-by: N Balachandran <nithya.balachandran@ibm.com>

Merge pull request #62601 from rhcs-dashboard/smb-tabs-route

mgr/dashboard: make smb tabs routable

rgw: Fix compilation issue with clang on centos

We are getting a compilation issue with accessing an incomplete type.

This fix moves some code around to avoid this issue.

I don't fully understand why we don't see this issue everywhere!

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

mgr/dashboard: RGW - Reatin Head Object Fix

Fixes: https://tracker.ceph.com/issues/70871
Signed-off-by: Dnyaneshwari Talwekar <dtalwekar@redhat.com>

mgr/dashboard: RGW - Tiering form - secret key fix

Fixes: https://tracker.ceph.com/issues/70888
Signed-off-by: Dnyaneshwari Talwekar <dtalwekar@redhat.com>

Merge pull request #62697 from Hezko/nvmeof-response-fix

mgr/dashboard: fix subsystems list api response

test: Add unittests for pgtemp_primaryfirst/pgtemp_undo_primaryfirst

Add unittests for pgtemp_primaryfirst and pgtemp_undo_primaryfirst
to prove the later is a reverse transform and that neither has any
effect until an optimized EC pool configures non-primary shards.

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

osd: Restrict choice of primary shard for ec_optimizations pools

Pools with ec_optimizations enabled have restrictions on which
shards are permitted to become the primary because not all shards
are updated for every I/O.

To preserve backwards compatibility with downlevel clients
pg_temp is used as the method to override the selection of
primary by OSDMap. Directly changing the logic in OSDMap
would have meant that all clients need to be upgraded to
tentacle before using optimized EC pools, so was discounted.
Using primary_temp to set the primary for an EC pool is
not reliable because under error conditions an OSD can store
multiple shards for the same PG and primary_temp cannot
define which of these shards will be choosen.

For optimized EC pools pg_temp is shuffled so that the
non-primary shards are listed last. This means that the
existing logic in OSDMap that picks the first available
shard as the primary will avoid selecting a non-primary
shard. OSDMonitor applies the shuffle when pg_temp is set,
this is then reverted in PeeringState when initializing the
acting set after OSDMap has selected the primary.

PeeringState::choose_acting is modified to set pg_temp if
OSDMap has selected a non-primary shard, this will cause
a new OSDMAP to be published which will persuade
OSDMap to select a primary shard instead.

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

mgr/dashboard: rgw accounts UI e2e tests

Fixes: https://tracker.ceph.com/issues/69674
Signed-off-by: Naman Munet <naman.munet@ibm.com>

Merge pull request #62527 from rhcs-dashboard/smb-file-uploader

mgr/dashboard: add file uploader for smb usersandgroups

Reviewed-by: nizamial09 <nia@redhat.com>

Merge pull request #62673 from adamemerson/wip-test-osd-signed-comparison

test/osd: Fix signed comparison warning

Merge pull request #62163 from zmc/bwc-debs

Enabling containerized debian pipeline builds

mgr/dashboard: add traddr param to nvmeof cli/api endpoints

Signed-off-by: Tomer Haskalovitch <il033030@Tomers-MBP.lan>

Merge pull request #62626 from aainscow/have_significant_feature

osd: Introduce macro to police use of significant features.

cls/rbd: drop overzealous CLS_ERR message in mirror_remote_namespace_get()

Currently it unnecessarily floods the log of the OSD which hosts
rbd_mirroring object with "No such file or directory" errors.  Just
drop it as read_key() already logs all errors except ENOENT.

Credit to N Balachandran <nibalach@redhat.com> for spotting this.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #62589 from ceph/wip-yuriw-18.2.5-release.notes-main

doc: reef 18.2.5 release notes

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #62734 from ceph/wip-yuriw-19.2.2-release.notes-main

doc: squid 19.2.2 release notes

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

buildcontainer-setup.sh: Fix curl package conflict

It looks like at some point the centos9 image started shipping with
curl-minimal, which conflicts with the regular curl package. Asking dnf to find
the binary avoids this, since both packages provide it. Since we were already
doing this with rpmbuild, we can go ahead and loop wget into that in case
something similar happens there.

Signed-off-by: Zack Cerza <zack@cerza.org>

build-with-container.py: Pass distro to make-debs

Signed-off-by: Zack Cerza <zack@cerza.org>

make-debs.sh: Optionally take debian version

Our existing CI builds have names like:
  ceph-base_20.0.0-194-g6efaea33-1jammy_amd64.deb
Before this change, they are like:
  ceph-base_20.0.0-158-gb0de3a42-1_amd64.deb
This way we can pass e.g. "jammy" to end up with names compatible with our CI
builds.

Signed-off-by: Zack Cerza <zack@cerza.org>

debian/rules: Optionally enable sccache

Signed-off-by: Zack Cerza <zack@cerza.org>

debian/rules: Disable dh_dwz if DWZ=false

Signed-off-by: Zack Cerza <zack@cerza.org>

make-debs.sh: Don't use ccache if using sccache

Signed-off-by: Zack Cerza <zack@cerza.org>

build-with-container.py: Pass version to make-debs

Signed-off-by: Zack Cerza <zack@cerza.org>

make-debs.sh: Optionally avoid make-dist

Signed-off-by: Zack Cerza <zack@cerza.org>

doc: squid 19.2.2 release notes

https://tracker.ceph.com/issues/70822
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #62759 from bluikko/doc-prompts-ceph-volume

doc/ceph-volume: Promptify commands and fix formatting

Merge pull request #62663 from adamemerson/wip-tbb-O0

build: Pull TBB into `$EXTRALIBS` if it is available

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #62669 from adamemerson/wip-allocsim-replayer-assert

test/objectstore/allocsim: Fix link error

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62753 from gbregman/main

mgr/cephadm/nvmeof: Add OMAP read lock parameters to NVMEoF configuration

Merge pull request #62762 from bluikko/patch-23

doc/cephfs: Fix double prompts in commands in multimds.rst

doc/cephfs: Fix double prompts in commands in multimds.rst

Some example commands had double prompts: one `#`
from the `.. prompt` and after that another hardcoded
`$` character.

Remove the extra user prompt and leave only the root
prompt from the `.. prompt` block.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/ceph-volume: Promptify commands and fix formatting

Use the more modern prompt block for CLI
commands, fix missing newline and messed up
line breaks.
Also change existing prompts to all indent with
same amount of spaces.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

mgr/dashboard: add description for default Pool Application select options

Add a description for the default pool application values in the pool creation form i.e cephfs <-> FileSystem, rbd <-> Block, rgw <-> Object.

Fixes: https://tracker.ceph.com/issues/70866
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #62633 from cyx1231st/wip-seastore-le-map-cleanup

crimson/os/seastore: cleanups related to lba and cache

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Zhang Song <zhangsong02@qianxin.com>

Merge pull request #62662 from adamemerson/wip-amqp-deprecation

rgw: Fix deprecated amqp includes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62580 from Matan-B/wip-matanb-findboost-warning

cmake/modules/FindBoost.cmake: fix Boost_NO_WARN_NEW_VERSIONS

Reviewed-by: Adam Emerson <aemerson@redhat.com>

doc/releases: fix formatting for backticks

Signed-off-by: Laura Flores <lflores@ibm.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #62451 from bill-scales/ec_data_structs

osd: EC optimizations: new types and additions to data structures

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #61652 from ifed01/wip-ifed-print-extra-level

os/bluestore: print the first RocksDB level which doesn't fit into fast

Reviewed-by: Md Mahamudur Rahaman Sajib <mahamudur.sajib@croit.io>

mgr/cephadm/nvmeof: Add OMAP read lock parameters to NVMEoF configuration

Fixes https://tracker.ceph.com/issues/70861

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #57206 from clwluvw/restrict-public-buckets

rgw: implement RestrictPublicBuckets from PublicAccessBlock

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62538 from cbodley/wip-rgw-control-watch-par

rgw: send concurrent watch/unwatch operations

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #62159 from cbodley/wip-qa-23264

qa/rgw: use storage classes for rgw/crypt

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #62706 from cbodley/wip-70729

test/rgw/multisite: fix parsing of 'topic list' output

Reviewed-by: Ali Masarwa <amasarwa@redhat.com>

Merge pull request #62686 from Cybertinus/main

documentation:remove deprecated usage apt-key

documentation: Update get-packages.rst

The usage of apt-key is deprecated, thus updated the documentation to reference the new way of installing a repo key.

Signed-off-by: Cybertinus <github@cybertinus.nl>

Merge pull request #62727 from idryomov/wip-70831

test/librbd/test_notify.py: force line-buffered output

Reviewed-by: Mykola Golub <mykola.golub@clyso.com>
Reviewed-by: Vinay Bhaskar Varada <vvarada@redhat.com>

Merge pull request #57448 from aclamk/wip-aclamk-bs-recompression-segmented-data

os/bluestore: Recompression, part 3. Segmented onode.

Merge pull request #62224 from aclamk/wip-aclamk-pere-wal-fsync

os/bluestore: Fast WAL for RocksDB

Merge pull request #62588 from aclamk/wip-aclamk-bluefs-remove-truncate

os/bluestore: Fix race in BlueFS truncate / remove

mgr/dashboard: Add RGW topic listing in dashboard

Fixes: https://tracker.ceph.com/issues/69143
Signed-off-by: pujaoshahu <pshahu@redhat.com>

Merge pull request #61909 from bazaah/chore/deprecated-iterator-impl

include,dpdk: fix two usages of std::iterator (deprecated)

qa/rgw: use storage classes for rgw/crypt

enable compression on one storage class so we can test copy to/from
compressed objects

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: storage classes specify extra args in yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60670 from cbodley/wip-cls-rgw-shard-io

rgw/rados: add async algorithms for concurrent index operations

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #62720 from tchaikov/wip-kefu-mailmap

mailmap: update affiliation for Kefu Chai

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62716 from afreen23/wip-fix-empty-ver

mgr/dashboard: Fix empty ceph version in GET api/hosts

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

cls/rgw: remove unused CLSRGWConcurrentIO and subclasses

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/bluestore/bluefs: Fix race condition between truncate() and unlink()

It was possible for unlink() to interrupt ongoing truncate().
As the result, unlink() finishes properly, but truncate() is not aware
of it and does:
1) updates file that is already removed
2) releases same allocations again

Now fixed by checking if file is deleted under FILE lock.

https://tracker.ceph.com/issues/70747

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

rgw/rados: index operations use async_reads/writes()

replace the classes derived from CLSRGWConcurrentIO with classes
derived from Reader/Writer/RevertibleWriter and use the async algorithms

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add concurrent io algorithms for sharded data

cls/rgw provides the base class CLSRGWConcurrentIO as a swiss army knife
for bucket index operations that visit every shard object. while it uses
asynchronous librados requests to perform the io, it blocks on a
condition variable when waiting for the AioCompletions.

for use in coroutines, we need a version of this that suspends instead
of blocking. and to support both stackful and stackless coroutines, we
want a fully-generic async inferface templated on CompletionToken.

while the CLSRGWConcurrentIO algorithm works for all current uses
(reads and writes, with/without retries, with/without cleanup), i chose
to break this into 3 algorithms with well-defined semantics:

1. reads: to produce a successful result, all shard operations must
   succeed. so any shard's failure causes the rest to be cancelled or
   skipped. supports retries for ListBucket (RGWBIAdvanceAndRetryError).
2. writes: even if some shards fail, we still want to visit every shard
   before returning the error. supports retries for log trimming
   operations (repeat until ENODATA).
3. revertible writes: similar to reads, requires all shard operations to
   succeed. on any failure, the side effects of any successful writes
   must be reverted before returning. only used by IndexInit (any created
   shards are removed on failure).

each algorithm provides a pure virtual base class that must be
implemented for each type of operation, similar to how existing
operations inherit from CLSRGWConcurrentIO.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/releases: fix formatting and wording

Signed-off-by: Laura Flores <lflores@ibm.com>

rgw: pass optional_yield to all index ops

prepare for coroutine support by making the yield context available to
all index ops

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62571 from clwluvw/trim-log-cleanup

rgw: BucketTrimShardCollectCR expect ENODATA as empty

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61966 from mkogan1/wip-qat-logs-crypto

rgw: qat: improve QAT logs when HW is miss-configured for QAT encryption

Reviewed-by: Casey Bodley <cbodley@redhat.com>