Merge PR #49766 into main

* refs/pull/49766/head:
mds: record and dump last tid for trimming completed requests (or flushes)

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #49844 into main

* refs/pull/49844/head:
qa: enforce string client-id check for testing oldest client tid warning
qa: optionally check pattern string match in `health detail`

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge PR #49912 into main

* refs/pull/49912/head:
qa: avoid explicit set to client mountpoint as "/"

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #50146 from adk3798/container-selinux-conflict

qa/distros: pass --allowerasing --nobest when installing container-tools

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #50133 from adk3798/fqdn-stray

mgr/cephadm: be aware of host's shortname and FQDN

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #50244 from phlogistonjohn/jjm-cephadm-test-call

cephadm: test call function & fix timeout argument

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #50101 from adk3798/all-devices-raw

mgr/orchestrator: allow deploying raw mode OSDs with --all-available-devices

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #45469 from zaitcev/srbac-bz1901857-3

RGW: Add a reader feature

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm/tests: enable timeout test cases for call function

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: fix timeout argument to call function

The timeout argument to call function, for executing sub-processes, did
not function - this patch makes timeout work as (probably) intended.
Use the `process.communicate()` method rather than `tee` functions to
handle IO collection. Since no logging is done until after the exit code
is known the tee calls are not necessary. Add calls to kill the child
process when the time out occurs. This helps prevent event loop "leaks"
that generate python warnings.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #50270 from mnasiadka/cephadm_ulimit_docker

cephadm: set --ulimit nofiles with Docker

Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: be aware of host's shortname and FQDN

The idea is to gether the shortname and FQDN as part
of gather-facts, and then if we ever try to check if a certain
host is in our internal inventory by hostname, we can check
these other known names. This should avoid issues where
we think a hostname specified by FQDN is not in our
inventory because we know the host by the shortname
or vice versa.

Fixes: https://tracker.ceph.com/issues/58738
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #50100 from adk3798/set-unmanaged

mgr/cephadm: add commands to set services to managed/unmanaged

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #50167 from trociny/wip-58792

mgr/cephadm: don't add mgr into iscsi trusted_ip_list if it's already there

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #50275 from rkachach/fix_issue_58856

mgr/rgw: adding mgr rgw module to ceph image

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #48911 from yuvalif/test-kafka-sasl-scram

rgw: test kafka sasl scram

reviewed-by: anthonyeleven, cbodley

Merge pull request #50311 from trociny/wip-58051-1

mgr/cephadm: try to avoid pull when getting container image info

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #50403 from yuvalif/wip-yuval-dont-start-notif-thread

rgw: don't start the bucket notification thread in radosgw-admin

reviewewd-by: cbodley

Merge pull request #49116 from Matan-B/wip-matanb-c-balanced-reads

crimson/osd: Support balance reads

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50410 from ljflores/wip-tracker-58925

qa: suppress memory leak in rocksdb

Merge pull request #49779 from Matan-B/wip-matanb-c-lru-comments

common/intrusive_lru: Improve readability

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

rgw: unify data member initialization in RGWRados

https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#c48-prefer-in-class-initializers-to-member-initializers-in-constructors-for-constant-initializers

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw: don't start the bucket notification thread in radosgw-admin

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

crimson/osd/object_context_loader: Hide private methods

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: Introduce with_head_and_clone_obc()

In continuation to 7ca2690be956a36f61c7729946b94ccd970dd9c7:
Now that the head ref is no longer a member of obc, we need a new
substitute way to get the head when needed.

When loading a clone object, the head object is loaded
first (See with_clone_obc). Therefore we can make use of this design
to move the loaded head forward to the relevant func (See with_head_and_clone_obc).
Usually, we wouldn't need to make use of both the head and the clone obc in the
same function. However, SnapTrimObjSubEvent::remove_or_update is an abnormal usage.

Note: We want to avoid holding any unneeded references to obcs
to allow the obc_registery to evict no longer valid obc.
Therefore, with_obc() which references only a single obc is the
preferred entry point for loading obcs.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/object_context_loader: with_clone_obc to move head

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/object_conetxt*: Rename get_ro_ss to get_head_ss

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: Remove head memeber from ObjectContext

Before this patch, ObjectContext had a head member which was used
to get the head obc of a clone object.
This member caused the head object to being referenced while
attempting to 'clear_replica_obc' (Since we only evict un-referenced
obc from the obc_registery).
This mechanism, of obtaining the head, is no longer needed since
'with_clone_obc' loads the head object context first (using
oid.get_head).

In this commit, head is removed from ObjectContext class and users
are removed as well.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: Add can_discard_op case

discard op in the case where same_primary_since is later than
the MOSDOp's map epoch

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: Implement context_registry_on_change()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

common/instrusive_lru: Implement for_each

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/*/crimson: Seperate Crimson's rbd api testing

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations: Add RepRequest::with_pg() to the pipeline

Wait on a PG to advance to the request's map epoch.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: add replica_clear_repop_obc() and log_operation()

Clear invalid obc from cache.

Fixes: https://tracker.ceph.com/issues/58089
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

common/intrusive_lru: Add clear_range()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/osd_types_fmt: Add fmt::formatter<pg_log_entry_t>

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados/thrash/workloads: Enable small-objects-balanced/localized

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operation/client_request: do_process() reorder

* Move error checking to the beginning of do_process()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/client_request: fix is_misdirected()

In the case of balanced read the op is not misdirected.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/client_request: Add logs around do_process()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/client_request: Skip do_recover_missing() on replica

* assert internal_client_request is on primary since
  do_recover_missing is also called by internal requests.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #50380 from rhcs-dashboard/script-reef-fix

script: update backport scripts for reef

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: neha-ojha <NOT@FOUND>

Merge pull request #50397 from soumyakoduri/wip-skoduri-multisite

rgw/tests: Reset endpoint before creating additional zones

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50384 from anthonyeleven/anthonyeleven/tweak-50374

doc/radosgw: Redd up s3select.rst as followup to #49411

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/radosgw: Redd up s3select.rst as followup to #49411

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

qa: suppress memory leak in rocksdb

Fixes: https://tracker.ceph.com/issues/58925
Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #50398 from zdover23/wip-doc-2023-03-07-dev-guide-basic-workflow-full-stop

doc/dev: add full stop to sentence in basic-wo

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: add full stop to sentence in basic-wo

Add a full stop to a sentence in basic-workflow.rst. I looked at this
document and noticed that it wasn't there, and it was just bugging me.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #48697 from phlogistonjohn/jjm-cleaner-build-sh

build scripts: various cleanups and minor changes preparing for containerized build and test

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

cephadm/tests: add initial test coverage for call function

The call function provides the ability to run subprocesses, log output,
and provides an optional timeout parameter. This timeout parameter does
not appear to function correctly today, so we make use of
pytest.param/pytest.mark.xfail to mark these cases as already known to
fail.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: disable coverage for some compatibility blocks

This change disables reporting missing coverage for blocks that
contain copy and pasted code from other python versions and exist
to make those functions available to older python versions.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

rgw/tests: Reset endpoint before creating additional zones

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Fixes: https://tracker.ceph.com/issues/58822

Merge pull request #50382 from zdover23/wip-doc-2023-03-05-radosgw-multisite-caption-3

doc/rgw: fix caption

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge PR #50310 into main

* refs/pull/50310/head:
client: remove unused method

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #48894 into main

* refs/pull/48894/head:
qa/tasks/cephfs: add "extra data pool" cephfs-data-scan tests
qa/tasks/cephfs: use cephfs tags when recreating osd caps
tools/cephfs-data-scan: make data pool command args optional
tools/cephfs-data-scan: support for multi-datapool
cls/cephfs: extend accumulate_inode_metadata client method to store pool id

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50386 from zdover23/wip-doc-2023-03-06-radosgw-s3-table

doc/rgw - fix grammar in table in s3.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw - fix grammar in table in s3.rst

Put an "only" in the correct place. Also, this is a very small change
that will allow me to see if Jenkins is working (it probably isn't).

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50003 from mkogan1/wip-fix-t58552

qa: d3n: fix test failure when "inline_data": false

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/osd/pg: Expand can_discard_op on BALANCE/LOCALIZE reads

See: 69def1433d49bd8543e26e9f1ebf00dfbc5059bd

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

doc/rgw: fix caption

Rewrite the third paragraph of the caption to the figure depicting how a
zonegroup works.

Follows https://github.com/ceph/ceph/pull/50316.

Signed-off-by: Zac Dover <zac.dover@proton.me>

script: update backport scripts for reef

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #50373 from zmc/nop-suite

Merge pull request #50375 from zdover23/wip-doc-2023-03-04-radosgw-keycloak-rewrite

doc/rgw: refine keycloak.rst

Merge pull request #50329 from cbodley/wip-58891

rgw/reshard: treat old RGWBucketInfo::num_shards=0 as 1 shard

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

doc/rgw: refine keycloak.rst

Full line-edit of doc/radosgw/keycloak.rst. Corrected syntax, grammar,
RST, and broken links.

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #49411 from ceph/json_rgw_integration

rgw/s3select: json integration: identifying new data-source and execute a JSON flow.

Reviewed-by: Albin Antony <aantony@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/teuthology/nop: Add 2nd job, with a node

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #49969 from guits/fix-drive-group-limit

drive_group: fix limit filter in drive_selection.selector

Merge pull request #50364 from anthonyeleven/anthonyeleven/fix-caps

doc/cephadm: Redd up compatibility.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephadm: Redd up compatibility.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #50354 from zdover23/wip-doc-2023-03-03-multisite-master-zonegroup-image-refinement

doc/rgw: clean zone-sync.svg

json integration: identifying new data-source and execute json flow.
s3select submodule

integrate the limit-operator into RGW. current changes is for CSV format

integrate the limit-operator with CSV-flow, and JSON-flow. an update of s3select submodule

a fix for the input-serialization-type selection

debug functionality.

fix for error handling

adding the scan-range feature. i.e. enables the user to define the range of processing(text only)

remove the temp variable(the s3select-layer handles the sql-result setting upon JSON flow)

adding documentation. for JSON, SQL limit-operator, scan-range option.
a fix for JSON test framework in s3select module

investigate crash. replace len with it.length()

editorial changes; non-initialized variables

adding validation for offset&length. (it seems that the offset&length causes Invalid reads, in some cases)

changes related to trino integration.(1) syntax issues related to trino-engine statement conbersions(s3select parser handles that) (2).
(2). trino rejects(HIVE_CURSUR_ERROR) some of the s3select responses do to its size, the change is about control the size of the respond.

update JSON documentation

trino changes: response in paging (limitation on size), trino syntax issues.

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

Merge pull request #46150 from dang/wip-dang-bucket-owner

RGW - Replace set_owner workaround with new fix

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

doc/rgw: clean zone-sync.svg

Refine zone-sync.svg (fix kerning and use fonts that render properly).

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50351 from zdover23/wip-doc-2023-03-03-radosgw-multisite-admonition-formatting

doc/radosgw: format admonitions

doc/radosgw: format admonitions

Break up the text of two similar admonitions into three paragraphs (in
each of the two instances). This makes the content of the admonition
much easier to read at a glance.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50345 from cbodley/wip-redmine-reef

script: add 'reef' release to backport-create-issue

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #49756 into main

* refs/pull/49756/head:
qa: add test for 'dump dir'
mds: add a command to dump directory information

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50172 from cbodley/wip-58794

rgw: catch manifest decode errors in GetObj

Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

script: add 'reef' release to backport-create-issue

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50171 from cbodley/wip-58793

rgw: don't dereference nullopt in DeleteMultiObj

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #50169 from cbodley/wip-rgw-unsal-rados

rgw/rados: remove sal handles from RGWRados::Object ops

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge PR #50217 into main

* refs/pull/50217/head:
qa: enable the xfstests generic/317 test case
qa: make the xfstests tmp directories to be readable and excutable

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #50317 from xxhdx1985126/wip-rbm-prepare-write

crimson/os/seastore/epm: also do prepare_write for rbm

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

rgw/reshard: avoid infinite loop if bi_list() returns -ENOENT

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50335 from adk3798/cephadm-compatability-update

doc/cephadm: update cephadm compatability and stability page

Reviewed-by: Zac Dover <zac.dover@proton.me>

rgw/rados: RGWSI_BucketIndex_RADOS requires layouts, not num_shards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: use rgw::num_shards() helper

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/reshard: refuse to reshard to new_num_shards=0

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/reshard: use the rgw::num_shards() helper

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/cephadm: update cephadm compatability and stability page

This page is very out of date. This commit probably doesn't
cover everything there is to say about stability and compatability
in cephadm, but it at least gets it noticeably closer to reality

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #50316 from zdover23/wip-doc-2023-03-01-radosgw-multisite-diagram-object-replication-text-revision

doc/rgw: improve diagram caption

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw: improve diagram caption

Improve the syntax and semantics of the caption of the diagram
"Replication of Object Data Between Zones" in doc/radosgw/multisite.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/kafka/tests: add SASL mechanism tests

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

RGW - Replace set_owner workaround with new fix

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

rgw: rgw::num_shards() returns 1 instead of 0

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50256 from yuvalif/wip-yuval-zipper-notifications

rgw/notifications: add bucket notification configuration to zipper

Reviewed-By: dang, cbodley

Merge pull request #48181 from huynp1999/wip-rgw-kafka-add-sasl-mechanism

src/rgw: SASL mechanism implem

Reviewed-By: ivancich, anthonyeleven, yuvalif

qa/tasks/cephfs: add "extra data pool" cephfs-data-scan tests

Signed-off-by: Mykola Golub <mykola.golub@clyso.com>

qa/tasks/cephfs: use cephfs tags when recreating osd caps

to make it work when an extra data pool is added

Signed-off-by: Mykola Golub <mykola.golub@clyso.com>