crimson/os/seastore/object_data_handler: adjust code indention of read()

To make it easier to read, make sure there are 2 spaces per nesting.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/object_data_handler: cleanup read() and logs

Specifically, refactor based on read range, pin range and extent range.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/object_data_handler: adjust l_current asserts for read()

l_current is only allowed to be larger than pin_key for the first pin.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/object_data_handler: cleanup and renames

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57434 from cyx1231st/wip-fix-seastore-capture

crimson/os/seastore/record_scanner: replace [=] capturing

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #55488 from cyx1231st/wip-crimson-load-obc

crimson/osd/osd_operations/client_request: make loading-obc concurrent

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57356 from soumyakoduri/wip-skoduri-lc-notify

rgw/cloud-transition: fix the crash with publish_commit

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56574 from badone/wip-tracker-59196-lazy-omap-regex-segfault

test/lazy-omap-stats: Convert to boost::regex

Merge pull request #56979 from yuvalif/wip-yuval-65337

rgw/notification: start/stop endpoint managers in notification manager

Reviewed-By: cbodley@ibm.com , kchheda3@bloomberg.net

Merge PR #56148 into main

* refs/pull/56148/head:
cephfs_mirror: check m_instance_watcher and m_mirror_watcher before using them

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56699 into main

* refs/pull/56699/head:
qa: ignore `Invalid tag char` warning
qa: ignore `object missing on disk` warning

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #56525 into main

* refs/pull/56525/head:
mgr/stats: initialize mx_last_updated in FSPerfStats

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56135 into main

* refs/pull/56135/head:
cephfs_mirror: increment sync_failures when sync_perms() and sync_snaps() fails

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/record_scanner: replace [=] capturing

Capturing with = is error prone and causes compile warning.

And it is wrong:
* cursor should be captured by reference
* mutable keyword is unnecessary

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #55923 from NitzanMordhai/wip-nitzan-add-deprecate-cls-gather

objclass: deprecate cls_cxx_gather

Merge pull request #56983 from NitzanMordhai/wip-nitzan-thrash-erasure-code-crush-4-nodes-8-6

suites/ec-rados-plugin=jerasure-k=8-m=6-crush: roles set

test/rgw/notifications: make sure that http handler is running

fail test if not. to indicate this is a test issue
and not a product bug

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: prevent flooding the http server

in tests that require retries

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: avoid BlockingIOError when logging errors

for example. job: 7697397
in test: yuvalif-2024-05-08_09:55:02-rgw:notifications-wip-yuval-65337-distro-default-smithi

also reduce the side of the error log by sending less objects to the
test_ps_s3_persistent_topic_stats test

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notifications: fix lifecycle tests

* tests were passing only because they were not performings their asserts
* tests are now separated with their own attribute
* their topics are now marked "persistent" to workaround the issue in:
  https://tracker.ceph.com/issues/65645

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: start/stop endpoint managers in notification manager

Fixes: https://tracker.ceph.com/issues/65337
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rados/test: Remove cls_remote_reade since gather deprecated

https://tracker.ceph.com/issues/64258
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

rgw/notifications: cleanup includes and unused parameters

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: enable notifications debug logs in vstart

as well as in:
- multisite tests (used for notification v2 migration tests)
- the qa suites running notifications

enable lifecycle logs in notification tests: for the lc notification test cases

this is needed after: 429967917b4f9b5f71214fa3c7718e780728249b

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #57421 from zdover23/wip-doc-2024-05-12-dev-developer-guide-dash-devel

doc/dev: s/to asses/to assess/

Merge pull request #57420 from zdover23/wip-doc-2024-05-11-cephfs-fs-volumes-1-of-x-followup

doc/cephfs: edit fs-volumes.rst (1 of x) followup

doc/dev: s/to asses/to assess/

s/to asses/to assess/ in doc/deve/developer_guide/dash-devel.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: edit fs-volumes.rst (1 of x) followup

Include the suggestions for improving doc/cephfs/fs-volumes.rst made by
Anthony D'Atri here
https://github.com/ceph/ceph/pull/57415#discussion_r1597362110

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57415 from zdover23/wip-doc-2024-05-11-cephfs-fs-volumes-1-of-x

doc/cephfs: edit fs-volumes.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: edit fs-volumes.rst (1 of x)

Edit doc/cephfs/fs-volumes to the section "FS Subvolumes" (but not
including the section "FS Subvolumes").

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57214 from adk3798/cephadm-ignore-stray-on-upgrade

qa/cephadm: ignore CEPHADM_STRAY_DAEMON in upgrade tests

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57205 from SidGilad/bug/65739_support_cgroup_split_removal_for_adopt

mgr/cephadm: make setting --cgroups=split configurable for adopted daemons

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56735 from samarahu/wip-rgw-d4n-long-descriptions

d4n: Add more in-depth config descriptions and improve naming

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Samarah Uriarte <samarah.uriarte@ibm.com>

Merge pull request #56585 from ivancich/wip-upgrade-arrow+flight-15.0.2

rgw: upgrade Apache Arrow submodule to 15.0.0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57259 from pdvian/wip-fix-unfoundev-progress

mgr/cephadm: Fix unfound progress events

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57182 from adk3798/nvmeof-config-fix

mgr/cephadm:  update default NVMEoF container image version

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57181 from adk3798/cleanup-iscsi-nvmeof-keyring

mgr/cephadm: cleanup iscsi and nvmeof keyrings upon daemon removal

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57096 from phlogistonjohn/jjm-smb-output-fmt

mgr/smb: disable JSON/YAML key sorting in favor of ordered dicts

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57089 from adk3798/test-cephadm-timeout-ignore-refresh-failed

qa/cephadm: ignore CEPHADM_REFRESH_FAILED on timeout test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57080 from phlogistonjohn/jjm-teuth-pull-img

qa/tasks/cephadm: fix pulling containers from private registries

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57042 from adk3798/host-drain-test-ignore-stray-host

qa/cephadm: ignore stray warnings on host drain test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56970 from adk3798/cephadm-ignore-orch-paused

qa/cephadm: ignore CEPHADM_PAUSED on test_orch_cli test

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #56963 from adk3798/nfs-cluster-ls-json-pretty

mgr/pybind/object_format: fix json-pretty being marked invalid

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #56957 from adk3798/no-stray-recent-removal

 mgr/cephadm: don't mark daemons created/removed in the last minute as stray

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56823 from adk3798/autotune-testing

mgr/cephadm: additional debug logging for autotuner

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56802 from clwluvw/rgw-mdsearch

rgw: allow disabling mdsearch APIs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57372 from ceph/wip-lusov-quiesce-docs

doc/cephfs: expand the role of quiesce in consistent backups

Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57277 from tchaikov/test_election-free-tracker

test/mon/test_election: free ConnectionTracker if blocked

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #56493 from igomon-bloomberg/wip_kafka_sasl_username_password_attrs

rgw/s3-notifications: use user-name/password topic attributes for SASL authentication

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #57075 from cbodley/wip-rgw-ldap-derr

rgw: don't print error when ldap isn't configured

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #57083 from ivancich/wip-optimize-bucket-check-mulitparts

rgw: make incomplete multipart upload part of bucket check efficient

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57098 from kchheda3/wip-dont-log-passwd

rgw/notification: Do not log endpoint as it could contain broker user-id & password.

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw: update s3select submodule addressing arrow 15.0.0 compilation issues

Update arrow submodule.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

doc/cephfs: expand the role of quiesce in consistent backups

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Fixes: https://tracker.ceph.com/issues/65881

Merge pull request #57331 from ronen-fr/wip-rf-tier0

tests/librados: do not use '0' as a string

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57155 from mkogan1/wip-rgw-ssl-ec

rgw/beast: fix crash observed in SSL stream.async_shutdown()

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57360 from cbodley/wip-neorados-rm-completion.h

neorados: remove unused symlink to completion.h

Reviewed-by: Adam Emerson <aemerson@redhat.com>

objclass: deprecate cls_cxx_gather

cls_cxx_gather is not maintained and having issues with retry.
since there is no current use of it, we will deprecate it.

Fixes: https://tracker.ceph.com/issues/64258
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #55773 from weirdwiz/main

exporter: handle exceptions gracefully

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw/cloud-transition: Use set_atomic() while updating HEAD

LC Cloud transition should use set_atomic() to prevent any overwrite
while updating the HEAD object.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/cloud-transition: fix the crash with accessing invalid attrs

The obj_state may not be valid anymore post LC operations (esp.,
cloud-transition). Hence read and store etag prior to them to be used
later by notification (publish_commit).

Fixes: https://tracker.ceph.com/issues/65862
Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #56865 from zmc/cmake-sccache-max-jobs

cmake: Intelligently set job limits for sccache

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson/common/operation: distruct barrier after the wait future is resolved

Specifically, OrderedExclusivePhaseT::mutex must be unlocked only after
it is locked. Otherwise it can unlock other client requests
unexpectedly, causing incorrect order upon exit.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/common/operation: cleanup, merge exit() into ~PipelineExitBarrierI()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations: introduce concurrent lock_obc phase

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations/client_request: make loading-obc concurrent during recover missing

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/.../client_request: introduce instance_handle_t::enter_stage_sync()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operation: introduce PhasedOperationT::enter_stage_sync()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/common/operation: cleanup around PipelineHandle::enter()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations/client_request: cleanup, drop CommonClientRequest::recover_missings

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations/client_request: cleanup, simplify recover_missings

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations/client_request: skip recover missings if snaps are empty

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

neorados: remove unused symlink to completion.h

this header is unused after ea67f3dee2a3f8fcdcbb0bc0e80e38ec70378f05
replaced uses of ceph::async::Completion with
boost::asio::any_completion_handler

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55437 from tchaikov/ehh-upstart

debian: do not use upstart to start/stop services

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56238 from SMIL-Infra/journald-fix

common/Journald: fix alias and mis-align

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55560 from tchaikov/systemd-sans-clsuter-name

systemd: do not pass --cluster option

Reviewed-by: John Mulligan <jmulligan@redhat.com>

exporter: handle exceptions gracefully

ceph exporter crahes, and fails to handle exceptions in
dump_asok_metrics(). add try and catch blocks to handle the exceptions gracefully.

Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>

Merge pull request #55007 from shminjs/fix-wrong-osd-op-flag-name

osd/osd_types: fix spelling of osd_op_flag_name

Merge PR #56810 into main

* refs/pull/56810/head:
common/StackStringStream: update pointer to newly allocated memory in overflow()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #57330 from ronen-fr/wip-rf-iam-rgw0

tests/rgw: fix ints returned where a string is expected

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57258 from cbodley/wip-qa-rgw-d4n-tests

qa/rgw/d4n: run ceph_test_d4n_ tests

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge PR #57192 into main

* refs/pull/57192/head:
PendingReleaseNotes: add note on the client incompatibility health warning and feature bit
doc/cephfs: add client_mds_auth_caps client feature bit
doc/cephfs: add missing client feature bits
doc/cephfs: document MDS_CLIENTS_BROKEN_ROOTSQUASH health error
qa: add tests for MDS_CLIENTS_BROKEN_ROOTSQUASH
mds: raise health warning if client lacks feature for root_squash
mon/MDSMonitor: add note about missing metadata inclusion
mds: check relevant caps for fs include root_squash
mds: refactor out fs_name match in MDSAuthCaps
qa: test for root_squash with multiple caps
qa: pass kwargs to mount from remount
qa: simplify update_attrs and only update relevant keys
client: allow overriding client features

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #57298 from zdover23/wip-doc-2024-05-07-mgr-dashboard-overview

doc/mgr: edit "Overview" in dashboard.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge PR #57166 into main

* refs/pull/57166/head:
qa: make quiesce ops dump world readable
qa: use specific ops/cache dump file names

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #57165 into main

* refs/pull/57165/head:
qa: ignore variation of PG_DEGRADED health warning

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57226 from ronen-fr/wip-rf-unused-cache

common: mark assert-only variables as unused

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

tests/librados: do not use '0' as a string

not even as a default string parameter to a
function.

C++23 does not allow int-to-string conversions,

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

tests/rgw: fix ints returned where a string is expected

Modifying test_rgw_iam_policy.cc to avoid C++23
compilation errors. C++23 does not allow int-to-string
conversions, and '0' cannot be returned where a string
is expected.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

PendingReleaseNotes: add note on the client incompatibility health warning and feature bit

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: add client_mds_auth_caps client feature bit

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: add missing client feature bits

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: document MDS_CLIENTS_BROKEN_ROOTSQUASH health error

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add tests for MDS_CLIENTS_BROKEN_ROOTSQUASH

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: raise health warning if client lacks feature for root_squash

Rather than evict all clients lacking this feature bit, raise a health error
that pushes the administrator to address it. This avoids the surprise of having
all affected clients suddenly evicted in the cluster.

Fixes: https://tracker.ceph.com/issues/65733
Fixes: 954ed30
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/MDSMonitor: add note about missing metadata inclusion

There is a "client_count" metadata on the health warning that apparently was
intended to be used for aggregating warnings but never was. Add a TODO item for
that.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: check relevant caps for fs include root_squash

When denying client reconnects because the MDS caps include root_squash and the
client features do not include CEPHFS_FEATURE_MDS_AUTH_CAPS_CHECK, ensure those
caps are only for the file system the MDS is joined to.

Fixes: https://tracker.ceph.com/issues/65733
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: refactor out fs_name match in MDSAuthCaps

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: test for root_squash with multiple caps

Where the client has root_squash for one cap but not for another. The fs
without root_squash should not necessarily reject the client.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: pass kwargs to mount from remount

So we can pass mntargs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: simplify update_attrs and only update relevant keys

So we can just pass the caller's kwargs to update_attrs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>