crimson/os/seastore: set last_committed_crc for rewritten extents

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: correct the offset of the data
copied from the original extents

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #57164 from guits/bump-promtail

cephadm: change loki/promtail default image tags

Merge pull request #57000 from chenyuanrun/fix-mkdir-p

ceph-volume: use os.makedirs for mkdir_p

Merge PR #57453 into main

* refs/pull/57453/head:
doc: add status badge for backport creation
.github: use shorter name for backport tracker action
.github: document where runs/output can be examined

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #57391 from soumyakoduri/wip-skoduri-sync-policy

Add new sync-policy related params to boto3 extension

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53859 from rhcs-dashboard/add-pseudo-validation

mgr/dashboard: add absolute path validation for pseudo path of nfs export

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc: add status badge for backport creation

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

.github: use shorter name for backport tracker action

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

.github: document where runs/output can be examined

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57368 from myoungwon/wip-seastore-is-data-stable

crimson/os/seastore: add is_data_stable() to allow delta-overwrite on EXIST_CLEAN

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57444 from idryomov/wip-51845

qa/suites/krbd: drop pre-single-major and move "layering only" coverage

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #57434 from cyx1231st/wip-fix-seastore-capture

crimson/os/seastore/record_scanner: replace [=] capturing

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: add comments to add a note that seastore_obj_data_write_amplification needs to be reconsidered

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: replace is_left_stable/is_right_stable with is_left_fresh/is_right_fresh

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #55488 from cyx1231st/wip-crimson-load-obc

crimson/osd/osd_operations/client_request: make loading-obc concurrent

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57356 from soumyakoduri/wip-skoduri-lc-notify

rgw/cloud-transition: fix the crash with publish_commit

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56574 from badone/wip-tracker-59196-lazy-omap-regex-segfault

test/lazy-omap-stats: Convert to boost::regex

Add new sync-policy related params to boto3 extension

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #56979 from yuvalif/wip-yuval-65337

rgw/notification: start/stop endpoint managers in notification manager

Reviewed-By: cbodley@ibm.com , kchheda3@bloomberg.net

qa/suites/krbd: rename no-object-map to no-exclusive-lock

Exclusive lock has always been disabled by this facet, so it might as
well be reflected in its name.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: move "layering only" coverage to fsx

It makes much more sense there since it's where we actually create
clones and flatten them a lot.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: drop pre-single-major test

Single-major mapping scheme was introduced in 2014 and became the
default in 2017.  It's getting increasingly difficult to build and,
more importantly, to boot a 10 year old kernel with recent userspace
(systemd, etc).  If someone is still running such a kernel, it's
really unlikely that they would have the most recent rbd CLI tool
installed.

Fixes: https://tracker.ceph.com/issues/51845
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #56148 into main

* refs/pull/56148/head:
cephfs_mirror: check m_instance_watcher and m_mirror_watcher before using them

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56699 into main

* refs/pull/56699/head:
qa: ignore `Invalid tag char` warning
qa: ignore `object missing on disk` warning

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #56525 into main

* refs/pull/56525/head:
mgr/stats: initialize mx_last_updated in FSPerfStats

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56135 into main

* refs/pull/56135/head:
cephfs_mirror: increment sync_failures when sync_perms() and sync_snaps() fails

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/record_scanner: replace [=] capturing

Capturing with = is error prone and causes compile warning.

And it is wrong:
* cursor should be captured by reference
* mutable keyword is unnecessary

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

cephadm: change loki/promtail default image tags

This commit changes the default images for both loki and promtail
containers.
Also, to allow this update we need to update the configuration of loki
in order to add a new storage schema configuration:

```
    - from: 2024-05-03
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
```

This is because 3.0.0 uses new structured metadata.

See [1][2].

[1] https://grafana.com/docs/loki/latest/operations/storage/schema/
[2] https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/#when-to-use-structured-metadata

Fixes: https://tracker.ceph.com/issues/65784
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #55923 from NitzanMordhai/wip-nitzan-add-deprecate-cls-gather

objclass: deprecate cls_cxx_gather

Merge pull request #56983 from NitzanMordhai/wip-nitzan-thrash-erasure-code-crush-4-nodes-8-6

suites/ec-rados-plugin=jerasure-k=8-m=6-crush: roles set

test/rgw/notifications: make sure that http handler is running

fail test if not. to indicate this is a test issue
and not a product bug

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: prevent flooding the http server

in tests that require retries

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: avoid BlockingIOError when logging errors

for example. job: 7697397
in test: yuvalif-2024-05-08_09:55:02-rgw:notifications-wip-yuval-65337-distro-default-smithi

also reduce the side of the error log by sending less objects to the
test_ps_s3_persistent_topic_stats test

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notifications: fix lifecycle tests

* tests were passing only because they were not performings their asserts
* tests are now separated with their own attribute
* their topics are now marked "persistent" to workaround the issue in:
  https://tracker.ceph.com/issues/65645

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: start/stop endpoint managers in notification manager

Fixes: https://tracker.ceph.com/issues/65337
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rados/test: Remove cls_remote_reade since gather deprecated

https://tracker.ceph.com/issues/64258
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

rgw/notifications: cleanup includes and unused parameters

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: enable notifications debug logs in vstart

as well as in:
- multisite tests (used for notification v2 migration tests)
- the qa suites running notifications

enable lifecycle logs in notification tests: for the lc notification test cases

this is needed after: 429967917b4f9b5f71214fa3c7718e780728249b

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #57421 from zdover23/wip-doc-2024-05-12-dev-developer-guide-dash-devel

doc/dev: s/to asses/to assess/

Merge pull request #57420 from zdover23/wip-doc-2024-05-11-cephfs-fs-volumes-1-of-x-followup

doc/cephfs: edit fs-volumes.rst (1 of x) followup

doc/dev: s/to asses/to assess/

s/to asses/to assess/ in doc/deve/developer_guide/dash-devel.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: edit fs-volumes.rst (1 of x) followup

Include the suggestions for improving doc/cephfs/fs-volumes.rst made by
Anthony D'Atri here
https://github.com/ceph/ceph/pull/57415#discussion_r1597362110

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57415 from zdover23/wip-doc-2024-05-11-cephfs-fs-volumes-1-of-x

doc/cephfs: edit fs-volumes.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: edit fs-volumes.rst (1 of x)

Edit doc/cephfs/fs-volumes to the section "FS Subvolumes" (but not
including the section "FS Subvolumes").

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57214 from adk3798/cephadm-ignore-stray-on-upgrade

qa/cephadm: ignore CEPHADM_STRAY_DAEMON in upgrade tests

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57205 from SidGilad/bug/65739_support_cgroup_split_removal_for_adopt

mgr/cephadm: make setting --cgroups=split configurable for adopted daemons

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56735 from samarahu/wip-rgw-d4n-long-descriptions

d4n: Add more in-depth config descriptions and improve naming

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Samarah Uriarte <samarah.uriarte@ibm.com>

Merge pull request #56585 from ivancich/wip-upgrade-arrow+flight-15.0.2

rgw: upgrade Apache Arrow submodule to 15.0.0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57259 from pdvian/wip-fix-unfoundev-progress

mgr/cephadm: Fix unfound progress events

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57182 from adk3798/nvmeof-config-fix

mgr/cephadm:  update default NVMEoF container image version

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57181 from adk3798/cleanup-iscsi-nvmeof-keyring

mgr/cephadm: cleanup iscsi and nvmeof keyrings upon daemon removal

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57096 from phlogistonjohn/jjm-smb-output-fmt

mgr/smb: disable JSON/YAML key sorting in favor of ordered dicts

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57089 from adk3798/test-cephadm-timeout-ignore-refresh-failed

qa/cephadm: ignore CEPHADM_REFRESH_FAILED on timeout test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57080 from phlogistonjohn/jjm-teuth-pull-img

qa/tasks/cephadm: fix pulling containers from private registries

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57042 from adk3798/host-drain-test-ignore-stray-host

qa/cephadm: ignore stray warnings on host drain test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56970 from adk3798/cephadm-ignore-orch-paused

qa/cephadm: ignore CEPHADM_PAUSED on test_orch_cli test

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #56963 from adk3798/nfs-cluster-ls-json-pretty

mgr/pybind/object_format: fix json-pretty being marked invalid

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #56957 from adk3798/no-stray-recent-removal

 mgr/cephadm: don't mark daemons created/removed in the last minute as stray

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56823 from adk3798/autotune-testing

mgr/cephadm: additional debug logging for autotuner

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56802 from clwluvw/rgw-mdsearch

rgw: allow disabling mdsearch APIs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57372 from ceph/wip-lusov-quiesce-docs

doc/cephfs: expand the role of quiesce in consistent backups

Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os/seastore: add is_data_stable() to allow delta-overwrite on EXIST_CLEAN

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #57277 from tchaikov/test_election-free-tracker

test/mon/test_election: free ConnectionTracker if blocked

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #56493 from igomon-bloomberg/wip_kafka_sasl_username_password_attrs

rgw/s3-notifications: use user-name/password topic attributes for SASL authentication

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #57075 from cbodley/wip-rgw-ldap-derr

rgw: don't print error when ldap isn't configured

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #57083 from ivancich/wip-optimize-bucket-check-mulitparts

rgw: make incomplete multipart upload part of bucket check efficient

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57098 from kchheda3/wip-dont-log-passwd

rgw/notification: Do not log endpoint as it could contain broker user-id & password.

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw: update s3select submodule addressing arrow 15.0.0 compilation issues

Update arrow submodule.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

doc/cephfs: expand the role of quiesce in consistent backups

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Fixes: https://tracker.ceph.com/issues/65881

Merge pull request #57331 from ronen-fr/wip-rf-tier0

tests/librados: do not use '0' as a string

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57155 from mkogan1/wip-rgw-ssl-ec

rgw/beast: fix crash observed in SSL stream.async_shutdown()

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57360 from cbodley/wip-neorados-rm-completion.h

neorados: remove unused symlink to completion.h

Reviewed-by: Adam Emerson <aemerson@redhat.com>

objclass: deprecate cls_cxx_gather

cls_cxx_gather is not maintained and having issues with retry.
since there is no current use of it, we will deprecate it.

Fixes: https://tracker.ceph.com/issues/64258
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #55773 from weirdwiz/main

exporter: handle exceptions gracefully

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw/cloud-transition: Use set_atomic() while updating HEAD

LC Cloud transition should use set_atomic() to prevent any overwrite
while updating the HEAD object.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/cloud-transition: fix the crash with accessing invalid attrs

The obj_state may not be valid anymore post LC operations (esp.,
cloud-transition). Hence read and store etag prior to them to be used
later by notification (publish_commit).

Fixes: https://tracker.ceph.com/issues/65862
Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #56865 from zmc/cmake-sccache-max-jobs

cmake: Intelligently set job limits for sccache

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson/common/operation: distruct barrier after the wait future is resolved

Specifically, OrderedExclusivePhaseT::mutex must be unlocked only after
it is locked. Otherwise it can unlock other client requests
unexpectedly, causing incorrect order upon exit.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/common/operation: cleanup, merge exit() into ~PipelineExitBarrierI()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations: introduce concurrent lock_obc phase

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations/client_request: make loading-obc concurrent during recover missing

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/.../client_request: introduce instance_handle_t::enter_stage_sync()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operation: introduce PhasedOperationT::enter_stage_sync()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/common/operation: cleanup around PipelineHandle::enter()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations/client_request: cleanup, drop CommonClientRequest::recover_missings

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations/client_request: cleanup, simplify recover_missings

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations/client_request: skip recover missings if snaps are empty

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

neorados: remove unused symlink to completion.h

this header is unused after ea67f3dee2a3f8fcdcbb0bc0e80e38ec70378f05
replaced uses of ceph::async::Completion with
boost::asio::any_completion_handler

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55437 from tchaikov/ehh-upstart

debian: do not use upstart to start/stop services

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56238 from SMIL-Infra/journald-fix

common/Journald: fix alias and mis-align

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55560 from tchaikov/systemd-sans-clsuter-name

systemd: do not pass --cluster option

Reviewed-by: John Mulligan <jmulligan@redhat.com>

exporter: handle exceptions gracefully

ceph exporter crahes, and fails to handle exceptions in
dump_asok_metrics(). add try and catch blocks to handle the exceptions gracefully.

Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>

Merge pull request #55007 from shminjs/fix-wrong-osd-op-flag-name

osd/osd_types: fix spelling of osd_op_flag_name

Merge PR #56810 into main

* refs/pull/56810/head:
common/StackStringStream: update pointer to newly allocated memory in overflow()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #57330 from ronen-fr/wip-rf-iam-rgw0

tests/rgw: fix ints returned where a string is expected

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57258 from cbodley/wip-qa-rgw-d4n-tests

qa/rgw/d4n: run ceph_test_d4n_ tests

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge PR #57192 into main

* refs/pull/57192/head:
PendingReleaseNotes: add note on the client incompatibility health warning and feature bit
doc/cephfs: add client_mds_auth_caps client feature bit
doc/cephfs: add missing client feature bits
doc/cephfs: document MDS_CLIENTS_BROKEN_ROOTSQUASH health error
qa: add tests for MDS_CLIENTS_BROKEN_ROOTSQUASH
mds: raise health warning if client lacks feature for root_squash
mon/MDSMonitor: add note about missing metadata inclusion
mds: check relevant caps for fs include root_squash
mds: refactor out fs_name match in MDSAuthCaps
qa: test for root_squash with multiple caps
qa: pass kwargs to mount from remount
qa: simplify update_attrs and only update relevant keys
client: allow overriding client features

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #57298 from zdover23/wip-doc-2024-05-07-mgr-dashboard-overview

doc/mgr: edit "Overview" in dashboard.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge PR #57166 into main

* refs/pull/57166/head:
qa: make quiesce ops dump world readable
qa: use specific ops/cache dump file names

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>