rgw/notification: remove non x-amz-meta-* attributes from the metadata section of the regular notifications

Signed-off-by: Juan Zhu <jzhu4@dev-10-34-20-139.pw1.bcc.bloomberg.com>

Merge pull request #51299 from zdover23/wip-doc-2023-05-02-radosgw-notifications-rabbitmq-push-endpoint-directive-removal

doc/radosgw: rabbitmq - push-endpoint edit

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #50698 from linuxkidd/wip-rgw-restore-bucket-index-update

rgw: add use of cached `rados ls` output to rgw-restore-bucket-index script

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #51270 from ljflores/wip-telemetry-selftest

mgr/telemetry: compile all channels and collections in selftest

Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #51296 from zdover23/wip-doc-2023-05-01-rados-operations-stretch-mode-top-matter

doc/rados: edit stretch-mode.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51298 from ljflores/wip-urllib3-version

mgr: add urllib3==1.26.15 to mgr/requirements.txt

doc/rados: edit stretch-mode.rst

Refine and supplement the introductory and explanatory text at the top
of the /doc/rados/operations/stretch-mode.rst file.

Co-authored-by: Josh Durgin <jdurgin@redhat.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/radosgw: rabbitmq - push-endpoint edit

Remove a note that directed users to change "push-endpoint" (with a
hyphen) to "push_endpoint" (with an underscore) when using rabbitmq.

Re: https://github.com/ceph/ceph/pull/48486#issuecomment-1529925389

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr: add urllib3==1.26.15 to mgr/requirements.txt

We do not depend on any particular version of
urllib3, but as a workaround to the incompatibility
of urllib3 constraints between kubernetes and
requests, we need to pin it temporarily to
the version both are happy with.

Fixes: https://tracker.ceph.com/issues/59591
Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #51284 from zdover23/wip-doc-2023-04-29-cephfs-nfs

doc/cephfs: improve sentence in nfs.rst

Merge pull request #51138 from AliMasarweh/wip-alimasa-pubsub-hide-autogen-topics

RGW: bucket notification - hide auto generated topics when listing topics

Merge pull request #51293 from yuvalif/wip-yuval-revert-47457

revert PRs #47457 and #51031

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Revert "tracer/osd/librados/build/rgw: rgw and osd end2end tracing using opentelemetry"

This reverts commit 5d4c5e65be94a346e62b9f8ac63241779c4c7a6a.

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Revert "rgw/aio: fix recursion from tracing changes"

This reverts commit 9f160e4f1aa385f39f1cd55eb42f278c94dfd85c.

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #51285 from zdover23/wip-doc-2023-04-29-rados-operations-stretch-mode-stretch-mode

doc/rados: edit stretch-mode procedure

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit stretch-mode procedure

Edit the "stretch mode" section in doc/rados/operations/stretch-mode.rst
so that the procedure is formatted as a procedure and the sentences
correctly have heads.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51276 from pcuzner/fix-prom-docs

docs: Update the cephadm docs for Prometheus endpoint info

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: improve sentence in nfs.rst

Corect a sentence that should have been corrected in
https://github.com/ceph/ceph/pull/51204.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51069 from rhcs-dashboard/exporter-prom

exporter: promethize counter names

docs: Update the Prometheus endpoint info

This patch just tidies up some of the links and adds
an example showing how the http_sd_configs option
may be used.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #51271 from zdover23/wip-doc-2023-04-28-rados-configuration-mon-config-ref-background

doc/rados: m-config-ref: edit "background"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: m-config-ref: edit "background"

Edit the "Background" section of doc/rados/monitor/config-ref.rst

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/telemetry: compile all channels and collections in selftest

The selftest currently compiles only channels and collections that
are on by default. This change tests the compilation of all channels
and collections.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #50530 from Matan-B/wip-matanb-crimson-only-new-rbd-v2

crimson/osd: Handle rollback to head obejct

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50584 from Matan-B/wip-matanb-crimson-only-add-thrash

qa/suites/crimson-rados/thrash: Enable supported tests and ops

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51198 from Matan-B/wip-matanb-snap-trimmer-loop

crimson/osd/snaptrim_event: Handle missing clone obc case

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #51189 from rhcs-dashboard/fix-cephpgimbalance-alert

mgr/dashboard: fix CephPGImbalance alert

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51154 from zdover23/wip-doc-2023-04-20-rados-operations-3-of-x

doc/rados/ops: edit user-management.rst (3 of x)

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #51204 from zdover23/wip-doc-2023-04-25-cephfs-nfs-ganesha-permissions

doc/cephfs: explain cephfs data and metadata set

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: explain cephfs data and metadata set

Explain how to set application metadata for the CephFS data pool and the
CephFS metadata pool.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51228 from jsoref/spelling-rbd

rbd: fix spelling errors

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51030 from batrick/i59425

qa: check each fs for health

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rbd: fix spelling errors

* acquire
* are
* asynchronous
* attempt
* bootstrap
* concurrent
* consume
* couldn't
* cumulative
* disable
* disabling
* disaster
* disconnected
* endianness
* entries
* exclusive
* filesystem
* flag
* generic
* github
* image
* information
* initiating
* latency
* limitations
* metadata
* modify
* namespace
* noautoconsole
* ourselves
* prefetch
* propagate
* protection
* recorder
* recover
* release
* replicated
* reserved
* selection
* sentinel
* several
* snapshot
* source
* specifying
* suppress
* synchronize
* the
* transfer
* triggering
* unknown
* validation
* version
* visible
* write log entries

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

Merge pull request #50927 from rhcs-dashboard/export-authx

mgr/dashboard: import/export ceph authx users

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: import/export authx users

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
Fixes: https://tracker.ceph.com/issues/59486

Merge PR #51223 into main

* refs/pull/51223/head:
Revert "tools/cephfs/top/CMakeList.txt: check the minimum compatible python version for cephfs-top"

Reviewed-by: Jos Collin <jcollin@redhat.com>

Revert "tools/cephfs/top/CMakeList.txt: check the minimum compatible python version for cephfs-top"

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51193 from zdover23/wip-doc-2023-04-24-start-intro-first-para-rewrite

doc/start: rewrite intro paragraph

doc/start: rewrite intro paragraph

Rewrite the first paragraph in doc/start/intro.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50722 from adk3798/asyncio-cephadm-timeout

mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #49103 from adk3798/mon-crush-location

mgr/cephadm: allow setting mon crush locations through mon service spec

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #51120 from mgfritch/cephadm-haproxy-offline-host

cephadm: reschedule haproxy from an offline host

Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands

Since we already have make use of asyncio for our ssh commands,
we can use asyncio's timeout on waiting for concurrent futures to complete
as a way to have universal timeouts on our cephadm commands.
This change also creates a contextmanager that will catch any asyncio.TimeoutError.
Using the contextmanager along with calls to the wait_async function
will catch any timeout exception raised and convert it into an appropriate
OrchetratorError including information about what and where for the timeout
if it was provided (host where run, what command). This allows us to guarantee a
background ssh command eventually returns and inform users of any
timeouts by raising a health warning or logging the error instead
of sitting idle indefinitely

Fixes: https://tracker.ceph.com/issues/54024
Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: pass --timeout to cephadm binary commands

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #50719 from guits/update-dockerfile-keepalived

contrib: update keepalived Dockerfile

Reviewed-by: Adam King <adking@redhat.com>

qa/suites/crimson-rados/thrash: Enable supported thrashers

Balanced/Localized reads are now supported.
snap_remove and rollback are supported as well.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados/thrash: Add snap_remove/create weights

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: PG::on_active_actmap() clear PG_STATE_SNAPTRIM_ERROR

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: PG::on_active_actmap() fix SnapTrimEvent stop condition

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/snaptrim_event: Handle missing clone obc

The case where resolve_oid is not able to return the clone oid (in order
to load the clone obc) should result in an SNAPTRIM_ERROR and stop the loop
iteration. Without taking this case into account an infinite loop of snap
trim events will be created.
The ENOENT error is delegated upwards to be handled by PG::on_active_actmap().

Fixes: https://tracker.ceph.com/issues/59231
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #49980 from myoungwon/wip-generalize-recordsubmitter

crimson/os/seastore/journal: generalize recordsubmitter

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

exporter: promethize counter names

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>
Fixes: https://tracker.ceph.com/issues/59475

crimson/osd/osd_operations/snaptrim_event: Add log lines and formatting

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #51067 from guits/fix-backward-compatible-issue-cv

ceph-volume: fix batch refactor issue

Merge pull request #50838 from guits/osd-replacement-improvements

orchestrator: add `--no-destroy` arg to `ceph orch osd rm`

Merge pull request #50745 from guits/quick-fix-cv

ceph-volume: quick fix in zap.py

mgr/dashboard: fix CephPGImbalance alert

Fixes: https://tracker.ceph.com/issues/55568
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #51178 from zdover23/wip-doc-2023-04-22-glossary-placement-groups

doc/glossary: add "Placement Groups" definition

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51177 from zdover23/wip-doc-2023-04-22-start-documenting-ceph-first-150-lines

doc/start: edit first 150 lines of documenting-ceph

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: edit first 50 lines of documenting-ceph

Edit the first 150 lines of doc/start/documenting-ceph.rst. This is part
of an initiative to harvest the fruits of Cephalocon 2023, at which
documentation proved to be in demand to a surprising degree.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/glossary: add "Placement Groups" definition

Add a definition of "Placement Groups" to the Glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51175 from zdover23/wip-doc-2023-04-21-main-page-docs-link

doc: add link to "documenting ceph" to index.rst

doc: add link to "documenting ceph" to index.rst

Add a link to the landing page of docs.ceph.com to direct documentation
contributors to documentation-related information.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50324 from pkalever/scrub_incomplete

librbd: remove previous incomplete primary snapshot after successfully creating a new one

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #48717 from liu-chunmei/multicore-seastore

crimson/os: support multicore seastore

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os: make ZNS support shard metadata

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/os: fix formatter build error

Signed-off-by: chunmei <chunmei.liu@intel.com>

crimson/os: fix unittest error for multicore seastore

Signed-off-by: chunmei <chunmei.liu@intel.com>

crimson/os: static assign segments to each shard

and make device sharded

Signed-off-by: chunmei <chunmei.liu@intel.com>

cirmson/os: seperate futurizedStore interface

into futurizedStore and futurizedShardStore
and modification the other parts in osd and objectstore

Signed-off-by: chunmei <chunmei.liu@intel.com>

crimson/seastore: fix heap-used-after-free bug

Signed-off-by: chunmei <chunmei.liu@intel.com>

Merge pull request #51153 from rhcs-dashboard/applitools-fix

mgr/dashboard: ignore Details card in applitools vrt

Reviewed-by: Pegonzal <NOT@FOUND>

Merge pull request #51163 from rhcs-dashboard/hide-rbd-force-promote-notification

mgr/dashboard: hide notification on force promote

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #50835 from cyx1231st/wip-seastar-adjust-conn-ref

crimson/net: change ConnectionRef to be a local_shared_foreign_ptr

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: hide notification on force promote

Fixes: https://tracker.ceph.com/issues/59500
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

crimson/os/seastore/cbjournal: reset start address for replay if committed_to exceeds cbjournal_end()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test/crimson/seastore/cbjournal: call close after a unittest is done and relocate open and close related functions

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/journal: add CircularJournalSpace

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #51127 from rishabh-d-dave/fs-qa-caps_helper-bug

qa/cephfs/caps_helper: fix a bug in methods that generate cap string

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50882 from rishabh-d-dave/fs-qa-CapTester

qa/cephfs: improvements in caps_helper

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/journal: introduce JournalAllocator to generalize SegmentAllocator

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/journal: relocate RecordSubmitter related classes

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #51159 from idryomov/wip-rbd-man-filestore

doc/man/rbd: resurrect filestore alloc_size note

Reviewed-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

doc/man/rbd: resurrect filestore alloc_size note

Mistakenly removed in commit d79f2a81541c ("docs: warning and remove
few docs section for Filestore Update docs after filestore removal.").
The kernel client, however new, will continue to be able to talk to
FileStore OSDs for as long as they exist.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50580 from ceph/dependabot-npm_and_yarn-src-pybind-mgr-dashboard-frontend-webpack-and-angular-devkit-build-angular-5.76.1

mgr/dashboard: bump webpack and @angular-devkit/build-angular in /src/pybind/mgr/dashboard/frontend

Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/rados/ops: edit user-management.rst (3 of x)

Line-edit doc/rados/user-management.rst (3 of x).

https://tracker.ceph.com/issues/58485

Follows https://github.com/ceph/ceph/pull/51140.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: ignore Details card in applitools vrt

because the details card includes the FSID and Ceph Version which can
change on different clusters.

also don't run the 800x600 resolution because for new dashboard it looks
really weird.

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51140 from zdover23/wip-doc-2023-04-18-rados-operations-2-of-x

doc/rados: edit user-management (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit user-management (2 of x)

Line-edit doc/rados/user-management.rst (2 of x). Some internal
references had to be removed, but these will be repaired when the next
part of this file is updated in a future PR.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51143 from NitzanMordhai/wip-nitzan-filestore-removal-docs-update

docs: warning and remove few docs section for Filestore

Reviewed-by: Zac Dover <zac.dover@proton.me>

docs: warning and remove few docs section for Filestore
Update docs after filestore removal.

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #51068 from cbodley/wip-59269

cmake/rgw: librgw tests depend on ALLOC_LIBS

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #51031 from cbodley/wip-59423

rgw/aio: fix recursion from tracing changes

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #50975 from yuvalif/wip-yuval-fix-59381

rgw/lua: fix CopyFrom crash

Merge pull request #50965 from yuvalif/wip-yuval-fix-59383

rgw/kafka: failed to reconnect to broker after idle timeout

RGW: notification | hide auto generated topics when listing topics

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

librbd: always refresh after creating snapshot in CreatePrimaryRequest

Up until now this was conditioned on whether the caller expressed
interest in the ID of the created snapshot and happened to work only
because CreatePrimaryRequest wasn't actually consulting any mirror
snapshot metadata.  This has just changed with unlink_peer() needing to
see an up-to-date complete flag which is set in SetImageStateRequest
following the write out of image state object(s).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50918 from rhcs-dashboard/delete-ceph-authx

mgr/dashboard: delete-ceph-authx

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

librbd: remove previous incomplete primary snapshot after successfully creating a new one

Problem:
-------
At a high level, creating a primary snapshot consists of three steps:

1. actually creating a snapshot in the mirror namespace
2. generating a set of image state objects with additional metadata for
   the snapshot
3. marking the snapshot as complete after the image state objects are
   written out

Depending on the circumstances, a request to create a primary snapshot
can be forwarded to rbd-mirror daemon.  If that happens and rbd-mirror
daemon gets axed for some practical reason after completing steps (1)
and/or (2) but before completing step (3), we are left with a
permanently incomplete primary snapshot because upon retrying that
primary snapshot creation request, librbd notices that such snapshot
already exists.  It does not check whether this "pre-existing" snapshot
is complete.

Solution:
--------
As part of the next mirror snapshot create (say triggered by the
scheduler) the unlink_peer() is called, it checks if there exists any
incomplete snapshot and delete them accordingly.

Fixes: https://tracker.ceph.com/issues/58887
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

Merge PR #50089 into main

* refs/pull/50089/head:
doc: add a note for minimum compatible python version and supported distros
tools/cephfs/top/CMakeList.txt: check the minimum compatible python version for cephfs-top

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>