Merge pull request #47637 from rzarzynski/wip-crimson-blocklist

crimson/osd: bring support for client blocklisting

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Sam Just <sjust@redhat.com>

Merge PR #51158 into main

* refs/pull/51158/head:
cephfs-top: fix help text for delay

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #51005 into main

* refs/pull/51005/head:
qa: fix test_nfs_export_creation_at_symlink
qa: update test cases to check for ENOTDIR instead of EINVAL
qa: fix test_nfs_export_with_invalid_path
mgr/nfs: handle exceptions for cephfs_path_is_dir()
mgr/nfs/utils: changes to helper func to check cephfs path

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51133 from zdover23/wip-doc-2023-04-18-mgr-prometheus-prompts-update

doc/mgr: update prompts in prometheus.rst

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #51299 from zdover23/wip-doc-2023-05-02-radosgw-notifications-rabbitmq-push-endpoint-directive-removal

doc/radosgw: rabbitmq - push-endpoint edit

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #50698 from linuxkidd/wip-rgw-restore-bucket-index-update

rgw: add use of cached `rados ls` output to rgw-restore-bucket-index script

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #51270 from ljflores/wip-telemetry-selftest

mgr/telemetry: compile all channels and collections in selftest

Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #51296 from zdover23/wip-doc-2023-05-01-rados-operations-stretch-mode-top-matter

doc/rados: edit stretch-mode.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51298 from ljflores/wip-urllib3-version

mgr: add urllib3==1.26.15 to mgr/requirements.txt

doc/rados: edit stretch-mode.rst

Refine and supplement the introductory and explanatory text at the top
of the /doc/rados/operations/stretch-mode.rst file.

Co-authored-by: Josh Durgin <jdurgin@redhat.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/radosgw: rabbitmq - push-endpoint edit

Remove a note that directed users to change "push-endpoint" (with a
hyphen) to "push_endpoint" (with an underscore) when using rabbitmq.

Re: https://github.com/ceph/ceph/pull/48486#issuecomment-1529925389

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr: add urllib3==1.26.15 to mgr/requirements.txt

We do not depend on any particular version of
urllib3, but as a workaround to the incompatibility
of urllib3 constraints between kubernetes and
requests, we need to pin it temporarily to
the version both are happy with.

Fixes: https://tracker.ceph.com/issues/59591
Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #51284 from zdover23/wip-doc-2023-04-29-cephfs-nfs

doc/cephfs: improve sentence in nfs.rst

Merge pull request #51138 from AliMasarweh/wip-alimasa-pubsub-hide-autogen-topics

RGW: bucket notification - hide auto generated topics when listing topics

Merge pull request #51293 from yuvalif/wip-yuval-revert-47457

revert PRs #47457 and #51031

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Revert "tracer/osd/librados/build/rgw: rgw and osd end2end tracing using opentelemetry"

This reverts commit 5d4c5e65be94a346e62b9f8ac63241779c4c7a6a.

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Revert "rgw/aio: fix recursion from tracing changes"

This reverts commit 9f160e4f1aa385f39f1cd55eb42f278c94dfd85c.

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #51285 from zdover23/wip-doc-2023-04-29-rados-operations-stretch-mode-stretch-mode

doc/rados: edit stretch-mode procedure

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit stretch-mode procedure

Edit the "stretch mode" section in doc/rados/operations/stretch-mode.rst
so that the procedure is formatted as a procedure and the sentences
correctly have heads.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51276 from pcuzner/fix-prom-docs

docs: Update the cephadm docs for Prometheus endpoint info

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: improve sentence in nfs.rst

Corect a sentence that should have been corrected in
https://github.com/ceph/ceph/pull/51204.

Signed-off-by: Zac Dover <zac.dover@proton.me>

cephfs-top: fix help text for delay

Updates:

`
  -d {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25}, --delay {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25}
                        Refresh interval in seconds (default: 1)
`

to

`
  -d DELAY, --delay DELAY
                        Refresh interval in seconds (default: 1, range: 1 - 25)
`

Fixes: https://tracker.ceph.com/issues/59553
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #51069 from rhcs-dashboard/exporter-prom

exporter: promethize counter names

docs: Update the Prometheus endpoint info

This patch just tidies up some of the links and adds
an example showing how the http_sd_configs option
may be used.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #51271 from zdover23/wip-doc-2023-04-28-rados-configuration-mon-config-ref-background

doc/rados: m-config-ref: edit "background"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: m-config-ref: edit "background"

Edit the "Background" section of doc/rados/monitor/config-ref.rst

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/mgr: update prompts in prometheus.rst

Update prompts in prometheus.rst so that they're unselectable.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/telemetry: compile all channels and collections in selftest

The selftest currently compiles only channels and collections that
are on by default. This change tests the compilation of all channels
and collections.

Signed-off-by: Laura Flores <lflores@redhat.com>

crimson/osd: drop the OSDMap caching from PG

`PG` directly aggregates both `OSDMap` and `PeeringState` which
maintains its own `OSDMap` instance. This duplication leads to
issues as the `PG::osdmap` never gets updated.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement watcher blocklisting

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: check for blocklisted clients

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #50530 from Matan-B/wip-matanb-crimson-only-new-rbd-v2

crimson/osd: Handle rollback to head obejct

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50584 from Matan-B/wip-matanb-crimson-only-add-thrash

qa/suites/crimson-rados/thrash: Enable supported tests and ops

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51198 from Matan-B/wip-matanb-snap-trimmer-loop

crimson/osd/snaptrim_event: Handle missing clone obc case

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #51189 from rhcs-dashboard/fix-cephpgimbalance-alert

mgr/dashboard: fix CephPGImbalance alert

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51154 from zdover23/wip-doc-2023-04-20-rados-operations-3-of-x

doc/rados/ops: edit user-management.rst (3 of x)

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #51204 from zdover23/wip-doc-2023-04-25-cephfs-nfs-ganesha-permissions

doc/cephfs: explain cephfs data and metadata set

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: explain cephfs data and metadata set

Explain how to set application metadata for the CephFS data pool and the
CephFS metadata pool.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51228 from jsoref/spelling-rbd

rbd: fix spelling errors

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51030 from batrick/i59425

qa: check each fs for health

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rbd: fix spelling errors

* acquire
* are
* asynchronous
* attempt
* bootstrap
* concurrent
* consume
* couldn't
* cumulative
* disable
* disabling
* disaster
* disconnected
* endianness
* entries
* exclusive
* filesystem
* flag
* generic
* github
* image
* information
* initiating
* latency
* limitations
* metadata
* modify
* namespace
* noautoconsole
* ourselves
* prefetch
* propagate
* protection
* recorder
* recover
* release
* replicated
* reserved
* selection
* sentinel
* several
* snapshot
* source
* specifying
* suppress
* synchronize
* the
* transfer
* triggering
* unknown
* validation
* version
* visible
* write log entries

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

Merge pull request #50927 from rhcs-dashboard/export-authx

mgr/dashboard: import/export ceph authx users

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

qa: fix test_nfs_export_creation_at_symlink

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

qa: update test cases to check for ENOTDIR instead of EINVAL

- test_nfs_export_creation_at_filepath:
ENOTDIR is raised instead of EINVAL which is better
aligned with the nature of the failure

- test_nfs_export_creation_at_symlink:
ENOTDIR is raised instead of ENOENT since the code
can now check if the path is symlink but won't follow
it.

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

qa: fix test_nfs_export_with_invalid_path

It actually didn't test the invalid path but still ended with
ENOENT(which is expected in case path is invalid) as the test
didn't create a fs, and it failed saying "FS nfs-cephfs not found"
which too raises ENOENT and thus it always passed.

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

mgr/nfs: handle exceptions for cephfs_path_is_dir()
in create_cephfs_export()

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

mgr/nfs/utils: changes to helper func to check cephfs path

- Renamed to cephfs_path_is_dir

- Removed exception handling to prevent redundant log statements like:
   "No such file or directory error in stat: b'/mnt/testdir_symlink': No such file or directory [Errno 2]"

  Exceptions handled inside caller eliminates this redundancy

- Set modifier flag AT_SYMLINK_NOFOLLOW

- Removed string "{path} is not a dir" when raising NotADirectoryError
  Rationale: will be handled in export.py

- change mock to cephfs_path_is_dir

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

mgr/dashboard: import/export authx users

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
Fixes: https://tracker.ceph.com/issues/59486

Merge PR #51223 into main

* refs/pull/51223/head:
Revert "tools/cephfs/top/CMakeList.txt: check the minimum compatible python version for cephfs-top"

Reviewed-by: Jos Collin <jcollin@redhat.com>

Revert "tools/cephfs/top/CMakeList.txt: check the minimum compatible python version for cephfs-top"

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51193 from zdover23/wip-doc-2023-04-24-start-intro-first-para-rewrite

doc/start: rewrite intro paragraph

doc/start: rewrite intro paragraph

Rewrite the first paragraph in doc/start/intro.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50722 from adk3798/asyncio-cephadm-timeout

mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #49103 from adk3798/mon-crush-location

mgr/cephadm: allow setting mon crush locations through mon service spec

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #51120 from mgfritch/cephadm-haproxy-offline-host

cephadm: reschedule haproxy from an offline host

Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands

Since we already have make use of asyncio for our ssh commands,
we can use asyncio's timeout on waiting for concurrent futures to complete
as a way to have universal timeouts on our cephadm commands.
This change also creates a contextmanager that will catch any asyncio.TimeoutError.
Using the contextmanager along with calls to the wait_async function
will catch any timeout exception raised and convert it into an appropriate
OrchetratorError including information about what and where for the timeout
if it was provided (host where run, what command). This allows us to guarantee a
background ssh command eventually returns and inform users of any
timeouts by raising a health warning or logging the error instead
of sitting idle indefinitely

Fixes: https://tracker.ceph.com/issues/54024
Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: pass --timeout to cephadm binary commands

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #50719 from guits/update-dockerfile-keepalived

contrib: update keepalived Dockerfile

Reviewed-by: Adam King <adking@redhat.com>

qa/suites/crimson-rados/thrash: Enable supported thrashers

Balanced/Localized reads are now supported.
snap_remove and rollback are supported as well.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados/thrash: Add snap_remove/create weights

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: PG::on_active_actmap() clear PG_STATE_SNAPTRIM_ERROR

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: PG::on_active_actmap() fix SnapTrimEvent stop condition

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/snaptrim_event: Handle missing clone obc

The case where resolve_oid is not able to return the clone oid (in order
to load the clone obc) should result in an SNAPTRIM_ERROR and stop the loop
iteration. Without taking this case into account an infinite loop of snap
trim events will be created.
The ENOENT error is delegated upwards to be handled by PG::on_active_actmap().

Fixes: https://tracker.ceph.com/issues/59231
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #49980 from myoungwon/wip-generalize-recordsubmitter

crimson/os/seastore/journal: generalize recordsubmitter

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

exporter: promethize counter names

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>
Fixes: https://tracker.ceph.com/issues/59475

crimson/osd/osd_operations/snaptrim_event: Add log lines and formatting

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #51067 from guits/fix-backward-compatible-issue-cv

ceph-volume: fix batch refactor issue

Merge pull request #50838 from guits/osd-replacement-improvements

orchestrator: add `--no-destroy` arg to `ceph orch osd rm`

Merge pull request #50745 from guits/quick-fix-cv

ceph-volume: quick fix in zap.py

mgr/dashboard: fix CephPGImbalance alert

Fixes: https://tracker.ceph.com/issues/55568
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #51178 from zdover23/wip-doc-2023-04-22-glossary-placement-groups

doc/glossary: add "Placement Groups" definition

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51177 from zdover23/wip-doc-2023-04-22-start-documenting-ceph-first-150-lines

doc/start: edit first 150 lines of documenting-ceph

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: edit first 50 lines of documenting-ceph

Edit the first 150 lines of doc/start/documenting-ceph.rst. This is part
of an initiative to harvest the fruits of Cephalocon 2023, at which
documentation proved to be in demand to a surprising degree.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/glossary: add "Placement Groups" definition

Add a definition of "Placement Groups" to the Glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51175 from zdover23/wip-doc-2023-04-21-main-page-docs-link

doc: add link to "documenting ceph" to index.rst

doc: add link to "documenting ceph" to index.rst

Add a link to the landing page of docs.ceph.com to direct documentation
contributors to documentation-related information.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50324 from pkalever/scrub_incomplete

librbd: remove previous incomplete primary snapshot after successfully creating a new one

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #48717 from liu-chunmei/multicore-seastore

crimson/os: support multicore seastore

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os: make ZNS support shard metadata

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/os: fix formatter build error

Signed-off-by: chunmei <chunmei.liu@intel.com>

crimson/os: fix unittest error for multicore seastore

Signed-off-by: chunmei <chunmei.liu@intel.com>

crimson/os: static assign segments to each shard

and make device sharded

Signed-off-by: chunmei <chunmei.liu@intel.com>

cirmson/os: seperate futurizedStore interface

into futurizedStore and futurizedShardStore
and modification the other parts in osd and objectstore

Signed-off-by: chunmei <chunmei.liu@intel.com>

crimson/seastore: fix heap-used-after-free bug

Signed-off-by: chunmei <chunmei.liu@intel.com>

Merge pull request #51153 from rhcs-dashboard/applitools-fix

mgr/dashboard: ignore Details card in applitools vrt

Reviewed-by: Pegonzal <NOT@FOUND>

Merge pull request #51163 from rhcs-dashboard/hide-rbd-force-promote-notification

mgr/dashboard: hide notification on force promote

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #50835 from cyx1231st/wip-seastar-adjust-conn-ref

crimson/net: change ConnectionRef to be a local_shared_foreign_ptr

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: hide notification on force promote

Fixes: https://tracker.ceph.com/issues/59500
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

crimson/os/seastore/cbjournal: reset start address for replay if committed_to exceeds cbjournal_end()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test/crimson/seastore/cbjournal: call close after a unittest is done and relocate open and close related functions

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/journal: add CircularJournalSpace

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #51127 from rishabh-d-dave/fs-qa-caps_helper-bug

qa/cephfs/caps_helper: fix a bug in methods that generate cap string

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50882 from rishabh-d-dave/fs-qa-CapTester

qa/cephfs: improvements in caps_helper

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/journal: introduce JournalAllocator to generalize SegmentAllocator

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/journal: relocate RecordSubmitter related classes

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #51159 from idryomov/wip-rbd-man-filestore

doc/man/rbd: resurrect filestore alloc_size note

Reviewed-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

doc/man/rbd: resurrect filestore alloc_size note

Mistakenly removed in commit d79f2a81541c ("docs: warning and remove
few docs section for Filestore Update docs after filestore removal.").
The kernel client, however new, will continue to be able to talk to
FileStore OSDs for as long as they exist.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50580 from ceph/dependabot-npm_and_yarn-src-pybind-mgr-dashboard-frontend-webpack-and-angular-devkit-build-angular-5.76.1

mgr/dashboard: bump webpack and @angular-devkit/build-angular in /src/pybind/mgr/dashboard/frontend

Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/rados/ops: edit user-management.rst (3 of x)

Line-edit doc/rados/user-management.rst (3 of x).

https://tracker.ceph.com/issues/58485

Follows https://github.com/ceph/ceph/pull/51140.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: ignore Details card in applitools vrt

because the details card includes the FSID and Ceph Version which can
change on different clusters.

also don't run the 800x600 resolution because for new dashboard it looks
really weird.

Signed-off-by: Nizamudeen A <nia@redhat.com>