doc/glossary: add "RADOS Cluster"

Define "RADOS Cluster" in the glossary.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48682 from Matan-B/wip-matanb-obc-loader

crimson/osd: Introduce ObjectContextLoader

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #48831 from rzarzynski/wip-crimson-list_snaps-fix-errmsg

crimson/osd: fix error msg in OpsExecuter::do_list_snaps()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #48527 from rzarzynski/wip-crimson-simpler-cloning

crimson/osd: simplify cloning

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #49119 from zdover23/wip-doc-2022-11-29-rados-ops-change-mon-elections-prompts

doc/rados: add prompts to change-mon-elections.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49120 from zdover23/wip-doc-2022-11-29-rados-ops-control-prompts

doc/rados: add prompts to control.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49118 from zdover23/wip-doc-2022-11-29-rados-cache-tiering-prompts

doc/rados: add prompts to cache-tiering.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49114 from zdover23/wip-doc-2022-11-29-rados-bluestore-migration-prompts

doc/rados: add prompts to bluestore-migration.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: add prompts to control.rst

Add unselectable prompts to doc/rados/operations/control.rst.

https://tracker.ceph.com/issues/57108

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/rados: add prompts to change-mon-elections.rst

Add unselectable prompts to
doc/rados/operations/change-mon-elections.rst.

https://tracker.ceph.com/issues/57108

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48525 from rhcs-dashboard/fix-PgImbalance-alert

mgr/dashboard: fix CephPGImbalance alert

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #48526 from rhcs-dashboard/fix-cephPoolGrowth-alert

mgr/dashboard: Fix CephPoolGrowthWarning alert

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/rados: add prompts to cache-tiering.rst

Add unselectable prompts to doc/rados/operations/cache-tiering.rst.

https://tracker.ceph.com/issues/57108

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/rados: add prompts to bluestore-migration.rst

Add unselectable prompts to
doc/rados/operations/bluestore-migration.rst.

https://tracker.ceph.com/issues/57108

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49107 from zdover23/wip-doc-2022-11-29-rados-balancer-prompts

doc/rados: add prompts to balancer.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: add prompts to balancer.rst

Add unselectable prompts to doc/rados/operations/balancer.rst.

https://tracker.ceph.com/issues/57108

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49015 from batrick/meta-doc

doc/cephfs: correct names for fs/meta

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48490 from aclamk/wip-aclamk-fix-deferred

[bluestore] Improve deferred write decision

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #49053 from tchaikov/wip-crimson-fmt-v9

crimson: various fmt related changes and fixes

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #47221 from markhpc/wip-faster-rocksdb

[WIP] kv/RocksDBStore: Improved RocksDB Settings and Tombstone behavior

Merge pull request #48875 from yuvalif/wip-yuval-fix-58014

rgw/notifications: sending metadata in COPY and CompleteMultipartUpload

Merge pull request #49082 from zdover23/wip-doc-2022-11-28-rbd-isci-monitoring-prompts

doc/rbd: add prompts to iscsi-monitoring.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #48598 from yuvalif/wip-yuval-fix-57899

rgw/admin/notifications: support admin operations on topics with tenants

Merge pull request #49079 from zdover23/wip-doc-2022-11-28-glossary-ceph-system-remove

doc/glossary: s/Ceph System/Ceph Cluster/

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rbd: add prompts to iscsi-monitoring.rst

Add unselectable prompts to doc/rbd/iscsi-monitoring.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: s/Ceph System/Ceph Cluster/

Remove "Ceph System" and replace it with "Ceph Cluster".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49074 from zdover23/wip-doc-2022-11-27-dev-guide-basic-workflow-local-working-copy

doc/dev: improve Basic Workflow wording

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: improve Basic Workflow wording

Make several semantic and syntactic improvements to basic-workflow.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48680 from pkalever/snap-id

rbd: add --snap-id option to "rbd device map" to allow mapping arbitrary snapshots

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49066 from zdover23/wip-doc-2022-11-26-glossary-ceph-system-remove

doc/glossary: remove "Ceph System"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49065 from zdover23/wip-doc-2022-11-26-rados-ops-add-or-rm-osds-prompts

doc/rados: add prompts to add-or-remove-osds

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49045 from zdover23/wip-doc-2022-11-25-rbd-iscsi-initiator

doc/rbd: update iSCSI gateway info

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/rbd: update iSCSI gateway info

Update iSCSI gateway info in the RBD documentation.

https://tracker.ceph.com/issues/58080

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: remove "Ceph System"

"Ceph System" appears only in definitions where "Ceph Cluster" should
be, so I'm removing the headword. A future commit will scrub the
glossary of the phrase "Ceph System".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

qa/workunits/rbd-nbd: clear DEV after detach tests

Otherwise we attempt to unmap it in cleanup(), needlessly.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/rados: add prompts to add-or-remove-osds

Add unselectable prompts to add-or-rm-osds.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49062 from zdover23/wip-doc-2022-11-26-glossary-ceph-manager

doc/glossary: link to "Ceph Manager"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: link to "Ceph Manager"

Add link to "Ceph Manager" material.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48761 from soumyakoduri/wip-skoduri-cloudtier-fixes

rgw/cloud-transition: Fix issues with MCG endpoint

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/osd: correct the fmt::formatter<WatchTimeoutRequest>

it was introduced by c9569f8eb027b9e08c8eeb59ecb37378ef66e5c0

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: print exceptions using e.what()

see also e6223872c58e492d3570335fd7f05bb52641837c

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: specialize fmt::formatter<> for StagedIterator

StagedIterator is a nested class of stage_t, which is in turn a template
class. it would be impossible to partial specialize fmt::formatter<> for
a nested class of a template class. to workaround this, we specialize
fmt::formatter<> for a type which has `do_format_to()` method, and
define this method for StagedIterator.

since seastar::logger is the only user of the operator<<() operator,
the original operator<<() operator is replaced with this new helper
method.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49036 from idryomov/wip-52932

mgr/rbd_support: avoid wedging the task queue if pool is removed

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #49042 from idryomov/wip-rbd-support-flake8

mgr/rbd_support: clean up and enable flake8

Reviewed-by: Mykola Golub <mgolub@suse.com>

crimson/os: include fmt/os.h for printing std::error_code

this is required by {fmt} v9 and up.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: specialize fmt::formatter<> for formatting using {fmt}

so the specialized types can be formatted using {fmt} v9.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49046 from zdover23/wip-doc-2022-11-25-rados-ops-erasure-code-profiles

doc/rados: cleanup "erasure code profiles"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: cleanup "erasure code profiles"

Improve the grammar of "Erasure Code Profiles".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48927 from neesingh-rh/fix_58028

cephfs-top: Fix unresposive sorting in some cases

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #48542 from CongMinYin/fix-inconsistent-pwl-status

librbd/cache/pwl: fix clean vs bytes_dirty cache state inconsistency

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cephfs-top: Fix unresposive sorting in some cases

Sorting is not working properly in case of change
in the number of clients or after removing and
creating the same filesystem again.

Fixes: https://tracker.ceph.com/issues/58028
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #49026 from neesingh-rh/fix_58031

cephfs-top: sorting excepts when the filesystems are removed and created

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

mgr/rbd_support: clean up and enable flake8

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49029 from Matan-B/wip-matanb-EIO-flag

crimson/osd: Support Pool EIO flag

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48932 from aisakaki/wip-crimson-loop

crimson/common: re-implement do_for_each

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49040 from tchaikov/wip-cephadm-flake8

cephadm: pin flake8 to 5.0.4

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

pybind/mgr: use annotation style without comment

* use annotation style without comment
* add noqa to import statement to prevent flake8 from complaining
  those which import typing modules

to silence the warnings like:

```
flake8 run-test: commands[0] | flake8 --config=tox.ini alerts balancer cephadm cli_api crash devicehealth diskprediction_local hello iostat localpool nfs orchestrator prometheus selftest
cephadm/schedule.py:5:1: F401 'typing.Callable' imported but unused
cephadm/schedule.py:8:1: F401 'ceph.deployment.service_spec.ServiceSpec' imported but unused
cephadm/tests/fixtures.py:17:1: F401 'orchestrator.OrchResult' imported but unused
orchestrator/module.py:4:1: F401 'typing.Set' imported but unused
orchestrator/module.py:17:1: F401 'ceph.deployment.inventory.Device' imported but unused
prometheus/module.py:17:1: F401 'typing.DefaultDict' imported but unused
6     F401 'typing.Callable' imported but unused
ERROR: InvocationError for command /home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/.tox/flake8/bin/flake8 --config=tox.ini alerts
```

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

qa/tasks/cephfs: annotate variable without comment

to silence the warning like
```
flake8 run-test: commands[0] | flake8 --select=F,E9 --exclude=venv,.tox
./tasks/cephfs/test_full.py:6:5: F401 'typing.Optional' imported but unused
```

since we've migrated to python3.6 and up, there is no need to be
compatible with Python 2 anymore.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cephadm: pin flake8 to 5.0.4

to workaround https://github.com/zheller/flake8-quotes/issues/110
flake8 v5.0.4 is the last known-to-work flake8 before flake8 v6.0.0.

Fixes: https://tracker.ceph.com/issues/58069
Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cephfs-top: sorting excepts when the filesystems are removed and created

Fixes: https://tracker.ceph.com/issues/58031
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #49022 from cyx1231st/wip-seastar-msgr-fix-yielded-closing

crimson/net: fix use-after-free during yielded closing

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

mgr/rbd_support: avoid wedging the task queue if pool is removed

rados.ObjectNotFound exception handler was referencing ioctx variable
which is assigned only if the pool exists and rados.open_ioctx() call
succeeds.  This lead to a fatal error

  mgr[rbd_support] Failed to locate pool mypool
  mgr[rbd_support] execute_task: [errno 2] error opening pool 'b'mypool''
  mgr[rbd_support] Fatal runtime error: local variable 'ioctx' referenced before assignment

and wedged the task queue.  No other commands were processed until
ceph-mgr daemon restart.

Fixes: https://tracker.ceph.com/issues/52932
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49037 from zdover23/wip-doc-2022-11-24-container-install-consume-44310

doc/install: update "Official Releases" sources

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/install: update "Official Releases" sources

Remove "Docker Hub" from the list of locations from which Ceph can be
acquired.

This commit represents a second attempt to get
https://github.com/ceph/ceph/pull/49034 past the Jenkins CI tests, and
that PR was itself an attempt to incorporate the change in
https://github.com/ceph/ceph/pull/44310, which could not be merged
because the commit message associated with that PR did not begin with
the string "doc", which meant that the "Signed-off-by" check failed.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

test/librados/aio_cxx: use fmt formatting

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/client_request: Support FLAG_EIO

See 5ac9f523ea263eeb747e2543a667a25de67d07b8

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #49025 from zdover23/wip-doc-2022-11-23-glossary-ceph-client

doc/glossary: fix "Ceph Client"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49027 from zdover23/wip-doc-2022-11-23-glossary-ceph-object-store

doc/glossary: add "Ceph Object Store"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48859 from yanghonggang/yhg-code-cleanup

osd/ECBackend: remove unused variables

Reviewed-by: Omri Zeneva <ozeneva@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49010 from tchaikov/more-than-future

crimson/common: use concept for type constraints

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #49005 from CHOKYUJIN/main

doc/dev: fix a typo in deduplication.rst

test/librados/aio_cxx: Simple PoolEIOFlag is added

Useful for debugging.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/librados/aio_cxx: Fix PoolEIOFlag test

* wait_for_complete is added before getting the return value.

* for loop iteration limit is added to avoid infinite run.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

doc/glossary: add "Ceph Object Store"

Add a definition for "Ceph Object Store".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/common: use concept for type constraints

instead of using SFINAE, use concept for specifying the type constraints
for better readability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48896 from yuvalif/wip-yuval-fix-50076

rgw/kafka: route librdkafka log messages to rgw log

Merge pull request #48877 from yuvalif/wip-yuval-fix-58020

rgw/notifications: send mtime in complete multipart upload event

Merge pull request #48758 from yuvalif/wip-yuval-kafka-tests

rgw/notification/test: add support for testing kafka security

doc/glossary: fix "Ceph Client"

Add detail to and clean the syntactics and semantics of "Ceph Client".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/dev: fix a typo in deduplication.rst

Signed-off-by: Kyujin Cho <bori19960@snu.ac.kr>

crimson/net: fix use-after-free during yielded closing

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/common: re-implement do_for_each

The current implementation of crimson::do_for_each might meet
stack overflow when future is available but seastar::need_preempt
is true. This new implementation mirror to the seastar::do_for_each
with crimson errorator mechanism will solve this problem.

Fixes: https://tracker.ceph.com/issues/58005.
Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #49011 from tchaikov/wip-crimson-fmt-v9

crimson: various fmt related changes and fixes.

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #49017 from zdover23/wip-doc-2022-11-23-glossary-cluster-map

doc/glossary: s/comprising/consisting of/

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: s/comprising/consisting of/

s/comprising/consisting of/ - Make the text easier to read for
non-native or weak readers of English.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/cephfs: correct names for fs/meta

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #49001 from tchaikov/wip-fmt-v9

*: specialize fmt::formatter<> for formatting using {fmt}

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os: print exception using e.what()

see also e6223872c58e492d3570335fd7f05bb52641837c

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: print pointer using fmt::ptr()

also, partial specialize fmt::format<> for boost::instrusive_ptr<>,
so we can use fmt::ptr() to print boost::intrusive_ptr<>, just like
how we print std::shared_ptr<>.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48984 from Matan-B/wip-matanb-unit-debug-lvl

doc/dev/developer_guide/tests-unit-tests: Add unit test caveat

Reviewed-by: Zac Dover <zac.dover@gmail.com>

crimson/os: print the value of optional<>

if an optional is known to have value, we can just print its value.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: specialize fmt::formatter<>

so the specialized types can be formatted using {fmt} v9

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

*: specialize fmt::formatter<> for formatting using {fmt}

in {fmt} v9, fmt::formatter<> is not defined automatically for
the types with operator<<, we need to define them explictly. so in this
change, the types printed using `fmt::print()` or `fmt::format()` are
equiped with fmt::formatter<> to appease {fmt} v9.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/osd/pg: with_locked_obc() avoid head/clone distinguish

The head or clone branch seperation can be done later on
(in with_obc()).

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/object_context_loader: Introduce with_clone_obc_only()

A with_clone_obc() variant, in PGBackend::rollback() with_locked_obc
is locking the head obc used, we should not lock the head object
obc again (as in by calling with_head_obc()).

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg*: Move resolve_oid() to object_context

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: Use ObjectContextLoader::notify_on_change()

obc_set_accessing is owned by ObjectContextLoader and is
removed from PG.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: Introduce ObjectContextLoader

* obc load logic is moved to a seperate class for better reusability.

* with_existing_* logic is removed. (not being used)

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/os/seastore: add reference to the iterator of segment_map_t

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #49000 from tchaikov/wip-crimson-fmt-v9

crimson: various fmt related changes and fixes.

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>