doc/dev: improve Basic Workflow wording

Make several semantic and syntactic improvements to basic-workflow.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49066 from zdover23/wip-doc-2022-11-26-glossary-ceph-system-remove

doc/glossary: remove "Ceph System"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49065 from zdover23/wip-doc-2022-11-26-rados-ops-add-or-rm-osds-prompts

doc/rados: add prompts to add-or-remove-osds

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49045 from zdover23/wip-doc-2022-11-25-rbd-iscsi-initiator

doc/rbd: update iSCSI gateway info

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/rbd: update iSCSI gateway info

Update iSCSI gateway info in the RBD documentation.

https://tracker.ceph.com/issues/58080

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: remove "Ceph System"

"Ceph System" appears only in definitions where "Ceph Cluster" should
be, so I'm removing the headword. A future commit will scrub the
glossary of the phrase "Ceph System".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/rados: add prompts to add-or-remove-osds

Add unselectable prompts to add-or-rm-osds.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49062 from zdover23/wip-doc-2022-11-26-glossary-ceph-manager

doc/glossary: link to "Ceph Manager"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: link to "Ceph Manager"

Add link to "Ceph Manager" material.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48761 from soumyakoduri/wip-skoduri-cloudtier-fixes

rgw/cloud-transition: Fix issues with MCG endpoint

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49036 from idryomov/wip-52932

mgr/rbd_support: avoid wedging the task queue if pool is removed

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #49042 from idryomov/wip-rbd-support-flake8

mgr/rbd_support: clean up and enable flake8

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #49046 from zdover23/wip-doc-2022-11-25-rados-ops-erasure-code-profiles

doc/rados: cleanup "erasure code profiles"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: cleanup "erasure code profiles"

Improve the grammar of "Erasure Code Profiles".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48927 from neesingh-rh/fix_58028

cephfs-top: Fix unresposive sorting in some cases

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #48542 from CongMinYin/fix-inconsistent-pwl-status

librbd/cache/pwl: fix clean vs bytes_dirty cache state inconsistency

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cephfs-top: Fix unresposive sorting in some cases

Sorting is not working properly in case of change
in the number of clients or after removing and
creating the same filesystem again.

Fixes: https://tracker.ceph.com/issues/58028
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #49026 from neesingh-rh/fix_58031

cephfs-top: sorting excepts when the filesystems are removed and created

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

mgr/rbd_support: clean up and enable flake8

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49029 from Matan-B/wip-matanb-EIO-flag

crimson/osd: Support Pool EIO flag

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48932 from aisakaki/wip-crimson-loop

crimson/common: re-implement do_for_each

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49040 from tchaikov/wip-cephadm-flake8

cephadm: pin flake8 to 5.0.4

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

pybind/mgr: use annotation style without comment

* use annotation style without comment
* add noqa to import statement to prevent flake8 from complaining
  those which import typing modules

to silence the warnings like:

```
flake8 run-test: commands[0] | flake8 --config=tox.ini alerts balancer cephadm cli_api crash devicehealth diskprediction_local hello iostat localpool nfs orchestrator prometheus selftest
cephadm/schedule.py:5:1: F401 'typing.Callable' imported but unused
cephadm/schedule.py:8:1: F401 'ceph.deployment.service_spec.ServiceSpec' imported but unused
cephadm/tests/fixtures.py:17:1: F401 'orchestrator.OrchResult' imported but unused
orchestrator/module.py:4:1: F401 'typing.Set' imported but unused
orchestrator/module.py:17:1: F401 'ceph.deployment.inventory.Device' imported but unused
prometheus/module.py:17:1: F401 'typing.DefaultDict' imported but unused
6     F401 'typing.Callable' imported but unused
ERROR: InvocationError for command /home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/.tox/flake8/bin/flake8 --config=tox.ini alerts
```

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

qa/tasks/cephfs: annotate variable without comment

to silence the warning like
```
flake8 run-test: commands[0] | flake8 --select=F,E9 --exclude=venv,.tox
./tasks/cephfs/test_full.py:6:5: F401 'typing.Optional' imported but unused
```

since we've migrated to python3.6 and up, there is no need to be
compatible with Python 2 anymore.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cephadm: pin flake8 to 5.0.4

to workaround https://github.com/zheller/flake8-quotes/issues/110
flake8 v5.0.4 is the last known-to-work flake8 before flake8 v6.0.0.

Fixes: https://tracker.ceph.com/issues/58069
Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cephfs-top: sorting excepts when the filesystems are removed and created

Fixes: https://tracker.ceph.com/issues/58031
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #49022 from cyx1231st/wip-seastar-msgr-fix-yielded-closing

crimson/net: fix use-after-free during yielded closing

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

mgr/rbd_support: avoid wedging the task queue if pool is removed

rados.ObjectNotFound exception handler was referencing ioctx variable
which is assigned only if the pool exists and rados.open_ioctx() call
succeeds.  This lead to a fatal error

  mgr[rbd_support] Failed to locate pool mypool
  mgr[rbd_support] execute_task: [errno 2] error opening pool 'b'mypool''
  mgr[rbd_support] Fatal runtime error: local variable 'ioctx' referenced before assignment

and wedged the task queue.  No other commands were processed until
ceph-mgr daemon restart.

Fixes: https://tracker.ceph.com/issues/52932
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49037 from zdover23/wip-doc-2022-11-24-container-install-consume-44310

doc/install: update "Official Releases" sources

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/install: update "Official Releases" sources

Remove "Docker Hub" from the list of locations from which Ceph can be
acquired.

This commit represents a second attempt to get
https://github.com/ceph/ceph/pull/49034 past the Jenkins CI tests, and
that PR was itself an attempt to incorporate the change in
https://github.com/ceph/ceph/pull/44310, which could not be merged
because the commit message associated with that PR did not begin with
the string "doc", which meant that the "Signed-off-by" check failed.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

test/librados/aio_cxx: use fmt formatting

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/client_request: Support FLAG_EIO

See 5ac9f523ea263eeb747e2543a667a25de67d07b8

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #49025 from zdover23/wip-doc-2022-11-23-glossary-ceph-client

doc/glossary: fix "Ceph Client"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49027 from zdover23/wip-doc-2022-11-23-glossary-ceph-object-store

doc/glossary: add "Ceph Object Store"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48859 from yanghonggang/yhg-code-cleanup

osd/ECBackend: remove unused variables

Reviewed-by: Omri Zeneva <ozeneva@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49010 from tchaikov/more-than-future

crimson/common: use concept for type constraints

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #49005 from CHOKYUJIN/main

doc/dev: fix a typo in deduplication.rst

test/librados/aio_cxx: Simple PoolEIOFlag is added

Useful for debugging.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/librados/aio_cxx: Fix PoolEIOFlag test

* wait_for_complete is added before getting the return value.

* for loop iteration limit is added to avoid infinite run.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

doc/glossary: add "Ceph Object Store"

Add a definition for "Ceph Object Store".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/common: use concept for type constraints

instead of using SFINAE, use concept for specifying the type constraints
for better readability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48896 from yuvalif/wip-yuval-fix-50076

rgw/kafka: route librdkafka log messages to rgw log

Merge pull request #48877 from yuvalif/wip-yuval-fix-58020

rgw/notifications: send mtime in complete multipart upload event

Merge pull request #48758 from yuvalif/wip-yuval-kafka-tests

rgw/notification/test: add support for testing kafka security

doc/glossary: fix "Ceph Client"

Add detail to and clean the syntactics and semantics of "Ceph Client".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/dev: fix a typo in deduplication.rst

Signed-off-by: Kyujin Cho <bori19960@snu.ac.kr>

crimson/net: fix use-after-free during yielded closing

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/common: re-implement do_for_each

The current implementation of crimson::do_for_each might meet
stack overflow when future is available but seastar::need_preempt
is true. This new implementation mirror to the seastar::do_for_each
with crimson errorator mechanism will solve this problem.

Fixes: https://tracker.ceph.com/issues/58005.
Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #49011 from tchaikov/wip-crimson-fmt-v9

crimson: various fmt related changes and fixes.

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #49017 from zdover23/wip-doc-2022-11-23-glossary-cluster-map

doc/glossary: s/comprising/consisting of/

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: s/comprising/consisting of/

s/comprising/consisting of/ - Make the text easier to read for
non-native or weak readers of English.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49001 from tchaikov/wip-fmt-v9

*: specialize fmt::formatter<> for formatting using {fmt}

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os: print exception using e.what()

see also e6223872c58e492d3570335fd7f05bb52641837c

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: print pointer using fmt::ptr()

also, partial specialize fmt::format<> for boost::instrusive_ptr<>,
so we can use fmt::ptr() to print boost::intrusive_ptr<>, just like
how we print std::shared_ptr<>.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48984 from Matan-B/wip-matanb-unit-debug-lvl

doc/dev/developer_guide/tests-unit-tests: Add unit test caveat

Reviewed-by: Zac Dover <zac.dover@gmail.com>

crimson/os: print the value of optional<>

if an optional is known to have value, we can just print its value.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: specialize fmt::formatter<>

so the specialized types can be formatted using {fmt} v9

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

*: specialize fmt::formatter<> for formatting using {fmt}

in {fmt} v9, fmt::formatter<> is not defined automatically for
the types with operator<<, we need to define them explictly. so in this
change, the types printed using `fmt::print()` or `fmt::format()` are
equiped with fmt::formatter<> to appease {fmt} v9.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os/seastore: add reference to the iterator of segment_map_t

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #49000 from tchaikov/wip-crimson-fmt-v9

crimson: various fmt related changes and fixes.

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: print oid of missing clone

instead of printing the empty optional<>, print the oid of missing
clone. the oid is more interesting in this context.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: specialize fmt::formatter<> to print using {fmt}

in {fmt} v9, fmt::formatter<> is not defined automatically for
the types with operator<<, we need to define them explictly. so in this
change, the types printed using `fmt::print()` or `fmt::format()` are
equiped with fmt::formatter<> to appease {fmt} v9.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/net: print shared_ptr using fmt::ptr()

in {fmt} v9, only `void*` pointers can be printed directly. when it
comes to typed pointer or smart pointers, we need to print them using
`fmt::ptr()`. in this change, an overload is added so we can print
`seastar::shared_ptr<>` via `fmt::ptr()` as well, just like how we
print `std::shared_ptr<>`.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/osd: add formatter for std::optional<>

in `PG::do_update_log_missing()`, `std::optional<eversion_t>` is printed
using {fmt}, but {fmt} does not support formatting `std::optional<>`. so
we need to swing our own formatter. we can promote this formatter to
a more common place if it is proved to be useful otherwhere in the
source tree.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: print exceptions using exception.what()

{fmt} does not support formatting exceptions until
https://github.com/fmtlib/fmt/pull/3062/commits/76adb05a78a026b94c2ecadd20c00ae0c33c5a37
so we need to print them via exception.what().

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/common: print std::exception_ptr after conveting it to exception

fmt does not support formatting `std::exception_ptr`, so we need to
rethrow it, so we can convert it to a known exception type.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48987 from tchaikov/cmake-cares

cmake: s/c-ares::c-ares/c-ares::cares/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48998 from zdover23/wip-doc-2022-11-22-glossary-ceph-storage-cluster

doc/glossary: define "Ceph Storage Cluster"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

rgw/cloud-transition: Fix issues with MCG endpoint

Few s3 endpoints (like Noobaa MCG or AWS) may return BucketAlreadyOwned
or BucketAlreadyExists error if the target path (configured for
cloud transition) already exists. Do not fail transition in those cases.

Also fixed below issues -
* with racing lc threads, when one lc thread is trying to check and create the
target bucket, others may go ahead with the object transition assuming target path
exists.
* initialize few fields to avoid setting wrong content_len during transition

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

doc/glossary: define "Ceph Storage Cluster"

Add "Ceph Storage Cluster" to the glossary.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48773 from myoungwon/wip-fix-out-date-delta

crimson/os/seastore: add deallocation map during replay to filter out out-dated delta

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #48837 from badone/wip-ceph-debug-docker-ubuntu-src.list

scripts: Add Ubuntu source repo entry to ceph-debug-docker

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #48989 from zdover23/wip-doc-2022-11-22-glossary-node

doc/glossary: Define "Ceph Node"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48990 from zdover23/wip-doc-2022-11-22-glossary-cluster-map-oxford-comma

doc/glossary: add oxford comma to "Cluster Map"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

cmake: s/c-ares::c-ares/c-ares::cares/

to be aligned with the c-ares upstream, which provides
c-ares-config.cmake in c-ares v1.17 and up. this cmake config file
is also packaged by fedora and CentOS stream in c-ares-devel-1.17.1 and
up.

this change prepares us for bumping up Seastar submodule, which also
has a similar change: https://github.com/scylladb/seastar/commit/aea45c98f255e2013d4b9b399944e9810ee1d972

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

doc/glossary: add oxford comma to "Cluster Map"

Add an Oxford comma to a list under the "Cluster Map" headword in the
glossary, so that it doesn't read like "I'd like to thank my parents,
Ayn Rand and God."

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: Define "Ceph Node"

Add a definition of "Ceph Node" to the glossary.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48983 from zdover23/wip-doc-2022-11-21-rados-ops-add-or-rm-mons-prompts-2-of-2

doc/rados: add prompts to add-or-rm-prompts.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev/developer_guide/tests-unit-tests: Add unit test caveat

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #48810 from stefan-chivu/max-path-len-increase

dokan: Configurable maximum path length

Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

doc/rados: add prompts to add-or-rm-prompts.rst

Add unselectable prompts to add-or-rm-prompts.rst. This is part 2 of 2.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48664 from joscollin/wip-cephfs-top-drop-x-coord-map

cephfs-top: drop x_coord_map

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #48850 from joscollin/wip-fix-clear-screen-no-fs

cephfs-top: run_display: clear the screen and block commands when no fs

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #48956 from zdover23/wip-doc-2022-11-19-src-pybind-cephfs-cephfs-grammar

src/pybind/cephfs: fix grammar

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

rgw/notification/test: avoid kafka ssl tests in teuthology

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

test/dokan: Added dokan max path test

Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

ceph-dokan: Made max path length configurable

Added ceph-dokan CLI optarg for configuring the value of
the maximum path length. By default, it will be set to 256 and
it will have a maximum value of 32767.

Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

cephfs-top: drop x_coord_map

As we started using curses pads in PR#48090, we don't need x_coord_map anymore.
In this patch, the metrics are positioned in cephfs-top based on padding (ITEMS_PAD).

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #48910 from joscollin/wip-fix-header-show-filters

cephfs-top: display filters in the header

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #48978 from zdover23/wip-doc-2022-11-21-rados-ops-add-or-rm-mons-prompts-1-of-2

doc/rados: add prompts to add-or-rm-prompts.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

crimson/os/seastore/cbj: add deallocation map during replay to filter out out-dated delta

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

doc/rados: add prompts to add-or-rm-prompts.rst

Add unselectable prompts to add-or-rm-prompts.rst. This commit covers
the first 300 lines of the file. This is part 1 of 2.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48975 from zdover23/wip-doc-2022-11-21-glossary-SDS

doc/glossary: Add "SDS"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: Add "SDS"

Add "SDS" to the glossary.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48949 from ktdreyer/doc-windows-drive-letter

doc: display Windows RBD drive letter

Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #48968 from zdover23/wip-doc-2022-11-20-rbd-immutable-object-cache-daemon-grammar

doc/rbd: improve grammar in "immutable object..."

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/rbd: improve grammar in "immutable object..."

Improve the grammar in the "Immutable Object Cache Daemon" section.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48962 from zdover23/wip-doc-2022-11-20-radosgw-object-gateway-grammar

doc/radosw: improve radosgw text

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #48963 from zdover23/wip-doc-2022-11-20-glossary-ceph-block-storage

doc/glossary: beef up "Ceph Block Storage"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: beef up "Ceph Block Storage"

Beef up the "Ceph Block Storage" entry.

Signed-off-by: Zac Dover <zac.dover@gmail.com>