crimson/os/seastore/omap: adjust omap_list_config_t

Add inclusive behavior for both first and last in omap_list_config_t,
and optimize the usage.
Set first closed last open for range for omap range remove.

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore: introduce omap range remove

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore/omap: add tests for range remove

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore/omap: introduce omap range remove

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore/omap: add tests to range list

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/os/seastore/omap: support two borders list method

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #48927 from neesingh-rh/fix_58028

cephfs-top: Fix unresposive sorting in some cases

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #48542 from CongMinYin/fix-inconsistent-pwl-status

librbd/cache/pwl: fix clean vs bytes_dirty cache state inconsistency

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cephfs-top: Fix unresposive sorting in some cases

Sorting is not working properly in case of change
in the number of clients or after removing and
creating the same filesystem again.

Fixes: https://tracker.ceph.com/issues/58028
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #49026 from neesingh-rh/fix_58031

cephfs-top: sorting excepts when the filesystems are removed and created

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #49029 from Matan-B/wip-matanb-EIO-flag

crimson/osd: Support Pool EIO flag

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48932 from aisakaki/wip-crimson-loop

crimson/common: re-implement do_for_each

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49040 from tchaikov/wip-cephadm-flake8

cephadm: pin flake8 to 5.0.4

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

pybind/mgr: use annotation style without comment

* use annotation style without comment
* add noqa to import statement to prevent flake8 from complaining
  those which import typing modules

to silence the warnings like:

```
flake8 run-test: commands[0] | flake8 --config=tox.ini alerts balancer cephadm cli_api crash devicehealth diskprediction_local hello iostat localpool nfs orchestrator prometheus selftest
cephadm/schedule.py:5:1: F401 'typing.Callable' imported but unused
cephadm/schedule.py:8:1: F401 'ceph.deployment.service_spec.ServiceSpec' imported but unused
cephadm/tests/fixtures.py:17:1: F401 'orchestrator.OrchResult' imported but unused
orchestrator/module.py:4:1: F401 'typing.Set' imported but unused
orchestrator/module.py:17:1: F401 'ceph.deployment.inventory.Device' imported but unused
prometheus/module.py:17:1: F401 'typing.DefaultDict' imported but unused
6     F401 'typing.Callable' imported but unused
ERROR: InvocationError for command /home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/.tox/flake8/bin/flake8 --config=tox.ini alerts
```

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

qa/tasks/cephfs: annotate variable without comment

to silence the warning like
```
flake8 run-test: commands[0] | flake8 --select=F,E9 --exclude=venv,.tox
./tasks/cephfs/test_full.py:6:5: F401 'typing.Optional' imported but unused
```

since we've migrated to python3.6 and up, there is no need to be
compatible with Python 2 anymore.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cephadm: pin flake8 to 5.0.4

to workaround https://github.com/zheller/flake8-quotes/issues/110
flake8 v5.0.4 is the last known-to-work flake8 before flake8 v6.0.0.

Fixes: https://tracker.ceph.com/issues/58069
Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cephfs-top: sorting excepts when the filesystems are removed and created

Fixes: https://tracker.ceph.com/issues/58031
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #49022 from cyx1231st/wip-seastar-msgr-fix-yielded-closing

crimson/net: fix use-after-free during yielded closing

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49037 from zdover23/wip-doc-2022-11-24-container-install-consume-44310

doc/install: update "Official Releases" sources

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/install: update "Official Releases" sources

Remove "Docker Hub" from the list of locations from which Ceph can be
acquired.

This commit represents a second attempt to get
https://github.com/ceph/ceph/pull/49034 past the Jenkins CI tests, and
that PR was itself an attempt to incorporate the change in
https://github.com/ceph/ceph/pull/44310, which could not be merged
because the commit message associated with that PR did not begin with
the string "doc", which meant that the "Signed-off-by" check failed.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

test/librados/aio_cxx: use fmt formatting

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/client_request: Support FLAG_EIO

See 5ac9f523ea263eeb747e2543a667a25de67d07b8

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #49025 from zdover23/wip-doc-2022-11-23-glossary-ceph-client

doc/glossary: fix "Ceph Client"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49027 from zdover23/wip-doc-2022-11-23-glossary-ceph-object-store

doc/glossary: add "Ceph Object Store"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48859 from yanghonggang/yhg-code-cleanup

osd/ECBackend: remove unused variables

Reviewed-by: Omri Zeneva <ozeneva@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #49010 from tchaikov/more-than-future

crimson/common: use concept for type constraints

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #49005 from CHOKYUJIN/main

doc/dev: fix a typo in deduplication.rst

test/librados/aio_cxx: Simple PoolEIOFlag is added

Useful for debugging.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/librados/aio_cxx: Fix PoolEIOFlag test

* wait_for_complete is added before getting the return value.

* for loop iteration limit is added to avoid infinite run.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

doc/glossary: add "Ceph Object Store"

Add a definition for "Ceph Object Store".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/common: use concept for type constraints

instead of using SFINAE, use concept for specifying the type constraints
for better readability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48896 from yuvalif/wip-yuval-fix-50076

rgw/kafka: route librdkafka log messages to rgw log

Merge pull request #48877 from yuvalif/wip-yuval-fix-58020

rgw/notifications: send mtime in complete multipart upload event

Merge pull request #48758 from yuvalif/wip-yuval-kafka-tests

rgw/notification/test: add support for testing kafka security

doc/glossary: fix "Ceph Client"

Add detail to and clean the syntactics and semantics of "Ceph Client".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/dev: fix a typo in deduplication.rst

Signed-off-by: Kyujin Cho <bori19960@snu.ac.kr>

crimson/net: fix use-after-free during yielded closing

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/common: re-implement do_for_each

The current implementation of crimson::do_for_each might meet
stack overflow when future is available but seastar::need_preempt
is true. This new implementation mirror to the seastar::do_for_each
with crimson errorator mechanism will solve this problem.

Fixes: https://tracker.ceph.com/issues/58005.
Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #49011 from tchaikov/wip-crimson-fmt-v9

crimson: various fmt related changes and fixes.

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #49017 from zdover23/wip-doc-2022-11-23-glossary-cluster-map

doc/glossary: s/comprising/consisting of/

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: s/comprising/consisting of/

s/comprising/consisting of/ - Make the text easier to read for
non-native or weak readers of English.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49001 from tchaikov/wip-fmt-v9

*: specialize fmt::formatter<> for formatting using {fmt}

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os: print exception using e.what()

see also e6223872c58e492d3570335fd7f05bb52641837c

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os: print pointer using fmt::ptr()

also, partial specialize fmt::format<> for boost::instrusive_ptr<>,
so we can use fmt::ptr() to print boost::intrusive_ptr<>, just like
how we print std::shared_ptr<>.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48984 from Matan-B/wip-matanb-unit-debug-lvl

doc/dev/developer_guide/tests-unit-tests: Add unit test caveat

Reviewed-by: Zac Dover <zac.dover@gmail.com>

crimson/os: print the value of optional<>

if an optional is known to have value, we can just print its value.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: specialize fmt::formatter<>

so the specialized types can be formatted using {fmt} v9

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

*: specialize fmt::formatter<> for formatting using {fmt}

in {fmt} v9, fmt::formatter<> is not defined automatically for
the types with operator<<, we need to define them explictly. so in this
change, the types printed using `fmt::print()` or `fmt::format()` are
equiped with fmt::formatter<> to appease {fmt} v9.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os/seastore: add reference to the iterator of segment_map_t

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #49000 from tchaikov/wip-crimson-fmt-v9

crimson: various fmt related changes and fixes.

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: print oid of missing clone

instead of printing the empty optional<>, print the oid of missing
clone. the oid is more interesting in this context.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: specialize fmt::formatter<> to print using {fmt}

in {fmt} v9, fmt::formatter<> is not defined automatically for
the types with operator<<, we need to define them explictly. so in this
change, the types printed using `fmt::print()` or `fmt::format()` are
equiped with fmt::formatter<> to appease {fmt} v9.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/net: print shared_ptr using fmt::ptr()

in {fmt} v9, only `void*` pointers can be printed directly. when it
comes to typed pointer or smart pointers, we need to print them using
`fmt::ptr()`. in this change, an overload is added so we can print
`seastar::shared_ptr<>` via `fmt::ptr()` as well, just like how we
print `std::shared_ptr<>`.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/osd: add formatter for std::optional<>

in `PG::do_update_log_missing()`, `std::optional<eversion_t>` is printed
using {fmt}, but {fmt} does not support formatting `std::optional<>`. so
we need to swing our own formatter. we can promote this formatter to
a more common place if it is proved to be useful otherwhere in the
source tree.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: print exceptions using exception.what()

{fmt} does not support formatting exceptions until
https://github.com/fmtlib/fmt/pull/3062/commits/76adb05a78a026b94c2ecadd20c00ae0c33c5a37
so we need to print them via exception.what().

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/common: print std::exception_ptr after conveting it to exception

fmt does not support formatting `std::exception_ptr`, so we need to
rethrow it, so we can convert it to a known exception type.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48987 from tchaikov/cmake-cares

cmake: s/c-ares::c-ares/c-ares::cares/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48998 from zdover23/wip-doc-2022-11-22-glossary-ceph-storage-cluster

doc/glossary: define "Ceph Storage Cluster"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: define "Ceph Storage Cluster"

Add "Ceph Storage Cluster" to the glossary.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48773 from myoungwon/wip-fix-out-date-delta

crimson/os/seastore: add deallocation map during replay to filter out out-dated delta

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #48837 from badone/wip-ceph-debug-docker-ubuntu-src.list

scripts: Add Ubuntu source repo entry to ceph-debug-docker

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #48989 from zdover23/wip-doc-2022-11-22-glossary-node

doc/glossary: Define "Ceph Node"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48990 from zdover23/wip-doc-2022-11-22-glossary-cluster-map-oxford-comma

doc/glossary: add oxford comma to "Cluster Map"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

cmake: s/c-ares::c-ares/c-ares::cares/

to be aligned with the c-ares upstream, which provides
c-ares-config.cmake in c-ares v1.17 and up. this cmake config file
is also packaged by fedora and CentOS stream in c-ares-devel-1.17.1 and
up.

this change prepares us for bumping up Seastar submodule, which also
has a similar change: https://github.com/scylladb/seastar/commit/aea45c98f255e2013d4b9b399944e9810ee1d972

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

doc/glossary: add oxford comma to "Cluster Map"

Add an Oxford comma to a list under the "Cluster Map" headword in the
glossary, so that it doesn't read like "I'd like to thank my parents,
Ayn Rand and God."

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: Define "Ceph Node"

Add a definition of "Ceph Node" to the glossary.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48983 from zdover23/wip-doc-2022-11-21-rados-ops-add-or-rm-mons-prompts-2-of-2

doc/rados: add prompts to add-or-rm-prompts.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev/developer_guide/tests-unit-tests: Add unit test caveat

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #48810 from stefan-chivu/max-path-len-increase

dokan: Configurable maximum path length

Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

doc/rados: add prompts to add-or-rm-prompts.rst

Add unselectable prompts to add-or-rm-prompts.rst. This is part 2 of 2.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48664 from joscollin/wip-cephfs-top-drop-x-coord-map

cephfs-top: drop x_coord_map

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #48850 from joscollin/wip-fix-clear-screen-no-fs

cephfs-top: run_display: clear the screen and block commands when no fs

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #48956 from zdover23/wip-doc-2022-11-19-src-pybind-cephfs-cephfs-grammar

src/pybind/cephfs: fix grammar

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

rgw/notification/test: avoid kafka ssl tests in teuthology

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

test/dokan: Added dokan max path test

Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

ceph-dokan: Made max path length configurable

Added ceph-dokan CLI optarg for configuring the value of
the maximum path length. By default, it will be set to 256 and
it will have a maximum value of 32767.

Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

cephfs-top: drop x_coord_map

As we started using curses pads in PR#48090, we don't need x_coord_map anymore.
In this patch, the metrics are positioned in cephfs-top based on padding (ITEMS_PAD).

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #48910 from joscollin/wip-fix-header-show-filters

cephfs-top: display filters in the header

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #48978 from zdover23/wip-doc-2022-11-21-rados-ops-add-or-rm-mons-prompts-1-of-2

doc/rados: add prompts to add-or-rm-prompts.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

crimson/os/seastore/cbj: add deallocation map during replay to filter out out-dated delta

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

doc/rados: add prompts to add-or-rm-prompts.rst

Add unselectable prompts to add-or-rm-prompts.rst. This commit covers
the first 300 lines of the file. This is part 1 of 2.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48975 from zdover23/wip-doc-2022-11-21-glossary-SDS

doc/glossary: Add "SDS"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: Add "SDS"

Add "SDS" to the glossary.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48949 from ktdreyer/doc-windows-drive-letter

doc: display Windows RBD drive letter

Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #48968 from zdover23/wip-doc-2022-11-20-rbd-immutable-object-cache-daemon-grammar

doc/rbd: improve grammar in "immutable object..."

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/rbd: improve grammar in "immutable object..."

Improve the grammar in the "Immutable Object Cache Daemon" section.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48962 from zdover23/wip-doc-2022-11-20-radosgw-object-gateway-grammar

doc/radosw: improve radosgw text

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #48963 from zdover23/wip-doc-2022-11-20-glossary-ceph-block-storage

doc/glossary: beef up "Ceph Block Storage"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: beef up "Ceph Block Storage"

Beef up the "Ceph Block Storage" entry.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/radosw: improve radosgw text

Punch up the paragraph about the Ceph Object Gateway daemon (radosgw).

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48959 from zdover23/wip-doc-2022-11-20-glossary-radosgw

doc/glossary: define "RGW"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: define "RGW"

Add definition under the headword "RGW".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

src/pybind/cephfs: fix grammar

Fix the grammar in the explanation of the lstat() API call.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48942 from zdover23/wip-doc-2022-11-18-src-pybind-typo

src/pybind: fix typo in cephfs.pyx

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #48939 from zdover23/wip-doc-2022-11-18-glossary-rados

doc/glossary: add "RADOS" definition

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: add "RADOS" definition

Define "RADOS" in the glossary. We will return in the next month to beef
this entry up.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48941 from zdover23/wip-doc-2022-11-18-releases-quincy

doc/releases: fix prompts in quincy.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc: display Windows RBD drive letter

Give the Powershell command that identifies the Windows drive letter
when partitioning a new RBD image.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #48938 from ljflores/wip-curl-error

qa/workunits/rados: specify redirect in curl command

Merge pull request #48940 from kelubo/main

doc: Delete a duplicate row.