ceph-mixing: fix ceph_hosts variable

Do only use `instance` to query for hostnames in single-cluster-mode.
Consider the cluster matcher only in multi-cluster-mode. In this case
the query will look like:
`"label_values({cluster=~\"$cluster\"}, instance)"`.

Fixes: https://tracker.ceph.com/issues/57987
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #48840 from zdover23/wip-doc-2022-11-11-glossary-remove-test-framework

doc/glossary: remove "Ceph Test Framework"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: remove "Ceph Test Framework"

Remove "Ceph Test Framework" because it doesn't actually refer to
anything.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48832 from zdover23/wip-doc-2022-11-11-glossary-ceph-OSD-Daemon-Daemons-redirect

doc/glossary: redirect entries to "Ceph OSD"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: redirect entries to "Ceph OSD"

Redirect "Ceph OSD Daemon" and "Ceph OSD Daemons" to the entry for "Ceph
OSD", which is already defined.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48298 from alram/alram/osd-meta-created_at

osd: add created_at meta

Merge pull request #48044 from nkshirsagar/main

mon: bail from handle_command() if _generate_command_map() fails

Merge pull request #46036 from ifed01/wip-ifed-new-statfs-update

os/bluestore: get rid off statfs update on each txc

Merge pull request #48826 from jecluis/wip-rgw-fix-lua-ifdef

rgw/rados: drop unnecessary ifdef

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48694 from gaord/FIX_BUG_57954

mgr/rook:NFSRados constructor expects type of rados as a parameter instead of MgrModule.

Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

rgw/rados: drop unnecessary ifdef

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

Merge pull request #48823 from zdover23/wip-doc-2022-11-10-glossary-ceph-manager-dashboard

doc/glossary: improve "Ceph Manager Dashboard"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: improve "Ceph Manager Dashboard"

Redirect "Ceph Manager Dashboard" to the glossary term "Ceph Dashboard".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48819 from zdover23/wip-doc-2022-11-10-glossary-ceph

doc/glossary: improve "ceph" term

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: improve "ceph" term

Define "ceph" in glossary.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48809 from zdover23/wip-doc-2022-11-09-glossary-cluster-map-link-improvement

doc/glossary: improve "Ceph Manager" term

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: improve "Ceph Manager" term

Improve the glossary entry "Ceph Manager" by correcting the grammar and
properly linking to the "Ceph Manager" section of the Architecture
Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48735 from stefan-chivu/ceph-dokan-tests

test/dokan: Added ceph-dokan tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #48762 from cyx1231st/wip-seastore-define-generation

crimson/os/seastore: define the usage of generation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Zhang Song <zhangsong325@gmail.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #48793 from joscollin/wip-cephfs-top-version-revert

cephfs-top: revert add version in header

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #48713 from SUSE/wip-fix-CVE-2022-3650

ceph-crash: drop privleges to run as "ceph" user, rather than root (CVE-2022-3650)

Merge pull request #48792 from zdover23/wip-doc-2022-11-08-glossary-ceph-cluster-map

doc/glossary: update "Cluster Map"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48787 from zdover23/wip-doc-2022-11-08-architecture-RADOS-pdf-URL-correction

doc/architecture: correct PDF link

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Revert "doc: update cephfs-top.png"

This reverts commit 2233b68fe0efa3666f90296fd5df249a3f56bbe5.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Revert "cephfs-top: add version in header"

This reverts commit 7d2deda2b212a4b5c1b81737321fb4ac2ef10347.

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc/glossary: update "Cluster Map"

Improve "Cluster Map" in the glossary.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48769 from zdover23/wip-doc-2022-11-08-glossary-OSD

doc/glossary: disambiguate "OSD"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/architecture: correct PDF link

Correct PDF link from ceph.com website (old) to ceph.io website
(current).

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48137 from nSedrickm/color-contrast-accessibility-update

mgr/dashboard: improve level AA color contrast accessibility for dashboard pages and components

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48776 from zdover23/wip-doc-2022-11-08-rados-config-storage-devices-backend-correction

doc/rados: s/backend/back end/

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #48775 from zdover23/wip-doc-2022-11-08-glossary-OSD-link

doc/glossary: link to OSD material

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #48774 from zdover23/wip-doc-2022-11-08-glossary-BlueStore

doc/glossary: add "BlueStore"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/rados: s/backend/back end/

s/backend/back end/ in storage-devices.rst.
(IBM Style Guide, 305).

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: link to OSD material

Link to the OSD material in RADOS Configuration
"Storage Devices".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: add "BlueStore"

Define "BlueStore" in the glossary and link to the BlueStore
section in the RADOS Configuration "Storage Devices" chapter.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48768 from zdover23/wip-doc-2022-11-07-glossary-ceph-osd

doc/glossary: define "Ceph OSD"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: disambiguate "OSD"

Distinguish "object storage device" from "object storage daemon".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: define "Ceph OSD"

Define "Ceph OSD", using Sage's own description.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48022 from rhcs-dashboard/paginate-services

mgr/dashboard: paginate services

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48763 from zdover23/wip-doc-2022-11-07-glossary-ceph-manager

doc/glossary.rst: define "Ceph Manager"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary.rst: define "Ceph Manager"

Add a definition for Ceph Manager (ceph-mgr).

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/os/seastore: rename reclaim_gen_t to rewrite_gen_t

Generation is counting rewrites, not reclaims.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: define the usage of generation

Unify the definition of the reclaim generation, cleanup and explain its
intentions and usages. Please refer to the comments in seastore_types.h.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

ceph-dokan: Exposed --win-vol-serial CLI option for ceph-dokan

Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

test: Added ceph-dokan tests

Added tests for the following ceph-dokan functionalities:

- mounting a CephFS volume
- creating files & folders
- r/w from files
- find files
- move files
- get file type
- get/set file timestamps
- set file eof/allocation size
- get volume information
- get volume space information
- file create dispositions
- file delete disposition

Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

Merge pull request #48616 from rhcs-dashboard/mirror-promote-demote-bug

mgr/dashboard: bug fixes for rbd mirroring edit and promotion/demotion

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #48538 from Syphdias/add-bonding-alert

ceph-mixin: Add Prometheus Alert for Degraded Bond

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #48693 from ljx023/seastore-device-size-detect

crimson/os/seastore/segment_manager/block: get block device size

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #48757 from zdover23/wip-doc-2022-11-07-glossary-OSD

doc/glossary: define "OSD"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: define "OSD"

Define "OSD" as "Object Storage Device", distinct from
"object storage daemon".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/dashboard: improve level AA color contrast accessibility for dashboard pages and components
Use more accessible colors for dashboard components, text and table columns

Fixes: https://tracker.ceph.com/issues/56023
Signed-off-by: nsedrickm <nsedrick101@gmail.com>

Merge pull request #48753 from zdover23/wip-doc-2022-11-06-glossary-remove-front-matter

doc/glossary.rst: remove old front matter

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary.rst: remove old front matter

Remove two paragraphs from May 2013 that no longer apply.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48750 from zdover23/wip-doc-2022-11-06-glossary-various

doc/glossary: improve wording

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: improve wording

Remove "e.g." and "etc." and improve the wording of a definition.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48747 from zdover23/wip-doc-2022-11-05-glossary-dashboard

doc/glossary.rst: add "Ceph Dashboard" term

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary.rst: add "Ceph Dashboard" term

Add a definition for "Ceph Dashboard".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48744 from zdover23/wip-doc-2022-11-05-glossary-rbd

doc/glosary.rst: add "Ceph Block Device" term

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glosary.rst: add "Ceph Block Device" term

Add a definition for "Ceph Block Device".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48666 from joscollin/wip-cephfs-top-version

cephfs-top: add version in header

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #48291 from ljflores/wip-upgrade-telemetry

doc: update cephfs-top.png

Signed-off-by: Jos Collin <jcollin@redhat.com>

cephfs-top: add version in header

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #48739 from zdover23/wip-doc-2022-11-05-dev-guide-lone-sentence

doc/dev: improve lone sentence

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: improve lone sentence

Improve a lone sentence to make it less ambiguous, and at the same
time permit the testing of a script.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48736 from zdover23/wip-doc-2022-11-05-cephadm-host-management-removing-hosts

doc/cephadm: improve lone sentence

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/cephadm: improve lone sentence

Improve a lone sentence, both to make the cephadm documentation better
and to test a script that is under development.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48716 from tchaikov/crimson-cpuset

crimson/os/alienstore: parse crimson_alien_thread_cpu_cores in cpuset(7)'s list format

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #48111 from neesingh-rh/feature_55121

cephfs-top: addition of sort feature and limit option

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #48722 from zdover23/wip-doc-2022-11-04-cephadm-index-list-alphabetizing

doc/cephadm: alphabetize external tools list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48721 from zdover23/wip-doc-2022-11-04-cephadm-host-management-listing-hosts

doc/cephadm: arrange "listing hosts" section

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephadm: alphabetize external tools list

Alphabetize "Ansible", "Rook", and "Salt", so that
the items in the list match the items in the sentence
that precedes the list.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/cephadm: arrange "listing hosts" section

Collect the material at the top of this page into a
"Listing Hosts" section and clean the English so that
it is clearer.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc: addition of sort and limit features in doc

Fixes: https://tracker.ceph.com/issues/55121
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

cephfs-top: addition of sort and limit feature

This commit intends to add:
- sort-by field value feature to cephfs-top.
- feature to limit number of clients displayed

Fixes: https://tracker.ceph.com/issues/55121
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #48661 from thesamesam/gcc-13

common: Fix build with GCC 13 (missing <cstdint> include)

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: bug fixes for rbd mirroring edit and promotion/demotion

Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

crimson/os/alienstore: use boost::counting_iterator for setting cpuset

less repeatings this way.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os/alienstore: parse crimson_alien_thread_cpu_cores as a cpuset(7)

the "List format" listed in cpuset(7) allows us to specify a range of
CPU cores in a comma-separated list. and the upper bound of the range
is optional. before this change, the upper bound is not optional.

before this change, the upper bound of the range is not optional. and
the upper bound is not inclusive. so we don't support the list format
of cpuset(7). Take cores "1,2,3,5,7,8" for example, we need to set the
option to "1-4,5-6,7-9" to represent this cpuset.
after this change, the upper bound is optional, so we can properly
support the list format defined by cpuset(7). and the upper bound is
inclusive. so we can use "1-3,5,7-8", which is compatible with
notation defined by cpuset(7).

in this change, the cpuset option is parsed using a seastar helper,
which is implemented using a regex. so we don't need to manually parse
it. as Seastar's parser returns an `std::optional<std::set<unsigned>>`.
if the string does not match with the regex of comma-separated list,
the returned cpuset does not have a value. this design is more explicit.
so in this change, instead of using `std::vector<uint64_t>`,
`std::optional<cpuset>` is used.

Signed-off-by: Jianxin Li <jianxin1.li@intel.com>
Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48712 from zdover23/wip-doc-2022-11-03-cephadm-compatibility-grammar

doc/cephadm: fix grammar in compatibility.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/cephadm: fix grammar in compatibility.rst

Fix grammar in doc/cephadm/compatibility.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

ceph-crash: drop privleges to run as "ceph" user, rather than root

If privileges cannot be dropped, log an error and exit.  This commit
also catches and logs exceptions when scraping the crash path, without
which ceph-crash would just exit if it encountered an error.

Fixes: CVE-2022-3650
Fixes: https://tracker.ceph.com/issues/57967
Signed-off-by: Tim Serong <tserong@suse.com>

ceph-crash: fix stderr handling

Popen.communicate() returns a tuple (stdout, stderr), and stderr
will be of type bytes, hence the need to decode it before checking
if it's an empty string or not.

Fixes: a77b47eeeb5770eeefcf4619ab2105ee7a6a003e
Signed-off-by: Tim Serong <tserong@suse.com>

ceph-mixin: Add Prometheus Alert for Degraded Bond

Currently there is no alert for a network interface card to be misconfigured or
failed which is part of a network bond.

This could lead to redundancies and performance being degraded unnoticed.

To solve this, I use node exporter metrics to look at the number of total peers
of the bond and the ones that are active. If the numbers differ, something is up
and should be looked at.

Fixes: https://tracker.ceph.com/issues/57962
Signed-off-by: Christian Kugler <syphdias+git@gmail.com>

crimson/os/seastore/segment_manager/block: get block device size

Previously the file_stat(::stat) can not get correct size for block devices.
Now call the file.size(using ::lseek) to replace it.

Signed-off-by: Jianxin Li <jianxin1.li@intel.com>

Merge pull request #48699 from zdover23/wip-doc-2022-11-02-dev-osd-internals-ec-glossary-fixes

doc/dev: refine erasure_coding.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: refine erasure_coding.rst

Improve the readability and clarity of
erasure_coding.rst.

Co-author: Cole Mitchell <cole.mitchell@gmail.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/rook:NFSRados constructor expects type of rados as a parameter instead of MgrModule. This change should fix nfs cluster creation bug 57954.

Signed-off-by: Ben Gao <bengao168@msn.com>

Merge pull request #48687 from StevenWash/bugfix/fix_command_error_blank

doc/radosgw: fix command error blank

doc/radosgw: fix command error blank

Merge pull request #48684 from zdover23/wip-doc-2022-11-01-dev-osd-internals-ec-glossary-alphabetize

doc/dev: alphabetize EC glossary

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: alphabetize EC glossary

Alphabetize the glossary in
doc/dev/osd_internals/erasure_coding.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48673 from idryomov/wip-49947

doc: note architecture availability for PMEM mode of PWL cache

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #48548 from dparmar18/update_debug_script

script/ceph-debug-docker.sh: change default branch to main

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #48674 from zdover23/doc-2022-10-31-dev-osd-internals-ec-glossary

doc/dev: improve EC glossary

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: improve EC glossary

Improve the clarity and syntax of the text in
doc/dev/osd_internals/erasure_coding.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc: note architecture availability for PMEM mode of PWL cache

Fixes: https://tracker.ceph.com/issues/49947
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #48667 from zdover23/wip-doc-2022-10-30-ceph-volume-lvm-prepare-colemitchell-notes

doc/ceph-volume: refine Filestore docs

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/ceph-volume: refine Filestore docs

Add Cole Mitchell's improvements to the filestore section.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48665 from zdover23/wip-doc-2022-10-30-ceph-volume-lvm-prepare-datri-notes

doc/ceph-volume: improve prepare.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>