test/libcephfs: fix rebasing issues

Signed-off-by: Rishabh Dave <ridave@redhat.com>

libcephfs: replace errno.h errors with CEPHFS_E*

libcephfs and the associated tests were still using errno.h
values in a few places, which we're now going to replace with
portable CEPHFS_E* values.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

messages: avoid converting ceph errors on Windows

libclient and libcephfs return CEPHFS_E* errors, which are basically
Linux errno codes. If we convert mds errors to host errno values, we
end up mixing error codes.

For Windows, we'll preserve the original error value, which is expected
to be a Linux (CEPHFS_E*) error.

It may be worth doing the same for other platforms eventually.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

client: use CEPHFS_E*

The cephfs client uses CEPHFS_E* errors, except for a few methods
that are still using errno values.

We're addressing this inconsistency.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

test/libcephfs: use CEPHFS_E* errors

The cephfs client now uses CEPHFS_E errors (based on Linux)
instead of the platform specific errno values.

This change updates the libcephfs tests accordingly.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

libcephfs: switch to CEPHFS_E* errors

The cephfs client errors have been switched from errno values
(OS specific) to CEPHFS_E* errors (based on the Linux codes) [1][2].

The issue is that libcephfs still uses errno, which means that
non-Linux clients will get inconsistent errors.

This change updates libcephfs to use CEPHFS_E* errors and stay
consistent with the underlying client library.

[1] https://github.com/ceph/ceph/blob/6ef176b6745199224b62482a1241752826a5be14/src/libcephfs.cc#L203
[2] https://tracker.ceph.com/issues/48802

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

test/libcephfs: disable flaky timestamp assertion on Windows

The LibCephFS.SnapXattrs test takes two snapshots and then
compares the timestamps, expecting the latter one to be newer.

On Windows, the snapshot timestamps are sometimes the same,
in which case the test fails.

For now, we'll disable this check on Windows.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

client: use _setattrx when changing timestamps

_setattrx uses the ceph_statx structure while _setattr uses "stat",
which on some platforms (Windows) has lower time precision.

We'll switch some client.cc methods to using _setattrx on all
platforms since there's no downside in doing so and it improves
Windows timestamp precision.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

client: set nsec to 0 when converting stat struct on Windows

The Windows stat structure has limitted timestamp precision.
We'll ensure that the nsec field is properly initialized with 0.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

test/libcephfs: skip dirent inode check on Windows

The LibCephFS.DirLs test calls ceph_readdirplus_r and expects
the returning dirent.d_ino value to match the ceph_statx.stx_ino
inode.

The issue is that on Windows, "dirent" uses long (4B) inodes, which
isn't large enough to hold the ceph inodes (uint64_t).

For now, we'll drop this specific assertion. On Windows, the
ceph_statx inode should be used instead of the dirent one.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

client: avoid trimming inodes on Windows

On Windows, ino_t is defined as "unsigned short" (2B), which
isn't large enough to hold the Ceph inodes (uint64_t).

For this reason, we'll avoid using ino_t on Windows.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

test/libcephfs: address windows issues

This commit addresses a few issues that prevent the libcephfs from running
on Windows:

* added missing definitions to fs_compat.h
* the "nobody" user is ignored on Windows, which is why we'll skip the tests
  that rely on it
* sys/mman.h and sys/resource.h are not available on Windows and will be skipped
* skip rlimit, not available on Windows
* fork is not available on Windows, for now we'll avoid compiling those tests
  when targetting Windows. For what is worth, those are skipped on other
  platforms anyway but currently fail at compile time when using mingw.
* use "strchr" instead of "index" or "strchrnul"

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

test/libcephfs: include compat.h

Some type definitions are missing on Windows, which is why we'll
have to include the ceph compat.h header.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

test/libcephfs: enable the tests on Windows

This patch series enables the libcephfs tests to run on Windows,
addressing all the incompatibilities that we have encountered.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #48823 from zdover23/wip-doc-2022-11-10-glossary-ceph-manager-dashboard

doc/glossary: improve "Ceph Manager Dashboard"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: improve "Ceph Manager Dashboard"

Redirect "Ceph Manager Dashboard" to the glossary term "Ceph Dashboard".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48819 from zdover23/wip-doc-2022-11-10-glossary-ceph

doc/glossary: improve "ceph" term

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: improve "ceph" term

Define "ceph" in glossary.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48809 from zdover23/wip-doc-2022-11-09-glossary-cluster-map-link-improvement

doc/glossary: improve "Ceph Manager" term

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: improve "Ceph Manager" term

Improve the glossary entry "Ceph Manager" by correcting the grammar and
properly linking to the "Ceph Manager" section of the Architecture
Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48735 from stefan-chivu/ceph-dokan-tests

test/dokan: Added ceph-dokan tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #48762 from cyx1231st/wip-seastore-define-generation

crimson/os/seastore: define the usage of generation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Zhang Song <zhangsong325@gmail.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #48793 from joscollin/wip-cephfs-top-version-revert

cephfs-top: revert add version in header

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #48713 from SUSE/wip-fix-CVE-2022-3650

ceph-crash: drop privleges to run as "ceph" user, rather than root (CVE-2022-3650)

Merge pull request #48792 from zdover23/wip-doc-2022-11-08-glossary-ceph-cluster-map

doc/glossary: update "Cluster Map"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48787 from zdover23/wip-doc-2022-11-08-architecture-RADOS-pdf-URL-correction

doc/architecture: correct PDF link

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Revert "doc: update cephfs-top.png"

This reverts commit 2233b68fe0efa3666f90296fd5df249a3f56bbe5.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Revert "cephfs-top: add version in header"

This reverts commit 7d2deda2b212a4b5c1b81737321fb4ac2ef10347.

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc/glossary: update "Cluster Map"

Improve "Cluster Map" in the glossary.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48769 from zdover23/wip-doc-2022-11-08-glossary-OSD

doc/glossary: disambiguate "OSD"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/architecture: correct PDF link

Correct PDF link from ceph.com website (old) to ceph.io website
(current).

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48137 from nSedrickm/color-contrast-accessibility-update

mgr/dashboard: improve level AA color contrast accessibility for dashboard pages and components

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48776 from zdover23/wip-doc-2022-11-08-rados-config-storage-devices-backend-correction

doc/rados: s/backend/back end/

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #48775 from zdover23/wip-doc-2022-11-08-glossary-OSD-link

doc/glossary: link to OSD material

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #48774 from zdover23/wip-doc-2022-11-08-glossary-BlueStore

doc/glossary: add "BlueStore"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/rados: s/backend/back end/

s/backend/back end/ in storage-devices.rst.
(IBM Style Guide, 305).

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: link to OSD material

Link to the OSD material in RADOS Configuration
"Storage Devices".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: add "BlueStore"

Define "BlueStore" in the glossary and link to the BlueStore
section in the RADOS Configuration "Storage Devices" chapter.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48768 from zdover23/wip-doc-2022-11-07-glossary-ceph-osd

doc/glossary: define "Ceph OSD"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: disambiguate "OSD"

Distinguish "object storage device" from "object storage daemon".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/glossary: define "Ceph OSD"

Define "Ceph OSD", using Sage's own description.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48022 from rhcs-dashboard/paginate-services

mgr/dashboard: paginate services

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48763 from zdover23/wip-doc-2022-11-07-glossary-ceph-manager

doc/glossary.rst: define "Ceph Manager"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary.rst: define "Ceph Manager"

Add a definition for Ceph Manager (ceph-mgr).

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/os/seastore: rename reclaim_gen_t to rewrite_gen_t

Generation is counting rewrites, not reclaims.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: define the usage of generation

Unify the definition of the reclaim generation, cleanup and explain its
intentions and usages. Please refer to the comments in seastore_types.h.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

ceph-dokan: Exposed --win-vol-serial CLI option for ceph-dokan

Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

test: Added ceph-dokan tests

Added tests for the following ceph-dokan functionalities:

- mounting a CephFS volume
- creating files & folders
- r/w from files
- find files
- move files
- get file type
- get/set file timestamps
- set file eof/allocation size
- get volume information
- get volume space information
- file create dispositions
- file delete disposition

Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>

Merge pull request #48616 from rhcs-dashboard/mirror-promote-demote-bug

mgr/dashboard: bug fixes for rbd mirroring edit and promotion/demotion

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #48538 from Syphdias/add-bonding-alert

ceph-mixin: Add Prometheus Alert for Degraded Bond

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #48693 from ljx023/seastore-device-size-detect

crimson/os/seastore/segment_manager/block: get block device size

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #48757 from zdover23/wip-doc-2022-11-07-glossary-OSD

doc/glossary: define "OSD"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: define "OSD"

Define "OSD" as "Object Storage Device", distinct from
"object storage daemon".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/dashboard: improve level AA color contrast accessibility for dashboard pages and components
Use more accessible colors for dashboard components, text and table columns

Fixes: https://tracker.ceph.com/issues/56023
Signed-off-by: nsedrickm <nsedrick101@gmail.com>

Merge pull request #48753 from zdover23/wip-doc-2022-11-06-glossary-remove-front-matter

doc/glossary.rst: remove old front matter

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary.rst: remove old front matter

Remove two paragraphs from May 2013 that no longer apply.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48750 from zdover23/wip-doc-2022-11-06-glossary-various

doc/glossary: improve wording

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: improve wording

Remove "e.g." and "etc." and improve the wording of a definition.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48747 from zdover23/wip-doc-2022-11-05-glossary-dashboard

doc/glossary.rst: add "Ceph Dashboard" term

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary.rst: add "Ceph Dashboard" term

Add a definition for "Ceph Dashboard".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48744 from zdover23/wip-doc-2022-11-05-glossary-rbd

doc/glosary.rst: add "Ceph Block Device" term

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glosary.rst: add "Ceph Block Device" term

Add a definition for "Ceph Block Device".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48666 from joscollin/wip-cephfs-top-version

cephfs-top: add version in header

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #48291 from ljflores/wip-upgrade-telemetry

doc: update cephfs-top.png

Signed-off-by: Jos Collin <jcollin@redhat.com>

cephfs-top: add version in header

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #48739 from zdover23/wip-doc-2022-11-05-dev-guide-lone-sentence

doc/dev: improve lone sentence

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: improve lone sentence

Improve a lone sentence to make it less ambiguous, and at the same
time permit the testing of a script.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48736 from zdover23/wip-doc-2022-11-05-cephadm-host-management-removing-hosts

doc/cephadm: improve lone sentence

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/cephadm: improve lone sentence

Improve a lone sentence, both to make the cephadm documentation better
and to test a script that is under development.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48716 from tchaikov/crimson-cpuset

crimson/os/alienstore: parse crimson_alien_thread_cpu_cores in cpuset(7)'s list format

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #48111 from neesingh-rh/feature_55121

cephfs-top: addition of sort feature and limit option

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #48722 from zdover23/wip-doc-2022-11-04-cephadm-index-list-alphabetizing

doc/cephadm: alphabetize external tools list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48721 from zdover23/wip-doc-2022-11-04-cephadm-host-management-listing-hosts

doc/cephadm: arrange "listing hosts" section

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephadm: alphabetize external tools list

Alphabetize "Ansible", "Rook", and "Salt", so that
the items in the list match the items in the sentence
that precedes the list.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/cephadm: arrange "listing hosts" section

Collect the material at the top of this page into a
"Listing Hosts" section and clean the English so that
it is clearer.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc: addition of sort and limit features in doc

Fixes: https://tracker.ceph.com/issues/55121
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

cephfs-top: addition of sort and limit feature

This commit intends to add:
- sort-by field value feature to cephfs-top.
- feature to limit number of clients displayed

Fixes: https://tracker.ceph.com/issues/55121
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #48661 from thesamesam/gcc-13

common: Fix build with GCC 13 (missing <cstdint> include)

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: bug fixes for rbd mirroring edit and promotion/demotion

Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

crimson/os/alienstore: use boost::counting_iterator for setting cpuset

less repeatings this way.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/os/alienstore: parse crimson_alien_thread_cpu_cores as a cpuset(7)

the "List format" listed in cpuset(7) allows us to specify a range of
CPU cores in a comma-separated list. and the upper bound of the range
is optional. before this change, the upper bound is not optional.

before this change, the upper bound of the range is not optional. and
the upper bound is not inclusive. so we don't support the list format
of cpuset(7). Take cores "1,2,3,5,7,8" for example, we need to set the
option to "1-4,5-6,7-9" to represent this cpuset.
after this change, the upper bound is optional, so we can properly
support the list format defined by cpuset(7). and the upper bound is
inclusive. so we can use "1-3,5,7-8", which is compatible with
notation defined by cpuset(7).

in this change, the cpuset option is parsed using a seastar helper,
which is implemented using a regex. so we don't need to manually parse
it. as Seastar's parser returns an `std::optional<std::set<unsigned>>`.
if the string does not match with the regex of comma-separated list,
the returned cpuset does not have a value. this design is more explicit.
so in this change, instead of using `std::vector<uint64_t>`,
`std::optional<cpuset>` is used.

Signed-off-by: Jianxin Li <jianxin1.li@intel.com>
Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #48712 from zdover23/wip-doc-2022-11-03-cephadm-compatibility-grammar

doc/cephadm: fix grammar in compatibility.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/cephadm: fix grammar in compatibility.rst

Fix grammar in doc/cephadm/compatibility.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

ceph-crash: drop privleges to run as "ceph" user, rather than root

If privileges cannot be dropped, log an error and exit.  This commit
also catches and logs exceptions when scraping the crash path, without
which ceph-crash would just exit if it encountered an error.

Fixes: CVE-2022-3650
Fixes: https://tracker.ceph.com/issues/57967
Signed-off-by: Tim Serong <tserong@suse.com>

ceph-crash: fix stderr handling

Popen.communicate() returns a tuple (stdout, stderr), and stderr
will be of type bytes, hence the need to decode it before checking
if it's an empty string or not.

Fixes: a77b47eeeb5770eeefcf4619ab2105ee7a6a003e
Signed-off-by: Tim Serong <tserong@suse.com>

ceph-mixin: Add Prometheus Alert for Degraded Bond

Currently there is no alert for a network interface card to be misconfigured or
failed which is part of a network bond.

This could lead to redundancies and performance being degraded unnoticed.

To solve this, I use node exporter metrics to look at the number of total peers
of the bond and the ones that are active. If the numbers differ, something is up
and should be looked at.

Fixes: https://tracker.ceph.com/issues/57962
Signed-off-by: Christian Kugler <syphdias+git@gmail.com>

crimson/os/seastore/segment_manager/block: get block device size

Previously the file_stat(::stat) can not get correct size for block devices.
Now call the file.size(using ::lseek) to replace it.

Signed-off-by: Jianxin Li <jianxin1.li@intel.com>

Merge pull request #48699 from zdover23/wip-doc-2022-11-02-dev-osd-internals-ec-glossary-fixes

doc/dev: refine erasure_coding.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: refine erasure_coding.rst

Improve the readability and clarity of
erasure_coding.rst.

Co-author: Cole Mitchell <cole.mitchell@gmail.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48687 from StevenWash/bugfix/fix_command_error_blank

doc/radosgw: fix command error blank

doc/radosgw: fix command error blank

Merge pull request #48684 from zdover23/wip-doc-2022-11-01-dev-osd-internals-ec-glossary-alphabetize

doc/dev: alphabetize EC glossary

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: alphabetize EC glossary

Alphabetize the glossary in
doc/dev/osd_internals/erasure_coding.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48673 from idryomov/wip-49947

doc: note architecture availability for PMEM mode of PWL cache

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #48548 from dparmar18/update_debug_script

script/ceph-debug-docker.sh: change default branch to main

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #48674 from zdover23/doc-2022-10-31-dev-osd-internals-ec-glossary

doc/dev: improve EC glossary

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: improve EC glossary

Improve the clarity and syntax of the text in
doc/dev/osd_internals/erasure_coding.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc: note architecture availability for PMEM mode of PWL cache

Fixes: https://tracker.ceph.com/issues/49947
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #48667 from zdover23/wip-doc-2022-10-30-ceph-volume-lvm-prepare-colemitchell-notes

doc/ceph-volume: refine Filestore docs

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>