Zac Dover [Thu, 25 Aug 2022 15:56:41 +0000 (01:56 +1000)]
doc/mgr: add prompt directives to dashboard.rst
This commit adds prompt directives (.. prompt:: bash $) to
the commands in dashboard.rst.
There are several ".. include::" directives in the dashboard.rst
file, which means that part of this page is sourced from elsewhere
than the dashboard.rst file. Because I have not yet added prompt
directives to those files, there is an inconsistency in the rendering
of this file. Most of the commands on this page have unselectable
prompts (unselectable prompts are the prompts that don't get added to
the buffer when you copy them to one of the clipboards). But the
commands on this page that come from those ".. include::" directives
do not yet have unselectable prompts.
This file is over 1600 lines long. It was perhaps not optimally wise
of me to have edited all of it in one fell swoop. It took many hours,
and carefully checking it will probably take at least one hour. I
suggest that whoever reviews this should not spend much time on it,
but should instead make a quick pass over the page and make sure that
it looks passable.
The English syntax on this page (and throughout the Dashboard doc-
umentation) will be tightened to remove ambiguity and to improve
readability in the near future, so hold all English-language-related
comments for a future pull request.
Adam King [Wed, 24 Aug 2022 14:36:53 +0000 (10:36 -0400)]
doc/cephadm: fix example for specifying networks for rgw
count_per_host must be used with underscores rather
than dashes to work, you need to pass service_id not
service_name and the option for the port is called
rgw_frontend_port not just "port"
Zac Dover [Tue, 23 Aug 2022 06:59:04 +0000 (16:59 +1000)]
doc/mgr: edit orchestrator.rst
This PR improves the English language in the "Orchestrator CLI"
section of the MGR documentation. It adds a couple of section
headers in order to signpost the information in the document
a bit more than had already been done, but it makes no major
structural changes to the presentation of the information here.
This PR was motivated by feedback from the 2022 Ceph User Survey
in which one of the respondents wrote "better ceph orch documen-
tation".
The final section on this page, "Current Implementation Status",
must be verified by someone who is familiar with the current state
of "ceph orch" and a date stamp should be applied to the top of
the section so that the word "current" has a meaningful referent.
Nizamudeen A [Tue, 26 Apr 2022 10:19:09 +0000 (15:49 +0530)]
mgr/dashboard: prometheus rules internal server error
After we increase/decrease the count of the node-exporter, we get a 500
- Internal server error from api/prometheus/rules endpoint. On further
debugging its caused by the jsonDecodder, because I guess the expected
input for the json.loads() is not a json formatted input. So to fix
that issue I can either do an error handling on the json.loads() or I
can move the json.loads() on the already existing try block. I went for
the second approach here.
qa: filter internal directories in 'subvolumegroup ls' command
Internal directories: '_nogroup', '_index', '_legacy', '_deleting'
1. Internal directories should be filtered in 'subvolmegroup ls' command.
2. Internal directories should not be accepted as a group name.
mgr/volumes: filter internal directories in 'subvolumegroup ls' command
Internal directories: '_nogroup', '_index', '_legacy', '_deleting'
1. Internal directories should be filtered in 'subvolmegroup ls' command.
2. Internal directories should not be accepted as a group name.
Used the https://www.npmjs.com/package/@grafana/e2e npm packages and
followed
https://github.com/grafana/grafana/blob/main/contribute/style-guides/e2e.md
to understand the style of the grafana e2e testing.
In this PR I introduces the tests for the Hosts Overall
Performance and also RGW per Daemon and Overall Performance
Xiubo Li [Mon, 7 Mar 2022 07:42:42 +0000 (15:42 +0800)]
mds: flush mdlog if locked and still has wanted caps not satisfied
In _do_cap_update() if one client is releasing the Fw caps the
relevant client range will be erased, and then new_max will be 0.
It will skip flushing the mdlog after it submitting a journal log,
which will keep holding the wrlock for the filelock.
So when a new client is trying to open the file for reading, since
the wrlock is locked for the filelock the file_eval() is possibly
couldn't changing the lock state and at the same time if the
filelock is in stable state, such as in EXECL, MIX. The mds may
skip flushing the mdlog in the open related code too.
We need to flush the mdlog if there still has any wanted caps
couldn't be satisfied and has any lock for the filelock after the
file_eval().
Kefu Chai [Mon, 8 Aug 2022 14:41:17 +0000 (22:41 +0800)]
pybind/mgr/dashboard: do not use distutils.version.StrictVersion
replace `distutils.version.StrictVersion` with
`pkg_resources.parse_version()`
as the former is deprecated, see https://peps.python.org/pep-0632/.
let's use `pkg_resources` instead. this change also addresses
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1010894.
we have this issue when testing with an ubuntu jammy test node.
see https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1967139
Kefu Chai [Mon, 8 Aug 2022 14:41:17 +0000 (22:41 +0800)]
pybind/mgr/dashboard: do not use distutils.version.StrictVersion
replace `distutils.version.StrictVersion` with
`pkg_resources.parse_version()`
as the former is deprecated, see https://peps.python.org/pep-0632/.
let's use `pkg_resources` instead. this change also addresses
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1010894.
we have this issue when testing with an ubuntu jammy test node.
see https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1967139
Kefu Chai [Fri, 12 Aug 2022 05:06:25 +0000 (13:06 +0800)]
mgr/dashboard: bump up more-itertools
before this change, more-itertools tries to import Sequence from
collections, this leads us to failures like:
```
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
__import__(pkg_name)
File
"/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/__init__.py",
line 9, in <module>
import cherrypy
File
"/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/.tox/openapi-check/lib/python3.10/site-packages/cherrypy/__init__.py",
line 76, in <module>
from . import _cprequest, _cpserver, _cptree, _cplogging, _cpconfig
File
"/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/.tox/openapi-check/lib/python3.10/site-packages/cherrypy/_cprequest.py",
line 11, in <module>
from cherrypy import _cpreqbody
File
"/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/.tox/openapi-check/lib/python3.10/site-packages/cherrypy/_cpreqbody.py",
line 135, in <module>
import cheroot.server
File
"/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/.tox/openapi-check/lib/python3.10/site-packages/cheroot/server.py",
line 96, in <module>
from .workers import threadpool
File
"/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/.tox/openapi-check/lib/python3.10/site-packages/cheroot/workers/threadpool.py",
line 20, in <module>
from jaraco.functools import pass_none
File
"/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/.tox/openapi-check/lib/python3.10/site-packages/jaraco/functools.py",
line 8, in <module>
import more_itertools
File
"/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/.tox/openapi-check/lib/python3.10/site-packages/more_itertools/__init__.py",
line 1, in <module>
from more_itertools.more import * # noqa
File
"/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/.tox/openapi-check/lib/python3.10/site-packages/more_itertools/more.py",
line 3, in <module>
from collections import Counter, defaultdict, deque, Sequence
ImportError: cannot import name 'Sequence' from 'collections'
(/usr/lib/python3.10/collections/__init__.py)
ERROR: InvocationError for command
/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/.tox/openapi-check/bin/python3
-m dashboard.controllers.docs
/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/.tox/openapi-check/tmp/openapi.yaml
(exited with code 1)
```
after this change, more-itertools is pin'ed at the latest stable
at the time of writing, which includes the fixes including
https://github.com/more-itertools/more-itertools/commit/30a861bc5a4f53a9ba73923c9048a3632a0f9d18
.
please note, more-itertools dropped python3.3 support. but neither
do us support this python version, so we should be safe.
Kefu Chai [Fri, 18 Feb 2022 15:23:54 +0000 (23:23 +0800)]
crimson: specialize fmt::formatter<>() for crimson types
otherwise we'd have FTBFS like
/usr/include/fmt/core.h:1727:3: error: static_assert failed due to requirement 'formattable' "Cannot format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api$
static_assert(
^
/usr/include/fmt/core.h:1853:23: note: in instantiation of function template specialization 'fmt::detail::make_arg<true, fmt::basic_format_context<fmt::appender, char>, fmt::detail::type::custom_type, crimson::os:$
data_{detail::make_arg<
please note, delta_op_t is lifted out of the templated outer class
to avoid the headache of specialization of template of template in
another namespace.
Kefu Chai [Fri, 18 Feb 2022 15:23:54 +0000 (23:23 +0800)]
crimson: specialize fmt::formatter<>() for crimson types
otherwise we'd have FTBFS like
/usr/include/fmt/core.h:1727:3: error: static_assert failed due to requirement 'formattable' "Cannot format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api$
static_assert(
^
/usr/include/fmt/core.h:1853:23: note: in instantiation of function template specialization 'fmt::detail::make_arg<true, fmt::basic_format_context<fmt::appender, char>, fmt::detail::type::custom_type, crimson::os:$
data_{detail::make_arg<
please note, delta_op_t is lifted out of the templated outer class
to avoid the headache of specialization of template of template in
another namespace.
Zac Dover [Sat, 13 Aug 2022 23:14:56 +0000 (09:14 +1000)]
doc/rados/operations: add prompts to operating.rst
This commit adds ".. prompt:: bash $"-style prompts to operating.rst.
This brings this file up to the standard established in 2020 when
Kefu added support for the ".. prompt::" directive.
This commit is a part of an initiative to modernize the presentation
of all BASH commands in the RADOS documentation.
The progress of this project can be tracked here:
https://tracker.ceph.com/issues/57108
Adam C. Emerson [Wed, 18 May 2022 22:06:01 +0000 (18:06 -0400)]
build: Silence deprecation warnings from OpenSSL 3
The OpenSSL developers suggest that anyone wishing to continue using
low-level functions may either live with the warnings, silence them,
or switch to high level functions.
As high level functions do their own memory allocation, switching to
them may lead to performance regressions.
We do not wish to have deprecation warnings filling up our compiler
outputs when searching for other messages.
So silencing the warnings, at least for now, seems the least bad option.
Ilya Dryomov [Fri, 12 Aug 2022 09:10:45 +0000 (11:10 +0200)]
rbd: remove incorrect use of std::includes()
- std::includes() requires sorted ranges but command specs aren't
sorted
- std::includes() purpose is to check whether the second range is
a subsequence of the first range but here the size of the second
range is always equal to the size of the first range, which means
that, had the ranges been sorted, std::includes() would have checked
straight equality
Ilya Dryomov [Fri, 12 Aug 2022 09:10:45 +0000 (11:10 +0200)]
rbd: find_action() should sort actions first
The order in which objects with static storage duration in
different TUs are initialized is undefined. If the compiler
chooses to initialize Shell::Action objects in action/Trash.cc
before Shell::Action objects in action/TrashPurgeSchedule.cc,
all "rbd trash purge schedule ..." commands get shadowed by
"rbd trash purge" command:
$ rbd trash purge schedule list
rbd: too many arguments
The confusing error arises because "rbd trash purge" takes a single
positional argument. "schedule" gets interpreted as <pool-spec> and
"list" generates an error.
Kotresh HR [Fri, 24 Jun 2022 07:17:57 +0000 (12:47 +0530)]
mds: Don't blocklist the clients in any replay state
The mds in 'standby-replay' state should not blocklist clients.
Attempting to do so would hit an assert while journaling.
Before the commit 79f7576 the 'Server:apply_blocklist' used to
fetch the blocklist addresses from 'Objecter::consume_blocklist_events'
which returns blocklist addresses only if 'blocklist_events_enabled'
are enabled. This is not enabled for mds in 'standby-replay' state.
Hence this issue wasn't seen.
After the commit 79f7576 the 'Server::apply_blocklist' fetches the
blocklist addresses from the osdmap. This would return the blocklist
addresses irrespective of the mds state. The mds in 'replay' state
doesn't blocklist the clients but if the mds is in 'standby-replay'
state, it tries to journal the blocklist and asserts. This patch
fixes the same by not allowing the blocklist if the mds is in any
replay state.
Kefu Chai [Wed, 10 Aug 2022 14:35:14 +0000 (22:35 +0800)]
ceph.spec.in: %enable_devtoolset11 only if the macro is defined
there is chance that we are using `yum-builddep` to prepare the
build dependencies. in that case, gcc-toolset-11-build is not
installed. it's like a chicken-egg dilemma, but the point is
`yum-builddep` is able to pull in the gcc-toolset-11-build. once
gcc-toolset-11-build is installed, we will have the %enable_devtoolset11
rpm macro.
Kefu Chai [Mon, 8 Aug 2022 12:40:52 +0000 (20:40 +0800)]
ceph.spec.in: add libatomic to BuildRequires on fedora
otherwise we'd have failures like
/opt/compiler-explorer/gcc-trunk-20220808/bin/../lib/gcc/x86_64-linux-gnu/13.0.0/../../../../x86_64-linux-gnu/bin/ld:
/tmp/ccVlMbVh.o: in function `std::atomic<tagged_ptr>::store(tagged_ptr,
std::memory_order)':
/opt/compiler-explorer/gcc-trunk-20220808/include/c++/13.0.0/atomic:273:
undefined reference to `__atomic_store_16'
when generating the building system using CMake on fedora 36.
Kefu Chai [Thu, 4 Aug 2022 03:57:05 +0000 (11:57 +0800)]
ceph.spec.in: BuildRequires gcc-toolset-11-libatomic-devel for aarch64
to address following failure when generating the building system
using CMake:
```
-- Performing Test HAVE_LIBATOMIC
-- Performing Test HAVE_LIBATOMIC - Failed
CMake Error at cmake/modules/CheckCxxAtomic.cmake:66 (message):
Host compiler /opt/rh/gcc-toolset-11/root/usr/bin/c++ requires libatomic,
but it is not found
```
Kefu Chai [Thu, 4 Aug 2022 01:00:29 +0000 (09:00 +0800)]
ceph.spec.in: call %enable_devtoolset11 on centos8
before this change %enable_devtoolset11 is called only when building
with crimson on centos8.
after this change %enable_devtoolset11 is called when building on
centos8. because we've started using gcc-toolset-11 for building
rpm packages on centos8 after the C++20 migration. so, to build
with gcc-11, we need to enable it.
also, because gcc-toolset-11 is used, we have to disable
annotated_build.
gcc-toolset-11-annobin is already installed, but ceph.spec.in adds
"-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1" which needs the gcc
plugin too
resolves this failure during the cmake configure step:
-- Check for working CXX compiler: /opt/rh/gcc-toolset-11/root/usr/bin/c++
-- Check for working CXX compiler: /opt/rh/gcc-toolset-11/root/usr/bin/c++ - broken
CMake Error at /usr/share/cmake/Modules/CMakeTestCXXCompiler.cmake:59 (message):
The C++ compiler
Kefu Chai [Mon, 27 Jun 2022 13:18:57 +0000 (21:18 +0800)]
ceph.spec.in: use %enable_devtoolset11 to enable GTS-11
%enable_devtoolset11 redefines %___build_pre by appending
`source scl_source enable gcc-toolset-11` to it. `___build_pre` should
be able to populate this setting to both %build and %install. and hence
address the FTBFS where we need to use the tool chain from GTS-11.
This can be surprising but we actually compile things during
the `install` stage of `rpm-build`. The example is the pybind's
`setup.py` which builds `rados_dummy.c`.
ceph.spec.in: use gcc-toolset-10 for building crimson
This commit bumps up the toolset version but only to build crimson.
That is, the classical OSD stays unaffected.
The reason behind the upgrade is the following FTBFS:
```
[ 32%] Building CXX object src/seastar/CMakeFiles/seastar.dir/src/core/reactor.cc.o
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11345-ga3bb1485/rpm/el8/BUILD/ceph-17.0.0-11345-ga3bb1485/src/seastar/src/core/reactor.cc: In constructor ‘seastar::reactor::reactor(std::shared_ptr<seastar::smp>, seastar::alien::instance&, unsigned int, seastar::reactor_backend_selector, seastar::reactor_config)’:
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11345-ga3bb1485/rpm/el8/BUILD/ceph-17.0.0-11345-ga3bb1485/src/seastar/src/core/reactor.cc:926:90: error: use of deleted function ‘seastar::condition_variable::condition_variable()’
926 | , _thread_pool(std::make_unique<thread_pool>(this, seastar::format("syscall-{}", id))) {
| ^
In file included from /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11345-ga3bb1485/rpm/el8/BUILD/ceph-17.0.0-11345-ga3bb1485/src/seastar/include/seastar/core/reactor.hh:74,
from /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11345-ga3bb1485/rpm/el8/BUILD/ceph-17.0.0-11345-ga3bb1485/src/seastar/src/core/reactor.cc:32:
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11345-ga3bb1485/rpm/el8/BUILD/ceph-17.0.0-11345-ga3bb1485/src/seastar/include/seastar/core/condition-variable.hh:157:5: note: ‘seastar::condition_variable::condition_variable() noexcept’ is implicitly deleted because its exception-specification does not match the implicit exception-specification ‘’
157 | condition_variable() noexcept = default;
```
projects / ceph.git / log