Laura Flores [Thu, 5 Aug 2021 22:11:25 +0000 (22:11 +0000)]
mgr/telemetry: add checks for histograms with unique schema
In some cases, the user might change the configuration of a certain histogram, such as its bucket size. To handle this, I wrote a check that separates histograms with unique configurations so there are no conflicts. Histograms with like configs are combined.
Laura Flores [Mon, 2 Aug 2021 23:05:26 +0000 (23:05 +0000)]
mgr/telemetry: improve the formatting of perf histograms in telemetry report
Currently, the JSON formatter displays lists vertically rahter than horizontally, which makes it particularly difficult for users to scroll through the perf histograms, much less read/interpret them. With this commit, I wrote a patch within the show() function (the function that is controlled by `ceph telemetry show`) which formats the the lists of ranges and values in the perf histograms as strings. This way, parts of the histogram output is displayed horizontally instead of vertically. This makes the Telemetry report much more human-readable.
Laura Flores [Fri, 30 Jul 2021 08:40:06 +0000 (08:40 +0000)]
mgr/telemetry: aggregate values and improve readability in get_osd_perf_histograms()
To condense histogram data, I merged all like histograms between osds under one osd category. I also used json.dumps() to improve the readability of the final output.
Laura Flores [Fri, 23 Jul 2021 17:40:27 +0000 (17:40 +0000)]
mgr, mgr/telemetry: add access to osd commands in mgr and workload metrics to telemetry
In the manager module (src/pybind/mgr/mgr_module.py), there was a command that allowed access to monitor commands, but there was nothing to access osd commands. I added a new method called osd_command that allows access to the output of commands like "ceph osd.0 perf histogram dump".
In the telemetry module (src/pybind/mgr/telemetry/module.py), I added information that may help to characterize a workload. These changes include:
- io rate: delta in pg stats since last report, including a pg stats summary and some store stats
- stats sum per pool: summed up stats from every pg in a pool
- perf histograms: histogram dump of latency and request size for each osd
cmake: use add_compile_options() to pass compile options
add_definitions() is for adding "-D" define flags, while "-include" does
not define flags, so let's be explicit and use add_compile_options()
instead.
this change partially reverts 5fb90bb79671b8e22f507728e4b4b948a591f98f.
back then, we were using ubuntu bionic for building windows packages.
but since we've switched to ubuntu focal, where cmake 3.16 is available,
there is no reason to stick with add_definitions() anymore.
since we dropped the support of bionic, we now have the luxury of using
newer CMake! and by using CMake 3.16, we now have access to
* add_compile_definitions(), which was introduced in cmake 3.12
also, drop the comment of
> remove cmake/modules/FindPython* once 3.12 is required
because we need to support python 3.10 in future, better off keeping
the FindPython files around, so we can backport cmake changes from
upstream when necessary.
we are allowed to bump up the CMake's required to 3.16, because:
- ubuntu focal ships with cmake 3.16.3
- RHEL/CentOS8's AppStream has cmake 3.18
test/crimson/test_alien_echo: msgr->stop() before shutdown
it is required to remove all registered dispatchers before shutting down a
messenger. otherwise assert(dispatchers.empty()) in SocketMessenger::shutdown() fails.
Sage Weil [Thu, 15 Jul 2021 20:17:23 +0000 (15:17 -0500)]
mon: allow a MON_DOWN grace period after cluster mkfs
During teuthology tests, the initial cluster bootstrap often starts up
the mon sbut doesn't include all mons in the initial quorum, due to
mon startup misalignment and random delays. Provide a short grace period
where we will not raise a MON_DOWN alert even though the quorum is not
complete.
Fixes: https://tracker.ceph.com/issues/43584 Signed-off-by: Sage Weil <sage@newdream.net>
Sage Weil [Thu, 15 Jul 2021 15:05:22 +0000 (11:05 -0400)]
Merge PR #42073 into master
* refs/pull/42073/head:
doc/mgr/nfs: fix 'export apply', pool name
PendingReleaseNotes: document workaround for NFS storage change
qa/tasks/mgr/test_orchestrator_cli: fix test
qa/suites/orch/cephadm/mgr-nfs-upgrade: add test for nfs migration
mgr/cephadm: migrate nfs grace file
mgr/nfs: migrate pre-pacific nfs.ganesha-foo clusters to nfs.foo
doc/cephfs/fs-nfs-exports: document new export apply capabilities
qa/tasks/cephfs/test_nfs: define NFS_POOL_NAME
mgr/nfs: use NFS_POOL_NAME in test_nfs.py
mgr/nfs: test export apply on JSON list
mgr/nfs: add test for ganesha conf apply/import
qa/tasks/cephfs/test_nfs: retry mount a few times
mgr/cephadm: migrate all legacy nfs exports to new .nfs pool
mgr/nfs: adjust cephfs export caps if necessary
python-common: don't accept pool/ns for NFSServiceSpec
mgr/orchestrator: drop rados_config_location ServiceDescription property
mgr/cephadm: move rados_config_location() out of NFSServiceSpec
mgr/nfs: change nfs pool to .nfs
mgr/nfs/export: accept a JSON or ganesha EXPORT config
mgr/nfs: allow 'nfs export apply' to take a list of exports
python-common: remove pool + namespace from NFSServiceSpec
mgr/nfs: used fixed pool + ns
mgr/rook: used fixed pool + ns
mgr/dashboard: use fixed pool + ns
mgr/cephadm: always use fixed pool and namespace
mgr/nfs: adjust test to match pool name
mgr/nfs: always create ganesha pool with well-defined name
cmake: add transitive dependency on legacy-option-headers
the legacy-option-headers target is only marked as a dependency of the
common-objs and common-common-objs. because those targets are OBJECT
libraries, their dependencies aren't inherited by the targets that link
common-objs or common-common-objs
this adds the dependencies manually, so that changes to the config
yaml files will cause legacy-option-headers to regenerate the headers
Sage Weil [Fri, 2 Jul 2021 19:53:15 +0000 (15:53 -0400)]
mgr/cephadm: migrate all legacy nfs exports to new .nfs pool
Migrate all past NFS pools, whether they were created by mgr/nfs or by
the dashboard, to the new mgr/nfs .nfs pool.
Since this migrations relies on RADOS being available, we have to be a bit
careful here: we only attempt the migration from serve(), not during
module init.
After the exports are re-imported, we destroy existing ganesha daemons so
that new ones will get recreated. This ensures the (new) daemons have
cephx keys to access the new pool.
Note that no attempt is made to clean up the old NFS pools. This is out
of paranoia: if something goes wrong, the old NFS configuration data will
still be there.
Sage Weil [Fri, 2 Jul 2021 16:39:29 +0000 (12:39 -0400)]
mgr/nfs: change nfs pool to .nfs
This is a new pool that we can migrate all past NFS configuration to,
simplifying the migration process (and also allowing us to pick a
.-prefixed name).
Sage Weil [Wed, 14 Jul 2021 18:38:59 +0000 (14:38 -0400)]
Merge PR #42041 into master
* refs/pull/42041/head:
mgr/restful: ignore min/max_size
test/crush: drop min/max_size refs
qa/workunits/mon/pool_ops: remove test for min/max_size check
qa: scrub a few remaining mentions of ruleset
qa/standalone/mon/osd-*: fix tests
PendingReleaseNotes: note min/max_size removal
mgr/dashboard: remove max/min_size and ruleset
mon/OSDMonitor: fix calls to CrushTester
crush: eliminate min_size and max_size
test/cli/crushtool: reunumber rulesets in test maps
crushtool: require min/max or num-rep for --test
crush: remove last traces of 'ruleset'
test/cli/crushtool: use 'id' instead of 'ruleset' in crush inputs
crushtool: take --min-rep and --max-rep explicitly
crush/CrushTester: drop --ruleset
doc: scrub 'ruleset' from docs
src/erasure-code: rule, not ruleset
mon/OSDMonitor: remove check_crush_rule() callers
mon/OSDMonitor: rule, not ruleset
crushtool: remove check for overlapped ruels
crush/CrushWrapper: get_osd_pool_default_crush_replicated_ruleset -> rule
crush: remove find_rule()
mon/OSDMonitor: use pool's crush rule directly
osd/OSDMap: drop checks for ruleset == ruleid
osd/OSDMap: use pool's crush rule_id directly
mon/PGMap: use pool's crush_rule directly
mon/OSDMonitor: remove crush ruleset->rule rewrite
Adam C. Emerson [Wed, 14 Jul 2021 14:57:02 +0000 (10:57 -0400)]
rgw: Rename REMOVE_OBJ to INVALIDATE_OBJ
Also rename ObjectCache::remove to ObjectCache::invalidate_remove
Since we're depending on these message types/functions having
invalidate semantics but NOT caching a negative result, rename and
leave a comment for clarity.
Fixes: https://tracker.ceph.com/issues/51674 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
projects / ceph.git / log