]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
3 years agocrimson/os/seastore/segment_manager: implement metrics in BlockSegmentManager 42457/head
Yingxin Cheng [Fri, 23 Jul 2021 02:28:58 +0000 (10:28 +0800)]
crimson/os/seastore/segment_manager: implement metrics in BlockSegmentManager

* data read/write effort;
* metadata write effort;
* opened/closed/released segments;
* unused bytes from closed segments;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
4 years agocrimson/os/seastore/segment_manager: respect len parameter of BlockSegmentManager...
Yingxin Cheng [Thu, 22 Jul 2021 07:28:40 +0000 (15:28 +0800)]
crimson/os/seastore/segment_manager: respect len parameter of BlockSegmentManager::read()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
4 years agoMerge pull request #42342 from dsavineau/cephadm_template_jinja2_lint
Kefu Chai [Thu, 22 Jul 2021 05:28:33 +0000 (13:28 +0800)]
Merge pull request #42342 from dsavineau/cephadm_template_jinja2_lint

mgr/cephadm/templates: add jinja2 lint

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
4 years agoMerge pull request #42284 from orozery/remove-calc-target-vector-copies
Kefu Chai [Thu, 22 Jul 2021 01:01:46 +0000 (09:01 +0800)]
Merge pull request #42284 from orozery/remove-calc-target-vector-copies

osdc/Objecter: avoid vector copies in _calc_target

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42401 from tchaikov/wip-cmake-version
Kefu Chai [Thu, 22 Jul 2021 00:58:10 +0000 (08:58 +0800)]
Merge pull request #42401 from tchaikov/wip-cmake-version

cmake: drop set(VERSION ...)

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
4 years agoMerge pull request #42408 from tchaikov/wip-mgr-always-on
Kefu Chai [Thu, 22 Jul 2021 00:56:43 +0000 (08:56 +0800)]
Merge pull request #42408 from tchaikov/wip-mgr-always-on

mon/MgrMonitor: drop nautilus modules from always_on_modules

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
4 years agoMerge pull request #42415 from sebastian-philipp/haproxy-user-root
Kefu Chai [Thu, 22 Jul 2021 00:55:46 +0000 (08:55 +0800)]
Merge pull request #42415 from sebastian-philipp/haproxy-user-root

cephadm: haproxy 2.4 defaults to a different container user.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
4 years agoMerge pull request #41907 from kamoltat/wip-ksirivad-progress-time-interval
Neha Ojha [Wed, 21 Jul 2021 23:53:38 +0000 (16:53 -0700)]
Merge pull request #41907 from kamoltat/wip-ksirivad-progress-time-interval

pybind/mgr/progress: introduce 5 second sleep interval

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
4 years agoMerge PR #42295 into master
Patrick Donnelly [Wed, 21 Jul 2021 20:13:43 +0000 (13:13 -0700)]
Merge PR #42295 into master

* refs/pull/42295/head:
doc/cephfs: recover file system after recovering

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
4 years agoMerge pull request #42402 from ljflores/perf-counter-docs
Neha Ojha [Wed, 21 Jul 2021 17:40:46 +0000 (10:40 -0700)]
Merge pull request #42402 from ljflores/perf-counter-docs

doc/dev/perf_counters: update docs to include more context about perf counter usage

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
4 years agoMerge pull request #42333 from tchaikov/wip-doc-program
Kefu Chai [Wed, 21 Jul 2021 17:35:34 +0000 (01:35 +0800)]
Merge pull request #42333 from tchaikov/wip-doc-program

doc/man/8/ceph-volume: specify "program" for subcommands

Reviewed-by: Neha Ojha <nojha@redhat.com>
4 years agoMerge pull request #42196 from cbodley/wip-qa-rgw-rm-cephadm
Casey Bodley [Wed, 21 Jul 2021 17:14:35 +0000 (13:14 -0400)]
Merge pull request #42196 from cbodley/wip-qa-rgw-rm-cephadm

qa/rgw: remove rgw_cephadm.yaml from rgw/singleton suite

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
4 years agodoc/man/8/ceph-volume: specify "program" for subcommands 42333/head
Kefu Chai [Wed, 14 Jul 2021 12:54:50 +0000 (20:54 +0800)]
doc/man/8/ceph-volume: specify "program" for subcommands

as per
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html

> Like py:currentmodule, this directive produces no output. Instead, it
> serves to notify Sphinx that all following option directives document
> options for the program called name.
> ...
> The program name may contain spaces (in case you want to document
> subcommands like svn add and svn commit separately).

and to avoid the warnings like:

doc/man/8/ceph-volume.rst:424: WARNING: Duplicate explicit target name:
"cmdoption-ceph-volume-h".

we should specify different "program" for different set of options.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42317 from cbodley/wip-39657
Casey Bodley [Wed, 21 Jul 2021 17:13:54 +0000 (13:13 -0400)]
Merge pull request #42317 from cbodley/wip-39657

rgw multisite: metadata sync treats all errors as 'transient' for retry

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
4 years agoMerge pull request #41800 from BryceCao/wip-modfiy-XML-when-deleterole-take-error
Casey Bodley [Wed, 21 Jul 2021 17:12:29 +0000 (13:12 -0400)]
Merge pull request #41800 from BryceCao/wip-modfiy-XML-when-deleterole-take-error

rgw : modfiy error XML for deleterole

Reviewed-by: Casey Bodley <cbodley@redhat.com>
4 years agoMerge pull request #41858 from BryceCao/wip-del-tenant-parse-in-createrole
Casey Bodley [Wed, 21 Jul 2021 17:11:44 +0000 (13:11 -0400)]
Merge pull request #41858 from BryceCao/wip-del-tenant-parse-in-createrole

rgw : add check for tenant provided in RGWCreateRole

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
4 years agoMerge pull request #42361 from cbodley/wip-49747
Casey Bodley [Wed, 21 Jul 2021 17:10:30 +0000 (13:10 -0400)]
Merge pull request #42361 from cbodley/wip-49747

qa/rgw: add failing tempest test to blocklist

Reviewed-by: Ali Maredia <amaredia@redhat.com>
4 years agoMerge pull request #35153 from BryceCao/wip-swift-create-bucket-constraint
Casey Bodley [Wed, 21 Jul 2021 17:09:10 +0000 (13:09 -0400)]
Merge pull request #35153 from BryceCao/wip-swift-create-bucket-constraint

rgw : skip bucket name detection for system request

Reviewed-by: Casey Bodley <cbodley@redhat.com>
4 years agoMerge pull request #35061 from joke-lee/rgw-multisite-forward-policy-accessblock
Casey Bodley [Wed, 21 Jul 2021 17:07:34 +0000 (13:07 -0400)]
Merge pull request #35061 from joke-lee/rgw-multisite-forward-policy-accessblock

rgw: forward request in multisite for RGWDeleteBucketPolicy and RGWDeleteBucketPublicAccessBlock

Reviewed-by: Casey Bodley <cbodley@redhat.com>
4 years agoMerge pull request #42273 from dabukalam/object-lock-retain-date-iso8601
Casey Bodley [Wed, 21 Jul 2021 17:06:47 +0000 (13:06 -0400)]
Merge pull request #42273 from dabukalam/object-lock-retain-date-iso8601

rgw: Dump Object Lock Retain Date as ISO 8601

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
4 years agoMerge pull request #42425 from cbodley/wip-42788
Kefu Chai [Wed, 21 Jul 2021 16:13:09 +0000 (00:13 +0800)]
Merge pull request #42425 from cbodley/wip-42788

test/rgw: fix use of poll() with timers in unittest_rgw_dmclock_scheduler

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42410 from ronen-fr/wip-ronenf-standalone-repair
Neha Ojha [Wed, 21 Jul 2021 13:57:41 +0000 (06:57 -0700)]
Merge pull request #42410 from ronen-fr/wip-ronenf-standalone-repair

qa/standalone: fixing the timings when waiting for deep-scrub to start

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
4 years agoMerge pull request #42430 from tchaikov/wip-cmake-qa-drop-mypy 41793/head
Sebastian Wagner [Wed, 21 Jul 2021 06:58:18 +0000 (08:58 +0200)]
Merge pull request #42430 from tchaikov/wip-cmake-qa-drop-mypy

cmake: drop "mypy" from tox envlist of "qa"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
4 years agoMerge pull request #42418 from tchaikov/wip-crimson-nbd
Kefu Chai [Wed, 21 Jul 2021 06:49:02 +0000 (14:49 +0800)]
Merge pull request #42418 from tchaikov/wip-crimson-nbd

crimson/tools/store_nbd: pass app.alien() down to FSDriver

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
4 years agoMerge pull request #42367 from tchaikov/wip-cmake-Og
Kefu Chai [Wed, 21 Jul 2021 03:38:22 +0000 (11:38 +0800)]
Merge pull request #42367 from tchaikov/wip-cmake-Og

cmake: add "-Og" to CMAKE_C_FLAGS_DEBUG

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
4 years agoMerge pull request #42427 from tchaikov/wip-seastore-metric-cleanup
Kefu Chai [Wed, 21 Jul 2021 03:25:40 +0000 (11:25 +0800)]
Merge pull request #42427 from tchaikov/wip-seastore-metric-cleanup

crimson/os/seastore: trade a map with a plain array

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
4 years agoMerge pull request #42429 from neha-ojha/wip-51638-cleanup
Kefu Chai [Wed, 21 Jul 2021 03:24:40 +0000 (11:24 +0800)]
Merge pull request #42429 from neha-ojha/wip-51638-cleanup

qa/*/test_envlibrados_for_rocksdb.sh: remove OS specific configuration

Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agocmake: drop "mypy" from tox envlist of "qa" 42430/head
Kefu Chai [Wed, 21 Jul 2021 02:56:21 +0000 (10:56 +0800)]
cmake: drop "mypy" from tox envlist of "qa"

this change partially reverts 81305b0da9a3e04633c022b12644e2975aa46a0d,
otherwise we have following errors:

tasks/vstart_runner.py:691: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"
tasks/vstart_runner.py:705: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/os/seastore: trade a map with a plain array 42427/head
Kefu Chai [Tue, 20 Jul 2021 17:09:40 +0000 (01:09 +0800)]
crimson/os/seastore: trade a map with a plain array

simpler this way, as we don't need a std::map<> at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42261 from cyx1231st/wip-seastore-metrics-2
Kefu Chai [Tue, 20 Jul 2021 17:06:51 +0000 (01:06 +0800)]
Merge pull request #42261 from cyx1231st/wip-seastore-metrics-2

crimson/os/seastore: implement metrics at cache level

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42386 from liu-chunmei/seastore-metrics
Kefu Chai [Tue, 20 Jul 2021 17:05:34 +0000 (01:05 +0800)]
Merge pull request #42386 from liu-chunmei/seastore-metrics

crimson/seastore: add seastore operation latency metrics

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agotest/rgw: use spawn library for test_rgw_dmclock_scheduler 42425/head
Casey Bodley [Tue, 20 Jul 2021 16:50:25 +0000 (12:50 -0400)]
test/rgw: use spawn library for test_rgw_dmclock_scheduler

Signed-off-by: Casey Bodley <cbodley@redhat.com>
4 years agoMerge pull request #41929 from sebastian-philipp/fix-qa-tox
Kefu Chai [Tue, 20 Jul 2021 16:36:59 +0000 (00:36 +0800)]
Merge pull request #41929 from sebastian-philipp/fix-qa-tox

qa: Various make check fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42419 from dsavineau/fix_typo_radosgw_admin_max_entries
Kefu Chai [Tue, 20 Jul 2021 15:40:13 +0000 (23:40 +0800)]
Merge pull request #42419 from dsavineau/fix_typo_radosgw_admin_max_entries

doc/man/8/radosgw-admin: fix typo on max entries

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42289 from mgfritch/cephadm-test-create-daemon-dirs-prom
Sebastian Wagner [Tue, 20 Jul 2021 15:35:09 +0000 (17:35 +0200)]
Merge pull request #42289 from mgfritch/cephadm-test-create-daemon-dirs-prom

cephadm: use pyfakefs during test_create_daemon_dirs_prometheus

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
4 years agoMerge pull request #42028 from mgfritch/cephadm-infer-mismatch-fsid
Sebastian Wagner [Tue, 20 Jul 2021 15:34:15 +0000 (17:34 +0200)]
Merge pull request #42028 from mgfritch/cephadm-infer-mismatch-fsid

cephadm: infer fsid from ceph.conf

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
4 years agoMerge pull request #41280 from sebastian-philipp/test_cephadm-stdin
Sebastian Wagner [Tue, 20 Jul 2021 15:33:20 +0000 (17:33 +0200)]
Merge pull request #41280 from sebastian-philipp/test_cephadm-stdin

qa/workunits/test_cephadm: Also test stdin

Reviewed-by: Michael Fritch <mfritch@suse.com>
4 years agodoc/man/8/radosgw-admin: fix typo on max entries 42419/head
Dimitri Savineau [Tue, 20 Jul 2021 14:49:02 +0000 (10:49 -0400)]
doc/man/8/radosgw-admin: fix typo on max entries

This fixes a typo for the --max-entries option.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
4 years agocrimson/tools/store_nbd: mark padding variable with [[maybe_unused]] 42418/head
Kefu Chai [Tue, 20 Jul 2021 14:47:07 +0000 (22:47 +0800)]
crimson/tools/store_nbd: mark padding variable with [[maybe_unused]]

also silences -Wunused-variable warning from GCC

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/tools/store_nbd: pass app.alien() down to FSDriver
Kefu Chai [Tue, 20 Jul 2021 14:45:50 +0000 (22:45 +0800)]
crimson/tools/store_nbd: pass app.alien() down to FSDriver

this was an oversight in 8fad523d41008727c9a952afcda15b77be50ba3c

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocephadm: haproxy 2.4 defaults to a different container user. 42415/head
Sebastian Wagner [Tue, 20 Jul 2021 14:09:57 +0000 (16:09 +0200)]
cephadm: haproxy 2.4 defaults to a different container user.

Another alternative would be to investigage a different setup
leverageing `--sysctl net.ipv4.ip_unprivileged_port_start=0`,
but that would be a larger PR.

Fixes: https://tracker.ceph.com/issues/51355
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
4 years agomgr/cephadm/templates: add jinja2 lint 42342/head
Dimitri Savineau [Wed, 14 Jul 2021 18:57:51 +0000 (14:57 -0400)]
mgr/cephadm/templates: add jinja2 lint

This adds a jinja2 lint environment in tox for testing the cephadm jinja2
templates.

This patch fixes some minor jinja2 syntax for ganesha and keepalived even if
the current templates work perfectly.

Tags should have one (and only one) space

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
4 years agocmake: use string(APPEND ...) to concat strings 42367/head
Kefu Chai [Tue, 20 Jul 2021 08:16:07 +0000 (16:16 +0800)]
cmake: use string(APPEND ...) to concat strings

less repeating this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocmake: add "-Og" to CMAKE_C_FLAGS_DEBUG
Kefu Chai [Fri, 16 Jul 2021 02:37:23 +0000 (10:37 +0800)]
cmake: add "-Og" to CMAKE_C_FLAGS_DEBUG

without specifying -Og, -O0 is used. as per
https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

> -O0
>  Reduce compilation time and make debugging produce the expected
>  results. This is the default.

and

> -Og
>  Optimize debugging experience. -Og should be the optimization level of
>  choice for the standard edit-compile-debug cycle, offering a reasonable
>  level of optimization while maintaining fast compilation and a good
>  debugging experience. It is a better choice than -O0 for producing
>  debuggable code because some compiler passes that collect debug
>  information are disabled at -O0.

and Debug is the default built type if .git directory is found under
the root source directory, so by adding "-Og -g" to CMAKE_C_FLAGS_DEBUG,
developers can have better debugging experience when testing Ceph
built from a git repo. but the downside is that it might take longer
to build the tree.

this change should also enable us to link crimson on aarch64.

the same applies to Clang,

> -O0 Means “no optimization”: this level compiles the fastest and generates the most debuggable code.
> -O1 Somewhere between -O0 and -O2.
> -O2 Moderate level of optimization which enables most optimizations.
> -Og Like -O1. In future versions, this option might disable different optimizations in order to improve debuggability.

see
https://clang.llvm.org/docs/CommandGuide/clang.html#cmdoption-o0

see also https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101448

Fixes: https://tracker.ceph.com/issues/51441
Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42413 from galsalomon66/fix_s3tests_yaml
Casey Bodley [Tue, 20 Jul 2021 12:04:59 +0000 (08:04 -0400)]
Merge pull request #42413 from galsalomon66/fix_s3tests_yaml

RGW:revert force-branch to be ceph-master (s3-tests)

Reviewed-by: Casey Bodley <cbodley@redhat.com>
4 years agoMerge pull request #42304 from sebastian-philipp/doc-dameon-add-note
Sebastian Wagner [Tue, 20 Jul 2021 10:09:33 +0000 (12:09 +0200)]
Merge pull request #42304 from sebastian-philipp/doc-dameon-add-note

doc/cephadm: add notes to `orch daemon add`

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>
4 years agoMerge pull request #42372 from sebastian-philipp/doc-cephadm-rgw-ssl
Sebastian Wagner [Tue, 20 Jul 2021 10:09:20 +0000 (12:09 +0200)]
Merge pull request #42372 from sebastian-philipp/doc-cephadm-rgw-ssl

doc/cephadm: Add RGW ssl

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>
4 years agorevert force-branch to be ceph-master 42413/head
galsalomon66 [Tue, 20 Jul 2021 09:48:00 +0000 (12:48 +0300)]
revert force-branch to be ceph-master

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
4 years agocrimson/os/seastore/cache: report dirty extents usage 42261/head
Yingxin Cheng [Thu, 15 Jul 2021 08:24:39 +0000 (16:24 +0800)]
crimson/os/seastore/cache: report dirty extents usage

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
4 years agocrimson/os/seastore/cache: report cached extents usage
Yingxin Cheng [Thu, 15 Jul 2021 08:09:51 +0000 (16:09 +0800)]
crimson/os/seastore/cache: report cached extents usage

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
4 years agocrimson/os/seastore: measure read transaction efforts that are successful
Yingxin Cheng [Thu, 15 Jul 2021 06:36:09 +0000 (14:36 +0800)]
crimson/os/seastore: measure read transaction efforts that are successful

In order to calculate the ratio of discarded vs successful effort from
read-only transactions.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
4 years agocrimson/os/seastore: measure transactional efforts that are discarded or committed
Yingxin Cheng [Thu, 15 Jul 2021 02:26:02 +0000 (10:26 +0800)]
crimson/os/seastore: measure transactional efforts that are discarded or committed

The efforts of a transaction include the number and bytes of its read,
mutate, retire and fresh extents, and the bytes of delta generated.

This helps to understand the following aspects:
* The ratio of discarded efforts vs committed efforts;
* The average efforts of a transaction;
* The distribution of read/mutate/delta/retire/fresh efforts;
* The memory overhead and potential disk overhead of a transaction;
* How early a transaction invalidation happens;
* The average extent length;

It is possible to extend the effort metrics to be labeled by extent
types, in case we want to distinguish and profile the efforts at the
sub-component level.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
4 years agocrimson/os/seastore: assert the committing delta is not empty
Yingxin Cheng [Thu, 15 Jul 2021 02:17:38 +0000 (10:17 +0800)]
crimson/os/seastore: assert the committing delta is not empty

It makes no sense to commit an empty delta. It is mostly an issue that
user forget to generate delta during mutation, or there are futile
copy-on-write operations.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
4 years agocrimson/os/seastore/cache: cleanup to use Cache::invalidate(t, conflicting_extent)
Yingxin Cheng [Wed, 14 Jul 2021 08:05:20 +0000 (16:05 +0800)]
crimson/os/seastore/cache: cleanup to use Cache::invalidate(t, conflicting_extent)

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
4 years agocrimson/os/seastore/cache: count transactional cache access/hit
Yingxin Cheng [Fri, 9 Jul 2021 07:09:55 +0000 (15:09 +0800)]
crimson/os/seastore/cache: count transactional cache access/hit

Labeled by source and extent-type.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
4 years agoMerge pull request #42396 from ronen-fr/wip-ronenf-debug-docker-1
Ronen Friedman [Tue, 20 Jul 2021 07:59:04 +0000 (10:59 +0300)]
Merge pull request #42396 from ronen-fr/wip-ronenf-debug-docker-1

script: fix a bug in naming the source branch in ceph-debug-docker.sh

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42355 from yuvalif/wip-yuval-fix-51681
Yuval Lifshitz [Tue, 20 Jul 2021 07:55:16 +0000 (10:55 +0300)]
Merge pull request #42355 from yuvalif/wip-yuval-fix-51681

rgw/notifications: send correct size in case of delete marker creation

4 years agoMerge pull request #42358 from yuvalif/wip-yuval-fix-51687
Yuval Lifshitz [Tue, 20 Jul 2021 07:54:55 +0000 (10:54 +0300)]
Merge pull request #42358 from yuvalif/wip-yuval-fix-51687

rgw/notification: send current time in complete multipart upload event

4 years agoMerge pull request #42189 from yuvalif/wip-yuval-fix-51530
Yuval Lifshitz [Tue, 20 Jul 2021 07:54:38 +0000 (10:54 +0300)]
Merge pull request #42189 from yuvalif/wip-yuval-fix-51530

rgw/http/notifications: support content type in HTTP POST messages

4 years agocrimson/seastore: add seastore operation latency metrics 42386/head
chunmei-liu [Thu, 15 Jul 2021 04:37:23 +0000 (21:37 -0700)]
crimson/seastore: add seastore operation latency metrics

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
4 years agoREADME.md: recommend -Og instead of -O0
Kefu Chai [Fri, 16 Jul 2021 02:32:33 +0000 (10:32 +0800)]
README.md: recommend -Og instead of -O0

per https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

> -Og
>  Optimize debugging experience. -Og should be the optimization level of
>  choice for the standard edit-compile-debug cycle, offering a reasonable
>  level of optimization while maintaining fast compilation and a good
>  debugging experience. It is a better choice than -O0 for producing
>  debuggable code because some compiler passes that collect debug
>  information are disabled at -O0.

so instead of using -O0, for better developer experience, -Og is
recommended.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoqa/standalone: fixing the timings when waiting for deep-scrub to start 42410/head
Ronen Friedman [Sun, 11 Jul 2021 07:43:03 +0000 (10:43 +0300)]
qa/standalone: fixing the timings when waiting for deep-scrub to start

initiate_and_fetch_state() initiates a scrub, then polls the published
PG state looking for 'scrubbing'. Calling flush_pg_stats() as part of
the polling process might cause the scrub and the following recovery to
be missed altogether.

Note: this polling mechanism is definitely not robust. Will be
redesigned in the future.

Fixes: https://tracker.ceph.com/issues/51581
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
4 years agomon/MgrMonitor: drop nautilus modules from always_on_modules 42408/head
Kefu Chai [Tue, 20 Jul 2021 03:01:56 +0000 (11:01 +0800)]
mon/MgrMonitor: drop nautilus modules from always_on_modules

the list of always_on_modules for the older versions is only for the
upgrade use case, where the monitor is still the old release, so
we have to keep an entry for the old release in the always_on_modules
map.

but since we don't upgrade from a release older than 2 releases, there
is no need to keep an entry for nautilus.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agodoc/mgr/administrator: document mgr_disabled_modules
Kefu Chai [Tue, 20 Jul 2021 02:51:06 +0000 (10:51 +0800)]
doc/mgr/administrator: document mgr_disabled_modules

this option prevent mgr from loading specified modules. which could
be handly when debugging issues with always_on_modules.

and document mgr_standby_modules as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42388 from tchaikov/wip-crimson-alien
Kefu Chai [Tue, 20 Jul 2021 00:08:21 +0000 (08:08 +0800)]
Merge pull request #42388 from tchaikov/wip-crimson-alien

crimson/osd: use app.alien() to initialize AlienStore::alien

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
4 years agoMerge pull request #42392 from tchaikov/wip-test-test_ipaddr
Kefu Chai [Tue, 20 Jul 2021 00:07:45 +0000 (08:07 +0800)]
Merge pull request #42392 from tchaikov/wip-test-test_ipaddr

test/test_ipaddr: always set ifa_flags

Reviewed-by: Neha Ojha <nojha@redhat.com>
4 years agotest/rgw: fix use of poll() with timers in unittest_rgw_dmclock_scheduler
Casey Bodley [Mon, 19 Jul 2021 22:07:47 +0000 (18:07 -0400)]
test/rgw: fix use of poll() with timers in unittest_rgw_dmclock_scheduler

the AsyncScheduler uses an asio timer to dispatch work to its executor
with an optional delay. when no delay is requested, it waits on the
timer with an expiration time in the past (crimson::dmclock::TimeZero)

tests are failing here because poll() is returning without executing the
handlers of those expired timers

asio implements these timers with timerfd and epoll. debugging with
strace, i see that these timers armed with timerfd_settime() are not
always immediately ready according to epoll_wait():

  eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK)   = 3
  epoll_create1(EPOLL_CLOEXEC)            = 4
  timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC) = 5
  epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLERR|EPOLLET, data={u32=14164052, u64=14164052}}) = 0
  epoll_ctl(4, EPOLL_CTL_ADD, 5, {events=EPOLLIN|EPOLLERR, data={u32=14164064, u64=14164064}}) = 0
  timerfd_settime(5, TFD_TIMER_ABSTIME, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=1}}, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=0}}) = 0
  epoll_wait(4, [{events=EPOLLIN, data={u32=14164052, u64=14164052}}], 128, 0) = 1
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [], 128, 0)               = 0
  epoll_wait(4, [{events=EPOLLIN, data={u32=14164064, u64=14164064}}], 128, 0) = 1

in this example, it took 6 calls to context.poll() before it was ready
to execute the timer's handler

to work around this, replace calls to context.poll() with calls to
context.run_for() with a very short duration

Fixes: https://tracker.ceph.com/issues/42788
Signed-off-by: Casey Bodley <cbodley@redhat.com>
4 years agodoc/dev/perf_counters: update docs to include more context about perf counter usage 42402/head
Laura Flores [Mon, 19 Jul 2021 17:17:58 +0000 (17:17 +0000)]
doc/dev/perf_counters: update docs to include more context about perf counter usage

The Perf Counters docs, although informative, are lacking for users and developers who are wondering what they can do with their perf counter data. I wrote an extra paragraph here that outlines some ways in which the counters can be used, including diagnosing problems in a cluster and identifying workload patterns.

Signed-off-by: Laura Flores <lflores@redhat.com>
4 years agoMerge pull request #42188 from votdev/issue_51408_motd
Ernesto Puerta [Mon, 19 Jul 2021 17:56:50 +0000 (19:56 +0200)]
Merge pull request #42188 from votdev/issue_51408_motd

mgr/dashboard: Add configurable MOTD or wall notification

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
4 years agocmake: drop set(VERSION ...) 42401/head
Kefu Chai [Mon, 19 Jul 2021 17:21:25 +0000 (01:21 +0800)]
cmake: drop set(VERSION ...)

project(...). this is the authentic place to set version number.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42102 from Matan-B/lua-request-userId
Yuval Lifshitz [Mon, 19 Jul 2021 15:01:27 +0000 (18:01 +0300)]
Merge pull request #42102 from Matan-B/lua-request-userId

rgw: Added request field to lua scripting

4 years agoMerge pull request #41927 from Matan-B/rfe-51193-add-luarocks-version
Yuval Lifshitz [Mon, 19 Jul 2021 15:00:51 +0000 (18:00 +0300)]
Merge pull request #41927 from Matan-B/rfe-51193-add-luarocks-version

rgw: add package version support to lua scripting

4 years agoMerge pull request #42074 from ljflores/wip-lflores-perf-channel
Josh Durgin [Mon, 19 Jul 2021 14:59:32 +0000 (07:59 -0700)]
Merge pull request #42074 from ljflores/wip-lflores-perf-channel

mgr/telemetry: add new 'perf' channel that shares aggregated perf counter metrics of a cluster

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>
4 years agoscript: fix a bug in naming the source branch in ceph-debug-docker.sh 42396/head
Ronen Friedman [Mon, 19 Jul 2021 09:57:33 +0000 (09:57 +0000)]
script: fix a bug in naming the source branch in ceph-debug-docker.sh

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
4 years agoMerge pull request #42391 from kelubo/master
Sebastian Wagner [Mon, 19 Jul 2021 09:07:34 +0000 (11:07 +0200)]
Merge pull request #42391 from kelubo/master

doc/cephadm: positions of words are interchanged

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
4 years agoMerge pull request #42393 from tchaikov/wip-cmake-cleanups
Kefu Chai [Mon, 19 Jul 2021 07:57:45 +0000 (15:57 +0800)]
Merge pull request #42393 from tchaikov/wip-cmake-cleanups

cmake: require CMake v3.16 and cleanups

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
4 years agocmake: do not check policies introduced before 3.16 42393/head
Kefu Chai [Wed, 10 Feb 2021 17:14:07 +0000 (01:14 +0800)]
cmake: do not check policies introduced before 3.16

see

- https://cmake.org/cmake/help/latest/policy/CMP0065.html
- https://cmake.org/cmake/help/latest/policy/CMP0074.html
...
- https://cmake.org/cmake/help/latest/policy/CMP0093.html

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocmake: use add_compile_options() to pass compile options
Kefu Chai [Mon, 19 Jul 2021 04:44:38 +0000 (12:44 +0800)]
cmake: use add_compile_options() to pass compile options

add_definitions() is for adding "-D" define flags, while "-include" does
not define flags, so let's be explicit and use add_compile_options()
instead.

this change partially reverts 5fb90bb79671b8e22f507728e4b4b948a591f98f.
back then, we were using ubuntu bionic for building windows packages.
but since we've switched to ubuntu focal, where cmake 3.16 is available,
there is no reason to stick with add_definitions() anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agodoc/cephadm: add missing "ceph" 42391/head
wangfei [Mon, 19 Jul 2021 04:03:30 +0000 (12:03 +0800)]
doc/cephadm: add missing "ceph"

The word ceph is missing.

Signed-off-by: "Wang,Fei" <wf.ab@126.com>
4 years agodoc/cephadm: correct a transposed word error
wangfei [Mon, 19 Jul 2021 01:53:40 +0000 (09:53 +0800)]
doc/cephadm: correct a transposed word error

The positions of two words are interchanged:
scans each cluster in the host ----> scans each host in the cluster

Signed-off-by: "Wang,Fei" <wf.ab@126.com>
4 years agocmake: require CMake v3.16
Kefu Chai [Mon, 19 Jul 2021 04:55:57 +0000 (12:55 +0800)]
cmake: require CMake v3.16

since we dropped the support of bionic, we now have the luxury of using
newer CMake! and by using CMake 3.16, we now have access to

* add_compile_definitions(), which was introduced in cmake 3.12

also, drop the comment of

> remove cmake/modules/FindPython* once 3.12 is required

because we need to support python 3.10 in future, better off keeping
the FindPython files around, so we can backport cmake changes from
upstream when necessary.

we are allowed to bump up the CMake's required to 3.16, because:

- ubuntu focal ships with cmake 3.16.3
- RHEL/CentOS8's AppStream has cmake 3.18

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agotest/test_ipaddr: avoid reinterpret cast 42392/head
Kefu Chai [Mon, 19 Jul 2021 03:26:50 +0000 (11:26 +0800)]
test/test_ipaddr: avoid reinterpret cast

drop couple occurrences of C-style cast, for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agotest/test_ipaddr: always set ifa_flags
Kefu Chai [Mon, 19 Jul 2021 03:22:18 +0000 (11:22 +0800)]
test/test_ipaddr: always set ifa_flags

do not rely on the uninitialized bits on stack for testing, otherwise
there is chance that we have following test failure:

64: ../src/test/test_ipaddr.cc:226: Failure
64: Expected equality of these values:
64:   two.ifa_addr
64:     Which is: 127.0.0.1:57677
64:   result
64:     Which is: 127.0.0.1:57677

the root cause is just that the "result" is actually `one.ifa_addr`,
which is marked UP as that's what the uninitialized bits on stack
indicate.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/osd: s/FuturizedStore*/FuturizedStore&/ in OSDMeta and PGMeta's ctors 42388/head
Kefu Chai [Mon, 19 Jul 2021 01:17:29 +0000 (09:17 +0800)]
crimson/osd: s/FuturizedStore*/FuturizedStore&/ in OSDMeta and PGMeta's ctors

since we always pass a valid store reference to these ctors, there is no
reason to use a pointer. let's use a reference instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/osd: create store in main()
Kefu Chai [Sun, 18 Jul 2021 14:44:59 +0000 (22:44 +0800)]
crimson/osd: create store in main()

to initialize an alienstore with a reference of
seastar::alien::instance is but the internal of AlienStore, would be
better if we could avoid exposing it as a part of the interface of
crimson::osd::OSD().

so, in this change, instead of creating FuturizedStore in OSD(), we
create it in main() where the app is available, so we can just create
FuturizedStore without passing the alien instance all the way down to
OSD().

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agocrimson/osd: use app.alien() to initialize AlienStore::alien
Kefu Chai [Sun, 18 Jul 2021 14:01:13 +0000 (22:01 +0800)]
crimson/osd: use app.alien() to initialize AlienStore::alien

in e53ea0886fba9073904f59ea85fb73d854565921, the new alien::submit_to() API
is used in the place of the old one. but seastar::alien::instance::_qs
should be initialized before we are able to use the alien instance, just
creating an instance of alien::instance is not enough.

in this change, instead of creating an instance of alien::instance using
make_unique<>, the return value of app.alien() is used to initialize the
alien member variable of AlienStore. app.alien() is always properly
initialized by reactor.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoosdc/Objecter: avoid vector copies in _calc_target 42284/head
Or Ozeri [Sun, 6 Jun 2021 09:34:05 +0000 (12:34 +0300)]
osdc/Objecter: avoid vector copies in _calc_target

Use std::move on temporary vector to claim its elements,
instead of copying them.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
4 years agoMerge pull request #42360 from noviicee/noviicee
Kefu Chai [Sun, 18 Jul 2021 03:16:39 +0000 (11:16 +0800)]
Merge pull request #42360 from noviicee/noviicee

mgr/devicehealth: replace CLICommand with CLIReadCommand

Reviewed-by: Varsha Rao <varao@redhat.com>
4 years agoMerge pull request #42379 from liewegas/cephadm-keyring-help
Kefu Chai [Sun, 18 Jul 2021 03:15:00 +0000 (11:15 +0800)]
Merge pull request #42379 from liewegas/cephadm-keyring-help

mgr/cephadm: add help strings for 'orch client-keyring ...' commands

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
4 years agoMerge pull request #42036 from kamoltat/wip-pg-autoscaler-overlap
Kefu Chai [Sun, 18 Jul 2021 03:07:42 +0000 (11:07 +0800)]
Merge pull request #42036 from kamoltat/wip-pg-autoscaler-overlap

pybind/mgr/autoscaler: don't scale pools with overlapping roots

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
4 years agoMerge pull request #42357 from cbodley/wip-cmake-legacy-option-headers-dep
Kefu Chai [Sun, 18 Jul 2021 03:01:54 +0000 (11:01 +0800)]
Merge pull request #42357 from cbodley/wip-cmake-legacy-option-headers-dep

cmake: add transitive dependency on legacy-option-headers

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42283 from orozery/op_target_t_constructor
Kefu Chai [Sun, 18 Jul 2021 03:01:25 +0000 (11:01 +0800)]
Merge pull request #42283 from orozery/op_target_t_constructor

common: optimize op_target_t and hobject_t constructors

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42366 from liewegas/mon-down-grace
Kefu Chai [Sun, 18 Jul 2021 02:59:54 +0000 (10:59 +0800)]
Merge pull request #42366 from liewegas/mon-down-grace

mon: allow a MON_DOWN grace period after cluster mkfs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
4 years agoMerge pull request #42315 from TheJJ/fix-op-duration-calculation
Kefu Chai [Sun, 18 Jul 2021 02:59:18 +0000 (10:59 +0800)]
Merge pull request #42315 from TheJJ/fix-op-duration-calculation

osd: fix op event duration calculation

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #42385 from tchaikov/wip-crimson-test-alien
Kefu Chai [Sat, 17 Jul 2021 05:35:33 +0000 (13:35 +0800)]
Merge pull request #42385 from tchaikov/wip-crimson-test-alien

test/crimson/test_alien_echo: fixes

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
4 years agoMerge pull request #42351 from SoftIron/softiron-distro
Kefu Chai [Sat, 17 Jul 2021 02:34:12 +0000 (10:34 +0800)]
Merge pull request #42351 from SoftIron/softiron-distro

build/qa: Add SoftIron Linux as valid distro in build and test scripts

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agotest/crimson/test_alien_echo: msgr->stop() before shutdown 42385/head
Kefu Chai [Sat, 17 Jul 2021 02:23:20 +0000 (10:23 +0800)]
test/crimson/test_alien_echo: msgr->stop() before shutdown

it is required to remove all registered dispatchers before shutting down a
messenger. otherwise assert(dispatchers.empty()) in SocketMessenger::shutdown() fails.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agotest/crimson/test_alien_echo: drop msgr v1 support
Kefu Chai [Sat, 17 Jul 2021 02:05:08 +0000 (10:05 +0800)]
test/crimson/test_alien_echo: drop msgr v1 support

crimson msgr dropped the v1 support back in
7a7e869c0a7d96e1e242929755c3af73085d00ac. so there is no point to offer
an option to use v1 protocol.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agodoc/cephfs: recover file system after recovering 42295/head
Ramana Raja [Tue, 13 Jul 2021 05:10:50 +0000 (01:10 -0400)]
doc/cephfs: recover file system after recovering

... monitor stores using OSDs. The steps are valid only to recover
single active MDS file systems.

Partially-fixes: https://tracker.ceph.com/issues/51341
Signed-off-by: Ramana Raja <rraja@redhat.com>
4 years agoqa/*/test_envlibrados_for_rocksdb.sh: remove OS specific configuration 42429/head
Neha Ojha [Fri, 16 Jul 2021 17:20:34 +0000 (17:20 +0000)]
qa/*/test_envlibrados_for_rocksdb.sh: remove OS specific configuration

Details in https://tracker.ceph.com/issues/51638#note-3

Signed-off-by: Neha Ojha <nojha@redhat.com>