]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
4 years agoMerge pull request #37954 from joscollin/wip-F46226-log-dq-throttle-limit-new
fullerdj [Wed, 3 Feb 2021 15:42:40 +0000 (10:42 -0500)]
Merge pull request #37954 from joscollin/wip-F46226-log-dq-throttle-limit-new

msg,mon,common: log when DispatchQueue throttle limit is reached

4 years agoMerge PR #39069 into master
Sage Weil [Wed, 3 Feb 2021 15:38:49 +0000 (10:38 -0500)]
Merge PR #39069 into master

* refs/pull/39069/head:
mgr/cephadm/upgrade: tolerate pre-pacific upgrade state
mgr/cephadm/upgrade: scale down MDS cluster(s) for major version upgrades
mgr/cephadm: fix capitalization, level; drop elipses of log msgs
mgr/cephadm/upgrade: match against any repo_digest, not image_id
cephadm: return repo_digests (plural) in pull/inspect output
mgr/cephadm: include container_image_digests in inventory
cephadm: include image_digests list in 'ls' output
vstart.sh: only extract first container digest
mgr/cephadm: move release -> major translation to helper
mgr/cephadm/upgrade: tolerate old upgrade_state.target_versoin
mgr/cephadm/upgrade: set require-osd-release when done with OSDs
mgr: add lookup_release_name(int) to mgr interface
mgr/cephadm: verify container image version after we pull it
mgr/cephadm: only save version portion of version string
cephadm: fix 'inspect' and 'pull'
mgr/cephadm/upgrade: implement N-2 version checks on upgrade start

Reviewed-by: Sebastian Wagner <swagner@suse.com>
4 years agoMerge pull request #39234 from neha-ojha/wip-49064
Neha Ojha [Wed, 3 Feb 2021 14:56:02 +0000 (06:56 -0800)]
Merge pull request #39234 from neha-ojha/wip-49064

qa/*/test_envlibrados_for_rocksdb: use osd_client_message_cap to prevent slow requests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
4 years agoMerge pull request #38955 from varshar16/wip-doc-update-nfs-ganesha-dynamic
Jeff Layton [Wed, 3 Feb 2021 12:45:38 +0000 (07:45 -0500)]
Merge pull request #38955 from varshar16/wip-doc-update-nfs-ganesha-dynamic

doc/cephfs/nfs: Add rook pod restart note, export and log block example

4 years agoMerge pull request #38911 from pcuzner/add-cluster-network-to-bootstrap
pcuzner [Wed, 3 Feb 2021 08:53:54 +0000 (21:53 +1300)]
Merge pull request #38911 from pcuzner/add-cluster-network-to-bootstrap

cephadm: Add cluster network to bootstrap

4 years agoMerge pull request #39239 from athanatos/sjust/wip-49114
Samuel Just [Wed, 3 Feb 2021 03:11:41 +0000 (19:11 -0800)]
Merge pull request #39239 from athanatos/sjust/wip-49114

crimson/os/seastore: fix staged-fltree build problems

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
4 years agocrimson/.../staged-fltree: propogate eagain in node, value 39239/head
Samuel Just [Wed, 3 Feb 2021 00:58:54 +0000 (16:58 -0800)]
crimson/.../staged-fltree: propogate eagain in node, value

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agocrimson/.../staged-fltree: fix tree_cursor_t::(trim|extend)_value
Samuel Just [Fri, 22 Jan 2021 00:46:13 +0000 (00:46 +0000)]
crimson/.../staged-fltree: fix tree_cursor_t::(trim|extend)_value

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agoMerge PR #39206 into master
Sage Weil [Tue, 2 Feb 2021 22:47:52 +0000 (17:47 -0500)]
Merge PR #39206 into master

* refs/pull/39206/head:
mgr/MetricTypes: condition encoding on feature bits

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
4 years agoMerge pull request #38924 from cyx1231st/wip-seastar-onode-tree-onode_t
Samuel Just [Tue, 2 Feb 2021 21:05:39 +0000 (13:05 -0800)]
Merge pull request #38924 from cyx1231st/wip-seastar-onode-tree-onode_t

crimson/onode-staged-tree: implement an extensive Value framework

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoqa/*/test_envlibrados_for_rocksdb: use osd_client_message_cap to prevent slow requests 39234/head
Neha Ojha [Tue, 2 Feb 2021 17:43:56 +0000 (17:43 +0000)]
qa/*/test_envlibrados_for_rocksdb: use osd_client_message_cap to prevent slow requests

EnvLibradosMutipoolTest.DBBulkLoadKeysInRandomOrder can overload OSDs and cause
heartbeat timeouts. Tests in test_envlibrados_for_rocksdb also generate slow
requests on OSDs. Use osd_client_message_cap to prevent this.
Since this option is disabled by default, this may be a good way to exercise it.

Fixes: https://tracker.ceph.com/issues/49064
Signed-off-by: Neha Ojha <nojha@redhat.com>
4 years agoMerge pull request #39209 from idryomov/wip-unbreak-rbd-task
Ilya Dryomov [Tue, 2 Feb 2021 17:16:56 +0000 (18:16 +0100)]
Merge pull request #39209 from idryomov/wip-unbreak-rbd-task

qa/tasks/rbd: fix regressions introduced with rbd encryption support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
4 years agoMerge PR #39212 into master
Sage Weil [Tue, 2 Feb 2021 16:49:07 +0000 (11:49 -0500)]
Merge PR #39212 into master

* refs/pull/39212/head:
cpatch: python 3.6 -> 3.8
cpatch: new default base image

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
4 years agoMerge pull request #39207 from zdover23/wip-doc-removing-workbench-docs
zdover23 [Tue, 2 Feb 2021 16:14:24 +0000 (02:14 +1000)]
Merge pull request #39207 from zdover23/wip-doc-removing-workbench-docs

doc/dev: remove workbench sections

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
4 years agoMerge pull request #39227 from tchaikov/wip-48825
Kefu Chai [Tue, 2 Feb 2021 16:11:41 +0000 (00:11 +0800)]
Merge pull request #39227 from tchaikov/wip-48825

mgr/zabbix: format ceph.[{#POOL},percent_used as float

Reviewed-by: Wido den Hollander <wido@widodh.nl>
4 years agoMerge pull request #39225 from tchaikov/wip-crimson-seastore-dec-ref
Kefu Chai [Tue, 2 Feb 2021 16:06:03 +0000 (00:06 +0800)]
Merge pull request #39225 from tchaikov/wip-crimson-seastore-dec-ref

crimson/os/seastore: don't use variadic template for dec_ref()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
4 years agoMerge pull request #38050 from ifed01/wip-ifed-fsck-spanning-blob
Igor Fedotov [Tue, 2 Feb 2021 15:09:08 +0000 (18:09 +0300)]
Merge pull request #38050 from ifed01/wip-ifed-fsck-spanning-blob

os/bluestore: detect and fix "zombie" spanning blobs using fsck.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
4 years agodoc/cephfs/nfs: Add rook pod restart note, export and log block example 38955/head
Varsha Rao [Mon, 18 Jan 2021 16:24:32 +0000 (21:54 +0530)]
doc/cephfs/nfs: Add rook pod restart note, export and log block example

Fixes: https://tracker.ceph.com/issues/48914
Signed-off-by: Varsha Rao <varao@redhat.com>
4 years agomgr/zabbix: format ceph.[{#POOL},percent_used as float 39227/head
Kefu Chai [Tue, 2 Feb 2021 13:36:18 +0000 (21:36 +0800)]
mgr/zabbix: format ceph.[{#POOL},percent_used as float

also, multiply it with 100, as we are using 1.0 for 100% here.

for more details regarding the mapping from value_type to its values.
see
https://www.zabbix.com/documentation/4.0/manual/api/reference/item/object

Fixes: https://tracker.ceph.com/issues/48825
Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoos/bluestore: detect and fix "zombie" spanning blobs using fsck. 38050/head
Igor Fedotov [Thu, 12 Nov 2020 17:04:45 +0000 (20:04 +0300)]
os/bluestore: detect and fix "zombie" spanning blobs using fsck.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
4 years agocrimson/os/seastore: don't use variadic template for dec_ref() 39225/head
Kefu Chai [Tue, 2 Feb 2021 12:08:41 +0000 (20:08 +0800)]
crimson/os/seastore: don't use variadic template for dec_ref()

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #39041 from ifed01/wip-ifed-fix-48966
Igor Fedotov [Tue, 2 Feb 2021 11:10:01 +0000 (14:10 +0300)]
Merge pull request #39041 from ifed01/wip-ifed-fix-48966

os/bluestore: fix a bug causing unexpected Onode's unpinned state.

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge PR #38869 into master
Jan Fajerski [Tue, 2 Feb 2021 09:46:35 +0000 (10:46 +0100)]
Merge PR #38869 into master

* refs/pull/38869/head:
ceph-volume: Fix usage of is_lv

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
4 years agoMerge pull request #38975 from athanatos/sjust/wip-seastore-pipelines
Kefu Chai [Tue, 2 Feb 2021 09:40:37 +0000 (17:40 +0800)]
Merge pull request #38975 from athanatos/sjust/wip-seastore-pipelines

seastore: update to support pipelined write operations

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoos/bluestore: fix a bug causing unexpected Onode's unpinned state. 39041/head
Igor Fedotov [Sat, 23 Jan 2021 17:33:13 +0000 (20:33 +0300)]
os/bluestore: fix a bug causing unexpected Onode's unpinned state.

There could be a race for Onodes put() and get() methods:

put()(pinned, nref=3)
  int n = --nref; (nref = 2)
  if (n == 2) {
    ..
    std::lock_guard l(ocs->lock);
    ...
    pinned = pinned && nref > 2; (= false)
    ...                                     get()
    if (r) {                                ++nref; (=3)
      n = --nref; (nref = 2)                return;
    }
    ...
    return

As a result nref = 2, pinned = false which is wrong

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
4 years agoMerge pull request #39205 from rzarzynski/wip-crimson-configurable-alien-threads-num
Kefu Chai [Tue, 2 Feb 2021 06:04:59 +0000 (14:04 +0800)]
Merge pull request #39205 from rzarzynski/wip-crimson-configurable-alien-threads-num

crimson: make the number of alien threads configurable.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
4 years agocephadm:updates following bootstrap code clean up 38911/head
Paul Cuzner [Tue, 2 Feb 2021 02:42:00 +0000 (15:42 +1300)]
cephadm:updates following bootstrap code clean up

Patch to align to new(ish) bootstrap code changes

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
4 years agocephadm: updated doc text
Paul Cuzner [Thu, 28 Jan 2021 03:35:57 +0000 (16:35 +1300)]
cephadm: updated doc text

The phrase "recommended"  has been removed.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
4 years agocephadm: unit test updates for cluster-network handling
Paul Cuzner [Sun, 17 Jan 2021 23:24:16 +0000 (12:24 +1300)]
cephadm: unit test updates for cluster-network handling

Tests updated to handle comma separated subnet lists

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
4 years agocephadm: updated handling of cluster_network parameter
Paul Cuzner [Sun, 17 Jan 2021 23:20:23 +0000 (12:20 +1300)]
cephadm: updated handling of cluster_network parameter

Parameter now supports a comma separated list of subnets and
detects the ipv4 or ipv6 setting on the cluster network to enable
osds to bind correctly.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
4 years agocephadm: install doc updated to include cluster-network parameter
Paul Cuzner [Thu, 14 Jan 2021 22:08:48 +0000 (11:08 +1300)]
cephadm: install doc updated to include cluster-network parameter

Install guide updated to include a description of the --cluster-network
parameter. The text also links to the complete definition for cluster-network
on the rados/configuration/network-config-ref page.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
4 years agocephadm: unit test updates for new subnet function
Paul Cuzner [Thu, 14 Jan 2021 20:56:15 +0000 (09:56 +1300)]
cephadm: unit test updates for new subnet function

Adds tests to validate the check_subnet function will work as
expected with various inputs

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
4 years agocephadm: Add cluster network parameter to bootstrap cmd
Paul Cuzner [Thu, 14 Jan 2021 20:53:34 +0000 (09:53 +1300)]
cephadm: Add cluster network parameter to bootstrap cmd

Adds a --cluster-network parameter to the bootstrap cmd to
set the internal cluster network.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
4 years agoMerge pull request #38956 from zdover23/2021_Jan_19_release_notes_grammar
Josh Durgin [Mon, 1 Feb 2021 23:31:48 +0000 (15:31 -0800)]
Merge pull request #38956 from zdover23/2021_Jan_19_release_notes_grammar

doc/PendingReleaseNotes: grammar and wording

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
4 years agomgr/cephadm/upgrade: tolerate pre-pacific upgrade state 39069/head
Sage Weil [Sat, 30 Jan 2021 22:35:21 +0000 (16:35 -0600)]
mgr/cephadm/upgrade: tolerate pre-pacific upgrade state

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agomgr/cephadm/upgrade: scale down MDS cluster(s) for major version upgrades
Sage Weil [Tue, 26 Jan 2021 22:10:13 +0000 (16:10 -0600)]
mgr/cephadm/upgrade: scale down MDS cluster(s) for major version upgrades

For octopus -> pacific, as with other recent releases, we need to scale
down the MDS cluster(s) to a single daemon before upgrading.  (This is
because the MDS intra-cluster protocols aren't fully versioned.)

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agomgr/cephadm: fix capitalization, level; drop elipses of log msgs
Sage Weil [Thu, 28 Jan 2021 15:02:27 +0000 (09:02 -0600)]
mgr/cephadm: fix capitalization, level; drop elipses of log msgs

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agomgr/cephadm/upgrade: match against any repo_digest, not image_id
Sage Weil [Wed, 27 Jan 2021 14:54:00 +0000 (08:54 -0600)]
mgr/cephadm/upgrade: match against any repo_digest, not image_id

The image id can vary across hosts and (most notably) docker vs podman.
Instead, use the repo_digest as an image identifier.

Unfortunately, a single image may have multiple digests, even within the
same registry, so keep a list of the digests for the image we are
upgrading to, and ensure that each container has a digest that matches at
least one of them.

This allows upgrade to proceed in mixed docker+podman clusters.  However,
it does not yet address a cluster with mixed CPU architectures, because
the container image will have different digest(s) for each architecture
build.

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agocephadm: return repo_digests (plural) in pull/inspect output
Sage Weil [Wed, 27 Jan 2021 14:12:00 +0000 (08:12 -0600)]
cephadm: return repo_digests (plural) in pull/inspect output

When we inspect a (pulled) image, return all of the repo digests.  Update
the mgr/cephadm code accordingly to match.

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agoMerge PR #39149 into master
Sage Weil [Mon, 1 Feb 2021 22:25:01 +0000 (16:25 -0600)]
Merge PR #39149 into master

* refs/pull/39149/head:
qa/workunits/cephtool/test.sh: tolerate some settling for osd info vs dump test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
4 years agomgr/cephadm: include container_image_digests in inventory
Sage Weil [Wed, 27 Jan 2021 14:02:25 +0000 (08:02 -0600)]
mgr/cephadm: include container_image_digests in inventory

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agocephadm: include image_digests list in 'ls' output
Sage Weil [Wed, 27 Jan 2021 13:50:14 +0000 (07:50 -0600)]
cephadm: include image_digests list in 'ls' output

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agovstart.sh: only extract first container digest
Sage Weil [Tue, 26 Jan 2021 19:11:13 +0000 (13:11 -0600)]
vstart.sh: only extract first container digest

Otherwise the container_image value includes the same hash twice, separated
by a newline, causing all sorts of confusion.

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agomgr/cephadm: move release -> major translation to helper
Sage Weil [Tue, 26 Jan 2021 15:24:54 +0000 (09:24 -0600)]
mgr/cephadm: move release -> major translation to helper

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agomgr/cephadm/upgrade: tolerate old upgrade_state.target_versoin
Sage Weil [Tue, 26 Jan 2021 15:24:38 +0000 (09:24 -0600)]
mgr/cephadm/upgrade: tolerate old upgrade_state.target_versoin

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agomgr/cephadm/upgrade: set require-osd-release when done with OSDs
Sage Weil [Mon, 25 Jan 2021 23:21:05 +0000 (17:21 -0600)]
mgr/cephadm/upgrade: set require-osd-release when done with OSDs

After we've upgraded all of the osds, complete the major version upgrade
for core RADOS by setting the OSD cluster min.

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agomgr: add lookup_release_name(int) to mgr interface
Sage Weil [Mon, 25 Jan 2021 23:19:59 +0000 (17:19 -0600)]
mgr: add lookup_release_name(int) to mgr interface

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agomgr/cephadm: verify container image version after we pull it
Sage Weil [Mon, 25 Jan 2021 22:47:16 +0000 (16:47 -0600)]
mgr/cephadm: verify container image version after we pull it

Apply the version checks to make sure this upgrade/downgrade is possible.

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agomgr/cephadm: only save version portion of version string
Sage Weil [Mon, 25 Jan 2021 22:46:43 +0000 (16:46 -0600)]
mgr/cephadm: only save version portion of version string

We don't want to 'ceph version ' prefix or sha1 suffix.  This matches us
up with 'orch upgrade start'

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agocephadm: fix 'inspect' and 'pull'
Sage Weil [Mon, 25 Jan 2021 22:32:45 +0000 (16:32 -0600)]
cephadm: fix 'inspect' and 'pull'

With podman 2.2.1 I would get

ERRO[0000] Error printing inspect output: template: all inspect:1: function "json" not defined

Removing the 'json' portion of the query resolves it.

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agomgr/cephadm/upgrade: implement N-2 version checks on upgrade start
Sage Weil [Mon, 25 Jan 2021 20:00:04 +0000 (14:00 -0600)]
mgr/cephadm/upgrade: implement N-2 version checks on upgrade start

- Prevent major upgrades that span > 2 releases
- Prevent downgrades to an rc or dev release

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agocpatch: python 3.6 -> 3.8 39212/head
Sage Weil [Mon, 1 Feb 2021 19:12:16 +0000 (13:12 -0600)]
cpatch: python 3.6 -> 3.8

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agocpatch: new default base image
Sage Weil [Mon, 1 Feb 2021 19:04:40 +0000 (13:04 -0600)]
cpatch: new default base image

Signed-off-by: Sage Weil <sage@newdream.net>
4 years agocrimson/os/seastore/segment_cleaner: tolerate mark_space_used on closed segment 38975/head
Samuel Just [Tue, 19 Jan 2021 20:49:49 +0000 (12:49 -0800)]
crimson/os/seastore/segment_cleaner: tolerate mark_space_used on closed segment

A journal segment can be closed while outstanding writes on that
segment have not yet updated metadata.

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agocrimson/os/seastore/journal: tolerate journal rolls during commit
Samuel Just [Tue, 19 Jan 2021 20:48:36 +0000 (12:48 -0800)]
crimson/os/seastore/journal: tolerate journal rolls during commit

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agocrimson/os/seastore: add basic pipeline phases to TransactionManager
Samuel Just [Mon, 11 Jan 2021 23:03:31 +0000 (15:03 -0800)]
crimson/os/seastore: add basic pipeline phases to TransactionManager

We need to ensure that the metadata preperation and completions happen
in order.

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agocrimson/common/operation: add OrderedConcurrentPhase
Samuel Just [Tue, 19 Jan 2021 03:46:30 +0000 (19:46 -0800)]
crimson/common/operation: add OrderedConcurrentPhase

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agosrc/crimson/common: decouple handle from stage
Samuel Just [Mon, 18 Jan 2021 22:06:18 +0000 (14:06 -0800)]
src/crimson/common: decouple handle from stage

We're going to introduce stages with other properties.

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agocrimson/os/seastore: fix find_hole
Samuel Just [Fri, 15 Jan 2021 01:26:13 +0000 (17:26 -0800)]
crimson/os/seastore: fix find_hole

We actually need to propogate eagain errors now, fix to use do_until.

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agocrimson/os/seastore: annotate all assert_all instances
Samuel Just [Fri, 15 Jan 2021 01:25:48 +0000 (17:25 -0800)]
crimson/os/seastore: annotate all assert_all instances

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agostore-nbd: support concurrent operations
Samuel Just [Tue, 8 Dec 2020 23:49:34 +0000 (15:49 -0800)]
store-nbd: support concurrent operations

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agocrimson/os/seastore: validate node immediately prior to adding pin
Samuel Just [Thu, 14 Jan 2021 03:43:07 +0000 (03:43 +0000)]
crimson/os/seastore: validate node immediately prior to adding pin

Mostly, operating on an invalid extent is harmless by construction --
it'll be caught by the read set check in submit_transaction.  However,
prior to adding an extent to the lba pin set, we need to check that it's
really still live, which we can do by checking that it and its immediate
parent are still valid.  Mutating the target extent would have
invalidated it, and replacing the target extent would necessarily have
involved a mutation to the parent, which would have invalidated it.

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agocrimson/os/seastore: expand error type for reads to include eagain
Samuel Just [Thu, 14 Jan 2021 07:42:36 +0000 (23:42 -0800)]
crimson/os/seastore: expand error type for reads to include eagain

There are cases where reads may detect an invalid cached extent and require
the user to restart the transaction.

As a side effect, clean up errorators to assert cases where construction
is invalid (invalid refcounts, enoent or invarg on node read, etc).  In
such cases, we may eventually want to introduce an internal_error error
type, but asserting is fine for now as we do not expect users to handle
these cases.

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agocrimson/common/errorator: add extend_ertr helper for merging errorators
Samuel Just [Thu, 14 Jan 2021 07:40:17 +0000 (23:40 -0800)]
crimson/common/errorator: add extend_ertr helper for merging errorators

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agocrimson/os/seastore/cache: put root in t.mutated_extents and avoid special behavior
Samuel Just [Thu, 14 Jan 2021 01:54:42 +0000 (17:54 -0800)]
crimson/os/seastore/cache: put root in t.mutated_extents and avoid special behavior

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agocrimson/common/operation.h: permit moving Handle objects
Samuel Just [Wed, 13 Jan 2021 06:00:57 +0000 (22:00 -0800)]
crimson/common/operation.h: permit moving Handle objects

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agotest/crimson/seastore/test_transaction_manager: add tests for concurrent IO
Samuel Just [Mon, 11 Jan 2021 23:03:24 +0000 (15:03 -0800)]
test/crimson/seastore/test_transaction_manager: add tests for concurrent IO

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agocrimson/os/seastore/cached_extent: add asserts for debugging
Samuel Just [Mon, 11 Jan 2021 23:01:26 +0000 (15:01 -0800)]
crimson/os/seastore/cached_extent: add asserts for debugging

Signed-off-by: Samuel Just <sjust@redhat.com>
4 years agoMerge pull request #36518 from yongseokoh/fix-dashboard
Ernesto Puerta [Mon, 1 Feb 2021 20:46:44 +0000 (21:46 +0100)]
Merge pull request #36518 from yongseokoh/fix-dashboard

mgr/dashboard: Fix missing root path of each session for CephFS

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
4 years agomgr/MetricTypes: condition encoding on feature bits 39206/head
Sage Weil [Sun, 31 Jan 2021 16:29:51 +0000 (10:29 -0600)]
mgr/MetricTypes: condition encoding on feature bits

Do not encode new metric|config types for older peers.

The combination of boost::optional and boost::variant make this super
awkward to condition on the features. :(

Fixes: https://tracker.ceph.com/issues/49069
Signed-off-by: Sage Weil <sage@newdream.net>
4 years agoMerge pull request #38805 from kamoltat/wip-mgr-pg-autoscaler-improve-out-of-box...
Josh Durgin [Mon, 1 Feb 2021 19:33:55 +0000 (11:33 -0800)]
Merge pull request #38805 from kamoltat/wip-mgr-pg-autoscaler-improve-out-of-box-experience

pybind/mgr/pg_autoscaler: avoid scale-down until there is pressure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
4 years agoqa/tasks/rbd: don't wait for krbd symlink to be created/removed 39209/head
Ilya Dryomov [Sun, 31 Jan 2021 14:24:32 +0000 (15:24 +0100)]
qa/tasks/rbd: don't wait for krbd symlink to be created/removed

Commit 3754c665a11e (":qa/tasks/rbd: test qemu on top of rbd
encryption") broke dev_create() for krbd by messing up the "wait
for the symlink to be created by udev" loop.  The rbd tool has been
synchronizing with udev internally since 2014, so rather than fixing
let's just drop it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
4 years agoqa/tasks/rbd: dev_create() expects a properties dict
Ilya Dryomov [Sun, 31 Jan 2021 13:27:52 +0000 (14:27 +0100)]
qa/tasks/rbd: dev_create() expects a properties dict

In order to add encryption_format property, commit 3754c665a11e
(":qa/tasks/rbd: test qemu on top of rbd encryption") changed the
dev_create() contract.  It now expects a properties dict, but the
main task routine wasn't updated and still passes role_images.

On top of that, role_images itself got broken.  It is supposed
to be a mapping from role to a corresponding image name, not to
a dict with an image_name key.  This affected generic_mount().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
4 years agodoc/PendingReleaseNotes: grammar and wording 38956/head
Zac Dover [Mon, 18 Jan 2021 16:38:59 +0000 (02:38 +1000)]
doc/PendingReleaseNotes: grammar and wording

This commit vets the PendingReleaseNotes for Octopus.
This might not be the only commit that updates the
release notes, but it is the first by me.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
4 years agodoc:dev remove workbench sections 39207/head
Zac Dover [Mon, 1 Feb 2021 15:01:25 +0000 (01:01 +1000)]
doc:dev remove workbench sections

This PR removes the following two sections:

1. Getting Ceph Workbench
2. Linking Ceph Workbench With Your OpenStack Tenant

Fixes: https://tracker.ceph.com/issues/48942
Signed-off-by: Zac Dover <zac.dover@gmail.com>
4 years agocrimson: make the number of alien threads configurable. 39205/head
Radoslaw Zarzynski [Mon, 1 Feb 2021 13:59:49 +0000 (14:59 +0100)]
crimson: make the number of alien threads configurable.

It's particularly important for reads to have large-enough
thread pool in `AlienStore` as they are synchronous; that
is, e.g. `BlueStore` blocks on IO when handling reads.

This commit introduces a configurable allowing operators
to increase the number from `1` which we were limited to
before the change.

Naming similarity with `osd_op_num_threads_per_shard` is
intentional.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
4 years agomgr/pg_autoscaler: avoid scale-down until there is pressure 38805/head
Kamoltat [Thu, 7 Jan 2021 15:39:19 +0000 (15:39 +0000)]
mgr/pg_autoscaler: avoid scale-down until there is pressure

The autoscaler will start out with scaling each
pools to have a full complements of pgs from the start
and will only decrease it when pools need more due to
increased usage.

Introduced a unit test that tests only the
function get_final_pg_target_and_ratio() which
deals with the distrubtion of pgs amongst the
pools

Edited workunit script to reflect the change
of how pgs are calculated and distrubted.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
4 years agoMerge pull request #39200 from mitsu-ko/mypatch
Sebastian Wagner [Mon, 1 Feb 2021 12:22:50 +0000 (13:22 +0100)]
Merge pull request #39200 from mitsu-ko/mypatch

doc/mgr/orchestrator: Unify the content of command and yaml

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #39201 from tchaikov/wip-sphinx-openapi
Kefu Chai [Mon, 1 Feb 2021 12:15:23 +0000 (20:15 +0800)]
Merge pull request #39201 from tchaikov/wip-sphinx-openapi

doc: silence warnings from openapi sphinx extension

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
4 years agoMerge pull request #39194 from tchaikov/wip-dashboard-enum
Kefu Chai [Mon, 1 Feb 2021 11:11:32 +0000 (19:11 +0800)]
Merge pull request #39194 from tchaikov/wip-dashboard-enum

pybind/mgr/dashboard: do not install enum34

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
4 years agodoc/mgr/orchestrator: Unify the content of command and yaml 39200/head
Mitsumasa KONDO [Mon, 1 Feb 2021 09:44:49 +0000 (18:44 +0900)]
doc/mgr/orchestrator: Unify the content of command and yaml

Signed-off-by: Mitsumasa KONDO <kondo.mitsumasa@gmail.com>
4 years agodoc: silence warnings from openapi sphinx extension 39201/head
Kefu Chai [Mon, 1 Feb 2021 09:53:27 +0000 (17:53 +0800)]
doc: silence warnings from openapi sphinx extension

it prints out

skipping non-JSON example generation.

if an API does not contains example while ":example:" is specified for
openapi extension, and if this API is not a GET request, openapi
extention complains.

see also
https://github.com/sphinx-contrib/openapi/blob/9dbae9c9a65291c44b27041c63ded1b20611a2fc/sphinxcontrib/openapi/openapi30.py#L191

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #39198 from tchaikov/wip-mypy.ini
Sebastian Wagner [Mon, 1 Feb 2021 08:48:04 +0000 (09:48 +0100)]
Merge pull request #39198 from tchaikov/wip-mypy.ini

mypy.ini: fix the unresolve conflict

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
4 years agoMerge pull request #39196 from badone/wip-doc-python3-api
zdover23 [Mon, 1 Feb 2021 07:18:35 +0000 (17:18 +1000)]
Merge pull request #39196 from badone/wip-doc-python3-api

doc/rados/api: Update code samples to python3

Reviewed-by: Zac Dover <zac.dover@gmail.com>
4 years agomypy.ini: fix the unresolved conflict 39198/head
Kefu Chai [Mon, 1 Feb 2021 06:53:03 +0000 (14:53 +0800)]
mypy.ini: fix the unresolved conflict

it was introduced by a0930a63106e9f0d556e18bd70ffca6566388742, i failed
to resolve this conflict when resolving the merge confliction.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agodoc/rados/api: Update code samples to python3 39196/head
Brad Hubbard [Mon, 1 Feb 2021 01:11:17 +0000 (11:11 +1000)]
doc/rados/api: Update code samples to python3

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
4 years agoMerge pull request #39106 from mgfritch/cephadm-apt-get
Kefu Chai [Sun, 31 Jan 2021 09:23:10 +0000 (17:23 +0800)]
Merge pull request #39106 from mgfritch/cephadm-apt-get

cephadm: use `apt-get` for package install/update

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
4 years agoMerge pull request #39131 from sebastian-philipp/rook-disable-untyped
Kefu Chai [Sun, 31 Jan 2021 09:22:20 +0000 (17:22 +0800)]
Merge pull request #39131 from sebastian-philipp/rook-disable-untyped

mgr/rook: disallow_untyped_defs = True

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
4 years agopybind/mgr/dashboard: do not install enum34 39194/head
Kefu Chai [Sun, 31 Jan 2021 04:03:07 +0000 (12:03 +0800)]
pybind/mgr/dashboard: do not install enum34

enum was introduced by python 3.4, but we require python3.6 and up.
so drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
4 years agodoc: add rook_client._helper to mock_imports 39131/head
Sebastian Wagner [Thu, 28 Jan 2021 14:13:53 +0000 (15:13 +0100)]
doc: add rook_client._helper to mock_imports

This is due to `mgr/rook` now also includes `._helper`

Co-authored-by Kefu Chai <kchai@redhat.com>

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
4 years agomgr/rook: disallow_untyped_defs = True
Sebastian Wagner [Thu, 28 Jan 2021 11:44:07 +0000 (12:44 +0100)]
mgr/rook: disallow_untyped_defs = True

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
4 years agomgr/rook: Fix possibly infinite loop in _execute_blight_job
Sebastian Wagner [Thu, 28 Jan 2021 12:13:20 +0000 (13:13 +0100)]
mgr/rook: Fix possibly infinite loop in _execute_blight_job

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
4 years agoMerge pull request #39143 from adk3798/ha-rgw-data-dir
Kefu Chai [Sun, 31 Jan 2021 02:34:46 +0000 (10:34 +0800)]
Merge pull request #39143 from adk3798/ha-rgw-data-dir

cephadm: fix get_data_dir calls for ha-rgw daemons

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
4 years agoMerge pull request #39081 from ifed01/wip-ifed-perpg-instrumentation
Kefu Chai [Sun, 31 Jan 2021 02:31:13 +0000 (10:31 +0800)]
Merge pull request #39081 from ifed01/wip-ifed-perpg-instrumentation

os/bluestore: be more verbose when fsck detects stray per-pg omaps

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
4 years agoMerge pull request #39095 from tchaikov/wip-cephadm-fsid
Kefu Chai [Sun, 31 Jan 2021 02:29:59 +0000 (10:29 +0800)]
Merge pull request #39095 from tchaikov/wip-cephadm-fsid

cephadm: add fsid if --name is not specified

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
4 years agoMerge pull request #39056 from sebastian-philipp/alerts-percent-d
Kefu Chai [Sun, 31 Jan 2021 02:28:54 +0000 (10:28 +0800)]
Merge pull request #39056 from sebastian-philipp/alerts-percent-d

mgr/alerts: interval can be None

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #38930 from vbashkirtsev/master
Kefu Chai [Sun, 31 Jan 2021 02:28:13 +0000 (10:28 +0800)]
Merge pull request #38930 from vbashkirtsev/master

client/fuse_ll.cc: use uint64_t for fuse_ll_forget() nlookup argument

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #39132 from rzarzynski/wip-blk-ptr_node-for-aio
Kefu Chai [Sun, 31 Jan 2021 02:26:38 +0000 (10:26 +0800)]
Merge pull request #39132 from rzarzynski/wip-blk-ptr_node-for-aio

blk: avoid temporary bptrs on aio paths; use ptr_node instead.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
4 years agoMerge pull request #39151 from batrick/radosstriper-refactor
Kefu Chai [Sun, 31 Jan 2021 02:25:46 +0000 (10:25 +0800)]
Merge pull request #39151 from batrick/radosstriper-refactor

libradosstriper: make default layout const and static

Reviewed-by: Kefu Chai <kchai@redhat.com>
4 years agoMerge pull request #39116 from tchaikov/wip-crimson-pgstat
Kefu Chai [Sun, 31 Jan 2021 02:24:36 +0000 (10:24 +0800)]
Merge pull request #39116 from tchaikov/wip-crimson-pgstat

crimson/osd: keep a local copy of pg_stat and cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>