]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
6 years agoMerge pull request #28203 from dzafman/wip-39699
Yuri Weinstein [Wed, 29 May 2019 22:07:24 +0000 (15:07 -0700)]
Merge pull request #28203 from dzafman/wip-39699

nautilus: osd: Don't include user changeable flag in snaptrim related assert

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
6 years agoMerge pull request #28258 from pdvian/wip-39672-nautilus
Yuri Weinstein [Wed, 29 May 2019 22:06:25 +0000 (15:06 -0700)]
Merge pull request #28258 from pdvian/wip-39672-nautilus

nautilus: os/bluestore: fix missing discard in BlueStore::_kv_sync_thread

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
6 years agoMerge pull request #27746 from LenzGr/wip-39345-nautilus
Yuri Weinstein [Wed, 29 May 2019 21:59:52 +0000 (14:59 -0700)]
Merge pull request #27746 from LenzGr/wip-39345-nautilus

nautilus: mgr/dashboard: code documentation

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
6 years agoMerge pull request #27760 from ricardoasmarques/wip-39454-nautilus
Yuri Weinstein [Wed, 29 May 2019 21:58:49 +0000 (14:58 -0700)]
Merge pull request #27760 from ricardoasmarques/wip-39454-nautilus

nautilus: mgr/dashboard: Validate if any client belongs to more than one group

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
6 years agoMerge pull request #27868 from LenzGr/wip-39467-nautilus
Yuri Weinstein [Wed, 29 May 2019 21:57:39 +0000 (14:57 -0700)]
Merge pull request #27868 from LenzGr/wip-39467-nautilus

nautilus: mgr/dashboard: Admin resource not honored

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Wido den Hollander <wido@widodh.nl>
6 years agoMerge pull request #28011 from ricardoasmarques/wip-39616-nautilus
Yuri Weinstein [Wed, 29 May 2019 21:54:56 +0000 (14:54 -0700)]
Merge pull request #28011 from ricardoasmarques/wip-39616-nautilus

nautilus: mgr/dashboard: iSCSI should allow exporting an RBD image with Journaling enabled

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
6 years agoMerge pull request #28026 from ricardoasmarques/wip-39631-nautilus
Yuri Weinstein [Wed, 29 May 2019 21:51:44 +0000 (14:51 -0700)]
Merge pull request #28026 from ricardoasmarques/wip-39631-nautilus

nautilus: mgr/dashboard: iSCSI form does not support IPv6

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
6 years agoMerge pull request #27920 from sebastian-philipp/wip-39536-nautilus
Nathan Cutler [Wed, 29 May 2019 10:16:42 +0000 (12:16 +0200)]
Merge pull request #27920 from sebastian-philipp/wip-39536-nautilus

nautilus: mgr/test_orchestrator: AttributeError: 'TestWriteCompletion' object has no attribute 'id'

Reviewed-by: Nathan Cutler <ncutler@suse.com>
6 years agoMerge pull request #28035 from pdvian/wip-39375-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:10:26 +0000 (09:10 -0700)]
Merge pull request #28035 from pdvian/wip-39375-nautilus

nautilus: osd/OSD.cc: make osd bench description consistent with parameters

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #28072 from pdvian/wip-39421-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:09:57 +0000 (09:09 -0700)]
Merge pull request #28072 from pdvian/wip-39421-nautilus

nautilus: osd/OSDMap: Replace get_out_osds with get_out_existing_osds

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
6 years agoMerge pull request #28088 from dzafman/wip-39721
Yuri Weinstein [Tue, 28 May 2019 16:09:23 +0000 (09:09 -0700)]
Merge pull request #28088 from dzafman/wip-39721

nautilus: osd: Include dups in copy_after() and copy_up_to()

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
6 years agoMerge pull request #28160 from pdvian/wip-39512-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:08:51 +0000 (09:08 -0700)]
Merge pull request #28160 from pdvian/wip-39512-nautilus

nautilus: osd/PG: do not use approx_missing_objects pre-nautilus

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #28164 from pdvian/wip-39514-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:08:17 +0000 (09:08 -0700)]
Merge pull request #28164 from pdvian/wip-39514-nautilus

nautilus: osd: take heartbeat_lock when calling heartbeat()

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
6 years agoMerge pull request #28205 from pdvian/wip-39519-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:07:47 +0000 (09:07 -0700)]
Merge pull request #28205 from pdvian/wip-39519-nautilus

nautilus: osd: Don't evict after a flush if intersecting scrub range

Reviewed-by: David Zafman <dzafman@redhat.com>
6 years agoMerge pull request #28219 from pdvian/wip-39539-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:07:15 +0000 (09:07 -0700)]
Merge pull request #28219 from pdvian/wip-39539-nautilus

nautilus: osd/PG: fix last_complete re-calculation on splitting

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #27632 from pdvian/wip-39043-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:06:08 +0000 (09:06 -0700)]
Merge pull request #27632 from pdvian/wip-39043-nautilus

nautilus: osd/PGLog: preserve original_crt to check rollbackability

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
6 years agoMerge pull request #27699 from liewegas/wip-39335-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:05:00 +0000 (09:05 -0700)]
Merge pull request #27699 from liewegas/wip-39335-nautilus

nautilus: mgr/BaseMgrModule: run MonCommandCompletion on the finisher

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
6 years agoMerge pull request #27711 from pdvian/wip-39184-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:04:34 +0000 (09:04 -0700)]
Merge pull request #27711 from pdvian/wip-39184-nautilus

nautilus: osd: Primary won't automatically repair replica on pulling error

Reviewed-by: David Zafman <dzafman@redhat.com>
6 years agoMerge pull request #27744 from ashishkumsingh/wip-39432-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:04:00 +0000 (09:04 -0700)]
Merge pull request #27744 from ashishkumsingh/wip-39432-nautilus

nautilus: osd/PG: discover missing objects when an OSD peers and PG is degraded

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #27771 from smithfarm/wip-39419-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:03:28 +0000 (09:03 -0700)]
Merge pull request #27771 from smithfarm/wip-39419-nautilus

nautilus: mon/Elector: format mon_release correctly

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #27839 from pdvian/wip-39219-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:02:55 +0000 (09:02 -0700)]
Merge pull request #27839 from pdvian/wip-39219-nautilus

nautilus: osd: FAILED ceph_assert(attrs || !pg_log.get_missing().is_missing(soid) || (it_objects != pg_log.get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT)) in PrimaryLogPG::get_object_context()

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #27876 from liewegas/wip-autoscaler-fixes-nautilus
Yuri Weinstein [Tue, 28 May 2019 16:02:28 +0000 (09:02 -0700)]
Merge pull request #27876 from liewegas/wip-autoscaler-fixes-nautilus

nautilus: mgr: fix pgp_num adjustments

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
6 years agoos/bluestore: fix missing discard in BlueStore::_kv_sync_thread 28258/head
Tang Junhui [Mon, 29 Apr 2019 08:31:55 +0000 (16:31 +0800)]
os/bluestore: fix missing discard in BlueStore::_kv_sync_thread

Discard comand should be issued before space released.

Signed-off-by: Junhui Tang <tangjunhui@sangfor.com.cn>
Fixes: https://tracker.ceph.com/issues/39621
(cherry picked from commit f6205d2eb2b8297b914afbaf54a8311f087eec90)

6 years agoMerge pull request #28215 from smithfarm/wip-40003-nautilus
Nathan Cutler [Mon, 27 May 2019 13:54:57 +0000 (15:54 +0200)]
Merge pull request #28215 from smithfarm/wip-40003-nautilus

nautilus: do_cmake.sh: source not found

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
6 years agoMerge pull request #28062 from bk201/wip-39664-nautilus
Lenz Grimmer [Mon, 27 May 2019 13:01:08 +0000 (15:01 +0200)]
Merge pull request #28062 from bk201/wip-39664-nautilus

nautilus: mgr/dashboard: incorrect help message for minimum blob size

Reviewed-by: Laura Paduano <lpaduano@suse.com>
6 years agoosd/PG: fix last_complete re-calculation on splitting 28219/head
xie xingguo [Sat, 20 Apr 2019 08:34:12 +0000 (16:34 +0800)]
osd/PG: fix last_complete re-calculation on splitting

We add hard-limit for pg_logs now, which means we might keep trimming
old log entries irrespective of pg's current missing_set. This as a
result can cause the last_complete pointer moving far ahead of the real
on-disk version (the oldest need of missing_set, for instance) the
corresponding pg should have on splitting:

```
2019-04-19 06:41:52.559247 7efd4725c700 10 osd.2 271 Splitting pg[5.6( v 270'943 lc 0'0 (238'300,270'943] local-lis/les=250/251 n=943 ec=223/223 lis/c 250/223 les/251/224/0 250/271/229) [5,2] r=1 lpr=271 pi=[223,271)/4 crt=270'943 unknown NOTIFY m=518 mbc={}] into 5.16
2019-04-19 06:41:52.561413 7efd4725c700 10 osd.2 pg_epoch: 271 pg[5.6( v 270'943 lc 238'300 (238'300,270'943] local-lis/les=250/251 n=943 ec=223/223 lis/c 250/223 c/f 251/224/0 250/271/229) [5,2] r=1 lpr=271 pi=[223,271)/4 crt=270'943 lcod 0'0 unknown NOTIFY m=261 mbc={}] release_backoffs [MIN,MAX)
```

For the above example, parent's last_complete cursor changed from **0'0** to
**238'300** directly due to the effort of trying to catch up the oldest
log entry changing when splitting was done. However, back into v12.2.9 primary
would still reference shard's last_complete field when trying to figure out all
possible locations of a currently missing object (see PG::MissingLoc::add_source_info):

```c++
  if (oinfo.last_complete < need) {
    if (omissing.is_missing(soid)) {
      ldout(pg->cct, 10) << "search_for_missing " << soid << " " << need
                         << " also missing on osd." << fromosd << dendl;
      continue;
    }
  }
```

Hence a wrongly calculated last_complete could then make primary mis-consider
that a specific shard might have the authoritative object it currently
looking for:

```
2019-04-19 06:41:52.904163 7fd4cfb5a700 10 osd.5 pg_epoch: 271 pg[5.6( v 270'943 lc 238'300 (238'300,270'943] local-lis/les=250/251 n=471 ec=223/223 lis/c 250/223 les/
c/f 251/224/0 250/271/229) [5,2] r=0 lpr=271 pi=[223,271)/4 crt=270'943 lcod 226'77 mlcod 0'0 peering m=16 mbc={}] proc_replica_log for osd.2: 5.6( v 270'943 lc 238'30
0 (238'300,270'943] local-lis/les=250/251 n=471 ec=223/223 lis/c 250/223 les/c/f 251/224/0 250/271/229) log((249'563,270'943], crt=270'943) missing(261 may_include_del
etes = 1)
2019-04-19 06:41:52.904645 7fd4cfb5a700 20 osd.5 pg_epoch: 271 pg[5.6( v 270'943 lc 238'300 (238'300,270'943] local-lis/les=250/251 n=471 ec=223/223 lis/c 250/223 les/
c/f 251/224/0 250/271/229) [5,2] r=0 lpr=271 pi=[223,271)/4 crt=270'943 lcod 226'77 mlcod 0'0 peering m=16 mbc={}]  after missing 5:624c3a7a:::benchmark_data_smithi190
_39968_object1382:head need 226'110 have 0'0
2019-04-19 06:41:53.567820 7fd4d035b700 10 osd.5 pg_epoch: 272 pg[5.6( v 270'943 lc 0'0 (238'300,270'943] local-lis/les=271/272 n=471 ec=223/223 lis/c 250/223 les/c/f
251/224/0 250/271/229) [5,2] r=0 lpr=271 pi=[223,271)/4 crt=270'943 lcod 226'77 mlcod 0'0 unknown m=16 u=13 mbc={255={(1+0)=220,(2+0)=28}}] search_for_missing 5:624c3a
7a:::benchmark_data_smithi190_39968_object1382:head 226'110 is on osd.2
```

note that ```5:624c3a7a:::benchmark_data_smithi190_39968_object1382:head 226'110```
was actually missing on both primary and shard osd.2 whereas primary insisted that
object should exist on shard osd.2!

https://github.com/ceph/ceph/pull/26175 posted an indirect fix
for the above problem by ignoring last_complete when checking the missing set,
but it should generally make more sense to fill in the last_complete field correctly
whenever possible.
Hence coming this additional fix.

Fixes: http://tracker.ceph.com/issues/26958
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit aad5d47be64ed7feba79f540ec1debc45625a74f)

6 years agoMerge pull request #27937 from pdvian/wip-39288-nautilus
Yuri Weinstein [Wed, 22 May 2019 19:49:12 +0000 (12:49 -0700)]
Merge pull request #27937 from pdvian/wip-39288-nautilus

nautilus: [rbd-mirror] image replayer should periodically flush IO and commit positions

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
6 years agoMerge pull request #27989 from smithfarm/wip-39590-nautilus
Nathan Cutler [Wed, 22 May 2019 16:15:59 +0000 (18:15 +0200)]
Merge pull request #27989 from smithfarm/wip-39590-nautilus

nautilus: qa/tasks/rbd_fio: fixed missing delimiter between 'cd' and 'configure'

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
6 years agoMerge pull request #27882 from pdvian/wip-39224-nautilus
Yuri Weinstein [Wed, 22 May 2019 16:15:09 +0000 (09:15 -0700)]
Merge pull request #27882 from pdvian/wip-39224-nautilus

nautlius: librbd: do not allow to deep copy migrating image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
6 years agoMerge pull request #27903 from pdvian/wip-39226-nautilus
Yuri Weinstein [Wed, 22 May 2019 16:14:39 +0000 (09:14 -0700)]
Merge pull request #27903 from pdvian/wip-39226-nautilus

nautilus: librbd: support EC data pool images sparsify

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
6 years agoMerge pull request #27988 from smithfarm/wip-39541-nautilus
Yuri Weinstein [Wed, 22 May 2019 16:13:25 +0000 (09:13 -0700)]
Merge pull request #27988 from smithfarm/wip-39541-nautilus

nautilus: tests: rbd: qemu-iotests tests fail under latest Ubuntu kernel

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
6 years agoMerge pull request #28019 from pdvian/wip-39315-nautilus
Yuri Weinstein [Wed, 22 May 2019 16:11:32 +0000 (09:11 -0700)]
Merge pull request #28019 from pdvian/wip-39315-nautilus

nautilus: krbd: fix rbd map hang due to udev return subsystem unordered

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
6 years agoMerge pull request #28128 from dillaman/wip-39729-nautilus
Yuri Weinstein [Wed, 22 May 2019 16:10:56 +0000 (09:10 -0700)]
Merge pull request #28128 from dillaman/wip-39729-nautilus

nautilus: qa/workunits/rbd: use https protocol for devstack git operations

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
6 years agoMerge pull request #28132 from dillaman/wip-39195-nautilus
Yuri Weinstein [Wed, 22 May 2019 16:10:18 +0000 (09:10 -0700)]
Merge pull request #28132 from dillaman/wip-39195-nautilus

nautilus: librbd: fixed several race conditions related to copyup

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
6 years agodo_cmake.sh: use bash 28215/head
Nathan Cutler [Mon, 20 May 2019 15:01:40 +0000 (17:01 +0200)]
do_cmake.sh: use bash

Fixes: http://tracker.ceph.com/issues/39981
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit ac8d7c5fd4045d79ab6caacc0b72d64866fcc5c8)

6 years agoosd: Don't evict after a flush if intersecting scrub range 28205/head
David Zafman [Tue, 26 Mar 2019 22:53:10 +0000 (15:53 -0700)]
osd: Don't evict after a flush if intersecting scrub range

Fixes: http://tracker.ceph.com/issues/38840
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 459cbb5a6ca3b521b20f36f328f25f398d0ef1c4)

6 years agoosd: Don't include user changeable flag in snaptrim related assert 28203/head
David Zafman [Tue, 30 Apr 2019 03:20:18 +0000 (20:20 -0700)]
osd: Don't include user changeable flag in snaptrim related assert

Caused by: a53ba7314c53e75d1e0b8a0edd29181db3c93863

Fixes: http://tracker.ceph.com/issues/38124
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 63c2060c2d9eef49dacf2ba240ccfbead9791f43)

6 years agoMerge pull request #27752 from liewegas/wip-devices-doc-nautilus
Nathan Cutler [Tue, 21 May 2019 08:11:02 +0000 (10:11 +0200)]
Merge pull request #27752 from liewegas/wip-devices-doc-nautilus

nautilus: doc/rados/operations/devices: document device prediction

Reviewed-by: Sébastien Han <seb@redhat.com>
6 years agoMerge pull request #27488 from sebastian-philipp/wip-39169-nautilus
Nathan Cutler [Mon, 20 May 2019 12:31:48 +0000 (14:31 +0200)]
Merge pull request #27488 from sebastian-philipp/wip-39169-nautilus

nautilus: doc/mgr/orchestrator_cli: Rook orch supports mon update

Reviewed-by: Nathan Cutler <ncutler@suse.com>
6 years agoMerge pull request #28163 from smithfarm/wip-39962-nautilus
Lenz Grimmer [Mon, 20 May 2019 10:56:19 +0000 (12:56 +0200)]
Merge pull request #28163 from smithfarm/wip-39962-nautilus

nautilus: mgr/dashboard: openssl exception when verifying certificates of HTTPS requests

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
6 years agoosd: take heartbeat_lock when calling heartbeat() 28164/head
Sage Weil [Tue, 23 Apr 2019 18:03:42 +0000 (13:03 -0500)]
osd: take heartbeat_lock when calling heartbeat()

Fixes: http://tracker.ceph.com/issues/39439
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7863d7c6c49347db97a6bca36f63fc35741ac0e5)

6 years agosystemd: ceph-mgr: set MemoryDenyWriteExecute to false 28163/head
Ricardo Dias [Wed, 8 May 2019 13:57:07 +0000 (14:57 +0100)]
systemd: ceph-mgr: set MemoryDenyWriteExecute to false

Fixes: http://tracker.ceph.com/issues/39628
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 1d7506fdce4924fb30bbabc65e26dafa06aab24f)

6 years agoosd/PG: do not use approx_missing_objects pre-nautilus 28160/head
Neha Ojha [Thu, 25 Apr 2019 02:15:27 +0000 (19:15 -0700)]
osd/PG: do not use approx_missing_objects pre-nautilus

We changed async recovery cost calculation in nautilus to also take
into account approx_missing_objects in ab241bf7e927cda2d0ed1698383d18dc4a4b601c
This commit depends on https://github.com/ceph/ceph/pull/23663, hence
wasn't backported to mimic.

Mimic only uses the difference in length of logs as the cost. Due to this,
the same OSD might have different costs in a mixed mimic and nautilus(or above)
cluster. This can lead to choose_acting() cycling between OSDs, when trying
to select the acting set and async_recovery_targets.

Fixes: https://tracker.ceph.com/issues/39441
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 4c617ecf1cf6f25cca42e5d59fa162dbc60f4de8)

Conflicts:
src/osd/PG.cc : Resolved in choose_async_recovery_ec and
choose_async_recovery_replicated

6 years agolibrbd: copyup read stats were incorrectly tied to child 28132/head
Jason Dillaman [Wed, 24 Apr 2019 15:37:41 +0000 (11:37 -0400)]
librbd: copyup read stats were incorrectly tied to child

The parent IO stats should be updated when reading from the parent
image for a copyup.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9b615c30753ad35636c875696ed951e807fd694c)

6 years agolibrbd: force a copyup object map update if migration completes
Jason Dillaman [Mon, 8 Apr 2019 17:42:16 +0000 (13:42 -0400)]
librbd: force a copyup object map update if migration completes

If a live-migration copyup was in-progress when the copyup completes,
force the object map to be updated.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e18e413b602208b38f432f4ba5bcb187fe102cb1)

6 years agolibrbd: cleaned up snap/parent lock release in read-from-parent path
Jason Dillaman [Thu, 4 Apr 2019 15:23:58 +0000 (11:23 -0400)]
librbd: cleaned up snap/parent lock release in read-from-parent path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b7f6ba710d2855ea8f223d205a55a1e425167afd)

6 years agotest/librbd: initial unit test for copyup state machine
Jason Dillaman [Tue, 2 Apr 2019 18:10:42 +0000 (14:10 -0400)]
test/librbd: initial unit test for copyup state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ea4901f539c7404c07e72f5c42fc8abcf840dafa)

6 years agolibrbd: copyup should restart delayed ops against the same object
Jason Dillaman [Thu, 4 Apr 2019 01:35:40 +0000 (21:35 -0400)]
librbd: copyup should restart delayed ops against the same object

This avoids the potential for a race condition where an in-flight
copyup is removed from the in-flight copyup list and a subsequent
IO against the same object causes a second in-flight copyup.

Fixes: http://tracker.ceph.com/issues/39021
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 31ed673a7484c8067dd928cabd5bcb923a35dbaf)

6 years agolibrbd: merge copyup object map update states
Jason Dillaman [Tue, 2 Apr 2019 15:51:36 +0000 (11:51 -0400)]
librbd: merge copyup object map update states

The object map HEAD and HEAD/snapshot update states have been
simplified and merged into a single state. This also fixes
several potential race conditions and an issue where CoR might
incorrectly mark the HEAD object has exists+dirty.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 5b90dbe0ae4be3dccfc3fbc0c8d5eb1ed7bb3168)

6 years agolibrbd: refactor copyup state machine to match current code style
Jason Dillaman [Fri, 29 Mar 2019 18:17:15 +0000 (14:17 -0400)]
librbd: refactor copyup state machine to match current code style

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 711a070fa086b499eeba9a4c62005456f406689b)

6 years agolibrbd: properly hold snap/parent locks during IO
Jason Dillaman [Fri, 29 Mar 2019 17:23:37 +0000 (13:23 -0400)]
librbd: properly hold snap/parent locks during IO

The ImageCtx::parent pointer was dereferenced without holding the lock
which could lead to a crash. The ImageCtx::migration_info structure
was also dereferenced without holding a lock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6dad1acee3f0a4b4b3c88e25c0cfafe21bc066c4)

6 years agolibrbd: empty IO requests should complete in new context
Jason Dillaman [Tue, 2 Apr 2019 19:33:00 +0000 (15:33 -0400)]
librbd: empty IO requests should complete in new context

This will avoid the possibility of a lock being held by the caller
and having the IO immediately complete back to the calling state
machine.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c9ff6cf1945b248b38e83484aa4a22c206da846b)

6 years agolibrbd: deep-copy object copy should register an in-flight op
Jason Dillaman [Mon, 1 Apr 2019 18:48:47 +0000 (14:48 -0400)]
librbd: deep-copy object copy should register an in-flight op

When handling live migrations, the source image is the parent
image of the destination image. To prevent the parent image from
being closed while a request is in-flight, the object copy
state machine now registers an async operation with the source
image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 89cfa6c8d328b9294c7f7efd53d82edcec59e66d)

6 years agolibrbd: deep-copy object copy should delay computing object existence
Jason Dillaman [Fri, 29 Mar 2019 20:33:21 +0000 (16:33 -0400)]
librbd: deep-copy object copy should delay computing object existence

This prevent a potential recursive lock on the destination image's
snap_lock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0c37066797737cd7c677b2649dae1e6422869afb)

6 years agoqa/workunits/rbd: use https protocol for devstack git operations 28128/head
Jason Dillaman [Thu, 9 May 2019 19:48:30 +0000 (15:48 -0400)]
qa/workunits/rbd: use https protocol for devstack git operations

Fixes: http://tracker.ceph.com/issues/39656
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fb4f9a8a08edd18b1a23e1be4c1285d0ec4d1de6)

6 years agoMerge pull request #27740 from ifed01/wip-ifed-fix-39446
Yuri Weinstein [Wed, 15 May 2019 16:05:06 +0000 (09:05 -0700)]
Merge pull request #27740 from ifed01/wip-ifed-fix-39446

nautilus: os/bluestore: fix out-of-bound access in bmap allocator.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoosd: Fix merge_log_dups() not remove dup corresponding to log.tail 28088/head
David Zafman [Thu, 9 May 2019 17:38:36 +0000 (10:38 -0700)]
osd: Fix merge_log_dups() not remove dup corresponding to log.tail

The log tail is not inclusive of the log, so not an overlap
Fix unittest_pglog to verify proper operation

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f526c6d55c8866712c10b64491fe9371f64cf4d5)

6 years agotest: unittest_pglog: Additional test cases for trim, copy_up_to and copy_after
David Zafman [Tue, 7 May 2019 04:19:59 +0000 (21:19 -0700)]
test: unittest_pglog: Additional test cases for trim, copy_up_to and copy_after

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit d0d244d222c7709a4a0688954816fd42e471b3c9)

6 years agotest: unittest_pglog: Remove irrelevant config values
David Zafman [Tue, 7 May 2019 02:57:55 +0000 (19:57 -0700)]
test: unittest_pglog: Remove irrelevant config values

osd_min_pg_log_entries and osd_max_pg_log_entries are not involved
in the log handling code being tested.

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a008bd8c18ee99f24db41c8e2044cd49c175f4b9)

6 years agotest: Adding standalone test of log copy handling
David Zafman [Fri, 3 May 2019 20:33:39 +0000 (13:33 -0700)]
test: Adding standalone test of log copy handling

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 7959159e830b39c69807c832bf3e01581a314d0a)

6 years agoosd: Include dups in copy_after() and copy_up_to()
David Zafman [Thu, 2 May 2019 02:47:55 +0000 (19:47 -0700)]
osd: Include dups in copy_after() and copy_up_to()

Client saw out of order results in a test with a very small
pg_log because a backfill/recovery sent a small number of
log entries for duplicate checks.

Since these copy operations are equivalent to both a copy
and trim (an argument controls how many log entries
transfer), we need to include any new dups with enough
existing dups up to the configured maximum.

Fixes: http://tracker.ceph.com/issues/39304
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 00f279db37fc648584c4c741e0453b1d4f4320e4)

Conflicts:
src/osd/PeeringState.cc contents in src/osd/PG.cc
src/osd/osd_types.cc (trivial)

6 years agoosd: Fix off by 1 error in trim handling of osd_pg_log_dups_tracked
David Zafman [Thu, 2 May 2019 02:44:04 +0000 (19:44 -0700)]
osd: Fix off by 1 error in trim handling of osd_pg_log_dups_tracked

Move log message out of loop so it doesn't repeat for no reason

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b698e34fc8e931623671b501855099050859a18a)

6 years agoosd: Remove unused pg_log_t::copy_range()
David Zafman [Tue, 30 Apr 2019 23:09:47 +0000 (16:09 -0700)]
osd: Remove unused pg_log_t::copy_range()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit d1ad129dd4b71640ae2e74d793e45146dca8bf26)

6 years agoosd/OSDMap: Replace get_out_osds with get_out_existing_osds 28072/head
Brad Hubbard [Wed, 17 Apr 2019 06:13:27 +0000 (16:13 +1000)]
osd/OSDMap: Replace get_out_osds with get_out_existing_osds

Fixes: http://tracker.ceph.com/issues/39154
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit adfb6a595363e2c085c6eac3001fcaa2dce007aa)

Conflicts:
src/osd/OSDMap.h : Resolved for get_out_existing_osds

6 years agoMerge pull request #27803 from pdvian/wip-39205-nautilus
Yuri Weinstein [Sat, 11 May 2019 15:43:00 +0000 (08:43 -0700)]
Merge pull request #27803 from pdvian/wip-39205-nautilus

nautilus: osd: shutdown recovery_request_timer earlier

Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #28065 from yuriw/wip-yuriw-fix-distros-nautilus
Yuri Weinstein [Fri, 10 May 2019 19:06:49 +0000 (12:06 -0700)]
Merge pull request #28065 from yuriw/wip-yuriw-fix-distros-nautilus

qa/tests: cleaned up supported distro for nautilus

6 years agoqa/tests: cleaned up supported disto for nautilus 28065/head
Yuri Weinstein [Fri, 10 May 2019 16:10:08 +0000 (09:10 -0700)]
qa/tests: cleaned up supported disto for nautilus

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoMerge pull request #27608 from liewegas/wip-38157-nautilus
Yuri Weinstein [Fri, 10 May 2019 15:20:46 +0000 (08:20 -0700)]
Merge pull request #27608 from liewegas/wip-38157-nautilus

nautilus: mgr/progress: behave if pgs disappear (due to a racing pg merge)

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
6 years agoMerge pull request #27654 from neha-ojha/wip-38881-nautilus
Yuri Weinstein [Fri, 10 May 2019 15:19:38 +0000 (08:19 -0700)]
Merge pull request #27654 from neha-ojha/wip-38881-nautilus

nautilus: osd/PG: skip rollforward when !transaction_applied during append_log()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
6 years agoMerge pull request #27854 from smithfarm/wip-39504-nautilus
Yuri Weinstein [Fri, 10 May 2019 15:18:47 +0000 (08:18 -0700)]
Merge pull request #27854 from smithfarm/wip-39504-nautilus

nautilus: core: Give recovery for inactive PGs a higher priority

Reviewed-by: David Zafman <dzafman@redhat.com>
6 years agoMerge pull request #27857 from ashishkumsingh/wip-39465-nautilus
Yuri Weinstein [Fri, 10 May 2019 15:18:13 +0000 (08:18 -0700)]
Merge pull request #27857 from ashishkumsingh/wip-39465-nautilus

nautilus: msg: output peer address when detecting bad CRCs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
6 years agoMerge pull request #27939 from pdvian/wip-39308-nautilus
Yuri Weinstein [Fri, 10 May 2019 15:17:32 +0000 (08:17 -0700)]
Merge pull request #27939 from pdvian/wip-39308-nautilus

nautilus: mgr/ActivePyModules: handle_command - fix broken lock

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
6 years agomgr/dashboard: fix help message for minimum blob size in pool form 28062/head
Kiefer Chang [Wed, 8 May 2019 04:15:43 +0000 (12:15 +0800)]
mgr/dashboard: fix help message for minimum blob size in pool form

A value less than maximum blob size is required.

Fixes: http://tracker.ceph.com/issues/39624
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
(cherry picked from commit c8c4ed692a2b6ac8bf2aa9337ed7d683187ff882)

6 years agoMerge pull request #27642 from LenzGr/nautilus-documentation
Ernesto Puerta [Fri, 10 May 2019 13:33:06 +0000 (15:33 +0200)]
Merge pull request #27642 from LenzGr/nautilus-documentation

nautilus: doc: Backported dashboard documentation changes

6 years agoMerge pull request #27936 from dillaman/wip-39450-nautilus
Yuri Weinstein [Thu, 9 May 2019 17:07:44 +0000 (10:07 -0700)]
Merge pull request #27936 from dillaman/wip-39450-nautilus

nautilus: librbd: re-add support for nautilus clients talking to jewel clusters

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
6 years agoMerge pull request #28046 from neha-ojha/wip-28036-nautilus
Neha Ojha [Thu, 9 May 2019 15:26:33 +0000 (08:26 -0700)]
Merge pull request #28046 from neha-ojha/wip-28036-nautilus

nautilus: doc/dev/osd_internals/async_recovery: update cost calculation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
6 years agodoc/dev/osd_internals/async_recovery: update cost calculation 28046/head
Neha Ojha [Thu, 9 May 2019 03:19:32 +0000 (20:19 -0700)]
doc/dev/osd_internals/async_recovery: update cost calculation

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 984073e44d096a3322d90b85da42c593fff94c6d)

6 years agoMerge pull request #28024 from ricardoasmarques/wip-39630-nautilus
Lenz Grimmer [Thu, 9 May 2019 10:33:00 +0000 (12:33 +0200)]
Merge pull request #28024 from ricardoasmarques/wip-39630-nautilus

nautilus: mgr/dashboard: iSCSI GET requests should not be logged

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
6 years agoMerge pull request #27942 from LenzGr/wip-39524-nautilus
Lenz Grimmer [Thu, 9 May 2019 07:52:56 +0000 (09:52 +0200)]
Merge pull request #27942 from LenzGr/wip-39524-nautilus

nautilus: mgr/dashboard: make auth token work with UTC times only

6 years agoosd/OSD.cc: make osd bench description consistent with parameters 28035/head
Neha Ojha [Tue, 16 Apr 2019 01:47:34 +0000 (18:47 -0700)]
osd/OSD.cc: make osd bench description consistent with parameters

Fixes: http://tracker.ceph.com/issues/39006
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 4b240cf86b1e739eb0c255e2f7f1018ea2bc97aa)

6 years agomgr/dashboard: Fix iSCSI form when using IPv6 28026/head
Ricardo Marques [Fri, 3 May 2019 10:23:12 +0000 (11:23 +0100)]
mgr/dashboard: Fix iSCSI form when using IPv6

Fixes: https://tracker.ceph.com/issues/39578
Signed-off-by: Ricardo Marques <rimarques@suse.com>
(cherry picked from commit f243f7a53170dd84a49348eb06fc566d163c6a47)

6 years agomgr/dashboard: iSCSI GET requests should not be logged 28024/head
Ricardo Marques [Fri, 26 Apr 2019 10:13:20 +0000 (11:13 +0100)]
mgr/dashboard: iSCSI GET requests should not be logged

Fixes: https://tracker.ceph.com/issues/39492
Signed-off-by: Ricardo Marques <rimarques@suse.com>
(cherry picked from commit 93d74cf9b9926c1638f15da6c8e6d3fd68d61831)

6 years agoMerge pull request #27758 from ricardoasmarques/wip-39452-nautilus
Lenz Grimmer [Wed, 8 May 2019 11:55:54 +0000 (13:55 +0200)]
Merge pull request #27758 from ricardoasmarques/wip-39452-nautilus

nautilus: mgr/dashboard: iSCSI form is showing a warning

Reviewed-by: Tiago Melo <tmelo@suse.com>
6 years agonautilus: mgr/dashboard: Adapt iSCSI discovery auth for read-only users (#27759)
Lenz Grimmer [Wed, 8 May 2019 11:29:35 +0000 (13:29 +0200)]
nautilus: mgr/dashboard: Adapt iSCSI discovery auth for read-only users (#27759)

nautilus: mgr/dashboard: Adapt iSCSI discovery auth for read-only users

Reviewed-by: Tiago Melo <tmelo@suse.com>
6 years agonautilus: mgr/dashboard: Upgrade to ceph-iscsi config v9 (#28001)
Lenz Grimmer [Wed, 8 May 2019 11:18:58 +0000 (13:18 +0200)]
nautilus: mgr/dashboard: Upgrade to ceph-iscsi config v9 (#28001)

nautilus: mgr/dashboard: Upgrade to ceph-iscsi config v9

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
6 years agokrbd: fix rbd map hang due to udev return subsystem unordered 28019/head
Zhi Zhang [Thu, 11 Apr 2019 05:09:01 +0000 (13:09 +0800)]
krbd: fix rbd map hang due to udev return subsystem unordered

The order of subsystem returned by udev_device_get_subsystem
might not be same order as adding subsystem by
udev_monitor_filter_add_match_subsystem_devtype. So if block
event is returned first and rbd event is returned next, then
further poll will get nothing back until timed-out.

Fixes: http://tracker.ceph.com/issues/39089
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>
(cherry picked from commit 42dd1eae630f31a628051049e4007175561f1939)

6 years agomgr/dashboard: All RBD features are supported by 'tcmu:runner' 28011/head
Ricardo Marques [Tue, 7 May 2019 11:28:19 +0000 (12:28 +0100)]
mgr/dashboard: All RBD features are supported by 'tcmu:runner'

Fixes: https://tracker.ceph.com/issues/39607
Signed-off-by: Ricardo Marques <rimarques@suse.com>
(cherry picked from commit f971270a22855753d33ac0dd0e14455297c637a2)

6 years agomgr/dashboard: Support ceph-iscsi config v9 28001/head
Ricardo Marques [Mon, 8 Apr 2019 20:22:25 +0000 (21:22 +0100)]
mgr/dashboard: Support ceph-iscsi config v9

Fixes: https://tracker.ceph.com/issues/39147
Signed-off-by: Ricardo Marques <rimarques@suse.com>
(cherry picked from commit 1b5bf05a2ce0590f034bbbc7ae750ed478f6ffe3)

6 years agonautilus: mgr/dashboard: Make all columns sortable (#27889)
Lenz Grimmer [Tue, 7 May 2019 09:18:59 +0000 (11:18 +0200)]
nautilus: mgr/dashboard: Make all columns sortable (#27889)

nautilus: mgr/dashboard: Make all columns sortable

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
6 years agonautilus: mgr/dashboard: Buggy data table search field (#27672)
Lenz Grimmer [Tue, 7 May 2019 08:25:31 +0000 (10:25 +0200)]
nautilus: mgr/dashboard: Buggy data table search field (#27672)

nautilus: mgr/dashboard: Buggy data table search field

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
6 years agoMerge pull request #27886 from rhcs-dashboard/39459-nautilus-prometheus-replace-white...
Lenz Grimmer [Tue, 7 May 2019 08:19:30 +0000 (10:19 +0200)]
Merge pull request #27886 from rhcs-dashboard/39459-nautilus-prometheus-replace-whitespaces

nautilus: mgr/prometheus: replace whitespaces in metrics' names

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
6 years agoMerge pull request #27673 from Devp00l/wip-39371-nautilus
Lenz Grimmer [Tue, 7 May 2019 07:57:36 +0000 (09:57 +0200)]
Merge pull request #27673 from Devp00l/wip-39371-nautilus

 nautilus: mgr/dashboard: Localization for date picker module

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
6 years agoqa/tasks/rbd_fio: fixed missing delimiter between 'cd' and 'configure' 27989/head
Jason Dillaman [Thu, 18 Apr 2019 12:55:19 +0000 (08:55 -0400)]
qa/tasks/rbd_fio: fixed missing delimiter between 'cd' and 'configure'

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 83c11efb98f28c19f70eac04a6a84fe820f968cb)

6 years agoqa/suites/rbd: add random distro selection to librbd tests
Jason Dillaman [Sat, 13 Apr 2019 16:53:35 +0000 (12:53 -0400)]
qa/suites/rbd: add random distro selection to librbd tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d83d598c121e41eb6c6903df794ea2bd15ed4b86)

6 years agoqa/workunits/rbd: use more recent qemu-iotests that support Bionic 27988/head
Jason Dillaman [Thu, 18 Apr 2019 18:02:45 +0000 (14:02 -0400)]
qa/workunits/rbd: use more recent qemu-iotests that support Bionic

Fixes: http://tracker.ceph.com/issues/24668
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b184acd218b36b85f5a25678d773c35407a2b745)

6 years agoPendingReleaseNotes: librbd can open images on Jewel clusters 27936/head
Jason Dillaman [Mon, 6 May 2019 12:20:42 +0000 (08:20 -0400)]
PendingReleaseNotes: librbd can open images on Jewel clusters

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
6 years agodoc: Fix small typo in dashboard documentation 27642/head
Tatjana Dehler [Mon, 29 Apr 2019 09:56:59 +0000 (11:56 +0200)]
doc: Fix small typo in dashboard documentation

Signed-off-by: Tatjana Dehler <tdehler@suse.com>
(cherry picked from commit 038ef6291d5f5728385683c693046f03184309fa)

6 years agodoc: Improved the dashboard proxy config section
Lenz Grimmer [Sun, 14 Apr 2019 14:28:30 +0000 (16:28 +0200)]
doc: Improved the dashboard proxy config section

Added paragraph that explains how standby dashboard instances perform
HTTP redirections to the currently active ceph-mgr node.
Added note about current limitations (see Bug#24662 for details).

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>
(cherry picked from commit 70a28a3e474177719fef250d0c84d28703946a2d)

6 years agodoc: dashboard: refined object gateway configuration
Lenz Grimmer [Fri, 12 Apr 2019 11:17:23 +0000 (13:17 +0200)]
doc: dashboard: refined object gateway configuration

As discussed in BUG#39252, the dashboard currently does not
support all possible RGW configuration possibilities when trying
to obtain hostname and port number. Update the docs to better
reflect this.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>
(cherry picked from commit 43c56e1a5a92fc876e6edc184459b665d270ce29)

6 years agodoc: Updated dashboard iSCSI configuration, added labels
Lenz Grimmer [Sun, 31 Mar 2019 18:32:15 +0000 (13:32 -0500)]
doc: Updated dashboard iSCSI configuration, added labels

Added note about the requirement for the latest ceph-iscsi version
3 to the dashboard documentation. Added some doc references
and replaced some URLs in the iSCSI docs with reST labels instead.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>
(cherry picked from commit 71e46887d4b0f1afd7dc8af461879dc0f9440d50)