]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
6 years agoqa/suites/rados/perf: add op_size to radosbench_omap_write.yaml 29120/head
Neha Ojha [Fri, 19 Jul 2019 20:16:50 +0000 (13:16 -0700)]
qa/suites/rados/perf: add op_size to radosbench_omap_write.yaml

Signed-off-by: Neha Ojha <nojha@redhat.com>
6 years agoqa/suites/rados/perf: add objectsize to radosbench_omap_write.yaml
Neha Ojha [Thu, 18 Jul 2019 23:56:07 +0000 (16:56 -0700)]
qa/suites/rados/perf: add objectsize to radosbench_omap_write.yaml

Signed-off-by: Neha Ojha <nojha@redhat.com>
6 years agoqa/suites/rados/perf: add radosbench_omap_write.yaml
Neha Ojha [Thu, 18 Jul 2019 18:09:29 +0000 (11:09 -0700)]
qa/suites/rados/perf: add radosbench_omap_write.yaml

This excercises the --write-omap option in the radosbench task.

Signed-off-by: Neha Ojha <nojha@redhat.com>
6 years agoqa: add basic omap testing capability
Neha Ojha [Fri, 28 Jun 2019 21:40:27 +0000 (14:40 -0700)]
qa: add basic omap testing capability

Signed-off-by: Neha Ojha <nojha@redhat.com>
6 years agoosd: add hdd and ssd variants for osd_recovery_max_active 28677/head
Sage Weil [Thu, 20 Jun 2019 15:50:13 +0000 (10:50 -0500)]
osd: add hdd and ssd variants for osd_recovery_max_active

Semi-arbitrarily set the SSD max to 10 (instead of 3).  This should be
tuned based on some real data.

Signed-off-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #28631 from cbodley/wip-17882
Casey Bodley [Wed, 19 Jun 2019 13:32:18 +0000 (09:32 -0400)]
Merge pull request #28631 from cbodley/wip-17882

qa/rgw: extra s3tests tasks use rgw endpoint configuration

Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agomgr/dashboard: Ensure E2E tests can be run independently (#28592)
Lenz Grimmer [Wed, 19 Jun 2019 13:07:39 +0000 (15:07 +0200)]
mgr/dashboard: Ensure E2E tests can be run independently (#28592)

mgr/dashboard: Ensure E2E tests can be run independently

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nathan Weinberg <nweinber@redhat.com>
Reviewed-by: Rafael Quintero <rquinter@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
6 years agoMerge PR #28453 into master
Sage Weil [Wed, 19 Jun 2019 12:59:45 +0000 (07:59 -0500)]
Merge PR #28453 into master

* refs/pull/28453/head:
qa/valgrind.supp: be slightly less specific on suppression
msg/async, v2: make the reset_recv_state() unconditional.

Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #28566 from liewegas/wip-40377
Kefu Chai [Wed, 19 Jun 2019 12:15:18 +0000 (20:15 +0800)]
Merge pull request #28566 from liewegas/wip-40377

osd: copy (don't move) pg list when sending beacon

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agoqa/rgw: update default port in perl workunits 28631/head
Casey Bodley [Tue, 18 Jun 2019 16:44:19 +0000 (12:44 -0400)]
qa/rgw: update default port in perl workunits

Signed-off-by: Casey Bodley <cbodley@redhat.com>
6 years agoqa/rgw: extra s3tests tasks use rgw endpoint configuration
Casey Bodley [Tue, 18 Jun 2019 13:07:33 +0000 (09:07 -0400)]
qa/rgw: extra s3tests tasks use rgw endpoint configuration

Signed-off-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge PR #28212 into master
Patrick Donnelly [Wed, 19 Jun 2019 01:31:29 +0000 (18:31 -0700)]
Merge PR #28212 into master

* refs/pull/28212/head:
mds: trim cache during standby replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
6 years agoMerge PR #28551 into master
Patrick Donnelly [Wed, 19 Jun 2019 01:30:25 +0000 (18:30 -0700)]
Merge PR #28551 into master

* refs/pull/28551/head:
mds: don't mark cap NEEDSNAPFLUSH if client has no pending capsnap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
6 years agoMerge PR #28545 into master
Patrick Donnelly [Wed, 19 Jun 2019 01:28:51 +0000 (18:28 -0700)]
Merge PR #28545 into master

* refs/pull/28545/head:
client: set snapdir's link count to 1

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
6 years agoMerge pull request #28496 from ifed01/wip-ifed-fix-40080
Kefu Chai [Tue, 18 Jun 2019 15:31:40 +0000 (23:31 +0800)]
Merge pull request #28496 from ifed01/wip-ifed-fix-40080

os/bluestore: fix duplicate allocations in bmap allocator

Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #28509 from tchaikov/wip-mon_clean_pg_upmaps_per_chunk
Kefu Chai [Tue, 18 Jun 2019 15:31:13 +0000 (23:31 +0800)]
Merge pull request #28509 from tchaikov/wip-mon_clean_pg_upmaps_per_chunk

common/options: make mon_clean_pg_upmaps_per_chunk unsigned

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #28418 from xiexingguo/wip-kick-recovery-priority
Kefu Chai [Tue, 18 Jun 2019 15:29:42 +0000 (23:29 +0800)]
Merge pull request #28418 from xiexingguo/wip-kick-recovery-priority

osd: give recovery ops initialized by client op a higher priority

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #25395 from rzarzynski/wip-build-compiletest_cxx11_client
Kefu Chai [Tue, 18 Jun 2019 15:12:35 +0000 (23:12 +0800)]
Merge pull request #25395 from rzarzynski/wip-build-compiletest_cxx11_client

tests: introduce compiletest_cxx11_client for C++11 conformity.

Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agoMerge PR #28229 into master
Sage Weil [Tue, 18 Jun 2019 15:02:40 +0000 (10:02 -0500)]
Merge PR #28229 into master

* refs/pull/28229/head:
os/bluestore: do garbage collection if blob count is too high.
common/perf_conters: make dump_formatted_xxx funcs as const.
os/bluestore: store extents for GC within WriteContext.
os/bluestore: GC class, make some members local.
os/bluestore: vector -> interval set in GC to track extents to collect.
tests/store_test: many-many spanning blobs test case

Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge PR #28630 into master
Sage Weil [Tue, 18 Jun 2019 14:58:22 +0000 (09:58 -0500)]
Merge PR #28630 into master

* refs/pull/28630/head:
os/bluestore: fix duplicative and misleading debug in KernelDevice::open().

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
6 years agomgr/dashboard: Ensure E2E tests can be run independently 28592/head
Patrick Nawracay [Mon, 17 Jun 2019 09:36:53 +0000 (11:36 +0200)]
mgr/dashboard: Ensure E2E tests can be run independently

Introduces the PageHelper.

- Reads class attributes from Helpers to reduce boilerplace code for
  navigation.
- The PageHelper is supposed to be the new class for code that's reused
  across all Helpers for page related code.
- The Helper class is by now meant to be used for non-page specific
  helper code, like used in the `checkConsole` method.
- The Helper class will act as central registry to enable all other
  tests to use helper functions of all derived PageHelper tests.

  Example:

  `Helper.pools.create('foobar', ...);`

Fixes: http://tracker.ceph.com/issues/40397
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
6 years agoos/bluestore: fix duplicative and misleading debug in KernelDevice::open(). 28630/head
Radoslaw Zarzynski [Tue, 18 Jun 2019 12:36:30 +0000 (14:36 +0200)]
os/bluestore: fix duplicative and misleading debug in KernelDevice::open().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
6 years agomgr/dashboard: Update translations (#28059)
Lenz Grimmer [Tue, 18 Jun 2019 11:20:20 +0000 (13:20 +0200)]
mgr/dashboard: Update translations (#28059)

mgr/dashboard: Update translations

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
6 years agoMerge pull request #28377 from p-na/wip-pna-message-xlf-opt
Lenz Grimmer [Tue, 18 Jun 2019 10:16:23 +0000 (12:16 +0200)]
Merge pull request #28377 from p-na/wip-pna-message-xlf-opt

mgr/dashboard: Improve hints shown when message.xlf is invalid

Reviewed-by: Tiago Melo <tmelo@suse.com>
6 years agomgr/dashboard: Log's time from UTC to local (#27777)
Lenz Grimmer [Tue, 18 Jun 2019 10:14:09 +0000 (12:14 +0200)]
mgr/dashboard: Log's time from UTC to local (#27777)

mgr/dashboard: Log's time from UTC to local

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
6 years agoMerge pull request #28264 from rhcs-dashboard/mypy-config-to-file
Sebastian Wagner [Tue, 18 Jun 2019 07:09:06 +0000 (09:09 +0200)]
Merge pull request #28264 from rhcs-dashboard/mypy-config-to-file

src/script: extract mypy config to mypy.ini

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
6 years agoMerge pull request #28605 from cbodley/wip-40304
Yuri Weinstein [Mon, 17 Jun 2019 20:22:49 +0000 (13:22 -0700)]
Merge pull request #28605 from cbodley/wip-40304

qa/rgw: fix import error in tasks/swift.py

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoqa/rgw: fix import error in tasks/swift.py 28605/head
Casey Bodley [Mon, 17 Jun 2019 15:54:36 +0000 (11:54 -0400)]
qa/rgw: fix import error in tasks/swift.py

> ImportError: No module named packaging

replaces packaging.version.Version with distutils.version.LooseVersion

Signed-off-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #28095 from Devp00l/wip-39702
Lenz Grimmer [Mon, 17 Jun 2019 13:28:48 +0000 (15:28 +0200)]
Merge pull request #28095 from Devp00l/wip-39702

mgr/dashboard: OSD custom action button removal

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
6 years agoMerge pull request #27670 from trociny/wip-rbd-mirror-fetch-autotune
Jason Dillaman [Mon, 17 Jun 2019 13:02:58 +0000 (09:02 -0400)]
Merge pull request #27670 from trociny/wip-rbd-mirror-fetch-autotune

 rbd-mirror: adjust journal fetch properties based on memory target

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
6 years agoMerge pull request #28524 from trociny/wip-qa-rbd-mirror-compare
Jason Dillaman [Mon, 17 Jun 2019 13:02:40 +0000 (09:02 -0400)]
Merge pull request #28524 from trociny/wip-qa-rbd-mirror-compare

qa/workunits/rbd: fix compare_images and compare_image_snapshots

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
6 years agoMerge pull request #28529 from trociny/wip-journal-in_flight_advance_sets
Jason Dillaman [Mon, 17 Jun 2019 13:01:04 +0000 (09:01 -0400)]
Merge pull request #28529 from trociny/wip-journal-in_flight_advance_sets

journal: wait for in flight advance sets on stopping recorder

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
6 years agomgr/dashboard: Add QA tests for RBD configuration (#27774)
Lenz Grimmer [Mon, 17 Jun 2019 11:29:00 +0000 (13:29 +0200)]
mgr/dashboard: Add QA tests for RBD configuration (#27774)

mgr/dashboard: Add QA tests for RBD configuration

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
6 years agomgr/dashboard: Cleanup HACKING.rst
Patrick Nawracay [Mon, 17 Jun 2019 11:01:33 +0000 (13:01 +0200)]
mgr/dashboard: Cleanup HACKING.rst

Rephrase explanations on how to write E2E tests with Protractor.

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
6 years agoMerge pull request #28586 from aclamk/fix-vstart-extra-conf
Kefu Chai [Mon, 17 Jun 2019 09:57:16 +0000 (17:57 +0800)]
Merge pull request #28586 from aclamk/fix-vstart-extra-conf

vstart.sh: Fix problem that all extra_conf got merged into single line.

Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agovstart.sh: Fix problem that all extra_conf got merged into single line. 28586/head
Adam Kupczyk [Mon, 17 Jun 2019 07:43:44 +0000 (09:43 +0200)]
vstart.sh: Fix problem that all extra_conf got merged into single line.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
6 years agoMerge pull request #28559 from dillaman/wip-40368
Mykola Golub [Sat, 15 Jun 2019 08:34:34 +0000 (11:34 +0300)]
Merge pull request #28559 from dillaman/wip-40368

librbd: avoid dereferencing an empty container during deep-copy

Reviewed-by: Mykola Golub <mgolub@suse.com>
6 years agoMerge pull request #28487 from xiexingguo/wip-objecter-pg-mapping-cache
Xie Xingguo [Sat, 15 Jun 2019 05:31:02 +0000 (13:31 +0800)]
Merge pull request #28487 from xiexingguo/wip-objecter-pg-mapping-cache

osdc/Objecter: pg-mapping cache

Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge PR #28082 into master
Patrick Donnelly [Fri, 14 Jun 2019 21:51:07 +0000 (14:51 -0700)]
Merge PR #28082 into master

* refs/pull/28082/head:
mgr / volumes: improve error handling
py / cephfs: invoke base class Error::__init__() from OSError::__init__()
mgr / volumes: carve out subvolume operations as a separate class
mgr / volumes: introduce subvolume specification class
mgr / volumes: carve of volume as a separate class

6 years agoosd: copy (don't move) pg list when sending beacon 28566/head
Sage Weil [Fri, 14 Jun 2019 21:23:21 +0000 (16:23 -0500)]
osd: copy (don't move) pg list when sending beacon

If we move the list, then a second beacon sent before the pg list refreshes
will be empty.

Fixes: http://tracker.ceph.com/issues/40377
Signed-off-by: Sage Weil <sage@redhat.com>
6 years agoMerge PR #28198 into master
Patrick Donnelly [Fri, 14 Jun 2019 18:44:17 +0000 (11:44 -0700)]
Merge PR #28198 into master

* refs/pull/28198/head:
qa/tasks: upgrade command arguments checks in vstart_runner.py

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
6 years agolibrbd: avoid dereferencing an empty container during deep-copy 28559/head
Jason Dillaman [Fri, 14 Jun 2019 17:34:24 +0000 (13:34 -0400)]
librbd: avoid dereferencing an empty container during deep-copy

While updating the image copy progress, a 'while' loop will pop the
last item from the queue and then deference the empty queue.

Fixes: http://tracker.ceph.com/issues/40368
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
6 years agoMerge pull request #27773 from istudko/patch-1
Kefu Chai [Fri, 14 Jun 2019 17:20:51 +0000 (01:20 +0800)]
Merge pull request #27773 from istudko/patch-1

doc/releases/nautilus: Correct a systemctl command in an upgrade guide

Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agoMerge pull request #27201 from bstillwell/bootstrap-keyring-caps-doc-fix
Kefu Chai [Fri, 14 Jun 2019 17:19:26 +0000 (01:19 +0800)]
Merge pull request #27201 from bstillwell/bootstrap-keyring-caps-doc-fix

doc: Add missing mgr cap for the bootstrap keyring

Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agoMerge pull request #28526 from Osa1989/patch-1
Kefu Chai [Fri, 14 Jun 2019 17:07:24 +0000 (01:07 +0800)]
Merge pull request #28526 from Osa1989/patch-1

doc/releases/nautilus.rst: fix command to check min_compat_client

Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agoMerge pull request #28522 from neha-ojha/wip-add-client-endpoints
Neha Ojha [Fri, 14 Jun 2019 16:34:57 +0000 (09:34 -0700)]
Merge pull request #28522 from neha-ojha/wip-add-client-endpoints

 qa/tasks/cbt.py: add support for client_endpoints

Reviewed-by: Mark Nelson <mnelson@redhat.com>
6 years agoMerge pull request #28346 from smithfarm/wip-39447-alt
Mykola Golub [Fri, 14 Jun 2019 15:19:31 +0000 (18:19 +0300)]
Merge pull request #28346 from smithfarm/wip-39447-alt

qa/rbd: add cram-based snap diff test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
6 years agomgr/dashboard: OSD custom action button removal 28095/head
Stephan Müller [Wed, 17 Apr 2019 12:17:44 +0000 (14:17 +0200)]
mgr/dashboard: OSD custom action button removal

Now the custom action dropdown is removed from the OSD component. It was
nearly a clone of the table actions button component, but it had a
different behavior as it did not show the main action inside the dropdown.

Fixes: https://tracker.ceph.com/issues/39702
Signed-off-by: Stephan Müller <smueller@suse.com>
6 years agoMerge pull request #28532 from cbodley/wip-40304
Yuri Weinstein [Fri, 14 Jun 2019 15:16:32 +0000 (08:16 -0700)]
Merge pull request #28532 from cbodley/wip-40304

qa/rgw: skip swift tests on rhel 7.6+

6 years agodoc/releases/nautilus.rst: fix command to check min_compat_client 28526/head
Osa1989 [Thu, 13 Jun 2019 08:31:02 +0000 (10:31 +0200)]
doc/releases/nautilus.rst: fix command to check min_compat_client

Command should be "ceph osd dump | grep min_compat_client"

Signed-off-by: Osama Elswah <osama.ibrahim.89@gmail.com>
6 years agoMerge pull request #27951 from ifed01/wip-ifed-fix-slowop-log
Igor Fedotov [Fri, 14 Jun 2019 10:43:49 +0000 (13:43 +0300)]
Merge pull request #27951 from ifed01/wip-ifed-fix-slowop-log

os/bluestore: fix origin reference in logging slow ops.

6 years agoMerge pull request #28490 from xiexingguo/wip-40193
Kefu Chai [Fri, 14 Jun 2019 10:29:38 +0000 (18:29 +0800)]
Merge pull request #28490 from xiexingguo/wip-40193

mon/OSDMonitor: report pg[pgp]_num_target instead of pg[pgp]_num

Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #28322 from liewegas/wip-preextend
Kefu Chai [Fri, 14 Jun 2019 10:27:44 +0000 (18:27 +0800)]
Merge pull request #28322 from liewegas/wip-preextend

os/bluestore: bluefs_preextend_wal_files=true

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
6 years agoMerge pull request #28260 from s0nea/wip-dashboard-pool-delete-flag
Laura Paduano [Fri, 14 Jun 2019 10:02:02 +0000 (12:02 +0200)]
Merge pull request #28260 from s0nea/wip-dashboard-pool-delete-flag

mgr/dashboard: consider 'mon_allow_pool_delete' flag

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
6 years agomds: don't mark cap NEEDSNAPFLUSH if client has no pending capsnap 28551/head
Yan, Zheng [Fri, 14 Jun 2019 09:32:42 +0000 (17:32 +0800)]
mds: don't mark cap NEEDSNAPFLUSH if client has no pending capsnap

This avoid getting wrlocks and setup client_snap_caps on snap inode.
Because there is no further snapflush that triggers releasing these
wrlocks.

Fixes: https://tracker.ceph.com/issues/40361
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
6 years agoMerge pull request #28534 from tchaikov/wip-crimson-mkkey
Kefu Chai [Fri, 14 Jun 2019 09:34:42 +0000 (17:34 +0800)]
Merge pull request #28534 from tchaikov/wip-crimson-mkkey

crimson/osd: add --mkkey support

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
6 years agoMerge pull request #28464 from majianpeng/fio-ceph-messenger-multi-client
Kefu Chai [Fri, 14 Jun 2019 09:14:43 +0000 (17:14 +0800)]
Merge pull request #28464 from majianpeng/fio-ceph-messenger-multi-client

test/fio/fio_ceph_messenger: make exec multi client on the same host.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agoMerge pull request #28404 from xiexingguo/wip-osd-noup-stall
Kefu Chai [Fri, 14 Jun 2019 08:37:33 +0000 (16:37 +0800)]
Merge pull request #28404 from xiexingguo/wip-osd-noup-stall

osd/OSD: keep synchronizing with mon if stuck at booting

Reviewed-by: Sage Weil <sage@redhat.com>
6 years agocrimson/osd: ignore exception when stopping osd 28534/head
Kefu Chai [Fri, 14 Jun 2019 06:57:55 +0000 (14:57 +0800)]
crimson/osd: ignore exception when stopping osd

OSD::stop() is a top-level method in the sense that it will be called
directly when engine stops. so it should not throw any exceptions.
instead, we will print out the error and ignore the exception if any.

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agocrimson/osd: add --mkkey support
Kefu Chai [Thu, 13 Jun 2019 14:48:48 +0000 (22:48 +0800)]
crimson/osd: add --mkkey support

see qa/tasks/ceph.py for how it is used, and why we need it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agocrimson/os: use buffer::write_file() helper
Kefu Chai [Thu, 13 Jun 2019 14:44:16 +0000 (22:44 +0800)]
crimson/os: use buffer::write_file() helper

instead of using the blocking calls

so we can write to file without blocking the reactor

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agocrimson: add buffer_io helpers
Kefu Chai [Fri, 14 Jun 2019 03:49:47 +0000 (11:49 +0800)]
crimson: add buffer_io helpers

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agoMerge pull request #28507 from tchaikov/wip-cmake-dpdk-boost
Kefu Chai [Fri, 14 Jun 2019 07:43:41 +0000 (15:43 +0800)]
Merge pull request #28507 from tchaikov/wip-cmake-dpdk-boost

cmake: update SPDK to build with GCC-9

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoqa/rbd: add cram-based snap diff test 28346/head
Nathan Cutler [Fri, 31 May 2019 10:57:15 +0000 (12:57 +0200)]
qa/rbd: add cram-based snap diff test

Fixes: http://tracker.ceph.com/issues/39447
Signed-off-by: Shyukri Shyukriev <shshyukriev@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
6 years agoauth/KeyRing: remove KeyRing::create_empty()
Kefu Chai [Thu, 13 Jun 2019 14:38:02 +0000 (22:38 +0800)]
auth/KeyRing: remove KeyRing::create_empty()

it never fails unless out of memory. and it would be simpler to just
allocate it on stack.

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agoosdc/Objecter: pg-mapping cache 28487/head
xie xingguo [Thu, 6 Jun 2019 08:12:27 +0000 (16:12 +0800)]
osdc/Objecter: pg-mapping cache

The CRUSH-based addressing is some kind of CPU intensive task
and hence should be avoid whenever possible.

The patch introduces a per objecter pg-mapping cache, which as
a result can saved us 10%+ CPU (fio, 4k randow read, 24k IOPS):

Was:
$ top -Hp 415864
top - 14:45:39 up 6 days,  5:45,  3 users,  load average: 9.67, 8.66, 8.45
Threads:  23 total,   3 running,  20 sleeping,   0 stopped,   0 zombie
%Cpu(s): 21.6 us, 14.5 sy,  0.0 ni, 59.6 id,  1.4 wa,  0.0 hi,  3.0 si,  0.0 st
KiB Mem : 19616576+total, 12773412+free, 50964508 used, 17467128 buff/cache
KiB Swap:  4194300 total,  4194300 free,        0 used. 14352892+avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 415942 root      20   0 1957800 492820 123880 R 80.0  0.3   0:13.75 tp_librbd

Now:
$ top -Hp 475779
top - 10:22:05 up  1:18,  4 users,  load average: 2.65, 1.44, 1.60
Threads:  23 total,   2 running,  21 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.1 us,  2.2 sy,  0.0 ni, 91.4 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem : 19616576+total, 13919555+free, 27280820 used, 29689392 buff/cache
KiB Swap:  4194300 total,  4194300 free,        0 used. 16798102+avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 476231 root      20   0 1957808 491712 123836 S 71.1  0.3   0:41.47 tp_librbd

And below is a more detailed report from the perf tool:

Was:
$ perf report -g graph
  Children      Self  Command    Shared Object        Symbol
+   29.33%     0.03%  tp_librbd  librados.so.2.0.0    [.] Objecter::op_submit
+   28.83%     0.09%  tp_librbd  librados.so.2.0.0    [.] Objecter::_op_submit_with_budget
+   27.12%     0.17%  tp_librbd  librados.so.2.0.0    [.] Objecter::_op_submit
+   16.47%     0.26%  tp_librbd  librados.so.2.0.0    [.] Objecter::_calc_target
+   15.04%     0.10%  tp_librbd  libceph-common.so.0  [.] OSDMap::_pg_to_up_acting_osds
+   13.52%     0.16%  tp_librbd  libceph-common.so.0  [.] OSDMap::_pg_to_raw_osds

Now:
$ perf report -g graph
  Children      Self  Command    Shared Object        Symbol
+   17.84%     0.04%  tp_librbd  librados.so.2.0.0    [.] Objecter::op_submit
+   17.34%     0.06%  tp_librbd  librados.so.2.0.0    [.] Objecter::_op_submit_with_budget
+   15.80%     0.17%  tp_librbd  librados.so.2.0.0    [.] Objecter::_op_submit
+    6.11%     2.02%  tp_librbd  librados.so.2.0.0    [.] Objecter::_calc_target

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
6 years agoMerge pull request #28535 from tchaikov/wip-crimson-packaging
Kefu Chai [Fri, 14 Jun 2019 05:27:45 +0000 (13:27 +0800)]
Merge pull request #28535 from tchaikov/wip-crimson-packaging

debian/: use ceph-osd for packaging crimson-osd

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
6 years agomgr / volumes: improve error handling 28082/head
Venky Shankar [Sun, 19 May 2019 17:34:57 +0000 (13:34 -0400)]
mgr / volumes: improve error handling

This cleans up lots of return-fu statements and make the source
look much more pythonic. Functions should raise an instance of
VolumeException() class wherever necessary (error handling).

Fixes: http://tracker.ceph.com/issues/39969
Signed-off-by: Venky Shankar <vshankar@redhat.com>
6 years agopy / cephfs: invoke base class Error::__init__() from OSError::__init__()
Venky Shankar [Sun, 19 May 2019 17:20:51 +0000 (13:20 -0400)]
py / cephfs: invoke base class Error::__init__() from OSError::__init__()

This will allow access to `args` when catching `Error`.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
6 years agomgr / volumes: carve out subvolume operations as a separate class
Venky Shankar [Fri, 17 May 2019 05:40:37 +0000 (01:40 -0400)]
mgr / volumes: carve out subvolume operations as a separate class

Move (and refactor) subvolume specific operations from main module
source to SubVolume class in a new subvolume source. Also, provide
hooks in VolumeClient to forward subvolume specific opertaions to
this class.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
6 years agomgr / volumes: introduce subvolume specification class
Venky Shankar [Thu, 16 May 2019 05:54:12 +0000 (01:54 -0400)]
mgr / volumes: introduce subvolume specification class

Specifications class that represents a unique subvolume
identified by (subvolume-id, group-id) tuple. Provide
heleper functions for fetching  various subvolume specific
attributes such as subvolume path, group path etc.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
6 years agomgr / volumes: carve of volume as a separate class
Venky Shankar [Mon, 13 May 2019 12:15:35 +0000 (17:45 +0530)]
mgr / volumes: carve of volume as a separate class

Move volume operations (create/delete) from main module source
to VolumeClient class in new volume source.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
6 years agoclient: set snapdir's link count to 1 28545/head
Yan, Zheng [Fri, 14 Jun 2019 04:01:31 +0000 (12:01 +0800)]
client: set snapdir's link count to 1

Otherwise nfs-ganesha treats snapdir as stale.

Fixes: https://tracker.ceph.com/issues/40101
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
6 years agoMerge PR #28456 into master
Patrick Donnelly [Thu, 13 Jun 2019 19:22:05 +0000 (12:22 -0700)]
Merge PR #28456 into master

* refs/pull/28456/head:
mds: ignore sessionmap version mismatch if mds_wipe_sessions is set
mds: fix corner case of replaying open sessions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
6 years agoMerge pull request #28521 from alimaredia/wip-qa-rgw-ssl
Casey Bodley [Thu, 13 Jun 2019 19:00:21 +0000 (15:00 -0400)]
Merge pull request #28521 from alimaredia/wip-qa-rgw-ssl

qa: use curl in wait_for_radosgw() in util/rgw.py

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #28415 from linuxbox2/wip-full-odsock
Matt Benjamin [Thu, 13 Jun 2019 17:57:25 +0000 (13:57 -0400)]
Merge pull request #28415 from linuxbox2/wip-full-odsock

rgw/OutputDataSocket: actually discard data on full buffer

6 years agoqa: use curl in wait_for_radosgw() in util/rgw.py 28521/head
Ali Maredia [Wed, 12 Jun 2019 20:12:47 +0000 (16:12 -0400)]
qa: use curl in wait_for_radosgw() in util/rgw.py

Signed-off-by: Ali Maredia <amaredia@redhat.com>
6 years agoMerge pull request #25496 from mkogan1/wip-rgw-dynamic-resharding-docs
Casey Bodley [Thu, 13 Jun 2019 16:01:16 +0000 (12:01 -0400)]
Merge pull request #25496 from mkogan1/wip-rgw-dynamic-resharding-docs

rgw: update the "radosgw-admin reshard status"

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoqa/rgw: skip swift tests on rhel 7.6+ 28532/head
Casey Bodley [Thu, 13 Jun 2019 13:45:29 +0000 (09:45 -0400)]
qa/rgw: skip swift tests on rhel 7.6+

swift bootstrap fails because libev-devel is not available

Fixes: http://tracker.ceph.com/issues/40304
Signed-off-by: Casey Bodley <cbodley@redhat.com>
6 years agodebian/: use ceph-osd for packaging crimson-osd 28535/head
Kefu Chai [Thu, 13 Jun 2019 08:28:29 +0000 (16:28 +0800)]
debian/: use ceph-osd for packaging crimson-osd

* debian/: remove ceph-crimson-osd package.
* debian/control: set `CEPH_OSD_BASENAME` env variable, which
  will be consumed by `ceph-osd.install`. alternatively, we could
  rename crimson-osd to ceph-osd in `override_dh_auto_install`,
  but let's go with this way at this moment, unless `mv` in
  `override_dh_auto_install` is proved to be better.
* ceph-osd.install: replace ceph-osd with crimson-osd if
  `CEPH_EXTRA_CMAKE_ARGS` has `WITH_SEASTAR` in it. this only
  happens when we are packaging the "crimson" flavor packages from
  jenkins.
* ceph-osd.install: `chmod +x` this file, as we need to use
  `/usr/bin/dh-exec` as the interpreter of it to perform variable
  substitution and install.

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agocmake: silence -Waddress-of-packed-member warnings from SPDK 28507/head
Kefu Chai [Wed, 12 Jun 2019 13:27:33 +0000 (21:27 +0800)]
cmake: silence -Waddress-of-packed-member warnings from SPDK

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agocmake: silence GCC-9 warnings from rocksdb
Kefu Chai [Wed, 12 Jun 2019 08:50:14 +0000 (16:50 +0800)]
cmake: silence GCC-9 warnings from rocksdb

GCC-9 complains:

rocksdb/db/version_edit.h:86:8: warning: implicitly-declared ‘constexpr
rocksdb::FileDescriptor::FileDescriptor(const rocksdb::FileDescriptor&)’
is deprecated [-Wdeprecated-copy]
   86 | struct FileMetaData {
      |        ^~~~~~~~~~~~
/var/ssd/ceph/src/rocksdb/db/version_edit.h:55:19: note: because
‘rocksdb::FileDescriptor’ has user-provided ‘rocksdb::FileDescriptor&
rocksdb::FileDescriptor::operator=(const rocksdb::FileDescriptor&)’
   55 |   FileDescriptor& operator=(const FileDescriptor& fd) {
      |                   ^~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agocmake: set ccache for rocksdb
Kefu Chai [Wed, 12 Jun 2019 10:54:26 +0000 (18:54 +0800)]
cmake: set ccache for rocksdb

should use launcher to set ccache

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agocmake: update built libs of SPDK and DPDK
Kefu Chai [Wed, 12 Jun 2019 11:18:53 +0000 (19:18 +0800)]
cmake: update built libs of SPDK and DPDK

we need to link against these new libs for fulfilling the dependencies
required by new SPDK.

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agocmake: disable isal and vhost support of SPDK
Kefu Chai [Wed, 12 Jun 2019 08:32:55 +0000 (16:32 +0800)]
cmake: disable isal and vhost support of SPDK

* isal needs nasm, which is a new dependency. so let's disable it
  at this moment.
* vhost requires DPDK compiled with vhost enabled. but it's disabled
  when we compile DPDK.

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agospdk: pick up latest spdk
Kefu Chai [Wed, 12 Jun 2019 08:04:41 +0000 (16:04 +0800)]
spdk: pick up latest spdk

to silence the warnings reported by GCC-9

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agojournal: wait for in flight advance sets on stopping recorder 28529/head
Mykola Golub [Thu, 13 Jun 2019 11:05:17 +0000 (12:05 +0100)]
journal: wait for in flight advance sets on stopping recorder

Before object overflow detection optimization (83461c42b) the
overflow was detected when trying to send data, so when an object
set was advanced we always had some data and flush always
completed later. But now we detect overflow on client side, it
may happen when buffer is empty and flush may complete
when "advance object set" is still in flight.

Signed-off-by: Mykola Golub <mgolub@suse.com>
6 years agorgw: provide admin friendly reshard status output 25496/head
Mark Kogan [Sun, 12 May 2019 13:45:30 +0000 (09:45 -0400)]
rgw: provide admin friendly reshard status output

Signed-off-by: Mark Kogan <mkogan@redhat.com>
6 years agoMerge pull request #28446 from tspmelo/wip-prepare-backport
Nathan Cutler [Thu, 13 Jun 2019 09:01:30 +0000 (11:01 +0200)]
Merge pull request #28446 from tspmelo/wip-prepare-backport

script/ceph-backport.sh: Add prepare function

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
6 years agoqa/workunits/rbd: fix compare_images and compare_image_snapshots 28524/head
Mykola Golub [Thu, 13 Jun 2019 07:26:29 +0000 (08:26 +0100)]
qa/workunits/rbd: fix compare_images and compare_image_snapshots

Due to pipe, sdiff return code was ignored. Also the compare functions
spent most of time in xxd, which was normally unnecessary.

Signed-off-by: Mykola Golub <mgolub@suse.com>
6 years agocommon/options: make mon_clean_pg_upmaps_per_chunk unsigned 28509/head
Kefu Chai [Wed, 12 Jun 2019 11:44:07 +0000 (19:44 +0800)]
common/options: make mon_clean_pg_upmaps_per_chunk unsigned

we don't use a negative number for some different purpose, for instance,
for disabling this option. and this helps to silence following warning:

/home/jenkins-build/build/workspace/ceph-pull-requests/src/mon/OSDMonitor.cc:
In member function 'virtual void
OSDMonitor::encode_pending(MonitorDBStore::TransactionRef)':
/home/jenkins-build/build/workspace/ceph-pull-requests/src/mon/OSDMonitor.cc:1195:31:
warning: comparison of integer expressions of different signedness:
'std::vector<pg_t>::size_type' {aka 'long unsigned int'} and 'int64_t'
{aka 'long int'} [-Wsign-compare]
       if (pgs_to_check.size() < g_conf()->mon_clean_pg_upmaps_per_chunk
* 2) {
           ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>
6 years agoMerge pull request #28515 from hvhaugwitz/mondb-recovery-script
Kefu Chai [Thu, 13 Jun 2019 00:36:16 +0000 (08:36 +0800)]
Merge pull request #28515 from hvhaugwitz/mondb-recovery-script

doc: update mondb recovery script

Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agoqa/tasks/cbt.py: add support for client_endpoints 28522/head
Neha Ojha [Wed, 12 Jun 2019 00:33:53 +0000 (00:33 +0000)]
qa/tasks/cbt.py: add support for client_endpoints

Note that this will only work for librbdfio for now.

Signed-off-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge PR #28473 into master
Patrick Donnelly [Wed, 12 Jun 2019 21:30:45 +0000 (14:30 -0700)]
Merge PR #28473 into master

* refs/pull/28473/head:
cephfs-shell: Catch OSError exceptions in lcd

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
6 years agoMerge PR #28475 into master
Patrick Donnelly [Wed, 12 Jun 2019 21:29:46 +0000 (14:29 -0700)]
Merge PR #28475 into master

* refs/pull/28475/head:
cephfs-shell: Fix 'lls' command errors

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
6 years agoMerge PR #28407 into master
Patrick Donnelly [Wed, 12 Jun 2019 21:27:53 +0000 (14:27 -0700)]
Merge PR #28407 into master

* refs/pull/28407/head:
mgr/volumes: fix error message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
6 years agoMerge PR #27856 into master
Patrick Donnelly [Wed, 12 Jun 2019 21:27:07 +0000 (14:27 -0700)]
Merge PR #27856 into master

* refs/pull/27856/head:
test: remove fs/test-volume.sh workunit
test: port fs/volume related tests to python

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
6 years agoMerge PR #28196 into master
Patrick Donnelly [Wed, 12 Jun 2019 21:16:31 +0000 (14:16 -0700)]
Merge PR #28196 into master

* refs/pull/28196/head:
ceph_volume_client: decode d_name before using it

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
6 years agoMerge PR #28221 into master
Patrick Donnelly [Wed, 12 Jun 2019 21:15:30 +0000 (14:15 -0700)]
Merge PR #28221 into master

* refs/pull/28221/head:
qa/tasks/cephfs/test_volume_client: print py2 or py3 which the test case runs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
6 years agoMerge PR #26737 into master
Patrick Donnelly [Wed, 12 Jun 2019 21:07:22 +0000 (14:07 -0700)]
Merge PR #26737 into master

* refs/pull/26737/head:
qa/cephfs: update tests for stale session handling
mds: change how mds revoke stale caps
mds: don't mark unresponsive sessions holding no caps stale

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>