]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
5 years agocommon/options: Set concurrent bluestore rocksdb compactions to 2 30150/head
Mark Nelson [Fri, 12 Jul 2019 16:45:17 +0000 (12:45 -0400)]
common/options: Set concurrent bluestore rocksdb compactions to 2

Signed-off-by: Mark Nelson <mnelson@redhat.com>
(cherry picked from commit 26dc5189e73a1f5a22f89ee67eef5430ef660b5d)

5 years agoMerge pull request #29900 from alfredodeza/mimic-rm41378
Andrew Schoen [Mon, 26 Aug 2019 17:21:33 +0000 (12:21 -0500)]
Merge pull request #29900 from alfredodeza/mimic-rm41378

mimic ceph-volume tests set the noninteractive flag for Debian

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoceph-volume tests set the noninteractive flag for Debian, to avoid prompts in apt 29900/head
Alfredo Deza [Wed, 21 Aug 2019 18:15:32 +0000 (14:15 -0400)]
ceph-volume tests set the noninteractive flag for Debian, to avoid prompts in apt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 89231c9a60a46ba0691f787efbfe5ab35f05c679)

5 years agoMerge pull request #29770 from alfredodeza/mimic-bz-1738379
Alfredo Deza [Wed, 21 Aug 2019 12:11:26 +0000 (08:11 -0400)]
Merge pull request #29770 from alfredodeza/mimic-bz-1738379

mimic ceph-volume: use the OSD identifier when reporting success

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
5 years agoceph-volume devices.lvm zap use the identifier to report success 29770/head
Alfredo Deza [Tue, 20 Aug 2019 12:04:56 +0000 (08:04 -0400)]
ceph-volume devices.lvm zap use the identifier to report success

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d0d5750c1ba508152c1d5111f2003a039a27145f)

5 years agoceph-volume tests check success message when zapping
Alfredo Deza [Tue, 20 Aug 2019 12:04:26 +0000 (08:04 -0400)]
ceph-volume tests check success message when zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 88c7429a60fdab6222d14132bea93b9bf773c720)

5 years agoMerge pull request #29661 from jan--f/wip-41246-mimic
Alfredo Deza [Tue, 20 Aug 2019 11:38:45 +0000 (07:38 -0400)]
Merge pull request #29661 from jan--f/wip-41246-mimic

mimic: simple functional tests drop test for lvm zap

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoMerge pull request #29688 from jan--f/wip-41298-mimic
Alfredo Deza [Tue, 20 Aug 2019 11:37:25 +0000 (07:37 -0400)]
Merge pull request #29688 from jan--f/wip-41298-mimic

mimic: batch functional idempotency test fails since message is now on stderr

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoMerge pull request #29618 from pdvian/wip-40943-mimic
Neha Ojha [Fri, 16 Aug 2019 21:07:51 +0000 (14:07 -0700)]
Merge pull request #29618 from pdvian/wip-40943-mimic

mimic: mon/OSDMonitor.cc: better error message about min_size

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #29699 from pdvian/wip-41086-mimic
Neha Ojha [Fri, 16 Aug 2019 16:31:51 +0000 (09:31 -0700)]
Merge pull request #29699 from pdvian/wip-41086-mimic

mimic: Change default for bluestore_fsck_on_mount_deep as false

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agocommon/options.cc: common/options.cc: change default value of bluestore_fsck_on_umoun... 29699/head
Neha Ojha [Wed, 31 Jul 2019 20:08:24 +0000 (13:08 -0700)]
common/options.cc: common/options.cc: change default value of bluestore_fsck_on_umount_deep to false

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 8c86c3a5b4bdca258016019f63359f52aa8cd84b)

5 years agocommon/options.cc: change default value of bluestore_fsck_on_mount_deep to false
Neha Ojha [Wed, 31 Jul 2019 00:05:18 +0000 (17:05 -0700)]
common/options.cc: change default value of bluestore_fsck_on_mount_deep to false

Fixes: https://tracker.ceph.com/issues/41017
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 7c03f088f1004d4e26b6bdea8fa58f8455f1d6a2)

5 years agoMerge pull request #29289 from smithfarm/wip-40497-mimic
Nathan Cutler [Thu, 15 Aug 2019 15:23:28 +0000 (17:23 +0200)]
Merge pull request #29289 from smithfarm/wip-40497-mimic

mimic: doc: Object Gateway multisite document read-only argument error

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoceph-volume: fix batch functional tests, idempotent test must check stderr 29688/head
Jan Fajerski [Thu, 15 Aug 2019 11:28:42 +0000 (13:28 +0200)]
ceph-volume: fix batch functional tests, idempotent test must check stderr

Fixes: https://tracker.ceph.com/issues/41295
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 88807110f3014e7a2a39c37ae5c64bb3ab107d80)

5 years agoceph-volume: don't try to test lvm zap on simple tests 29661/head
Jan Fajerski [Wed, 14 Aug 2019 12:36:58 +0000 (14:36 +0200)]
ceph-volume: don't try to test lvm zap on simple tests

Fixes: https://tracker.ceph.com/issues/41245
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 081109859a06f1b2ff6652dce07da4ddf6fb258e)

5 years agoMerge pull request #29631 from jan--f/backport-mimic-26826
Jan Fajerski [Wed, 14 Aug 2019 12:29:51 +0000 (14:29 +0200)]
Merge pull request #29631 from jan--f/backport-mimic-26826

ceph-volume: more mimic backports

5 years agoMerge pull request #29602 from jan--f/wip-41201-mimic
Jan Fajerski [Wed, 14 Aug 2019 12:28:33 +0000 (14:28 +0200)]
Merge pull request #29602 from jan--f/wip-41201-mimic

mimic: ceph-volume prints log messages to stdout

5 years agoceph-volume: make systemctl.get_running_osds resilient to garbage output 29631/head
Andrew Schoen [Fri, 8 Mar 2019 15:53:08 +0000 (09:53 -0600)]
ceph-volume: make systemctl.get_running_osds resilient to garbage output

Makes systemd.systemctl.get_running_osds more resilient to output
that doesn't match what we're expecting to parse. This also handles
the case of a nonzero exit status by retuning an empty list.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit f3707f1ad43513c2a6df17e9dce756533c3b4a1d)

5 years agoceph-volume: remove obsoleted simple unit test
Andrew Schoen [Thu, 7 Mar 2019 18:38:02 +0000 (12:38 -0600)]
ceph-volume: remove obsoleted simple unit test

This test is no longer valid because `simple scan`
without any arguments will now attempt to scan all
running OSDs.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit b59dcd01c2dd386ca1b63eb66ed243afffb80211)

5 years agoceph-volume: tox tests should ignore the tests/functional directory
Andrew Schoen [Thu, 7 Mar 2019 18:37:00 +0000 (12:37 -0600)]
ceph-volume: tox tests should ignore the tests/functional directory

Without this the functional testinfra tests try to run when we
are intending to only run the unit tests.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 1881f386d8207ee8c10f0863c4a1abf18cf3c5d9)

5 years agoceph-volume: use `simple scan` and `simple activate --all` in tests
Andrew Schoen [Thu, 7 Mar 2019 17:57:03 +0000 (11:57 -0600)]
ceph-volume: use `simple scan` and `simple activate --all` in tests

In a couple of the functional tests for simple we should run `simple
scan` without any arguments and `simple activate --all` to increase
the test coverage. We want to test pointing at a diretory and scanning
all running OSDs.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 87c33f149a83bb1175bfb473b226e8c46e4e2a6e)

5 years agoceph-volume: update docs and help menu for `simple scan`
Andrew Schoen [Thu, 7 Mar 2019 17:10:19 +0000 (11:10 -0600)]
ceph-volume: update docs and help menu for `simple scan`

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit f7943ef6007fb05d83602bc22cbacf3bb67d8625)

5 years agoceph-volume: adds unit tests for systemd.systemctl.get_running_osd_ids
Andrew Schoen [Thu, 7 Mar 2019 16:01:40 +0000 (10:01 -0600)]
ceph-volume: adds unit tests for systemd.systemctl.get_running_osd_ids

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 7cb098b6899542c875d911d28771a4c450d0bb3a)

5 years agoceph-volume: update `simple scan` to scan all running OSDs
Andrew Schoen [Wed, 6 Mar 2019 22:52:33 +0000 (16:52 -0600)]
ceph-volume: update `simple scan` to scan all running OSDs

If no argument is passed to `ceph-volume simple scan` it will
inspect any running osds and scan them if they were created by
ceph-disk.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 28e454ec30f867ae1229c66d263bcd608654553f)

5 years agoceph-volume: add systemd.systemctl.get_running_osd_ids
Andrew Schoen [Wed, 6 Mar 2019 22:51:06 +0000 (16:51 -0600)]
ceph-volume: add systemd.systemctl.get_running_osd_ids

This is used to retrieve a list of osd ids from the currently
running ceph-osd systemd units.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 15d20e29a2f2d74caf05dfec559e30941c7a46ab)

5 years agoceph-volume tests.functional.simple add zap by osd id to xenial
Alfredo Deza [Thu, 6 Dec 2018 20:18:06 +0000 (15:18 -0500)]
ceph-volume tests.functional.simple add zap by osd id to xenial

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f53f67ca3d25deb82c20d31df418a19948ba6f64)

5 years agoceph-volume tests.functional.simple add zap by osd id to centos7
Alfredo Deza [Thu, 6 Dec 2018 20:17:55 +0000 (15:17 -0500)]
ceph-volume tests.functional.simple add zap by osd id to centos7

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8955ec14566c48a08bb4826256b422aa29a2c16e)

Conflicts:
src/ceph-volume/ceph_volume/tests/functional/simple/centos7/filestore/activate/test.yml
        remove with_items key

5 years agomon/OSDMonitor.cc: better error message about min_size 29618/head
Neha Ojha [Tue, 23 Jul 2019 00:50:31 +0000 (17:50 -0700)]
mon/OSDMonitor.cc: better error message about min_size

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit ce25ac0a9196f90acef82915f122c5a06277993a)

5 years agoceph-volume: never log to stdout, use stderr instead 29602/head
Jan Fajerski [Thu, 8 Aug 2019 08:42:37 +0000 (10:42 +0200)]
ceph-volume: never log to stdout, use stderr instead

We should never print log messages to stdout, as this should be reserved
for output of ceph-volume.

Fixes: https://tracker.ceph.com/issues/41158
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit b8d6dcbe9f803c96c0af68da54f1262e9b6a9e77)

5 years agoceph-volume: terminal: encode unicode when writing to stdout
Kefu Chai [Sun, 24 Mar 2019 09:11:31 +0000 (17:11 +0800)]
ceph-volume: terminal: encode unicode when writing to stdout

python determins the encoding of stdout and stderr based on the LC_CTYPE
and PYTHONIOENCODING env variable, by default, python3's sys.stdout uses
'utf-8' as its encoding, so it will be able to write unicode string even
the stdout is not attached to a tty device. but when it comes to
python2, it will default to ascii if neither of these variabls is set.
so, if we are writing unicode using `_Write` in an environment where
LC_CTYPE and/or PYTHONIOENCODING are using non UTF-8 encoding, it chokes
by raising `UnicodeEncodeError` exception.

in this change, we add a wrapper around `_Write._writer` so it is able
to write unicode string in such a non-unicode-friendly environment.

for more info related the encoding of stdout and stderr, see
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONIOENCODING .

Signed-off-by: Alfredo Deza <adeza@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 77912c0c71874a23785d3bdd109ad61e4f4c0e28)

5 years agoMerge pull request #29507 from jan--f/wip-41138-mimic
Jan Fajerski [Wed, 7 Aug 2019 08:18:45 +0000 (10:18 +0200)]
Merge pull request #29507 from jan--f/wip-41138-mimic

mimic: ceph-volume prints errors to stdout with --format json

5 years agoceph-volume: print most logging messages to stderr 29507/head
Jan Fajerski [Mon, 4 Mar 2019 10:28:13 +0000 (11:28 +0100)]
ceph-volume: print most logging messages to stderr

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
Fixes: http://tracker.ceph.com/issues/38548
(cherry picked from commit 1f8f1e95dee88b4bab2d6931e720ed83584d2887)

 Conflicts:
src/pybind/mgr/test_orchestrator/module.py
        removed

5 years agoMerge pull request #29417 from jan--f/wip-41022-mimic
Jan Fajerski [Wed, 7 Aug 2019 06:44:00 +0000 (08:44 +0200)]
Merge pull request #29417 from jan--f/wip-41022-mimic

mimic: simple: when 'type' file is not present activate fails

5 years agoMerge pull request #29255 from smithfarm/wip-39274-mimic
Yuri Weinstein [Tue, 6 Aug 2019 21:19:37 +0000 (14:19 -0700)]
Merge pull request #29255 from smithfarm/wip-39274-mimic

mimic: rgw: S3 policy evaluated incorrectly

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #29256 from smithfarm/wip-39415-mimic
Yuri Weinstein [Tue, 6 Aug 2019 21:19:15 +0000 (14:19 -0700)]
Merge pull request #29256 from smithfarm/wip-39415-mimic

mimic: rgw: multisite: period pusher gets 403 Forbidden against other zonegroups

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #29262 from smithfarm/wip-39748-mimic
Yuri Weinstein [Tue, 6 Aug 2019 21:18:55 +0000 (14:18 -0700)]
Merge pull request #29262 from smithfarm/wip-39748-mimic

mimic: rgw: Add support for --bypass-gc flag of radosgw-admin bucket rm command in RGW Multi-site

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #29267 from smithfarm/wip-40128-mimic
Yuri Weinstein [Tue, 6 Aug 2019 21:18:30 +0000 (14:18 -0700)]
Merge pull request #29267 from smithfarm/wip-40128-mimic

mimic: rgw: Swift interface: server side copy fails if object name contains '?'

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #29268 from smithfarm/wip-40136-mimic
Yuri Weinstein [Tue, 6 Aug 2019 21:18:07 +0000 (14:18 -0700)]
Merge pull request #29268 from smithfarm/wip-40136-mimic

mimic: rgw: the Multi-Object Delete operation of S3 API wrongly handles the Code response element

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #29270 from smithfarm/wip-40141-mimic
Yuri Weinstein [Tue, 6 Aug 2019 21:17:35 +0000 (14:17 -0700)]
Merge pull request #29270 from smithfarm/wip-40141-mimic

mimic: rgw: multisite: 'radosgw-admin bucket sync status' should call syncs_from(source.name) instead of id

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #29273 from smithfarm/wip-40144-mimic
Yuri Weinstein [Tue, 6 Aug 2019 21:16:18 +0000 (14:16 -0700)]
Merge pull request #29273 from smithfarm/wip-40144-mimic

mimic: rgw: Multisite sync corruption for large multipart obj

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #29277 from smithfarm/wip-40225-mimic
Yuri Weinstein [Tue, 6 Aug 2019 21:15:35 +0000 (14:15 -0700)]
Merge pull request #29277 from smithfarm/wip-40225-mimic

mimic: rgw_file: include tenant when hashing bucket names

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
5 years agoMerge pull request #29279 from smithfarm/wip-40351-mimic
Yuri Weinstein [Tue, 6 Aug 2019 21:15:06 +0000 (14:15 -0700)]
Merge pull request #29279 from smithfarm/wip-40351-mimic

mimic: rgw/OutputDataSocket: append_output(buffer::list&) says it will (but does not) discard output at data_max_backlog

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #29284 from smithfarm/wip-40354-mimic
Yuri Weinstein [Tue, 6 Aug 2019 21:14:44 +0000 (14:14 -0700)]
Merge pull request #29284 from smithfarm/wip-40354-mimic

mimic: rgw: multisite: RGWListBucketIndexesCR for data full sync needs pagination

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #29285 from smithfarm/wip-40357-mimic
Yuri Weinstein [Tue, 6 Aug 2019 21:14:21 +0000 (14:14 -0700)]
Merge pull request #29285 from smithfarm/wip-40357-mimic

mimic: rgw: provide admin-friendly reshard status output

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #29288 from smithfarm/wip-40360-mimic
Yuri Weinstein [Tue, 6 Aug 2019 21:13:54 +0000 (14:13 -0700)]
Merge pull request #29288 from smithfarm/wip-40360-mimic

mimic: rgw: set null version object issues

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agomimic: ceph-volume broken assertion errors after pytest changes 29417/head
Jan Fajerski [Tue, 6 Aug 2019 13:34:12 +0000 (15:34 +0200)]
mimic: ceph-volume broken assertion errors after pytest changes

This fixes a backported test that was not fixed on a backport. Original
fix: https://github.com/ceph/ceph/pull/28948. Backport:
https://github.com/ceph/ceph/pull/29243.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
5 years agoMerge pull request #29463 from jan--f/wip-41056-mimic
Alfredo Deza [Mon, 5 Aug 2019 20:17:53 +0000 (16:17 -0400)]
Merge pull request #29463 from jan--f/wip-41056-mimic

mimic: ceph-volume does not recognize wal/db partitions created by ceph-disk

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume: fall back to PARTTYPE if PARTLABEL is empty 29463/head
Jan Fajerski [Tue, 30 Jul 2019 15:32:06 +0000 (17:32 +0200)]
ceph-volume: fall back to PARTTYPE if PARTLABEL is empty

In some cases ceph-disk does not populate PARTLABEL for wal and db
partitions. This commit adds the assumption that the empty string is a
valid label and falls back to identifying those disks by PARTTYPE.

Fixes: https://tracker.ceph.com/issues/40917
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 82d2ae7ffdf62ca93c7848fb62e506db4cda893c)

5 years agoceph-volume: adjust tests for empty PARTLABEL fields
Jan Fajerski [Tue, 30 Jul 2019 15:29:09 +0000 (17:29 +0200)]
ceph-volume: adjust tests for empty PARTLABEL fields

It seems like ceph-disk sometimes creates OSDs where the wal and db
partitions have no PARTLABEL set. This commit alters tests so that the
empty string is considered a valid label.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit bb29110c3c6e081568231131c70522dee6d12bb5)

5 years agoceph-volume: refactor ceph-disk_member unittests
Jan Fajerski [Tue, 30 Jul 2019 11:29:22 +0000 (13:29 +0200)]
ceph-volume: refactor ceph-disk_member unittests

This adds a parameterized fixture as input to the tests. Should make
writing new tests easier since only fixture parameters need to be added.
In the future we should be able to reuse the fixtures to build other
fixtures and reuse the parametrization.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit f42287c3416ad09c112e64756f5051c5c13aa344)

5 years agoceph-volume tests: ensure that better heuristics exist for objectstore detection
Alfredo Deza [Mon, 29 Jul 2019 19:50:14 +0000 (15:50 -0400)]
ceph-volume tests: ensure that better heuristics exist for objectstore detection

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e48ad3ab26fdab72755bc933781db3bfc6001a30)

5 years agoceph-volume simple.activate better detect bluestore/filestore when type is not found
Alfredo Deza [Mon, 29 Jul 2019 19:46:48 +0000 (15:46 -0400)]
ceph-volume simple.activate better detect bluestore/filestore when type is not found

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b6a449e2e3241aaa2c0eb8fa9f74b5d939c97852)

5 years agoMerge pull request #29029 from liewegas/wip-mgr-crash-py3-mimic
Yuri Weinstein [Tue, 30 Jul 2019 16:11:43 +0000 (09:11 -0700)]
Merge pull request #29029 from liewegas/wip-mgr-crash-py3-mimic

mimic: mgr/crash: fix python3 invalid syntax problems

Reviewed-by: Nathan Cutler <ncutler@suse.com>
5 years agoMerge pull request #29150 from pdvian/wip-40269-mimic
Yuri Weinstein [Tue, 30 Jul 2019 16:11:12 +0000 (09:11 -0700)]
Merge pull request #29150 from pdvian/wip-40269-mimic

mimic: build/ops: make "patch" build dependency explicit

Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoMerge pull request #29197 from smithfarm/wip-38963-mimic
Yuri Weinstein [Tue, 30 Jul 2019 16:10:43 +0000 (09:10 -0700)]
Merge pull request #29197 from smithfarm/wip-38963-mimic

mimic: mgr: DaemonServer::handle_conf_change - broken locking

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>
5 years agoMerge pull request #29201 from smithfarm/wip-40268-mimic
Yuri Weinstein [Tue, 30 Jul 2019 16:09:58 +0000 (09:09 -0700)]
Merge pull request #29201 from smithfarm/wip-40268-mimic

mimic: common: data race in OutputDataSocket

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #29217 from smithfarm/wip-39692-mimic
Yuri Weinstein [Tue, 30 Jul 2019 16:09:25 +0000 (09:09 -0700)]
Merge pull request #29217 from smithfarm/wip-39692-mimic

mimic: bluestore: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
5 years agoMerge pull request #29243 from jan--f/mimic-c-v-extend-batch
Jan Fajerski [Mon, 29 Jul 2019 15:57:41 +0000 (17:57 +0200)]
Merge pull request #29243 from jan--f/mimic-c-v-extend-batch

mimic: c-v extend batch

5 years agoceph-volume: create two NVME drives for testing 29243/head
Andrew Schoen [Tue, 27 Nov 2018 23:34:10 +0000 (17:34 -0600)]
ceph-volume: create two NVME drives for testing

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: bz#1650306
(cherry picked from commit 893b61b256bcff142dbd90b777b2c9df9e032e9d)

5 years agoceph-volume: set block_db_size to 9GB in centos7 mixed type test
Andrew Schoen [Wed, 28 Nov 2018 16:10:24 +0000 (10:10 -0600)]
ceph-volume: set block_db_size to 9GB in centos7 mixed type test

This is to enable testing of the --block-db-size flag

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit eab962c6ef68c588674bdac6fc4a86c84714f300)

5 years agoceph-volume: use two NVME drives in centos7 mixed_type tests
Andrew Schoen [Wed, 28 Nov 2018 16:05:45 +0000 (10:05 -0600)]
ceph-volume: use two NVME drives in centos7 mixed_type tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 934c32acb318aac9aff3c800cdbf99bbdee04d06)

5 years agoMerge pull request #29350 from jan--f/wip-40977-mimic
Alfredo Deza [Fri, 26 Jul 2019 18:33:21 +0000 (14:33 -0400)]
Merge pull request #29350 from jan--f/wip-40977-mimic

mimic: ceph-volume: missing string substitution when reporting mounts

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume:util: Use proper param substition 29350/head
Shyukri Shyukriev [Fri, 7 Jun 2019 12:09:19 +0000 (12:09 +0000)]
ceph-volume:util: Use proper param substition

Fixes: https://tracker.ceph.com/issues/25030
Signed-off-by: Shyukri Shyukriev <shshyukriev@suse.com>
(cherry picked from commit 654660cd89fe9d93087d390805651c860bf4e862)

5 years agoMerge pull request #26991 from ceph/backport-mimic-26957
Alfredo Deza [Thu, 25 Jul 2019 15:54:37 +0000 (11:54 -0400)]
Merge pull request #26991 from ceph/backport-mimic-26957

mimic: ceph-volume: look for rotational data in lsblk

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume: look for rotational data in lsblk 26991/head
Andrew Schoen [Thu, 14 Mar 2019 20:14:40 +0000 (15:14 -0500)]
ceph-volume: look for rotational data in lsblk

Also ask lsblk if a device is rotational if no information
is found in /sys/block, default to True if nothing is found.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit de748f9aef894b247622ee5dc49c1fbc31097ad0)

5 years agoMerge pull request #28918 from smithfarm/wip-40222-mimic
Yuri Weinstein [Thu, 25 Jul 2019 14:37:25 +0000 (07:37 -0700)]
Merge pull request #28918 from smithfarm/wip-40222-mimic

mimic: mds: reset heartbeat during long-running loops in recovery

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>
5 years agoMerge pull request #29187 from xiaoxichen/wip-40875-mimic
Yuri Weinstein [Thu, 25 Jul 2019 14:36:55 +0000 (07:36 -0700)]
Merge pull request #29187 from xiaoxichen/wip-40875-mimic

mimic: cephfs: client: bump ll_ref from int32 to uint64_t

Reviewed-by: Venky Shankar <vshankar@redhat.com>
5 years agoMerge pull request #29200 from smithfarm/wip-39685-mimic
Yuri Weinstein [Thu, 25 Jul 2019 14:36:25 +0000 (07:36 -0700)]
Merge pull request #29200 from smithfarm/wip-39685-mimic

mimic: cephfs: ceph-fuse: client hang because its bad session PipeConnection to mds

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>
5 years agoMerge pull request #29208 from smithfarm/wip-40317-mimic
Yuri Weinstein [Thu, 25 Jul 2019 14:35:41 +0000 (07:35 -0700)]
Merge pull request #29208 from smithfarm/wip-40317-mimic

mimic: tests: test_volume_client: print python version correctly

Reviewed-by: Venky Shankar <vshankar@redhat.com>
5 years agoMerge pull request #29210 from smithfarm/wip-38099-mimic
Yuri Weinstein [Thu, 25 Jul 2019 14:35:08 +0000 (07:35 -0700)]
Merge pull request #29210 from smithfarm/wip-38099-mimic

mimic: mds: remove cache drop admin socket command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
5 years agoMerge pull request #29211 from smithfarm/wip-38687-mimic
Yuri Weinstein [Thu, 25 Jul 2019 14:34:17 +0000 (07:34 -0700)]
Merge pull request #29211 from smithfarm/wip-38687-mimic

mimic: kcephfs TestClientLimits.test_client_pin fails with client caps fell below min

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>
6 years agoMerge pull request #27028 from mrandybu/mimic-alt
Yuri Weinstein [Wed, 24 Jul 2019 23:06:01 +0000 (16:06 -0700)]
Merge pull request #27028 from mrandybu/mimic-alt

mimic: ceph_detect_init: Add support for ALT Linux

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoMerge pull request #28210 from votdev/fix_rgw_issue
Yuri Weinstein [Wed, 24 Jul 2019 23:04:23 +0000 (16:04 -0700)]
Merge pull request #28210 from votdev/fix_rgw_issue

mimic: mgr/dashboard: Fix various RGW issues

Reviewed-by:  Tiago Melo <tmelo@suse.com>

6 years agoMerge pull request #28911 from bk201/wip-40660-mimic
Yuri Weinstein [Wed, 24 Jul 2019 23:03:24 +0000 (16:03 -0700)]
Merge pull request #28911 from bk201/wip-40660-mimic

mimic: mgr/dashboard: cephfs multimds graphs stack together

Reviewed-by: Ricardo Dias <rdias@suse.com>
6 years agoMerge pull request #29024 from ifed01/wip-ifed-fix-len0-mimic
Yuri Weinstein [Wed, 24 Jul 2019 23:02:19 +0000 (16:02 -0700)]
Merge pull request #29024 from ifed01/wip-ifed-fix-len0-mimic

mimic: os/bluestore: avoid length overflow in extents returned by Stupid Allocator

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
6 years agoMerge pull request #29174 from neha-ojha/wip-40583-mimic
Yuri Weinstein [Wed, 24 Jul 2019 23:00:55 +0000 (16:00 -0700)]
Merge pull request #29174 from neha-ojha/wip-40583-mimic

mimic: common/options.cc: Lower the default value of osd_deep_scrub_large_omap_object_key_threshold

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #29189 from badone/wip-mimic-lazy-omap-stats-backport-tracker...
Yuri Weinstein [Wed, 24 Jul 2019 22:59:50 +0000 (15:59 -0700)]
Merge pull request #29189 from badone/wip-mimic-lazy-omap-stats-backport-tracker-38552

mimic: core: lazy omap stat collection

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #29196 from smithfarm/wip-37749-mimic
Yuri Weinstein [Wed, 24 Jul 2019 22:58:23 +0000 (15:58 -0700)]
Merge pull request #29196 from smithfarm/wip-37749-mimic

mimic: tools: Add clear-data-digest command to objectstore tool

Reviewed-by: David Zafman <dzafman@redhat.com>
6 years agotest/rgw: test truncated reads in RGWHTTPManager 29273/head
Casey Bodley [Mon, 3 Jun 2019 14:59:00 +0000 (10:59 -0400)]
test/rgw: test truncated reads in RGWHTTPManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c669cf293f02b3c68a4c539d7accf7ddf35cb1ee)

Conflicts:
    src/test/rgw/test_http_manager.cc
- RGWHTTP::process() only takes one argument in mimic

6 years agodoc: fixed --read-only argument value in multisite doc 29289/head
Chenjiong Deng [Wed, 19 Jun 2019 10:17:43 +0000 (18:17 +0800)]
doc: fixed --read-only argument value in multisite doc

Fixes: https://tracker.ceph.com/issues/40458
Signed-off-by: Chenjiong Deng <dengchenjiong@umcloud.com>
(cherry picked from commit c65832c1ff03e802f98cb7f4bda620e1e515264f)

6 years agorgw: set null version object acl issues 29288/head
Tianshan Qu [Sun, 11 Nov 2018 11:56:51 +0000 (19:56 +0800)]
rgw: set null version object acl issues

1.set null version object acl will create empty index
RGWRados::set_attrs did not clear instance, so index prepare, complete got instance=null,
which lead to empty index 1000_<obj>_i_null.
there is no harm to create empty index, but listomapkeys to find that key.

2.if object is exist with versioned key, we can set none exists null version object
order:
1) enable bucket version
2) put obj
3) disable bucket version
4) set versoned_id=null acl will succeed which should not

Fixes: http://tracker.ceph.com/issues/36763
Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit ca6e5c89e8d6da498da15f0eea4bcd4d271b44e9)

6 years agorgw: provide admin friendly reshard status output 29285/head
Mark Kogan [Sun, 12 May 2019 13:45:30 +0000 (09:45 -0400)]
rgw: provide admin friendly reshard status output

Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit ba4167c2f30fbb868482d48aaada6388f58de920)

6 years agorgw: update the "radosgw-admin reshard status"
Mark Kogan [Tue, 11 Dec 2018 17:43:32 +0000 (12:43 -0500)]
rgw: update the "radosgw-admin reshard status"
command documentation with expected output examples

Fixes: http://tracker.ceph.com/issues/37615
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit 8565ec0db69223ae71766ba000eecc0d2f7d14c2)

6 years agorgw/multisite:RGWListBucketIndexesCR for data full sync pagination 29284/head
Shilpa Jagannath [Thu, 16 May 2019 09:15:46 +0000 (14:45 +0530)]
rgw/multisite:RGWListBucketIndexesCR for data full sync pagination

RGWListBucketIndexesCR uses the 'marker' from remote zone's RGWOp_Metadata_List
admin api to list all keys for data sync.

Fixes http://tracker.ceph.com/issues/39551

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit 708ba6dcc90525c8c4316760e01ee9b1e6da21b7)

Conflicts:
src/rgw/rgw_data_sync.cc
- trivial

6 years agoMerge pull request #29016 from ivancich/mimic-abort-multipart-fix
Yuri Weinstein [Wed, 24 Jul 2019 15:26:33 +0000 (08:26 -0700)]
Merge pull request #29016 from ivancich/mimic-abort-multipart-fix

mimic: rgw: abort multipart fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #29021 from smithfarm/wip-39074-mimic
Yuri Weinstein [Wed, 24 Jul 2019 15:26:07 +0000 (08:26 -0700)]
Merge pull request #29021 from smithfarm/wip-39074-mimic

mimic: rgw: multisite: data sync loops back to the start of the datalog after reaching the end

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #29139 from ivancich/mimic-fix-resharding-race
Yuri Weinstein [Wed, 24 Jul 2019 15:25:38 +0000 (08:25 -0700)]
Merge pull request #29139 from ivancich/mimic-fix-resharding-race

mimic: rgw: fix race b/w bucket reshard and ops waiting on reshard completion

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #29017 from smithfarm/wip-38080-mimic
Yuri Weinstein [Wed, 24 Jul 2019 15:22:59 +0000 (08:22 -0700)]
Merge pull request #29017 from smithfarm/wip-38080-mimic

mimic: multisite: overwrites in versioning-suspended buckets fail to sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agorgw/OutputDataSocket: actually discard data on full buffer 29279/head
Matt Benjamin [Wed, 5 Jun 2019 17:25:32 +0000 (13:25 -0400)]
rgw/OutputDataSocket: actually discard data on full buffer

A dout message in OutputDataSocket::append_output() states that
data will be dropped when appending would cause data_max_backlog
to be exceeded--but the method appends it anyway.

Log output discards at level 0, as messages will be lost.  Suppress
repeated warnings mod 100.  Switch to vector.

Fixes: http://tracker.ceph.com/issues/40178
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit c806b825dae649829de8847d36cb21ffd2bbee8e)

Conflicts:
src/common/OutputDataSocket.cc
src/common/OutputDataSocket.h

6 years agorgw_file: pretty-print fh_key 29277/head
Matt Benjamin [Wed, 5 Jun 2019 14:40:08 +0000 (10:40 -0400)]
rgw_file: pretty-print fh_key

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 84a2c22108a3d15b836680894ba750b637e49cd2)

6 years agorgw_file: include tenant in hashes of object
Matt Benjamin [Wed, 5 Jun 2019 11:25:35 +0000 (07:25 -0400)]
rgw_file: include tenant in hashes of object

Because bucket names are taken as object names in the top
of an export.  Make hashing by tenant general to avoid disjoint
hashing of bucket.

Fixes: http://tracker.ceph.com/issues/40118
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 8e0fd5fbfa7c770f6b668e79b772179946027bce)

6 years agorgw_file: include tenant when hashing bucket names
Matt Benjamin [Wed, 15 May 2019 19:53:32 +0000 (15:53 -0400)]
rgw_file: include tenant when hashing bucket names

Prevent identical paths from distinct tenants from colliding in
RGW NFS handle cache.

Fixes: http://tracker.ceph.com/issues/40118
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit b800a9de83dff23a150ed7d236cb61c8b7d971ae)

Conflicts:
src/rgw/rgw_file.cc
- ceph_assert

6 years agorgw: fix miss handle curl error return
Tianshan Qu [Fri, 31 May 2019 10:37:44 +0000 (18:37 +0800)]
rgw: fix miss handle curl error return

if meet tcp packet loss, curl return will be result=18(CURLE_PARTIAL_FILE)
and http_status=200, so sync will continue and cause content miss match.

Fixes: https://tracker.ceph.com/issues/39992
Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit 9b7025857026c64cf076e82d762556ac2bcd2625)

6 years agoceph-volume: add explicit mixed-type functional tests
Jan Fajerski [Wed, 16 Jan 2019 12:48:25 +0000 (13:48 +0100)]
ceph-volume: add explicit mixed-type functional tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit bea35c63c83fc2ab87fc983774ed8181cbe74e5b)

6 years agorgw: fetch_remote_obj() compares expected object size
Casey Bodley [Wed, 29 May 2019 21:11:13 +0000 (17:11 -0400)]
rgw: fetch_remote_obj() compares expected object size

if the size of object data received doesn't match the value returned in
the Rgwx-Object-Size header, fail with -EIO

Fixes: https://tracker.ceph.com/issues/39992
Signed-off-by: Xiaoxi CHEN <xiaoxchen@ebay.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c6c4c5639ea34c49329719c2725886e740acd0b3)

6 years agoradosgw-admin: fix syncs_from in 'bucket sync status' 29270/head
Casey Bodley [Fri, 24 May 2019 20:59:34 +0000 (16:59 -0400)]
radosgw-admin: fix syncs_from in 'bucket sync status'

If a zone is set up with sync_from_all=false, the bucket sync status
command will incorrectly report "not in sync_from" because it's calling
'zone.syncs_from(source.id)', where syncs_from() expects a zone name
instead of id

Fixes: http://tracker.ceph.com/issues/40022
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9da888686c3a8880976cd604bdafc4361068c985)

Conflicts:
src/rgw/rgw_zone.h
- in mimic, this code is in src/rgw/rgw_rados.h - made the change there instead

6 years agorgw: dump s3_code as the Code response element in RGWDeleteMultiObj_ObjStore_S3. 29268/head
Radoslaw Zarzynski [Tue, 13 Dec 2016 16:19:56 +0000 (17:19 +0100)]
rgw: dump s3_code as the Code response element in RGWDeleteMultiObj_ObjStore_S3.

Fixes: http://tracker.ceph.com/issues/18241
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
(cherry picked from commit 8406083fb14042cb461a3aaa84b82fd78df5168f)

6 years agoceph-volume: add tests for Device.lvm_size
Andrew Schoen [Thu, 6 Dec 2018 18:45:10 +0000 (12:45 -0600)]
ceph-volume: add tests for Device.lvm_size

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 01185fb32479f271beeb14dcf94e32b1401ad2d9)

6 years agoceph-volume: adds a lvm_size property to util.Device
Andrew Schoen [Wed, 5 Dec 2018 15:43:28 +0000 (09:43 -0600)]
ceph-volume: adds a lvm_size property to util.Device

This will give us the size lvm should report because it takes into
account the 1GB physical extent size we set.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5197c16b44fcf344862201549777dc2935fc19a3)

6 years agorgw: parse_copy_location defers url-decode 29267/head
Casey Bodley [Tue, 11 Dec 2018 21:13:58 +0000 (16:13 -0500)]
rgw: parse_copy_location defers url-decode

don't url-decode until after we search for the ?, or we'll truncate
object names that contain a url-encoded ?

Fixes: http://tracker.ceph.com/issues/27217
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 804bb853f1c7b5ce546c100d61c6fc526838c134)