]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
5 years agoosdc/Objecter: invalidate crcs on preallocated rx buffers 29248/head
Ilya Dryomov [Wed, 20 Feb 2019 21:30:29 +0000 (22:30 +0100)]
osdc/Objecter: invalidate crcs on preallocated rx buffers

Both simple and async messengers use c_str() when copying the data from
the socket into the receive buffer, going behind bufferlist's back.  If
the receive buffer is preallocated, we need to invalidate its crc cache
by hand to avoid possible data crc mismatches on the client side.

Fixes: https://tracker.ceph.com/issues/38416
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 3eeab975b604d9fc4290747f85a78d59a2452c2e)

Conflicts:
src/test/librados/io_cxx.cc
- In master, 3730d10623650ce8569be96b28cbba599a9a0db6 renamed this file from
  src/test/librados/io.cc but that commit is not being backported to luminous.
  Manually cherry-picked the test mods into src/test/librados/io.cc.

5 years agoMerge pull request #29901 from alfredodeza/luminous-rm41378
Alfredo Deza [Tue, 27 Aug 2019 11:49:26 +0000 (07:49 -0400)]
Merge pull request #29901 from alfredodeza/luminous-rm41378

luminous ceph-volume tests set the noninteractive flag for Debian

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoMerge pull request #29791 from jan--f/wip-41373-luminous
Alfredo Deza [Mon, 26 Aug 2019 15:53:49 +0000 (11:53 -0400)]
Merge pull request #29791 from jan--f/wip-41373-luminous

luminous: batch functional idempotency test fails since message is now on stderr

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume tests set the noninteractive flag for Debian, to avoid prompts in apt 29901/head
Alfredo Deza [Wed, 21 Aug 2019 18:15:32 +0000 (14:15 -0400)]
ceph-volume tests set the noninteractive flag for Debian, to avoid prompts in apt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 89231c9a60a46ba0691f787efbfe5ab35f05c679)

5 years agoMerge pull request #27551 from smithfarm/wip-osd-markdown-luminous
Gregory Farnum [Wed, 21 Aug 2019 21:08:13 +0000 (14:08 -0700)]
Merge pull request #27551 from smithfarm/wip-osd-markdown-luminous

luminous: qa/standalone/osd/osd-markdown: fix dup command disabling

5 years agoMerge pull request #27471 from dzafman/wip-38567
Gregory Farnum [Wed, 21 Aug 2019 21:00:12 +0000 (14:00 -0700)]
Merge pull request #27471 from dzafman/wip-38567

luminous: doc: osd_recovery_priority is not documented (but osd_recovery_op_priority is)

5 years agoceph-volume: fix batch functional tests, idempotent test must check stderr 29791/head
Jan Fajerski [Thu, 15 Aug 2019 11:28:42 +0000 (13:28 +0200)]
ceph-volume: fix batch functional tests, idempotent test must check stderr

Fixes: https://tracker.ceph.com/issues/41295
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 88807110f3014e7a2a39c37ae5c64bb3ab107d80)

 Conflicts:
src/ceph-volume/ceph_volume/tests/functional/batch/playbooks/test_explicit.yml
        removed

5 years agoMerge pull request #29603 from jan--f/wip-41202-luminous
Alfredo Deza [Tue, 13 Aug 2019 12:05:02 +0000 (08:05 -0400)]
Merge pull request #29603 from jan--f/wip-41202-luminous

luminous: ceph-volume prints log messages to stdout

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume: never log to stdout, use stderr instead 29603/head
Jan Fajerski [Thu, 8 Aug 2019 08:42:37 +0000 (10:42 +0200)]
ceph-volume: never log to stdout, use stderr instead

We should never print log messages to stdout, as this should be reserved
for output of ceph-volume.

Fixes: https://tracker.ceph.com/issues/41158
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit b8d6dcbe9f803c96c0af68da54f1262e9b6a9e77)

5 years agoceph-volume: terminal: encode unicode when writing to stdout
Kefu Chai [Sun, 24 Mar 2019 09:11:31 +0000 (17:11 +0800)]
ceph-volume: terminal: encode unicode when writing to stdout

python determins the encoding of stdout and stderr based on the LC_CTYPE
and PYTHONIOENCODING env variable, by default, python3's sys.stdout uses
'utf-8' as its encoding, so it will be able to write unicode string even
the stdout is not attached to a tty device. but when it comes to
python2, it will default to ascii if neither of these variabls is set.
so, if we are writing unicode using `_Write` in an environment where
LC_CTYPE and/or PYTHONIOENCODING are using non UTF-8 encoding, it chokes
by raising `UnicodeEncodeError` exception.

in this change, we add a wrapper around `_Write._writer` so it is able
to write unicode string in such a non-unicode-friendly environment.

for more info related the encoding of stdout and stderr, see
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONIOENCODING .

Signed-off-by: Alfredo Deza <adeza@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 77912c0c71874a23785d3bdd109ad61e4f4c0e28)

5 years agoMerge pull request #27637 from xiexingguo/wip-luminous-pr-21646
Xie Xingguo [Mon, 12 Aug 2019 00:32:46 +0000 (08:32 +0800)]
Merge pull request #27637 from xiexingguo/wip-luminous-pr-21646

luminous: qa/suites/rados/verify/tasks/rados_api_tests: whitelist OBJECT_MISPLACED

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #29539 from neha-ojha/wip-40704-luminous
Neha Ojha [Thu, 8 Aug 2019 17:55:37 +0000 (10:55 -0700)]
Merge pull request #29539 from neha-ojha/wip-40704-luminous

luminous: bluestore: Inspect allocations

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
5 years agoos/bluestore/BlueStore.cc: _open_fm() expects a bool 29539/head
Neha Ojha [Wed, 7 Aug 2019 20:33:56 +0000 (16:33 -0400)]
os/bluestore/BlueStore.cc: _open_fm() expects a bool

bug fix for 974eae57d55a0ea847b25b486b58540287292254

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agodoc/rados/operations/health-checks: document BlueStore fragmentation and BlueFS space...
Adam Kupczyk [Wed, 7 Aug 2019 09:52:29 +0000 (11:52 +0200)]
doc/rados/operations/health-checks: document BlueStore fragmentation and BlueFS space available features

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 713f9b4d093759d3541a62c8aa43ce0a655b2d64)

Conflicts:
doc/rados/operations/health-checks.rst - trivial resolution

5 years agotest/objectstore: Allocator_test. Add test for dumping free regions and fragmentation...
Adam Kupczyk [Tue, 6 Aug 2019 10:01:52 +0000 (12:01 +0200)]
test/objectstore: Allocator_test. Add test for dumping free regions and fragmentation_score.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 355dc85372a15713ab1cdd78f44ea02db8e08a62)

5 years agoBlueStore/allocator: Add command to inspect how much BlueStore's block can go to...
Adam Kupczyk [Mon, 5 Aug 2019 13:15:07 +0000 (15:15 +0200)]
BlueStore/allocator: Add command to inspect how much BlueStore's block can go to BlueFS.
                     Moved reduced BlueFSDeviceExpander interface.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 16a9dac1678613b45fab9576b8bc4368a58c7434)

Conflicts:
src/os/bluestore/BlueFS.cc - trivial resolution
src/os/bluestore/BlueFS.h
src/os/bluestore/BlueStore.cc
src/os/bluestore/BlueStore.h

5 years agoos/bluestore: Import necessary functions to fix rebase.
Adam Kupczyk [Wed, 7 Aug 2019 14:21:29 +0000 (10:21 -0400)]
os/bluestore: Import necessary functions to fix rebase.

This commit is specific to luminous to make the backport of
https://github.com/ceph/ceph/pull/29425 simpler.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
5 years agotools/ceph-bluestore-tool: add commands free-dump and free-score
Adam Kupczyk [Wed, 31 Jul 2019 11:31:04 +0000 (13:31 +0200)]
tools/ceph-bluestore-tool: add commands free-dump and free-score

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit e7f5e53cde3b586701ff8e759a941986c17af1a8)

5 years agocommon/admin_socket: Add 'execute_command' that allows for self-reflection.
Adam Kupczyk [Tue, 30 Jul 2019 11:54:58 +0000 (13:54 +0200)]
common/admin_socket: Add 'execute_command' that allows for self-reflection.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 838320a9b22c1749f609433843542690fccdfaee)

5 years agocommon/admin_socket: Adapted old protocol to use new protocol, simplifies handle...
Adam Kupczyk [Tue, 30 Jul 2019 10:26:33 +0000 (12:26 +0200)]
common/admin_socket: Adapted old protocol to use new protocol, simplifies handle path.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 2b5da1f671a76f77107b6eaefc67e0e9c9179ca7)

5 years agoBlueStore/allocator: Add ability to dump free allocations via admin socket interface.
Adam Kupczyk [Fri, 2 Aug 2019 11:54:13 +0000 (13:54 +0200)]
BlueStore/allocator: Add ability to dump free allocations via admin socket interface.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 278055d35ae3ca2eb06f59f4712ad9ce5d843d86)

5 years agoBlueStore/allocator: Give allocator names, so they can be distinguished.
Adam Kupczyk [Fri, 2 Aug 2019 11:53:37 +0000 (13:53 +0200)]
BlueStore/allocator: Give allocator names, so they can be distinguished.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 0d64893c3ebc8a56ceac61596efc6f75ed06fbaf)

5 years agoBlueStore/allocator: Improved (but slower) method of calculating fragmentation.
Adam Kupczyk [Thu, 21 Jun 2018 11:58:45 +0000 (13:58 +0200)]
BlueStore/allocator: Improved (but slower) method of calculating fragmentation.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit c1ac6a1207cdb71a471321e42ceca517383b36c8)

5 years agoMerge pull request #29508 from jan--f/wip-41139-luminous
Jan Fajerski [Wed, 7 Aug 2019 06:10:20 +0000 (08:10 +0200)]
Merge pull request #29508 from jan--f/wip-41139-luminous

luminous: ceph-volume prints errors to stdout with --format json

5 years agoceph-volume: print most logging messages to stderr 29508/head
Jan Fajerski [Mon, 4 Mar 2019 10:28:13 +0000 (11:28 +0100)]
ceph-volume: print most logging messages to stderr

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
Fixes: http://tracker.ceph.com/issues/38548
(cherry picked from commit 1f8f1e95dee88b4bab2d6931e720ed83584d2887)

 Conflicts:
src/pybind/mgr/test_orchestrator/module.py
        removed

5 years agoMerge pull request #29462 from jan--f/wip-41057-luminous
Jan Fajerski [Tue, 6 Aug 2019 11:35:34 +0000 (13:35 +0200)]
Merge pull request #29462 from jan--f/wip-41057-luminous

luminous: ceph-volume does not recognize wal/db partitions created by ceph-disk

5 years agoMerge pull request #29415 from jan--f/wip-41020-luminous
Alfredo Deza [Mon, 5 Aug 2019 14:52:33 +0000 (10:52 -0400)]
Merge pull request #29415 from jan--f/wip-41020-luminous

luminous: simple: when 'type' file is not present activate fails

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume: fall back to PARTTYPE if PARTLABEL is empty 29462/head
Jan Fajerski [Tue, 30 Jul 2019 15:32:06 +0000 (17:32 +0200)]
ceph-volume: fall back to PARTTYPE if PARTLABEL is empty

In some cases ceph-disk does not populate PARTLABEL for wal and db
partitions. This commit adds the assumption that the empty string is a
valid label and falls back to identifying those disks by PARTTYPE.

Fixes: https://tracker.ceph.com/issues/40917
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 82d2ae7ffdf62ca93c7848fb62e506db4cda893c)

5 years agoceph-volume: adjust tests for empty PARTLABEL fields
Jan Fajerski [Tue, 30 Jul 2019 15:29:09 +0000 (17:29 +0200)]
ceph-volume: adjust tests for empty PARTLABEL fields

It seems like ceph-disk sometimes creates OSDs where the wal and db
partitions have no PARTLABEL set. This commit alters tests so that the
empty string is considered a valid label.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit bb29110c3c6e081568231131c70522dee6d12bb5)

5 years agoceph-volume: refactor ceph-disk_member unittests
Jan Fajerski [Tue, 30 Jul 2019 11:29:22 +0000 (13:29 +0200)]
ceph-volume: refactor ceph-disk_member unittests

This adds a parameterized fixture as input to the tests. Should make
writing new tests easier since only fixture parameters need to be added.
In the future we should be able to reuse the fixtures to build other
fixtures and reuse the parametrization.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit f42287c3416ad09c112e64756f5051c5c13aa344)

5 years agoMerge pull request #28647 from dvanders/zstd
Josh Durgin [Wed, 31 Jul 2019 15:57:47 +0000 (08:57 -0700)]
Merge pull request #28647 from dvanders/zstd

luminous: compressor/zstd: improvements

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
5 years agoMerge pull request #26498 from k0ste/luminous_backports
Yuri Weinstein [Wed, 31 Jul 2019 15:35:09 +0000 (08:35 -0700)]
Merge pull request #26498 from k0ste/luminous_backports

luminous: mgr/balancer: blame if upmap won't actually work

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
5 years agoMerge pull request #28666 from varshar16/wip-fs-authorize-err
Yuri Weinstein [Wed, 31 Jul 2019 15:32:03 +0000 (08:32 -0700)]
Merge pull request #28666 from varshar16/wip-fs-authorize-err

luminous: mon/AuthMonitor: don't validate fs caps on authorize

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
5 years agoMerge pull request #28972 from neha-ojha/wip-40720-luminous
Yuri Weinstein [Wed, 31 Jul 2019 15:31:08 +0000 (08:31 -0700)]
Merge pull request #28972 from neha-ojha/wip-40720-luminous

[AFTER: #28644] luminous: os/bluestore: default to bitmap allocator for bluestore/bluefs

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
5 years agoMerge pull request #29213 from smithfarm/wip-38962-luminous
Yuri Weinstein [Wed, 31 Jul 2019 15:30:17 +0000 (08:30 -0700)]
Merge pull request #29213 from smithfarm/wip-38962-luminous

luminous: mgr: DaemonServer::handle_conf_change - broken locking

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoceph-volume tests: ensure that better heuristics exist for objectstore detection 29415/head
Alfredo Deza [Mon, 29 Jul 2019 19:50:14 +0000 (15:50 -0400)]
ceph-volume tests: ensure that better heuristics exist for objectstore detection

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e48ad3ab26fdab72755bc933781db3bfc6001a30)

5 years agoceph-volume simple.activate better detect bluestore/filestore when type is not found
Alfredo Deza [Mon, 29 Jul 2019 19:46:48 +0000 (15:46 -0400)]
ceph-volume simple.activate better detect bluestore/filestore when type is not found

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b6a449e2e3241aaa2c0eb8fa9f74b5d939c97852)

5 years agoMerge pull request #29351 from jan--f/wip-40978-luminous
Alfredo Deza [Fri, 26 Jul 2019 13:35:28 +0000 (09:35 -0400)]
Merge pull request #29351 from jan--f/wip-40978-luminous

luminous: ceph-volume: missing string substitution when reporting mounts

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume:util: Use proper param substition 29351/head
Shyukri Shyukriev [Fri, 7 Jun 2019 12:09:19 +0000 (12:09 +0000)]
ceph-volume:util: Use proper param substition

Fixes: https://tracker.ceph.com/issues/25030
Signed-off-by: Shyukri Shyukriev <shshyukriev@suse.com>
(cherry picked from commit 654660cd89fe9d93087d390805651c860bf4e862)

5 years agoMerge pull request #27056 from ifed01/wip-ifed-fix-storetest-luminous
Yuri Weinstein [Thu, 25 Jul 2019 21:50:52 +0000 (14:50 -0700)]
Merge pull request #27056 from ifed01/wip-ifed-fix-storetest-luminous

luminous: test/store_test: fix/workaround for BlobReuseOnOverwriteUT and garbageCollection

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #27618 from smithfarm/wip-39332-luminous
Yuri Weinstein [Thu, 25 Jul 2019 21:49:55 +0000 (14:49 -0700)]
Merge pull request #27618 from smithfarm/wip-39332-luminous

luminous: build/ops: rpm: Build with lttng on openSUSE

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoMerge pull request #27808 from xiexingguo/wip-39424
Yuri Weinstein [Thu, 25 Jul 2019 21:48:52 +0000 (14:48 -0700)]
Merge pull request #27808 from xiexingguo/wip-39424

luminous: mgr/BaseMgrModule: drop GIL in set_config

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #28644 from ifed01/wip-ifed-fix-40080-luminous
Yuri Weinstein [Thu, 25 Jul 2019 21:47:11 +0000 (14:47 -0700)]
Merge pull request #28644 from ifed01/wip-ifed-fix-40080-luminous

luminous: os/bluestore: fix duplicate allocations in bmap allocator

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #28742 from smithfarm/wip-40548-luminous
Yuri Weinstein [Thu, 25 Jul 2019 21:46:28 +0000 (14:46 -0700)]
Merge pull request #28742 from smithfarm/wip-40548-luminous

luminous: common: Keyrings created by ceph auth get are not suitable for ceph auth import

Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoMerge pull request #28851 from jdurgin/wip-18096-luminous
Yuri Weinstein [Thu, 25 Jul 2019 21:45:31 +0000 (14:45 -0700)]
Merge pull request #28851 from jdurgin/wip-18096-luminous

luminous: osd: report omap/data/metadata usage

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #28895 from ifed01/wip-ifed-fix-no-compress-lum
Yuri Weinstein [Thu, 25 Jul 2019 21:44:55 +0000 (14:44 -0700)]
Merge pull request #28895 from ifed01/wip-ifed-fix-no-compress-lum

luminous: os/bluestore: load OSD all compression settings unconditionally.

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #28933 from dzafman/wip-backport-24178
Yuri Weinstein [Thu, 25 Jul 2019 21:43:56 +0000 (14:43 -0700)]
Merge pull request #28933 from dzafman/wip-backport-24178

luminous: qa/workunits/rados/test_envlibrados_for_rocksdb: install g++ not g++-4.7

Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoMerge pull request #28594 from xiexingguo/wip-40229
Yuri Weinstein [Wed, 24 Jul 2019 23:43:03 +0000 (16:43 -0700)]
Merge pull request #28594 from xiexingguo/wip-40229

luminous: mon, osd: parallel clean_pg_upmaps

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #28110 from smithfarm/wip-38880-luminous
Yuri Weinstein [Wed, 24 Jul 2019 22:43:31 +0000 (15:43 -0700)]
Merge pull request #28110 from smithfarm/wip-38880-luminous

luminous: ENOENT in collection_move_rename on EC backfill target

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #28112 from smithfarm/wip-39373-luminous
Yuri Weinstein [Wed, 24 Jul 2019 22:42:52 +0000 (15:42 -0700)]
Merge pull request #28112 from smithfarm/wip-39373-luminous

luminous: core: ceph tell osd.xx bench help : gives wrong help

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #28811 from trociny/wip-40463-luminous
Yuri Weinstein [Wed, 24 Jul 2019 22:41:26 +0000 (15:41 -0700)]
Merge pull request #28811 from trociny/wip-40463-luminous

luminous: journal: properly advance read offset after skipping invalid range

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
5 years agoMerge pull request #27905 from smithfarm/wip-38954-luminous
Yuri Weinstein [Wed, 24 Jul 2019 15:20:27 +0000 (08:20 -0700)]
Merge pull request #27905 from smithfarm/wip-38954-luminous

luminous: tests: krbd discard qa fixes

5 years agoMerge pull request #28149 from trociny/wip-37692-luminous
Yuri Weinstein [Wed, 24 Jul 2019 15:19:52 +0000 (08:19 -0700)]
Merge pull request #28149 from trociny/wip-37692-luminous

luminous: librbd: disable image mirroring when moving to trash

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
5 years agoMerge pull request #28158 from dillaman/wip-38508-luminous
Yuri Weinstein [Wed, 24 Jul 2019 15:19:14 +0000 (08:19 -0700)]
Merge pull request #28158 from dillaman/wip-38508-luminous

luminous: librbd: add missing shutdown states to managed lock helper

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
5 years agoMerge pull request #28773 from dillaman/wip-40574-luminous
Yuri Weinstein [Wed, 24 Jul 2019 15:18:42 +0000 (08:18 -0700)]
Merge pull request #28773 from dillaman/wip-40574-luminous

luminous: librbd: properly track in-flight flush requests

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
5 years agoMerge pull request #28329 from smithfarm/wip-40092-luminous
Nathan Cutler [Wed, 24 Jul 2019 10:36:11 +0000 (12:36 +0200)]
Merge pull request #28329 from smithfarm/wip-40092-luminous

luminous: doc: Missing Documentation for radosgw-admin reshard commands (man pages)

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #27185 from smithfarm/wip-38902-luminous
Nathan Cutler [Wed, 24 Jul 2019 10:29:40 +0000 (12:29 +0200)]
Merge pull request #27185 from smithfarm/wip-38902-luminous

luminous: doc: Minor rados related documentation fixes

Reviewed-by: David Zafman <dzafman@redhat.com>
5 years agomgr/balancer: blame if upmap won't actually work 26498/head
xie xingguo [Mon, 14 Jan 2019 06:39:18 +0000 (14:39 +0800)]
mgr/balancer: blame if upmap won't actually work

With automatic balancing on, and if mode is set to upmap,
balancer will fail silently if min_compat_client is lower than
luminous.
You can't figure out that unless you take a closer look at the
mgr log, which is super annoying..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit c7356c66b0b0bb1a8a6197f84b26e6db0d192f60)

Conflicts:
- path: qa/suites/rados/thrash/d-balancer/upmap.yaml
  comment: changeset here -> qa/suites/rados/thrash/d-require-luminous/at-mkfs-balancer-upmap.yaml
- path: src/pybind/mgr/balancer/module.py
  comment: 'set_module_option()' -> 'set_config()' for mgr config-key

5 years agoMerge pull request #29075 from ceph/backport-luminous-26397
Alfredo Deza [Tue, 23 Jul 2019 20:40:41 +0000 (16:40 -0400)]
Merge pull request #29075 from ceph/backport-luminous-26397

luminous: ceph-volume: update testing playbook 'deploy.yml'

Reviewed-by: Alfredo Deza <adeza@redhat.com>
6 years agomgr/DaemonServer: handle_conf_change - fix broken locking 29213/head
xie xingguo [Tue, 26 Mar 2019 08:08:43 +0000 (16:08 +0800)]
mgr/DaemonServer: handle_conf_change - fix broken locking

The is_locked_by_me() is only meant to be safe if you know
you hold the lock. It's meant to be used in an assertion.

Fixes: http://tracker.ceph.com/issues/38899
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a929a5a01508e661fc7f3cea52822320d39983d2)

Conflicts:
src/mgr/DaemonServer.cc
- leave untouched the mysterious "ohai" log message
- use Mutex::Locker instead of std::lock_guard

6 years agoMerge pull request #29015 from smithfarm/wip-37892-luminous
Nathan Cutler [Tue, 23 Jul 2019 11:37:08 +0000 (13:37 +0200)]
Merge pull request #29015 from smithfarm/wip-37892-luminous

luminous: doc: wrong value of usage log default in logging section

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #27991 from smithfarm/wip-39497-luminous
Yuri Weinstein [Mon, 22 Jul 2019 19:55:54 +0000 (12:55 -0700)]
Merge pull request #27991 from smithfarm/wip-39497-luminous

luminous: rgw admin: object stat command output's delete_at not readable

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #28349 from smithfarm/wip-39177-luminous
Yuri Weinstein [Mon, 22 Jul 2019 19:55:26 +0000 (12:55 -0700)]
Merge pull request #28349 from smithfarm/wip-39177-luminous

luminous: rgw: remove_olh_pending_entries() does not limit the number of xattrs to remove

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #28408 from cbodley/wip-40149
Yuri Weinstein [Mon, 22 Jul 2019 19:55:01 +0000 (12:55 -0700)]
Merge pull request #28408 from cbodley/wip-40149

luminous: rgw: fix bucket may redundantly list keys after BI_PREFIX_CHAR

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #28717 from smithfarm/wip-40506-luminous
Yuri Weinstein [Mon, 22 Jul 2019 19:54:27 +0000 (12:54 -0700)]
Merge pull request #28717 from smithfarm/wip-40506-luminous

luminous: rgw: conditionally allow builtin users with non-unique email addresses

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #28784 from chardan/wip-23223-luminous
Yuri Weinstein [Mon, 22 Jul 2019 19:54:01 +0000 (12:54 -0700)]
Merge pull request #28784 from chardan/wip-23223-luminous

luminous: rgw: gc use aio

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoceph-volume: fix deploy playbook for ansible~=2.6,<2.7 29075/head
Andrew Schoen [Tue, 16 Jul 2019 15:29:40 +0000 (10:29 -0500)]
ceph-volume: fix deploy playbook for ansible~=2.6,<2.7

When testing the lumionus and mimic branch the import_role
ansible feature does not work as expected. Variables created by one
role is not available to the next role called when using
import_role. To fix this, we need to use the roles keyword instead.

This commit is only applies to luminous and mimic and should
only be applied there.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
6 years agoceph-volume: update deploy.yml
Guillaume Abrioux [Wed, 13 Feb 2019 10:27:12 +0000 (11:27 +0100)]
ceph-volume: update deploy.yml

update deploy.yml accordingly with recent changes in ceph-ansible.

Fixes: ceph/ceph-ansible#3602
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit da13da67bc1e8ba062c61a716eea485ae462db48)

6 years agoMerge pull request #26381 from ypdai/patch-1
Gregory Farnum [Fri, 12 Jul 2019 19:02:22 +0000 (12:02 -0700)]
Merge pull request #26381 from ypdai/patch-1

Update layout.rst

6 years agodoc: rgw: fix the default value of usage log setting 29015/head
Abhishek Lekshmanan [Thu, 10 Jan 2019 10:10:41 +0000 (11:10 +0100)]
doc: rgw: fix the default value of usage log setting

Fixes: http://tracker.ceph.com/issues/37856
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit b21c96ec69ec65f359473a0f193c716925086828)

6 years agoMerge pull request #28965 from liewegas/wip-set-aio-write-max-luminous
Neha Ojha [Thu, 11 Jul 2019 18:47:22 +0000 (11:47 -0700)]
Merge pull request #28965 from liewegas/wip-set-aio-write-max-luminous

luminous: os/bluestore: >2GB bluefs writes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agorgw: "radosgw-admin objects expire" always returns ok even if the process 28784/head
Zhang Shaowen [Wed, 20 Jun 2018 03:45:45 +0000 (11:45 +0800)]
rgw: "radosgw-admin objects expire" always returns ok even if the process
fails.

Fixes: http://tracker.ceph.com/issues/24592
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>
(cherry picked from commit f6fe07e8f549f6c54e4bc81a89558f14e71d3d9f)

6 years agorgw: use vector for remove_tags in gc aio
Yehuda Sadeh [Fri, 23 Feb 2018 16:41:19 +0000 (08:41 -0800)]
rgw: use vector for remove_tags in gc aio

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 9d37caccd907a7e58a836c5bcd9fb00a3dda8216)

6 years agorgw: gc aio, replace lists with other types
Yehuda Sadeh [Fri, 23 Feb 2018 16:43:16 +0000 (08:43 -0800)]
rgw: gc aio, replace lists with other types

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 278ca0352ac357426f435d1bb0343a6a815d1cdb)

6 years agorgw: make gc concurrenct io size configurable
Yehuda Sadeh [Thu, 22 Feb 2018 22:28:29 +0000 (14:28 -0800)]
rgw: make gc concurrenct io size configurable

and another tunable for log trim size

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1cceef22cf26e1a4a0234fb2f977a5fefae82905)

6 years agorgw: trim gc index using aio
Yehuda Sadeh [Thu, 22 Feb 2018 22:21:37 +0000 (14:21 -0800)]
rgw: trim gc index using aio

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit f00ee63e981646d0dc19bcb3d1e4165ef0a8cfce)
Signed-off-by: Jesse Williamson <jesse.williamson@canonical.com>
Conflicts: rgw_gc.cc (uses a kind of aggregate initialization not available in C++11)

6 years agorgw: use a single gc io manager for all shards
Yehuda Sadeh [Thu, 22 Feb 2018 21:57:33 +0000 (13:57 -0800)]
rgw: use a single gc io manager for all shards

to allow cross shards concurrency

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit e60404cb18682dcc6dad03ee1806a601d5c8b2e0)

6 years agorgw: use aio for gc processing
Yehuda Sadeh [Thu, 22 Feb 2018 21:31:59 +0000 (13:31 -0800)]
rgw: use aio for gc processing

still need to deal with index cleanup asynchronously

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1f6dc8c9b83fd0e74f7d524178fb37e90d06bbef)

6 years agorgw-admin: support for processing all gc objects including unexpired.
Yao Zongyou [Tue, 5 Sep 2017 09:33:04 +0000 (17:33 +0800)]
rgw-admin: support for processing all gc objects including unexpired.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>
(cherry picked from commit ceccb54f191182813cae70841e87b9969c3d4aeb)

6 years agodoc: modify the wrong word "defails" to "details". 26381/head
ypdai [Tue, 12 Feb 2019 09:50:07 +0000 (17:50 +0800)]
doc: modify the wrong word "defails" to "details".

Signed-off-by: ypdai <self19900924@gmail.com>
6 years agoos/bluestore: default to bitmap allocator for bluestore/bluefs 28972/head
Igor Fedotov [Fri, 18 May 2018 19:14:59 +0000 (22:14 +0300)]
os/bluestore: default to bitmap allocator for bluestore/bluefs

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 231b7dd9c5dc1d22e93a8f81d07e772a6615f9f0)

Conflicts:
src/common/options.cc: this a dev option in luminous

6 years agoceph_test_objectstore: add very_large_write test 28965/head
Sage Weil [Tue, 16 Apr 2019 13:53:56 +0000 (08:53 -0500)]
ceph_test_objectstore: add very_large_write test

Write 2GB to verify the blockdevice aio splitting.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1974f011137976edf277833d68bdcc4ff22779b0)
- fixed get_val/set_val calls
- add_block_device doesn't take third arg in luminous

6 years agoos/bluestore: fix aio pwritev lost data problem.
kungf [Tue, 16 Apr 2019 11:21:41 +0000 (19:21 +0800)]
os/bluestore: fix aio pwritev lost data problem.

On Linux, write() (and similar system calls) will transfer at most
0x7ffff000 (2,147,479,552) bytes, it will cap data if aio pwritev
more than 0x7ffff000, so we have the split the data to more aio submit.

Signed-off-by: kungf <yang.wang@easystack.cn>
(cherry picked from commit 4d33114a40d5ae0d541c36175977ca22789a3b88)

- conflict due to fd_direct instead of choose_fd()

6 years agobluestore: print aio in batch
Sage Weil [Wed, 10 Jul 2019 15:21:12 +0000 (10:21 -0500)]
bluestore: print aio in batch

KernelDevice::aio_{submit,write,read}() are critical paths. calling
cct->_conf->subsys.should_gather() multi-times is not optimal. the
downside of this issue is that if the aio is printed, the size of
buffer in PrebufferedStreambuf could be large if the number of iov is
large, that could be a heavy load to the memory subsystem.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 67c6f768add46d92d9403ca6b7709b892d8a686d)

# Conflicts:
# src/os/bluestore/aio.cc
 - adjacent to aio_queue_t::submit()

6 years agoMerge pull request #28926 from ceph/backport-luminous-28294
Alfredo Deza [Tue, 9 Jul 2019 18:37:29 +0000 (14:37 -0400)]
Merge pull request #28926 from ceph/backport-luminous-28294

luminous: tests: pass --ssh-config to pytest to resolve hosts when connecting

Reviewed-by: Alfredo Deza <adeza@redhat.com>
6 years agoMerge pull request #27365 from ifed01/wip-ifed-fix-alloc-overflow-luminous
Yuri Weinstein [Tue, 9 Jul 2019 16:32:57 +0000 (09:32 -0700)]
Merge pull request #27365 from ifed01/wip-ifed-fix-alloc-overflow-luminous

luminous: os/bluestore: fix length overflow.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jianpeng Ma <majianpeng@gmail.com>
6 years agoMerge pull request #28640 from tchaikov/luminous-38040
Yuri Weinstein [Tue, 9 Jul 2019 16:29:09 +0000 (09:29 -0700)]
Merge pull request #28640 from tchaikov/luminous-38040

luminous: mon,osd: limit MOSDMap messages by size as well as map count

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #27459 from ifed01/wip-ifed-no-verify-fm-luminous
Yuri Weinstein [Tue, 9 Jul 2019 16:08:04 +0000 (09:08 -0700)]
Merge pull request #27459 from ifed01/wip-ifed-no-verify-fm-luminous

luminous: os/bluestore/BitmapFreelistManager: disable bluestore_debug_freelist

Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #27529 from ifed01/wip-ifed-fix-21312-luminous
Yuri Weinstein [Tue, 9 Jul 2019 16:07:24 +0000 (09:07 -0700)]
Merge pull request #27529 from ifed01/wip-ifed-fix-21312-luminous

luminous: os/bluestore: call fault_range properly prior to looking for blob to …

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
6 years agoMerge pull request #27583 from xiexingguo/wip-38242
Yuri Weinstein [Tue, 9 Jul 2019 16:06:55 +0000 (09:06 -0700)]
Merge pull request #27583 from xiexingguo/wip-38242

luminous: backport recent messenger fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
6 years agoMerge pull request #27739 from ifed01/wip-ifed-fix-39444
Yuri Weinstein [Tue, 9 Jul 2019 16:06:13 +0000 (09:06 -0700)]
Merge pull request #27739 from ifed01/wip-ifed-fix-39444

luminous: os/bluestore: fix out-of-bound access in bmap allocator.

Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #27950 from smithfarm/wip-37612-luminous
Yuri Weinstein [Tue, 9 Jul 2019 16:05:39 +0000 (09:05 -0700)]
Merge pull request #27950 from smithfarm/wip-37612-luminous

luminous: build/ops: python-cephfs should depend on python-rados

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
6 years agoMerge pull request #28030 from ifed01/wip-ifed-dump-before-nospanid-luminous
Yuri Weinstein [Tue, 9 Jul 2019 16:05:07 +0000 (09:05 -0700)]
Merge pull request #28030 from ifed01/wip-ifed-dump-before-nospanid-luminous

luminous: os/bluestore: dump before "no spanning blob id" abort

Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #28111 from smithfarm/wip-38750-luminous
Yuri Weinstein [Tue, 9 Jul 2019 16:04:39 +0000 (09:04 -0700)]
Merge pull request #28111 from smithfarm/wip-38750-luminous

luminous: core: should report EINVAL in ErasureCode::parse() if m<=0

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #28929 from ceph/backport-luminous-28866
Alfredo Deza [Mon, 8 Jul 2019 17:26:47 +0000 (13:26 -0400)]
Merge pull request #28929 from ceph/backport-luminous-28866

luminous: ceph-volume broken assertion errors after pytest changes

Reviewed-by: Alfredo Deza <adeza@redhat.com>
6 years agoMerge pull request #28927 from ceph/backport-luminous-28836
Alfredo Deza [Mon, 8 Jul 2019 17:26:20 +0000 (13:26 -0400)]
Merge pull request #28927 from ceph/backport-luminous-28836

luminous: ceph-volume tests add a sleep in tox for slow OSDs after booting

Reviewed-by: Alfredo Deza <adeza@redhat.com>
6 years agoceph-volume api.lvm catch IndexError when parsing dmmapper output 28929/head
Alfredo Deza [Wed, 3 Jul 2019 18:59:18 +0000 (14:59 -0400)]
ceph-volume api.lvm catch IndexError when parsing dmmapper output

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bfff70d07a1ccf7fb11a96dd7c63335cfc9b5e26)

6 years agoceph-volume tests update to use error.value instead of str(error)
Alfredo Deza [Wed, 3 Jul 2019 18:58:31 +0000 (14:58 -0400)]
ceph-volume tests update to use error.value instead of str(error)

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 484be7ffb1e0865ff840090a73036567a1f4f5d2)

6 years agoceph-volume tests add a sleep in tox for slow OSDs after booting 28927/head
Alfredo Deza [Tue, 2 Jul 2019 12:34:01 +0000 (08:34 -0400)]
ceph-volume tests add a sleep in tox for slow OSDs after booting

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b398f99f810d97c94af6d943aed4c11a5fd69c19)

6 years agotests: pass --ssh-config to pytest to resolve hosts when connecting 28926/head
Alfredo Deza [Wed, 29 May 2019 12:56:37 +0000 (08:56 -0400)]
tests: pass --ssh-config to pytest to resolve hosts when connecting

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ed708e520999c145c3cf8a0c51e4605f275a1771)