]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
5 years agoceph-volume: do not fail when trying to remove crypt mapper 30556/head
Guillaume Abrioux [Fri, 20 Sep 2019 12:57:58 +0000 (14:57 +0200)]
ceph-volume: do not fail when trying to remove crypt mapper

In a containerized context, at some point, need to run `simple scan` on a device
from a separate container (not the existing and running corresponding container
to that device), but this can't work because when it tries to remove the
mapper which is still in use by the corresponding running osd container,
it fails.
This can be a bit more permissive and simply throw a warning.

Closes: https://tracker.ceph.com/issues/41392
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 80d3e4ca054b4d1334c2fb99fede46ab54ca4305)

5 years agoMerge pull request #29038 from dcasier/wip-40100-luminous
Alfredo Deza [Thu, 12 Sep 2019 21:52:54 +0000 (17:52 -0400)]
Merge pull request #29038 from dcasier/wip-40100-luminous

luminous: ceph-volume: lvm.activate: Return an error if WAL/DB devices absent

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume: lvm.activate: Return an error if WAL/DB devices absent 29038/head
David Casier [Wed, 3 Jul 2019 21:24:34 +0000 (23:24 +0200)]
ceph-volume: lvm.activate: Return an error if WAL/DB devices absent

Fixes: https://tracker.ceph.com/issues/40100
Signed-off-by: David Casier <david.casier@aevoo.fr>
(cherry picked from commit de439a0a236e4752e64a0cd63b7ce1ad3278eff3)

5 years agoMerge pull request #29771 from alfredodeza/luminous-bz-1738379
Jan Fajerski [Thu, 12 Sep 2019 13:00:16 +0000 (15:00 +0200)]
Merge pull request #29771 from alfredodeza/luminous-bz-1738379

luminous ceph-volume: use the OSD identifier when reporting success

5 years agoMerge pull request #30296 from alfredodeza/luminous-rm41378-2
Alfredo Deza [Thu, 12 Sep 2019 10:55:35 +0000 (06:55 -0400)]
Merge pull request #30296 from alfredodeza/luminous-rm41378-2

luminous ceph-volume: pre-install python-apt and its variants before test runs

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
5 years agoMerge pull request #30305 from alfredodeza/luminous-rm40664
Jan Fajerski [Thu, 12 Sep 2019 10:05:25 +0000 (12:05 +0200)]
Merge pull request #30305 from alfredodeza/luminous-rm40664

luminous ceph-volume  zap always skips block.db, leaves them around

5 years agoMerge pull request #30299 from alfredodeza/luminous-rm41660
Alfredo Deza [Wed, 11 Sep 2019 19:30:36 +0000 (15:30 -0400)]
Merge pull request #30299 from alfredodeza/luminous-rm41660

luminous ceph-volume: fix stderr failure to decode/encode when redirected

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoceph-volume tests create a test file for checking unicode output 30299/head
Alfredo Deza [Mon, 9 Sep 2019 22:55:13 +0000 (18:55 -0400)]
ceph-volume tests create a test file for checking unicode output

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 79196354aca3b3d30c7de1e2ac8b5a3c06ee8330)

5 years agoceph-volume tests create a shell test for functional unicode
Alfredo Deza [Mon, 9 Sep 2019 22:54:51 +0000 (18:54 -0400)]
ceph-volume tests create a shell test for functional unicode

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 595e492783283359b613d199a75f3e7495bfb9ed)

5 years agoceph-volume tests verify new logging fallback and encodings in terminal
Alfredo Deza [Mon, 9 Sep 2019 19:20:13 +0000 (15:20 -0400)]
ceph-volume tests verify new logging fallback and encodings in terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ee18ebc078b2d380b88f3726737184f34345e2f3)

5 years agoceph-volume terminal remove unicode stream handler
Alfredo Deza [Mon, 9 Sep 2019 17:45:08 +0000 (13:45 -0400)]
ceph-volume terminal remove unicode stream handler

This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fe66e01f6a4b19bb15695537713a629a951b28ce)

5 years agoceph-volume: instantiate the new terminal logger in main()
Alfredo Deza [Mon, 9 Sep 2019 17:44:36 +0000 (13:44 -0400)]
ceph-volume: instantiate the new terminal logger in main()

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bfb8422e262dcce49c50610a3977bfe1edc2a435)

5 years agoceph-volume create a logger for the terminal
Alfredo Deza [Mon, 9 Sep 2019 17:44:17 +0000 (13:44 -0400)]
ceph-volume create a logger for the terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 95b16b516c0736839fe50f12ecbeb7cd62c67335)

5 years agoceph-volume create a new tox.ini for shell-based tests
Alfredo Deza [Mon, 9 Sep 2019 17:40:06 +0000 (13:40 -0400)]
ceph-volume create a new tox.ini for shell-based tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4612ab3ae62a3e8e1cc9e51489feab0c938296d3)

5 years agoMerge pull request #30302 from alfredodeza/luminous-ceph-volume-zap-fix
Alfredo Deza [Wed, 11 Sep 2019 13:31:13 +0000 (09:31 -0400)]
Merge pull request #30302 from alfredodeza/luminous-ceph-volume-zap-fix

luminous ceph-volume lvm.zap fix cleanup for db partitions

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
5 years agoceph-volume lvm.zap fix cleanup for db partitions 30302/head
Dominik Csapak [Tue, 28 May 2019 14:29:21 +0000 (16:29 +0200)]
ceph-volume lvm.zap fix cleanup for db partitions

this uses the correct type 'db' for db type partitions, else
a block.db parition does not get cleaned up by ceph-volume zap

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
(cherry picked from commit 140af8a7eee8d230ac9d94ba100320a20c137a12)

5 years agoMerge pull request #30331 from jan--f/wip-9e2175c-luminous
Alfredo Deza [Wed, 11 Sep 2019 12:54:34 +0000 (08:54 -0400)]
Merge pull request #30331 from jan--f/wip-9e2175c-luminous

luminous: ceph-volume: set a lvm_size property on the fakedevice fixture

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume: set a lvm_size property on the fakedevice fixture 30331/head
Andrew Schoen [Thu, 6 Dec 2018 18:37:56 +0000 (12:37 -0600)]
ceph-volume: set a lvm_size property on the fakedevice fixture

This is so tests can continue to set sys_api['size'] and the code
can retrieve that as Device.lvm_size

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 9e2175c4001434f7b0ed8ce6f723e063cf001440)

5 years agoceph-volume tests: verify that wipefs tries several times 30305/head
Alfredo Deza [Thu, 11 Jul 2019 15:56:07 +0000 (11:56 -0400)]
ceph-volume tests: verify that wipefs tries several times

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 845e62cb4ba2636325304db2c2d65a185ace25cf)

5 years agoceph-volume lvm.zap: retry wipefs several times to prevent race condition failures
Alfredo Deza [Thu, 11 Jul 2019 13:34:10 +0000 (09:34 -0400)]
ceph-volume lvm.zap: retry wipefs several times to prevent race condition failures

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b7b8b106fa485d3b8345b583fbc7f74e30cbc03e)

5 years agoceph-volume tests pre-instrall python-apt to prevent auto-install failing later 30296/head
Alfredo Deza [Tue, 3 Sep 2019 14:00:14 +0000 (10:00 -0400)]
ceph-volume tests pre-instrall python-apt to prevent auto-install failing later

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 40cac7b9510f7904c2f9dfb1294d809e04bdcf7e)

5 years agoMerge pull request #30129 from smithfarm/wip-41644-luminous
Nathan Cutler [Mon, 9 Sep 2019 22:29:13 +0000 (00:29 +0200)]
Merge pull request #30129 from smithfarm/wip-41644-luminous

luminous: install-deps.sh: Remove CR repo

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
5 years agoMerge pull request #29910 from vumrao/wip-vumrao-bluefs-log-level
Neha Ojha [Mon, 9 Sep 2019 19:12:52 +0000 (12:12 -0700)]
Merge pull request #29910 from vumrao/wip-vumrao-bluefs-log-level

luminous: os/bluestore: apply shared_alloc_size to shared device with log level change

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
5 years agoMerge pull request #29190 from badone/wip-luminous-lazy-omap-stats-backport-tracker...
Neha Ojha [Mon, 9 Sep 2019 17:36:12 +0000 (10:36 -0700)]
Merge pull request #29190 from badone/wip-luminous-lazy-omap-stats-backport-tracker-38551

luminous: core: lazy omap stat collection

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
5 years agoMerge pull request #29175 from neha-ojha/wip-40583-luminous 30169/head
Neha Ojha [Sat, 7 Sep 2019 19:17:30 +0000 (12:17 -0700)]
Merge pull request #29175 from neha-ojha/wip-40583-luminous

luminous: common/options.cc: Lower the default value of osd_deep_scrub_large_omap_object_key_threshold

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
5 years agoMerge pull request #30149 from neha-ojha/wip-40769-luminous
Neha Ojha [Fri, 6 Sep 2019 17:10:42 +0000 (10:10 -0700)]
Merge pull request #30149 from neha-ojha/wip-40769-luminous

luminous: common/options: Set concurrent bluestore rocksdb compactions to 2

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
5 years agocommon/options: Set concurrent bluestore rocksdb compactions to 2 30149/head
Mark Nelson [Fri, 12 Jul 2019 16:45:17 +0000 (12:45 -0400)]
common/options: Set concurrent bluestore rocksdb compactions to 2

Signed-off-by: Mark Nelson <mnelson@redhat.com>
(cherry picked from commit 26dc5189e73a1f5a22f89ee67eef5430ef660b5d)

5 years agoinstall-deps.sh: Remove CR repo 30129/head
Brad Hubbard [Thu, 22 Nov 2018 00:07:22 +0000 (10:07 +1000)]
install-deps.sh: Remove CR repo

Remove the continuous release repos for CentOS and Virtuozzo 7 as they
should no longer be needed since http://tracker.ceph.com/issues/13997 is
no longer relevant and the newer versions of selinux packages pulled in
by the build system are causing problems for systems without CR repos
enabled.

Fixes: https://tracker.ceph.com/issues/37335
Signed-off-by: Alfredo Deza <adeza@redhat.com>
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit f39d9b7d1f4f2b63233802211a9ed0e641446ca1)

Conflicts:
install-deps.sh

5 years agoMerge pull request #30094 from jan--f/wip-41613-luminous
Alfredo Deza [Tue, 3 Sep 2019 17:51:42 +0000 (13:51 -0400)]
Merge pull request #30094 from jan--f/wip-41613-luminous

luminous: ceph-volume: lvm list is O(n^2)

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume: test number of times LVs list was created 30094/head
Rishabh Dave [Mon, 19 Aug 2019 13:33:21 +0000 (19:03 +0530)]
ceph-volume: test number of times LVs list was created

Add a test that cofirm that the LVs list was created only once when
calling devices.lvm.listing.list()

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 20d6264538dab444c0ef13f40c6c21ec3d2057f8)

5 years agoceph-volume: reuse list of LVs
Rishabh Dave [Wed, 14 Aug 2019 15:57:38 +0000 (21:27 +0530)]
ceph-volume: reuse list of LVs

Avoid creating of list of LVs in different methods of the same module
and resue them.

Fixes: https://tracker.ceph.com/issues/37490
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit ab3a58e78124a984d162b2f63992d01c5e59e2a7)

5 years agoceph-volume devices.lvm zap use the identifier to report success 29771/head
Alfredo Deza [Tue, 20 Aug 2019 12:04:56 +0000 (08:04 -0400)]
ceph-volume devices.lvm zap use the identifier to report success

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d0d5750c1ba508152c1d5111f2003a039a27145f)

5 years agoceph-volume tests check success message when zapping
Alfredo Deza [Tue, 20 Aug 2019 12:04:26 +0000 (08:04 -0400)]
ceph-volume tests check success message when zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 88c7429a60fdab6222d14132bea93b9bf773c720)

5 years agoMerge pull request #29901 from alfredodeza/luminous-rm41378
Alfredo Deza [Tue, 27 Aug 2019 11:49:26 +0000 (07:49 -0400)]
Merge pull request #29901 from alfredodeza/luminous-rm41378

luminous ceph-volume tests set the noninteractive flag for Debian

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoos/bluestore/BlueFS: Move bluefs alloc size initialization log message to log level 1 29910/head
Vikhyat Umrao [Thu, 22 Aug 2019 16:45:40 +0000 (09:45 -0700)]
os/bluestore/BlueFS: Move bluefs alloc size initialization log message to log level 1
Fixes: https://tracker.ceph.com/issues/41399
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
(cherry picked from commit 96d48af3252f0ec2c1864383ad5eee993ecfeb4f)

5 years agoMerge pull request #29791 from jan--f/wip-41373-luminous
Alfredo Deza [Mon, 26 Aug 2019 15:53:49 +0000 (11:53 -0400)]
Merge pull request #29791 from jan--f/wip-41373-luminous

luminous: batch functional idempotency test fails since message is now on stderr

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume tests set the noninteractive flag for Debian, to avoid prompts in apt 29901/head
Alfredo Deza [Wed, 21 Aug 2019 18:15:32 +0000 (14:15 -0400)]
ceph-volume tests set the noninteractive flag for Debian, to avoid prompts in apt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 89231c9a60a46ba0691f787efbfe5ab35f05c679)

5 years agoos/bluestore: record gifts < expected size in superblock 29860/head
Josh Durgin [Sat, 24 Aug 2019 02:52:52 +0000 (22:52 -0400)]
os/bluestore: record gifts < expected size in superblock

Returning 0 from _balance_bluefs_freespace() skips recording
allocations in the superblock, so we fail the consistency check on
startup. The elseif branch handles this case already, so just remove
it from the first branch. This is luminous/mimic specific, since
bluefs extents are not recorded in the superblock in later releases.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
5 years agoMerge pull request #27551 from smithfarm/wip-osd-markdown-luminous
Gregory Farnum [Wed, 21 Aug 2019 21:08:13 +0000 (14:08 -0700)]
Merge pull request #27551 from smithfarm/wip-osd-markdown-luminous

luminous: qa/standalone/osd/osd-markdown: fix dup command disabling

5 years agoMerge pull request #27471 from dzafman/wip-38567
Gregory Farnum [Wed, 21 Aug 2019 21:00:12 +0000 (14:00 -0700)]
Merge pull request #27471 from dzafman/wip-38567

luminous: doc: osd_recovery_priority is not documented (but osd_recovery_op_priority is)

5 years agoceph-volume: fix batch functional tests, idempotent test must check stderr 29791/head
Jan Fajerski [Thu, 15 Aug 2019 11:28:42 +0000 (13:28 +0200)]
ceph-volume: fix batch functional tests, idempotent test must check stderr

Fixes: https://tracker.ceph.com/issues/41295
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 88807110f3014e7a2a39c37ae5c64bb3ab107d80)

 Conflicts:
src/ceph-volume/ceph_volume/tests/functional/batch/playbooks/test_explicit.yml
        removed

5 years agoos/bluestore: try to gift large extents, then fall back to small extents 29740/head
Sage Weil [Mon, 19 Aug 2019 15:09:36 +0000 (10:09 -0500)]
os/bluestore: try to gift large extents, then fall back to small extents

First try to gift using the larger alloc_size (normally bluefs_alloc_size,
but here max(bluefs_alloc_size,bluefs_shared_alloc_size) just in case the
settings are weird.  If that fails, then try the shared_alloc_size.  If
that fails, fail and complain as before, with an more accurate error
message.

Signed-off-by: Sage Weil <sage@redhat.com>
5 years agoos/bluestore: gift freespace to bluefs based on shared_alloc_size
Sage Weil [Mon, 19 Aug 2019 13:47:40 +0000 (08:47 -0500)]
os/bluestore: gift freespace to bluefs based on shared_alloc_size

This does not appear in master because this code was removed with the
allocator fallback behavior, which was not backported.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 98c04c48c318ea15fdcbb331f879d9a962243559)

5 years agoos/bluestore/BlueFS: apply shared_alloc_size to shared device
Sage Weil [Mon, 19 Aug 2019 13:46:09 +0000 (08:46 -0500)]
os/bluestore/BlueFS: apply shared_alloc_size to shared device

Keep an alloc_size vector so that we have this value handy at all times.
Allow bluestore to fetch this value directly instead of looking at the
bluefs_* config options since this encapsulates things a bit better, and
also isn't vulnerable to the config setting changing at runtime.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e8b5a458c306dfe166b09239fc9f57c04fa47ead)

# Conflicts:
# src/os/bluestore/BlueFS.cc
- missing assert(got != 0) in luminous
- ROUND_UP_TO vs round_up_to
- missing fallback allocations
- no BDEV_NEW*
# src/os/bluestore/BlueFS.h
- adjacent declarations
# src/os/bluestore/BlueStore.cc
- P2ROUNDUP vs p2roundup

5 years agoos/bluestore/BlueFS: add bluefs_shared_alloc_size
Sage Weil [Wed, 7 Aug 2019 15:44:38 +0000 (10:44 -0500)]
os/bluestore/BlueFS: add bluefs_shared_alloc_size

Add a separate config option that controls the alloc_size for the shared
device (BDEV_SLOW).

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c69c9539f628c61a1cf4cae6aadd39cb48a8b022)

- no Option::TYPE_SIZE

5 years agoos/bluestore: cleanup around allocator calls
Sage Weil [Fri, 16 Aug 2019 21:44:57 +0000 (16:44 -0500)]
os/bluestore: cleanup around allocator calls

Both stupid and bitmap allocator returs -ENOSPC if they're
unable to allocate any space. Existing callers aren't always
respect this - hence doing some cleanup.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit a1246da5705e2a699656c2561cebc5ed29628c90)

- allocator fallback not there
- allocator fallback not there

5 years agoMerge pull request #29603 from jan--f/wip-41202-luminous
Alfredo Deza [Tue, 13 Aug 2019 12:05:02 +0000 (08:05 -0400)]
Merge pull request #29603 from jan--f/wip-41202-luminous

luminous: ceph-volume prints log messages to stdout

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume: never log to stdout, use stderr instead 29603/head
Jan Fajerski [Thu, 8 Aug 2019 08:42:37 +0000 (10:42 +0200)]
ceph-volume: never log to stdout, use stderr instead

We should never print log messages to stdout, as this should be reserved
for output of ceph-volume.

Fixes: https://tracker.ceph.com/issues/41158
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit b8d6dcbe9f803c96c0af68da54f1262e9b6a9e77)

5 years agoceph-volume: terminal: encode unicode when writing to stdout
Kefu Chai [Sun, 24 Mar 2019 09:11:31 +0000 (17:11 +0800)]
ceph-volume: terminal: encode unicode when writing to stdout

python determins the encoding of stdout and stderr based on the LC_CTYPE
and PYTHONIOENCODING env variable, by default, python3's sys.stdout uses
'utf-8' as its encoding, so it will be able to write unicode string even
the stdout is not attached to a tty device. but when it comes to
python2, it will default to ascii if neither of these variabls is set.
so, if we are writing unicode using `_Write` in an environment where
LC_CTYPE and/or PYTHONIOENCODING are using non UTF-8 encoding, it chokes
by raising `UnicodeEncodeError` exception.

in this change, we add a wrapper around `_Write._writer` so it is able
to write unicode string in such a non-unicode-friendly environment.

for more info related the encoding of stdout and stderr, see
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONIOENCODING .

Signed-off-by: Alfredo Deza <adeza@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 77912c0c71874a23785d3bdd109ad61e4f4c0e28)

5 years agoluminous: tests: Convert to boost::regex and add blocking scrub 29190/head
Brad Hubbard [Tue, 23 Jul 2019 07:42:54 +0000 (17:42 +1000)]
luminous: tests: Convert to boost::regex and add blocking scrub

Convert lazy_omap_stats_test to boost::regex since std::regex is not
fully implemented in the C++ libs used to compile luminous. Also add the
ability to block until scrub completes as that is not available in
luminous.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
5 years agoMerge pull request #27637 from xiexingguo/wip-luminous-pr-21646
Xie Xingguo [Mon, 12 Aug 2019 00:32:46 +0000 (08:32 +0800)]
Merge pull request #27637 from xiexingguo/wip-luminous-pr-21646

luminous: qa/suites/rados/verify/tasks/rados_api_tests: whitelist OBJECT_MISPLACED

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #29539 from neha-ojha/wip-40704-luminous
Neha Ojha [Thu, 8 Aug 2019 17:55:37 +0000 (10:55 -0700)]
Merge pull request #29539 from neha-ojha/wip-40704-luminous

luminous: bluestore: Inspect allocations

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
5 years agoos/bluestore/BlueStore.cc: _open_fm() expects a bool 29539/head
Neha Ojha [Wed, 7 Aug 2019 20:33:56 +0000 (16:33 -0400)]
os/bluestore/BlueStore.cc: _open_fm() expects a bool

bug fix for 974eae57d55a0ea847b25b486b58540287292254

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agodoc/rados/operations/health-checks: document BlueStore fragmentation and BlueFS space...
Adam Kupczyk [Wed, 7 Aug 2019 09:52:29 +0000 (11:52 +0200)]
doc/rados/operations/health-checks: document BlueStore fragmentation and BlueFS space available features

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 713f9b4d093759d3541a62c8aa43ce0a655b2d64)

Conflicts:
doc/rados/operations/health-checks.rst - trivial resolution

5 years agotest/objectstore: Allocator_test. Add test for dumping free regions and fragmentation...
Adam Kupczyk [Tue, 6 Aug 2019 10:01:52 +0000 (12:01 +0200)]
test/objectstore: Allocator_test. Add test for dumping free regions and fragmentation_score.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 355dc85372a15713ab1cdd78f44ea02db8e08a62)

5 years agoBlueStore/allocator: Add command to inspect how much BlueStore's block can go to...
Adam Kupczyk [Mon, 5 Aug 2019 13:15:07 +0000 (15:15 +0200)]
BlueStore/allocator: Add command to inspect how much BlueStore's block can go to BlueFS.
                     Moved reduced BlueFSDeviceExpander interface.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 16a9dac1678613b45fab9576b8bc4368a58c7434)

Conflicts:
src/os/bluestore/BlueFS.cc - trivial resolution
src/os/bluestore/BlueFS.h
src/os/bluestore/BlueStore.cc
src/os/bluestore/BlueStore.h

5 years agoos/bluestore: Import necessary functions to fix rebase.
Adam Kupczyk [Wed, 7 Aug 2019 14:21:29 +0000 (10:21 -0400)]
os/bluestore: Import necessary functions to fix rebase.

This commit is specific to luminous to make the backport of
https://github.com/ceph/ceph/pull/29425 simpler.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
5 years agotools/ceph-bluestore-tool: add commands free-dump and free-score
Adam Kupczyk [Wed, 31 Jul 2019 11:31:04 +0000 (13:31 +0200)]
tools/ceph-bluestore-tool: add commands free-dump and free-score

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit e7f5e53cde3b586701ff8e759a941986c17af1a8)

5 years agocommon/admin_socket: Add 'execute_command' that allows for self-reflection.
Adam Kupczyk [Tue, 30 Jul 2019 11:54:58 +0000 (13:54 +0200)]
common/admin_socket: Add 'execute_command' that allows for self-reflection.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 838320a9b22c1749f609433843542690fccdfaee)

5 years agocommon/admin_socket: Adapted old protocol to use new protocol, simplifies handle...
Adam Kupczyk [Tue, 30 Jul 2019 10:26:33 +0000 (12:26 +0200)]
common/admin_socket: Adapted old protocol to use new protocol, simplifies handle path.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 2b5da1f671a76f77107b6eaefc67e0e9c9179ca7)

5 years agoBlueStore/allocator: Add ability to dump free allocations via admin socket interface.
Adam Kupczyk [Fri, 2 Aug 2019 11:54:13 +0000 (13:54 +0200)]
BlueStore/allocator: Add ability to dump free allocations via admin socket interface.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 278055d35ae3ca2eb06f59f4712ad9ce5d843d86)

5 years agoBlueStore/allocator: Give allocator names, so they can be distinguished.
Adam Kupczyk [Fri, 2 Aug 2019 11:53:37 +0000 (13:53 +0200)]
BlueStore/allocator: Give allocator names, so they can be distinguished.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 0d64893c3ebc8a56ceac61596efc6f75ed06fbaf)

5 years agoBlueStore/allocator: Improved (but slower) method of calculating fragmentation.
Adam Kupczyk [Thu, 21 Jun 2018 11:58:45 +0000 (13:58 +0200)]
BlueStore/allocator: Improved (but slower) method of calculating fragmentation.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit c1ac6a1207cdb71a471321e42ceca517383b36c8)

5 years agoMerge pull request #29508 from jan--f/wip-41139-luminous
Jan Fajerski [Wed, 7 Aug 2019 06:10:20 +0000 (08:10 +0200)]
Merge pull request #29508 from jan--f/wip-41139-luminous

luminous: ceph-volume prints errors to stdout with --format json

5 years agoluminous: osd: Modify lazy omap stats to only show one copy
Brad Hubbard [Tue, 9 Jul 2019 05:03:24 +0000 (15:03 +1000)]
luminous: osd: Modify lazy omap stats to only show one copy

Only show stats from the primary to get more in line with other pg stats
representations.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 490829c8eaf14f41de978ac0f93ecda3df8bcdef)

5 years agoluminous: tests: Add test for lazy omap stat collection
Brad Hubbard [Mon, 4 Mar 2019 08:12:27 +0000 (18:12 +1000)]
luminous: tests: Add test for lazy omap stat collection

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 88e9ca58a06ba572ff4c6f6125a8ba7f8e0a7795)

5 years agoluminous: osd: Implement lazy omap usage statistics
Brad Hubbard [Tue, 11 Dec 2018 04:00:35 +0000 (14:00 +1000)]
luminous: osd: Implement lazy omap usage statistics

Opportunistic gathering of omap statistics during deep scrub.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 9e21ef0e40ff89b10dfdf857c703336496b3171f)

 Conflicts:
doc/dev/placement-group.rst: Trivial
src/osd/PGBackend.cc: Small difference in ScrubMap::object init
src/osd/osd_types.cc: Differences in structure versions and
        members
src/osd/osd_types.h: Differences in structure versions and
        members

5 years agoceph-volume: print most logging messages to stderr 29508/head
Jan Fajerski [Mon, 4 Mar 2019 10:28:13 +0000 (11:28 +0100)]
ceph-volume: print most logging messages to stderr

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
Fixes: http://tracker.ceph.com/issues/38548
(cherry picked from commit 1f8f1e95dee88b4bab2d6931e720ed83584d2887)

 Conflicts:
src/pybind/mgr/test_orchestrator/module.py
        removed

5 years agoMerge pull request #29462 from jan--f/wip-41057-luminous
Jan Fajerski [Tue, 6 Aug 2019 11:35:34 +0000 (13:35 +0200)]
Merge pull request #29462 from jan--f/wip-41057-luminous

luminous: ceph-volume does not recognize wal/db partitions created by ceph-disk

5 years agoMerge pull request #29415 from jan--f/wip-41020-luminous
Alfredo Deza [Mon, 5 Aug 2019 14:52:33 +0000 (10:52 -0400)]
Merge pull request #29415 from jan--f/wip-41020-luminous

luminous: simple: when 'type' file is not present activate fails

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume: fall back to PARTTYPE if PARTLABEL is empty 29462/head
Jan Fajerski [Tue, 30 Jul 2019 15:32:06 +0000 (17:32 +0200)]
ceph-volume: fall back to PARTTYPE if PARTLABEL is empty

In some cases ceph-disk does not populate PARTLABEL for wal and db
partitions. This commit adds the assumption that the empty string is a
valid label and falls back to identifying those disks by PARTTYPE.

Fixes: https://tracker.ceph.com/issues/40917
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 82d2ae7ffdf62ca93c7848fb62e506db4cda893c)

5 years agoceph-volume: adjust tests for empty PARTLABEL fields
Jan Fajerski [Tue, 30 Jul 2019 15:29:09 +0000 (17:29 +0200)]
ceph-volume: adjust tests for empty PARTLABEL fields

It seems like ceph-disk sometimes creates OSDs where the wal and db
partitions have no PARTLABEL set. This commit alters tests so that the
empty string is considered a valid label.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit bb29110c3c6e081568231131c70522dee6d12bb5)

5 years agoceph-volume: refactor ceph-disk_member unittests
Jan Fajerski [Tue, 30 Jul 2019 11:29:22 +0000 (13:29 +0200)]
ceph-volume: refactor ceph-disk_member unittests

This adds a parameterized fixture as input to the tests. Should make
writing new tests easier since only fixture parameters need to be added.
In the future we should be able to reuse the fixtures to build other
fixtures and reuse the parametrization.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit f42287c3416ad09c112e64756f5051c5c13aa344)

5 years agoMerge pull request #28647 from dvanders/zstd
Josh Durgin [Wed, 31 Jul 2019 15:57:47 +0000 (08:57 -0700)]
Merge pull request #28647 from dvanders/zstd

luminous: compressor/zstd: improvements

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
5 years agoMerge pull request #26498 from k0ste/luminous_backports
Yuri Weinstein [Wed, 31 Jul 2019 15:35:09 +0000 (08:35 -0700)]
Merge pull request #26498 from k0ste/luminous_backports

luminous: mgr/balancer: blame if upmap won't actually work

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
5 years agoMerge pull request #28666 from varshar16/wip-fs-authorize-err
Yuri Weinstein [Wed, 31 Jul 2019 15:32:03 +0000 (08:32 -0700)]
Merge pull request #28666 from varshar16/wip-fs-authorize-err

luminous: mon/AuthMonitor: don't validate fs caps on authorize

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
5 years agoMerge pull request #28972 from neha-ojha/wip-40720-luminous
Yuri Weinstein [Wed, 31 Jul 2019 15:31:08 +0000 (08:31 -0700)]
Merge pull request #28972 from neha-ojha/wip-40720-luminous

[AFTER: #28644] luminous: os/bluestore: default to bitmap allocator for bluestore/bluefs

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
5 years agoMerge pull request #29213 from smithfarm/wip-38962-luminous
Yuri Weinstein [Wed, 31 Jul 2019 15:30:17 +0000 (08:30 -0700)]
Merge pull request #29213 from smithfarm/wip-38962-luminous

luminous: mgr: DaemonServer::handle_conf_change - broken locking

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoceph-volume tests: ensure that better heuristics exist for objectstore detection 29415/head
Alfredo Deza [Mon, 29 Jul 2019 19:50:14 +0000 (15:50 -0400)]
ceph-volume tests: ensure that better heuristics exist for objectstore detection

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e48ad3ab26fdab72755bc933781db3bfc6001a30)

5 years agoceph-volume simple.activate better detect bluestore/filestore when type is not found
Alfredo Deza [Mon, 29 Jul 2019 19:46:48 +0000 (15:46 -0400)]
ceph-volume simple.activate better detect bluestore/filestore when type is not found

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b6a449e2e3241aaa2c0eb8fa9f74b5d939c97852)

5 years agoMerge pull request #29351 from jan--f/wip-40978-luminous
Alfredo Deza [Fri, 26 Jul 2019 13:35:28 +0000 (09:35 -0400)]
Merge pull request #29351 from jan--f/wip-40978-luminous

luminous: ceph-volume: missing string substitution when reporting mounts

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoceph-volume:util: Use proper param substition 29351/head
Shyukri Shyukriev [Fri, 7 Jun 2019 12:09:19 +0000 (12:09 +0000)]
ceph-volume:util: Use proper param substition

Fixes: https://tracker.ceph.com/issues/25030
Signed-off-by: Shyukri Shyukriev <shshyukriev@suse.com>
(cherry picked from commit 654660cd89fe9d93087d390805651c860bf4e862)

5 years agoMerge pull request #27056 from ifed01/wip-ifed-fix-storetest-luminous
Yuri Weinstein [Thu, 25 Jul 2019 21:50:52 +0000 (14:50 -0700)]
Merge pull request #27056 from ifed01/wip-ifed-fix-storetest-luminous

luminous: test/store_test: fix/workaround for BlobReuseOnOverwriteUT and garbageCollection

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #27618 from smithfarm/wip-39332-luminous
Yuri Weinstein [Thu, 25 Jul 2019 21:49:55 +0000 (14:49 -0700)]
Merge pull request #27618 from smithfarm/wip-39332-luminous

luminous: build/ops: rpm: Build with lttng on openSUSE

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoMerge pull request #27808 from xiexingguo/wip-39424
Yuri Weinstein [Thu, 25 Jul 2019 21:48:52 +0000 (14:48 -0700)]
Merge pull request #27808 from xiexingguo/wip-39424

luminous: mgr/BaseMgrModule: drop GIL in set_config

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #28644 from ifed01/wip-ifed-fix-40080-luminous
Yuri Weinstein [Thu, 25 Jul 2019 21:47:11 +0000 (14:47 -0700)]
Merge pull request #28644 from ifed01/wip-ifed-fix-40080-luminous

luminous: os/bluestore: fix duplicate allocations in bmap allocator

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #28742 from smithfarm/wip-40548-luminous
Yuri Weinstein [Thu, 25 Jul 2019 21:46:28 +0000 (14:46 -0700)]
Merge pull request #28742 from smithfarm/wip-40548-luminous

luminous: common: Keyrings created by ceph auth get are not suitable for ceph auth import

Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoMerge pull request #28851 from jdurgin/wip-18096-luminous
Yuri Weinstein [Thu, 25 Jul 2019 21:45:31 +0000 (14:45 -0700)]
Merge pull request #28851 from jdurgin/wip-18096-luminous

luminous: osd: report omap/data/metadata usage

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #28895 from ifed01/wip-ifed-fix-no-compress-lum
Yuri Weinstein [Thu, 25 Jul 2019 21:44:55 +0000 (14:44 -0700)]
Merge pull request #28895 from ifed01/wip-ifed-fix-no-compress-lum

luminous: os/bluestore: load OSD all compression settings unconditionally.

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #28933 from dzafman/wip-backport-24178
Yuri Weinstein [Thu, 25 Jul 2019 21:43:56 +0000 (14:43 -0700)]
Merge pull request #28933 from dzafman/wip-backport-24178

luminous: qa/workunits/rados/test_envlibrados_for_rocksdb: install g++ not g++-4.7

Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoMerge pull request #28594 from xiexingguo/wip-40229
Yuri Weinstein [Wed, 24 Jul 2019 23:43:03 +0000 (16:43 -0700)]
Merge pull request #28594 from xiexingguo/wip-40229

luminous: mon, osd: parallel clean_pg_upmaps

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #28110 from smithfarm/wip-38880-luminous
Yuri Weinstein [Wed, 24 Jul 2019 22:43:31 +0000 (15:43 -0700)]
Merge pull request #28110 from smithfarm/wip-38880-luminous

luminous: ENOENT in collection_move_rename on EC backfill target

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #28112 from smithfarm/wip-39373-luminous
Yuri Weinstein [Wed, 24 Jul 2019 22:42:52 +0000 (15:42 -0700)]
Merge pull request #28112 from smithfarm/wip-39373-luminous

luminous: core: ceph tell osd.xx bench help : gives wrong help

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #28811 from trociny/wip-40463-luminous
Yuri Weinstein [Wed, 24 Jul 2019 22:41:26 +0000 (15:41 -0700)]
Merge pull request #28811 from trociny/wip-40463-luminous

luminous: journal: properly advance read offset after skipping invalid range

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
5 years agoMerge pull request #27905 from smithfarm/wip-38954-luminous
Yuri Weinstein [Wed, 24 Jul 2019 15:20:27 +0000 (08:20 -0700)]
Merge pull request #27905 from smithfarm/wip-38954-luminous

luminous: tests: krbd discard qa fixes

5 years agoMerge pull request #28149 from trociny/wip-37692-luminous
Yuri Weinstein [Wed, 24 Jul 2019 15:19:52 +0000 (08:19 -0700)]
Merge pull request #28149 from trociny/wip-37692-luminous

luminous: librbd: disable image mirroring when moving to trash

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
5 years agoMerge pull request #28158 from dillaman/wip-38508-luminous
Yuri Weinstein [Wed, 24 Jul 2019 15:19:14 +0000 (08:19 -0700)]
Merge pull request #28158 from dillaman/wip-38508-luminous

luminous: librbd: add missing shutdown states to managed lock helper

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
5 years agoMerge pull request #28773 from dillaman/wip-40574-luminous
Yuri Weinstein [Wed, 24 Jul 2019 15:18:42 +0000 (08:18 -0700)]
Merge pull request #28773 from dillaman/wip-40574-luminous

luminous: librbd: properly track in-flight flush requests

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
5 years agoMerge pull request #28329 from smithfarm/wip-40092-luminous
Nathan Cutler [Wed, 24 Jul 2019 10:36:11 +0000 (12:36 +0200)]
Merge pull request #28329 from smithfarm/wip-40092-luminous

luminous: doc: Missing Documentation for radosgw-admin reshard commands (man pages)

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #27185 from smithfarm/wip-38902-luminous
Nathan Cutler [Wed, 24 Jul 2019 10:29:40 +0000 (12:29 +0200)]
Merge pull request #27185 from smithfarm/wip-38902-luminous

luminous: doc: Minor rados related documentation fixes

Reviewed-by: David Zafman <dzafman@redhat.com>