ceph.git
5 years agoMerge pull request #25117 from alfredodeza/luminous-rm26953 25124/head
Andrew Schoen [Thu, 15 Nov 2018 17:23:29 +0000 (11:23 -0600)]
Merge pull request #25117 from alfredodeza/luminous-rm26953

luminous ceph-volume: allow to specify --cluster-fsid instead of reading from ceph.conf

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoceph-volume tests.lvm verify get_cluster_fsid can read from args or ceph.conf 25117/head
Alfredo Deza [Wed, 3 Oct 2018 19:21:03 +0000 (15:21 -0400)]
ceph-volume tests.lvm verify get_cluster_fsid can read from args or ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 02ea6dc0663cdaf764fdb1a1a5b31a736b854e6e)

5 years agoceph-volume lvm.prepare consume the value from --cluster-fsid if available
Alfredo Deza [Wed, 3 Oct 2018 17:32:05 +0000 (13:32 -0400)]
ceph-volume lvm.prepare consume the value from --cluster-fsid if available

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a92883e60f3f2206a11b43762b3afcb84b05f07e)

5 years agoceph-volume lvm.common add the --cluster-fsid flag
Alfredo Deza [Wed, 3 Oct 2018 17:31:43 +0000 (13:31 -0400)]
ceph-volume lvm.common add the --cluster-fsid flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 15546ae6991c244197537d81092875b12613e20b)

5 years agoMerge pull request #25104 from alfredodeza/luminous-rm-27062
Alfredo Deza [Wed, 14 Nov 2018 23:17:29 +0000 (18:17 -0500)]
Merge pull request #25104 from alfredodeza/luminous-rm-27062

luminous ceph-volume: reject devices that have existing GPT headers

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoMerge pull request #25101 from alfredodeza/luminous-bz1644828
Alfredo Deza [Wed, 14 Nov 2018 21:34:27 +0000 (16:34 -0500)]
Merge pull request #25101 from alfredodeza/luminous-bz1644828

luminous ceph-volume: remove LVs when using zap --destroy

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoceph-volume: update tests after removing the LVPath arg validator 25104/head
Andrew Schoen [Wed, 14 Nov 2018 16:35:32 +0000 (10:35 -0600)]
ceph-volume: update tests after removing the LVPath arg validator

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: rm#27062
(cherry picked from commit 0cb95ace88a731f7c30adf3f2286691fa3b2c49c)

5 years agoceph-volume: return lvs as vg/lv from ValidDevice if as_string is True
Andrew Schoen [Tue, 13 Nov 2018 19:02:39 +0000 (13:02 -0600)]
ceph-volume: return lvs as vg/lv from ValidDevice if as_string is True

All code in prepare and activate expect lvs to be in that format.
Eventually we can refactor all that to use the Device class as well.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: rm#27062
(cherry picked from commit fdf7f3c575299b6be0ac40fb6eee5106cea010c9)

5 years agoceph-volume: adds lv_name property to Device class
Andrew Schoen [Tue, 13 Nov 2018 18:40:45 +0000 (12:40 -0600)]
ceph-volume: adds lv_name property to Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: rm#27062
(cherry picked from commit cc0317efac5a3a25edcdb48af2a63a969c2ac43a)

5 years agoceph-volume: combine the LVPath arg validator with ValidDevice
Andrew Schoen [Tue, 13 Nov 2018 18:28:23 +0000 (12:28 -0600)]
ceph-volume: combine the LVPath arg validator with ValidDevice

This combines the two arg validators and makes both batch and
prepare/active both use ValidDevice.

This will also allow us to use lvs with a full path, like
/dev/vg/lv instead of always enforcing vg/lv.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: rm#27062
(cherry picked from commit d6a5a7606c3a0c1f1624c5784abb02bb1a4e1620)

5 years agoceph-volume: adds has_gpt_headers property to Device class
Andrew Schoen [Tue, 13 Nov 2018 18:27:19 +0000 (12:27 -0600)]
ceph-volume: adds has_gpt_headers property to Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: rm#27062
(cherry picked from commit 568a0ce5be0161541772ceacb9b81652f51659f5)

5 years agoceph-volume: add PTTYPE to blkid parser
Andrew Schoen [Tue, 13 Nov 2018 15:35:21 +0000 (09:35 -0600)]
ceph-volume: add PTTYPE to blkid parser

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: rm#27062
(cherry picked from commit 3d86d2dda9fae87e143e5212d5f354a22f02909d)

5 years agoceph-volume tests update tests to handle new SystemExit exceptions in main 25101/head
Alfredo Deza [Tue, 13 Nov 2018 22:06:03 +0000 (17:06 -0500)]
ceph-volume tests update tests to handle new SystemExit exceptions in main

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 27d0a915d4a7f56aafc603a949b4e4c36772d236)

5 years agoceph-volume tests.api verify that Volume object can be removed
Alfredo Deza [Tue, 13 Nov 2018 22:05:31 +0000 (17:05 -0500)]
ceph-volume tests.api verify that Volume object can be removed

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9527cffbe335043f7c5a60162138dfe093ae51da)

5 years agoceph-volume api.lvm allow passing a Volume object to remove_lv
Alfredo Deza [Tue, 13 Nov 2018 21:56:02 +0000 (16:56 -0500)]
ceph-volume api.lvm allow passing a Volume object to remove_lv

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e7daa71604f4d08ace1125bb63dcfe909796d3ab)

5 years agoceph-volume tests.functional.lvm test full removal of LVs
Alfredo Deza [Tue, 13 Nov 2018 21:05:32 +0000 (16:05 -0500)]
ceph-volume tests.functional.lvm test full removal of LVs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4656bf0e333a50e0fca5fd0612cdda98ab5d70f9)

5 years agoceph-volume main raise sys.exit to prevent garbage in help
Alfredo Deza [Tue, 13 Nov 2018 13:32:16 +0000 (08:32 -0500)]
ceph-volume main raise sys.exit to prevent garbage in help

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f695d1716e9fc6252dd713eb8fb08c28b1284611)

5 years agoceph-volume lvm.zap remove vgs|lvs when --destroy is used
Alfredo Deza [Tue, 13 Nov 2018 13:30:23 +0000 (08:30 -0500)]
ceph-volume lvm.zap remove vgs|lvs when --destroy is used

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 677cbbb059185fe2808746b36bd3f988ff2af41c)

5 years agoMerge pull request #25067 from alfredodeza/luminous-rm36768
Andrew Schoen [Tue, 13 Nov 2018 16:12:35 +0000 (10:12 -0600)]
Merge pull request #25067 from alfredodeza/luminous-rm36768

luminous ceph-volume: patch Device when testing

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoceph-volume tests patch Device() by splitting parametrized method 25067/head
Alfredo Deza [Mon, 12 Nov 2018 17:57:26 +0000 (12:57 -0500)]
ceph-volume tests patch Device() by splitting parametrized method

This was causing failures on systems where there is no LVM or where the
device names don't match. Patching is always recommended to avoid
conflicts with the system testing

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 82d96f876484c9668e2fc4726057691ba2c50f06)

5 years agoceph-volume tests inventory should patch before using .get_devices()
Alfredo Deza [Mon, 12 Nov 2018 17:56:42 +0000 (12:56 -0500)]
ceph-volume tests inventory should patch before using .get_devices()

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7ee7a59910c9837659419c235f669a53cc9f613c)

5 years agoMerge pull request #25012 from alfredodeza/luminous-rm36470
Andrew Schoen [Mon, 12 Nov 2018 16:15:08 +0000 (10:15 -0600)]
Merge pull request #25012 from alfredodeza/luminous-rm36470

luminous ceph-volume enable  --no-systemd flag for simple sub-command

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoMerge pull request #25014 from alfredodeza/luminous-ceph-volume-inventory
Alfredo Deza [Mon, 12 Nov 2018 13:37:52 +0000 (08:37 -0500)]
Merge pull request #25014 from alfredodeza/luminous-ceph-volume-inventory

luminous ceph-volume: add inventory command

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
5 years agoceph-volume: add inventory command 25014/head
Jan Fajerski [Wed, 31 Oct 2018 13:59:05 +0000 (14:59 +0100)]
ceph-volume: add inventory command

The inventory command provides information about a nodes disk inventory.
Existing logical volumes on a disk or one of its partitions are scanned
and reported.
The output can be formatted as plain text or json.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 57adfc6bb8abd9fc84b1378d5f69a16994d19e11)

5 years agoceph-volume tests ensure activate behavior with systemd disabling 25012/head
Alfredo Deza [Thu, 8 Nov 2018 19:14:17 +0000 (14:14 -0500)]
ceph-volume tests ensure activate behavior with systemd disabling

Signed-off-by: Alfredo Deza <alfredo@deza.pe>
(cherry picked from commit 3e80118ccaccaa61e92b01f4f389d6056401d8a7)

5 years agoceph-volume simple.activate factor out systemd interactions for easier testing
Alfredo Deza [Thu, 8 Nov 2018 19:13:52 +0000 (14:13 -0500)]
ceph-volume simple.activate factor out systemd interactions for easier testing

Signed-off-by: Alfredo Deza <alfredo@deza.pe>
(cherry picked from commit 18ddd962116b5359af770088535f502797ac79c4)

5 years agoceph-volume simple.trigger replace systemd with from_trigger keyword argument
Alfredo Deza [Thu, 8 Nov 2018 14:14:24 +0000 (09:14 -0500)]
ceph-volume simple.trigger replace systemd with from_trigger keyword argument

Signed-off-by: Alfredo Deza <alfredo@deza.pe>
(cherry picked from commit 9526587403b76fa5397ae80b63328e3a59765c2e)

5 years agoceph-volume simple.activate allow a --no-systemd flag to skip systemctl actions
Alfredo Deza [Thu, 8 Nov 2018 14:13:31 +0000 (09:13 -0500)]
ceph-volume simple.activate allow a --no-systemd flag to skip systemctl actions

Signed-off-by: Alfredo Deza <alfredo@deza.pe>
(cherry picked from commit 4d9ad0d3d4bedd5efa95653fbb86607f0b469174)

5 years agoMerge pull request #24981 from alfredodeza/luminous-bz1644265
Alfredo Deza [Fri, 9 Nov 2018 14:07:22 +0000 (09:07 -0500)]
Merge pull request #24981 from alfredodeza/luminous-bz1644265

luminous ceph-volume util.encryption robust blkid+lsblk detection of lockbox

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoceph-volume util.encryption robust blkid+lsblk detection of lockbox 24981/head
Alfredo Deza [Fri, 2 Nov 2018 19:03:45 +0000 (15:03 -0400)]
ceph-volume util.encryption robust blkid+lsblk detection of lockbox

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fc3e6ea3d9200e0fe2a3aa2576fcd45db223c277)

5 years agoMerge pull request #24960 from alfredodeza/luminous-rm36672
Alfredo Deza [Wed, 7 Nov 2018 11:46:32 +0000 (06:46 -0500)]
Merge pull request #24960 from alfredodeza/luminous-rm36672

luminous ceph-volume tests install ceph-ansible's requirements.txt dependencies

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoMerge pull request #24958 from alfredodeza/luminous-rm36704
Andrew Schoen [Tue, 6 Nov 2018 21:46:05 +0000 (15:46 -0600)]
Merge pull request #24958 from alfredodeza/luminous-rm36704

luminous: ceph-volume tests.systemd update imports for systemd module

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoceph-volume tests install ceph-ansible's requirements.txt dependencies 24960/head
Alfredo Deza [Thu, 1 Nov 2018 12:48:28 +0000 (08:48 -0400)]
ceph-volume tests install ceph-ansible's requirements.txt dependencies

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 52ddb69813c5849506db5f850a3f2532ff1124b5)

5 years agoceph-volume tests.systemd update imports for systemd module 24958/head
Alfredo Deza [Tue, 6 Nov 2018 14:45:08 +0000 (09:45 -0500)]
ceph-volume tests.systemd update imports for systemd module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a63a6c86a03738bf50fa71a490951343a83afff6)

5 years agoceph-volume systemd import main so console_scripts work for executable
Alfredo Deza [Tue, 6 Nov 2018 14:44:39 +0000 (09:44 -0500)]
ceph-volume systemd import main so console_scripts work for executable

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 280e2baab27cc061d2d5e456f098cd3405f36873)

5 years agoMerge pull request #24827 from alfredodeza/luminous-rm36246
Andrew Schoen [Tue, 6 Nov 2018 21:05:48 +0000 (15:05 -0600)]
Merge pull request #24827 from alfredodeza/luminous-rm36246

luminous ceph-volume util.encryption don't push stderr to terminal

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoMerge pull request #24837 from alfredodeza/luminous-no-scripts
Andrew Schoen [Tue, 6 Nov 2018 21:04:30 +0000 (15:04 -0600)]
Merge pull request #24837 from alfredodeza/luminous-no-scripts

luminous ceph-volume: use console_scripts

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoMerge pull request #24853 from ceph/luminous-rm36648
Andrew Schoen [Tue, 6 Nov 2018 20:38:21 +0000 (14:38 -0600)]
Merge pull request #24853 from ceph/luminous-rm36648

luminous ceph-volume systemd import main so console_scripts work for executable

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoMerge pull request #24902 from neha-ojha/wip-revert-23493-luminous
Yuri Weinstein [Mon, 5 Nov 2018 22:14:30 +0000 (14:14 -0800)]
Merge pull request #24902 from neha-ojha/wip-revert-23493-luminous

luminous: Revert "PG: add custom_reaction Backfilled and release reservations after backfill

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
5 years agoMerge pull request #24903 from neha-ojha/wip-revert-pg-log-limit
Yuri Weinstein [Mon, 5 Nov 2018 22:12:42 +0000 (14:12 -0800)]
Merge pull request #24903 from neha-ojha/wip-revert-pg-log-limit

luminous: Revert pg log limit changes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
5 years agoMerge pull request #24432 from smithfarm/wip-35935-luminous
Yuri Weinstein [Mon, 5 Nov 2018 22:11:08 +0000 (14:11 -0800)]
Merge pull request #24432 from smithfarm/wip-35935-luminous

luminous: OSDMapMapping does not handle active.size() > pool size

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #24706 from smithfarm/wip-36552-luminous
Yuri Weinstein [Mon, 5 Nov 2018 22:10:12 +0000 (14:10 -0800)]
Merge pull request #24706 from smithfarm/wip-36552-luminous

luminous: gperftools-libs-2.6.1-1 or newer required for binaries linked against corresponding version at build time

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
5 years agoMerge pull request #24746 from smithfarm/wip-36591-luminous
Yuri Weinstein [Mon, 5 Nov 2018 22:08:23 +0000 (14:08 -0800)]
Merge pull request #24746 from smithfarm/wip-36591-luminous

luminous: segv in BlueStore::OldExtent::create

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #24802 from tchaikov/wip-luminous-35067
Yuri Weinstein [Mon, 5 Nov 2018 22:07:56 +0000 (14:07 -0800)]
Merge pull request #24802 from tchaikov/wip-luminous-35067

luminous: core: by pass cache if performing deep scrub

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agodoc: make note of pg log hard limit upgrade bug in PendingReleaseNotes 24903/head
Neha Ojha [Mon, 5 Nov 2018 19:47:26 +0000 (14:47 -0500)]
doc: make note of pg log hard limit upgrade bug in PendingReleaseNotes

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "osd: make calc_trim_to() independent of min_last_complete_ondisk"
Neha Ojha [Fri, 2 Nov 2018 23:51:03 +0000 (19:51 -0400)]
Revert "osd: make calc_trim_to() independent of min_last_complete_ondisk"

This reverts commit 3ecbf2a41fef3591ff247a212338b9a2d02a3468.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "osd: print pg log length and trim_to"
Neha Ojha [Fri, 2 Nov 2018 23:50:50 +0000 (19:50 -0400)]
Revert "osd: print pg log length and trim_to"

This reverts commit 21a72d2a52f8241ebe0a5455f030d2095e413025.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "osd: handle trim() during backfill"
Neha Ojha [Fri, 2 Nov 2018 23:50:38 +0000 (19:50 -0400)]
Revert "osd: handle trim() during backfill"

This reverts commit bdfc2a3b4fa8038ade31ccab5f5479508e9afff3.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "osd: allow trim() to proceed when there are missing items"
Neha Ojha [Fri, 2 Nov 2018 23:50:22 +0000 (19:50 -0400)]
Revert "osd: allow trim() to proceed when there are missing items"

This reverts commit d08f0aec86ecd9b23310f1c41f9cb9060f10bc6b.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "osd: reset complete_to when trimming the log past it"
Neha Ojha [Fri, 2 Nov 2018 23:50:03 +0000 (19:50 -0400)]
Revert "osd: reset complete_to when trimming the log past it"

This reverts commit 715d1fdbfc407de81686784d99fe28b51c6d97ef.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "osd/PGLog: allow pg log trim when complete_to is less than trim_to"
Neha Ojha [Fri, 2 Nov 2018 23:49:41 +0000 (19:49 -0400)]
Revert "osd/PGLog: allow pg log trim when complete_to is less than trim_to"

This reverts commit be8b406d9162a1f46b3eac2fca07b0cf20697b90.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "osd/PGLog.cc: use lgeneric_subdout instead of generic_dout"
Neha Ojha [Fri, 2 Nov 2018 23:49:21 +0000 (19:49 -0400)]
Revert "osd/PGLog.cc: use lgeneric_subdout instead of generic_dout"

This reverts commit f2ea4f8b67ac972cd428e8ea8f2ff062b1624677.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "src/osd/PG.cc: remove redundant call to trim_log()"
Neha Ojha [Fri, 2 Nov 2018 23:47:54 +0000 (19:47 -0400)]
Revert "src/osd/PG.cc: remove redundant call to trim_log()"

This reverts commit 31616627dcfb2b6ab7db7da025e191dc34094bb8.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "osd/PGLog.cc: check if complete_to points to log.end()"
Neha Ojha [Fri, 2 Nov 2018 23:47:37 +0000 (19:47 -0400)]
Revert "osd/PGLog.cc: check if complete_to points to log.end()"

This reverts commit ac772c90dfc3749527246242a3a84e5e48ff52e1.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "osd/PrimaryLogPG.cc: limit trimming at can_rollback_to"
Neha Ojha [Fri, 2 Nov 2018 23:46:22 +0000 (19:46 -0400)]
Revert "osd/PrimaryLogPG.cc: limit trimming at can_rollback_to"

This reverts commit 0e57c101c9dd51ee3a84123c96e626799ab6d21e.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "osd/PrimaryLogPG: avoid dereferencing invalid complete_to"
Neha Ojha [Fri, 2 Nov 2018 23:45:30 +0000 (19:45 -0400)]
Revert "osd/PrimaryLogPG: avoid dereferencing invalid complete_to"

This reverts commit eb3bdf0714b9c78c5b3634d4efe3b603824fea09.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "osd/PrimaryLogPG: fix potential pg-log overtrimming"
Neha Ojha [Fri, 2 Nov 2018 23:34:18 +0000 (19:34 -0400)]
Revert "osd/PrimaryLogPG: fix potential pg-log overtrimming"

This reverts commit 85a029a7242a1c6eef4e90ae246541e08207302a.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoRevert "PG: add custom_reaction Backfilled and release reservations after backfill" 24902/head
Neha Ojha [Fri, 2 Nov 2018 23:37:17 +0000 (19:37 -0400)]
Revert "PG: add custom_reaction Backfilled and release reservations after backfill"

This reverts commit f512f905b140bfa9cc3241793e441de2c377e60b.

Signed-off-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #24854 from cbodley/wip-36536
Yuri Weinstein [Thu, 1 Nov 2018 16:10:36 +0000 (09:10 -0700)]
Merge pull request #24854 from cbodley/wip-36536

luminous: rgw: user stats account for resharded buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #24855 from cbodley/wip-36534
Yuri Weinstein [Thu, 1 Nov 2018 16:10:00 +0000 (09:10 -0700)]
Merge pull request #24855 from cbodley/wip-36534

luminous: cls/user: cls_user_remove_bucket writes modified header

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #24862 from jonsger/luminous-backport-pr#23242
Yuri Weinstein [Thu, 1 Nov 2018 16:09:27 +0000 (09:09 -0700)]
Merge pull request #24862 from jonsger/luminous-backport-pr#23242

luminous: rgw: set default objecter_inflight_ops = 24576

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #24454 from croit/luminous-36041
Yuri Weinstein [Thu, 1 Nov 2018 16:06:06 +0000 (09:06 -0700)]
Merge pull request #24454 from croit/luminous-36041

luminous: rgw/beast: drop privileges after binding ports

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #24507 from pdvian/wip-36212-luminous
Yuri Weinstein [Thu, 1 Nov 2018 16:05:36 +0000 (09:05 -0700)]
Merge pull request #24507 from pdvian/wip-36212-luminous

luminous: rgw: RGWAsyncGetBucketInstanceInfo does not access coroutine memory

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #24519 from pdvian/wip-36214-luminous
Yuri Weinstein [Thu, 1 Nov 2018 16:05:10 +0000 (09:05 -0700)]
Merge pull request #24519 from pdvian/wip-36214-luminous

luminous: rgw: fix leak of curl handle on shutdown

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #24570 from joke-lee/luminous-list-RGWPostObj-when-enable-versioning
Yuri Weinstein [Thu, 1 Nov 2018 16:04:35 +0000 (09:04 -0700)]
Merge pull request #24570 from joke-lee/luminous-list-RGWPostObj-when-enable-versioning

luminous: rgw: list bucket can not show the object uploaded by RGWPostObj when enable bucket versioning

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #24857 from jonsger/luminous-backport-pr#23534
Yuri Weinstein [Thu, 1 Nov 2018 16:03:53 +0000 (09:03 -0700)]
Merge pull request #24857 from jonsger/luminous-backport-pr#23534

luminous: rgw: multisite: enforce spawn_window for data full sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agorgw: set default objecter_inflight_ops = 24576 24862/head
Matt Benjamin [Wed, 25 Jul 2018 20:06:33 +0000 (16:06 -0400)]
rgw: set default objecter_inflight_ops = 24576

Fixes: http://tracker.ceph.com/issues/25109
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 102c5bd)
Signed-off-by: Jonathan Brielmaier <jbrielmaier@suse.de>
Conflicts:
src/rgw/rgw_main.cc: use the existing vector, there is no map

5 years agoceph-volume systemd import main so console_scripts work for executable 24853/head
Alfredo Deza [Tue, 30 Oct 2018 19:50:28 +0000 (15:50 -0400)]
ceph-volume systemd import main so console_scripts work for executable

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0bcda52566279404b88184b2ce7b2200d581b0b7)

5 years agocls/user: cls_user_remove_bucket writes modified header 24855/head
Casey Bodley [Wed, 17 Oct 2018 18:50:18 +0000 (14:50 -0400)]
cls/user: cls_user_remove_bucket writes modified header

if cls_user_remove_bucket() modifies the header with dec_header_stats(),
it should write the updated header back

Fixes: http://tracker.ceph.com/issues/36496
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit dac26de1fa7b56ce0ec2fd43d26ab9a9b6a32e3d)

5 years agorgw: cls_user_set_buckets_info can update bucket id 24854/head
Casey Bodley [Mon, 15 Oct 2018 20:38:31 +0000 (16:38 -0400)]
rgw: cls_user_set_buckets_info can update bucket id

if cls_user_set_bucket_info() finds an existing bucket entry, it does
not update its bucket id

Fixes: https://tracker.ceph.com/issues/24505
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e1b97fc2e804d040e8aa6fc8a94e336222d1175e)

5 years agorgw: rgw_user_sync_all_stats uses bucket entrypoint
Casey Bodley [Mon, 15 Oct 2018 20:25:20 +0000 (16:25 -0400)]
rgw: rgw_user_sync_all_stats uses bucket entrypoint

when bucket reshard completes, rgw_link_bucket() passes the new bucket
instance id down to cls_user, but cls_user_set_buckets_info() does not
change the instance id when it's updating an existing bucket. so when
rgw_user_sync_all_stats() looks up each of the user's buckets, it uses
the original bucket instance id instead of the resharded one and
calculates user stats that may not match the current bucket stats

as a workaround, rgw_user_sync_all_stats() no longer relies on the
bucket instance id it gets from rgw_read_user_buckets(), and instead
calls get_bucket_info() to look up the current instance in the bucket
entrypoint

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b33328ff9bacd11d8449cb3b5c52f94d9dffec6e)

5 years agorgw: data sync checks whether the lease failed to renew 24857/head
Casey Bodley [Mon, 13 Aug 2018 19:47:29 +0000 (15:47 -0400)]
rgw: data sync checks whether the lease failed to renew

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry pick from commit 3754871)
Signed-off-by: Jonathan Brielmaier <jbrielmaier@suse.de>
Conflicts:
src/rgw/rgw_data_sync.cc: trivial

5 years agorgw: data sync holds lease over transition from full to incremental
Casey Bodley [Sat, 11 Aug 2018 15:39:35 +0000 (11:39 -0400)]
rgw: data sync holds lease over transition from full to incremental

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked commit from 3e9ac0f)
Signed-off-by: Jonathan Brielmaier <jbrielmaier@suse.de>
Conflicts:
src/rgw/rgw_data_sync.cc: use ldout instead of tn->log, reflect
        state of luminous in multiple places

5 years agoMerge pull request #24808 from ceph/wip-luminous-mclock-test
Gregory Farnum [Tue, 30 Oct 2018 22:26:41 +0000 (15:26 -0700)]
Merge pull request #24808 from ceph/wip-luminous-mclock-test

test/common: unittest_mclock_priority_queue builds with "make" command

5 years agoceph-volume: use console_scripts 24837/head
Mehdi Abaakouk [Fri, 26 Oct 2018 09:59:55 +0000 (11:59 +0200)]
ceph-volume: use console_scripts

Using "#!/usr/bin/env python" is not recommended as it's not portable.

setuptools provides an console_scripts entry_point that generates
scripts that always have the good sheban whatever the target operating
system and python version/distribution.

http://tracker.ceph.com/issues/36601

Signed-off-by: Mehdi Abaakouk <sileht@sileht.net>
(cherry picked from commit 6a2ed6c285b74f463f223d3501def8ef4e6283b1)

5 years agorgw: enforce spawn_window for data full sync
Casey Bodley [Fri, 10 Aug 2018 20:34:42 +0000 (16:34 -0400)]
rgw: enforce spawn_window for data full sync

limit the number of concurrent bucket sync crs spawned by data full sync

Fixes: http://tracker.ceph.com/issues/26897
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 895c28a)
Signed-off-by: Jonathan Brielmaier <jbrielmaier@suse.de>
Conflicts:
src/rgw/rgw_data_sync.cc: parameters in yield spawn

5 years agoceph-volume util.encryption don't push stderr to terminal when cryptsetup fails to... 24827/head
Alfredo Deza [Wed, 3 Oct 2018 12:31:20 +0000 (08:31 -0400)]
ceph-volume util.encryption don't push stderr to terminal when cryptsetup fails to find device

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 40f919850fddd14659243c47860224981439906a)

5 years agoMerge pull request #24783 from alfredodeza/luminous-evelu-ceph-volume-choose_disk
Andrew Schoen [Mon, 29 Oct 2018 15:43:58 +0000 (10:43 -0500)]
Merge pull request #24783 from alfredodeza/luminous-evelu-ceph-volume-choose_disk

luminous Additional work on ceph-volume to add some choose_disk capabilities

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
5 years agoos/bluestore: fix deep-scrub operation againest disk silent errors 24802/head
Xiaoguang Wang [Thu, 30 Aug 2018 02:26:41 +0000 (10:26 +0800)]
os/bluestore: fix deep-scrub operation againest disk silent errors

Say a object who has data caches, but in a while later, caches' underlying
physical device has silent disk erros accidentally, then caches and physical
data are not same. In such case, deep-scrub operation still tries to read
caches firstly and won't do crc checksum, then deep-scrub won't find such
data corruptions timely.

Here introduce a new flag 'CEPH_OSD_OP_FLAG_BYPASS_CLEAN_CACHE' which tells
deep-scrub to bypass object caches. Note that we only bypass cache who is in
STATE_CLEAN state. For STATE_WRITING caches, currently they are not written
to physical device, so deep-scrub operation can not read physical device and
can read these dirty caches safely. Once they are in STATE_CLEAN state(or not
added to bluestore cache), next round deep-scurb can check them correctly.

As to above discussions, I refactor BlueStore::BufferSpace::read sightly,
adding a new 'flags' argument, whose value will be 0 or:
     enum {
       BYPASS_CLEAN_CACHE = 0x1,     // bypass clean cache
     };

flags 0: normal read, do not bypass clean or dirty cache
flags BYPASS_CLEAN_CACHE: bypass clean cache, currently only for deep-scrube
                        operation

Test:
   I deliberately corrupt a object with cache, with this patch, deep-scrub
   can find data error very timely.

Signed-off-by: Xiaoguang Wang <xiaoguang.wang@easystack.cn>
(cherry picked from commit a7f1af25dd2ba88a322ed21828f073a277b09d02)

Conflicts:
src/include/rados.h
src/os/bluestore/BlueStore.cc: trivial resolution

5 years agotest/common: unittest_mclock_priority_queue builds with "make" command 24808/head
J. Eric Ivancich [Thu, 7 Sep 2017 21:47:00 +0000 (17:47 -0400)]
test/common: unittest_mclock_priority_queue builds with "make" command

Although it would be built with "make tests",
unittest_mclock_priority_queue was not built with a simple "make".

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 494b2f8f5ae11a689bb2032e4527968ff09bcdeb)

5 years agoceph_volume: Checking device validity at init time 24783/head
Erwan Velu [Wed, 10 Oct 2018 18:26:01 +0000 (20:26 +0200)]
ceph_volume: Checking device validity at init time

When initializing the Device structure, it have to run is_valid() to
ensure the data structures (_is_valid & rejected_reasons) to be
populated accordingly to the device state.

Signed-off-by: Erwan Velu <erwan@redhat.com>
(cherry picked from commit ef0ceef7c717ef615dc214299cb4a6b32c1edbe6)

5 years agoceph_volume: Rejecting locked devices
Erwan Velu [Tue, 9 Oct 2018 20:51:39 +0000 (22:51 +0200)]
ceph_volume: Rejecting locked devices

If we cannot open a block device in O_RDWR in exclusive mode, it means
someone is actually using it like a raw database or similar.

In that case, the device should be considered as unusable as OSDs will
not be in a position to use it.

Signed-off-by: Erwan Velu <erwan@redhat.com>
(cherry picked from commit d5de9583ee6c4d9ccaab2b1146606a0ee316cc27)

5 years agoceph_volume: Reporting nr_requests
Erwan Velu [Tue, 9 Oct 2018 20:28:19 +0000 (22:28 +0200)]
ceph_volume: Reporting nr_requests

We are already reporting the rotational & scheduler of a disk device.
Reporting the nr_requests could be useful to get how many concurrent IOs
the device supports/reports.

That could help detecting badly detected/configured devices.

Signed-off-by: Erwan Velu <erwan@redhat.com>
(cherry picked from commit e0ea3d475a85844afdcdf92cf638b8128c8e2d9e)

5 years agoceph_volume: Reporting firmware revision
Erwan Velu [Tue, 9 Oct 2018 20:26:28 +0000 (22:26 +0200)]
ceph_volume: Reporting firmware revision

We are already reporting model & vendor of a given disk, let's also
report the revision of the firmware. That is useful to filter-out some
known broken revisions.

Signed-off-by: Erwan Velu <erwan@redhat.com>
(cherry picked from commit 59720794079bef0ba7f8bc2d9dfc8aaddd210005)

5 years agoceph_volume: Rejecting Read-only devices
Erwan Velu [Tue, 9 Oct 2018 20:02:31 +0000 (22:02 +0200)]
ceph_volume: Rejecting Read-only devices

If a devices is said to be read-only, there is no chance we can actually
use it. So let's report it as unusable.

Signed-off-by: Erwan Velu <erwan@redhat.com>
(cherry picked from commit f1a94350064263f27f67d2c715c5ca5922481a01)

5 years agoceph_volume: Adding Device.is_valid()
Erwan Velu [Tue, 9 Oct 2018 19:04:02 +0000 (21:04 +0200)]
ceph_volume: Adding Device.is_valid()

A block device can be filtered-out/ignored because it have features that
doesn't match Ceph's expectations.

As of today, the current code was rejected removable devices but it was
pretty hidden from the user, and implicit in the get_devices() function.

This patch is creating a new is_valid() function to perform all the
rejection tests and returns if this device can be used in the Ceph
context or not.

If is_valid() is returning False, the 'rejected_reasons' list reports all
the reasons why that devices got rejected.

Signed-off-by: Erwan Velu <erwan@redhat.com>
(cherry picked from commit d8fdf0b7532cdffcae56511c377679e51841caec)

5 years agoMerge pull request #24759 from ceph/backport-luminous-24587
Alfredo Deza [Fri, 26 Oct 2018 13:46:27 +0000 (09:46 -0400)]
Merge pull request #24759 from ceph/backport-luminous-24587

luminous: ceph-volume: adds a --prepare flag to `lvm batch`

5 years agoMerge pull request #24482 from pdvian/wip-36149-luminous
Yuri Weinstein [Thu, 25 Oct 2018 22:22:14 +0000 (18:22 -0400)]
Merge pull request #24482 from pdvian/wip-36149-luminous

luminous: crush/CrushWrapper: fix crush tree json dumper

Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoMerge pull request #24532 from dzafman/wip-test-luminous
Yuri Weinstein [Thu, 25 Oct 2018 22:21:46 +0000 (18:21 -0400)]
Merge pull request #24532 from dzafman/wip-test-luminous

luminous: backport and other test fixes for osd-scrub-repair.sh

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
5 years agoMerge pull request #24538 from pdvian/wip-36229-luminous
Yuri Weinstein [Thu, 25 Oct 2018 22:21:09 +0000 (18:21 -0400)]
Merge pull request #24538 from pdvian/wip-36229-luminous

luminous: qa: add test that builds example librados programs

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #24574 from pdvian/wip-36295-luminous
Yuri Weinstein [Thu, 25 Oct 2018 22:20:47 +0000 (18:20 -0400)]
Merge pull request #24574 from pdvian/wip-36295-luminous

luminous: osdc/Objecter: possible race condition with connection reset

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #24582 from smithfarm/wip-36437-luminous
Yuri Weinstein [Thu, 25 Oct 2018 22:20:10 +0000 (18:20 -0400)]
Merge pull request #24582 from smithfarm/wip-36437-luminous

luminous: ec: src/common/interval_map.h: 161: FAILED assert(len > 0)

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #24593 from dzafman/wip-36419
Yuri Weinstein [Thu, 25 Oct 2018 22:19:46 +0000 (18:19 -0400)]
Merge pull request #24593 from dzafman/wip-36419

luminous: osd: get loadavg per cpu for scrub load threshold check

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #24602 from pdvian/wip-36297-luminous
Yuri Weinstein [Thu, 25 Oct 2018 22:19:20 +0000 (18:19 -0400)]
Merge pull request #24602 from pdvian/wip-36297-luminous

luminous: osd: add creating to pg_string_state

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #24659 from tchaikov/luminous-26890
Yuri Weinstein [Thu, 25 Oct 2018 22:18:10 +0000 (18:18 -0400)]
Merge pull request #24659 from tchaikov/luminous-26890

luminous: osd: cast 'whoami' to unsigned so it can be used as the seed for RNG

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
5 years agoMerge commit 'f0d49da1be98291f26f70b524f182eba4fe78cee' into luminous
David Galloway [Thu, 25 Oct 2018 21:47:04 +0000 (17:47 -0400)]
Merge commit 'f0d49da1be98291f26f70b524f182eba4fe78cee' into luminous

5 years agoceph-volume: update man page for batch --prepare 24758/head 24759/head
Andrew Schoen [Thu, 25 Oct 2018 12:51:41 +0000 (07:51 -0500)]
ceph-volume: update man page for batch --prepare

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit faa3aed4ad4df2dbdd853235e364d6e3b9b176b6)

5 years agoceph-volume: docs for --prepare flag of lvm batch
Andrew Schoen [Mon, 15 Oct 2018 15:38:43 +0000 (10:38 -0500)]
ceph-volume: docs for --prepare flag of lvm batch

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit b4825e1bd7b75a0a53ee5678230b262286293d75)

5 years agoceph-volume: when --prepare is added to batch the OSDs are only prepared
Andrew Schoen [Mon, 15 Oct 2018 15:35:12 +0000 (10:35 -0500)]
ceph-volume: when --prepare is added to batch the OSDs are only prepared

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit abd238dc1f33252a43adc8b86efb3d3b4a076964)

5 years agoceph-volume: adds a --prepare flag to ceph-volume lvm batch
Andrew Schoen [Mon, 15 Oct 2018 14:33:30 +0000 (09:33 -0500)]
ceph-volume: adds a --prepare flag to ceph-volume lvm batch

This flag will only prepare the OSDs, not activate them. This is useful
in our containerized ceph solution.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 4368ca7d48f4ce35f67854e8b0ca2be9babc5371)