]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
7 years agoMerge pull request #18741 from shinobu-x/wip-22019-luminous
Kefu Chai [Thu, 9 Nov 2017 14:27:47 +0000 (22:27 +0800)]
Merge pull request #18741 from shinobu-x/wip-22019-luminous

luminous: tests: "ceph osd create" is not idempotent

Reviewed-by: Kefu Chai <kchai@redhat.com>
7 years agoMerge pull request #18800 from linuxbox2/luminous-rgw_file-fsid-mount
Abhishek L [Thu, 9 Nov 2017 09:10:53 +0000 (10:10 +0100)]
Merge pull request #18800 from linuxbox2/luminous-rgw_file-fsid-mount

Luminous rgw file fsid mount -- pullup 1 of 2

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
7 years agoMerge pull request #18767 from smithfarm/wip-22020-luminous
Abhishek L [Thu, 9 Nov 2017 09:04:08 +0000 (10:04 +0100)]
Merge pull request #18767 from smithfarm/wip-22020-luminous

luminous: multisite: race between sync of bucket and bucket instance metadata

Reviewed-By: Casey Bodley <cbodley@redhat.com>
7 years agoMerge pull request #18794 from theanalyst/wip-18709
Abhishek L [Thu, 9 Nov 2017 08:56:16 +0000 (09:56 +0100)]
Merge pull request #18794 from theanalyst/wip-18709

luminous: rgw: Stale bucket index entry remains after object deletion

Reviewed-By: Casey Bodley <cbodley@redhat.com>
7 years agoMerge pull request #18726 from shinobu-x/wip-luminous-22035
Kefu Chai [Wed, 8 Nov 2017 10:19:27 +0000 (18:19 +0800)]
Merge pull request #18726 from shinobu-x/wip-luminous-22035

luminous: Spurious ceph-mgr failovers during mon elections

Reviewed-by: John Spray <john.spray@redhat.com>
7 years agoMerge pull request #18738 from shinobu-x/wip-21547-luminous
Kefu Chai [Wed, 8 Nov 2017 10:18:39 +0000 (18:18 +0800)]
Merge pull request #18738 from shinobu-x/wip-21547-luminous

luminous: ceph-mgr gets process called "exe" after respawn

Reviewed-by: John Spray <john.spray@redhat.com>
7 years agoMerge pull request #18727 from shinobu-x/wip-luminous-22034
Kefu Chai [Wed, 8 Nov 2017 10:17:54 +0000 (18:17 +0800)]
Merge pull request #18727 from shinobu-x/wip-luminous-22034

luminous: key mismatch for mgr after upgrade from jewel to luminous(dev)

Reviewed-by: John Spray <john.spray@redhat.com>
7 years agoMerge pull request #18723 from shinobu-x/wip-luminous-22023
Kefu Chai [Wed, 8 Nov 2017 06:44:41 +0000 (14:44 +0800)]
Merge pull request #18723 from shinobu-x/wip-luminous-22023

luminous: osd: make stat_bytes and stat_bytes_used counters PRIO_USEFUL

7 years agoMerge pull request #18737 from shinobu-x/wip-21549-luminous
Kefu Chai [Wed, 8 Nov 2017 06:38:10 +0000 (14:38 +0800)]
Merge pull request #18737 from shinobu-x/wip-21549-luminous

luminous: the dashboard uses absolute links for filesystems and clients

Reviewed-by: John Spray <john.spray@redhat.com>
7 years agoMerge pull request #18736 from shinobu-x/wip-21638-luminous
Kefu Chai [Wed, 8 Nov 2017 06:37:19 +0000 (14:37 +0800)]
Merge pull request #18736 from shinobu-x/wip-21638-luminous

luminous: dashboard OSD list has servers and osds in arbitrary order

Reviewed-by: John Spray <john.spray@redhat.com>
7 years agoMerge pull request #18728 from shinobu-x/wip-luminous-22032
Kefu Chai [Wed, 8 Nov 2017 06:36:40 +0000 (14:36 +0800)]
Merge pull request #18728 from shinobu-x/wip-luminous-22032

luminous: dashboard barfs on nulls where it expects numbers

Reviewed-by: John Spray <john.spray@redhat.com>
7 years agoMerge pull request #18699 from shinobu-x/wip-luminous-21538
Kefu Chai [Wed, 8 Nov 2017 06:34:35 +0000 (14:34 +0800)]
Merge pull request #18699 from shinobu-x/wip-luminous-21538

luminous: upmap does not respect osd reweights

Reviewed-by: Sage Weil <sage@redhat.com>
7 years agoMerge pull request #18719 from tchaikov/wip-pr-18373-luminous
Kefu Chai [Wed, 8 Nov 2017 06:30:42 +0000 (14:30 +0800)]
Merge pull request #18719 from tchaikov/wip-pr-18373-luminous

luminous: osd,mon: add max-pg-per-osd limit

Reviewed-by: Sage Weil <sage@redhat.com>
7 years agodoc: fix the desc of "osd max pg per osd hard ratio" 18719/head
Kefu Chai [Wed, 18 Oct 2017 08:07:50 +0000 (16:07 +0800)]
doc: fix the desc of "osd max pg per osd hard ratio"

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d940f6220346ad21dfc4f351c57389e8430c0257)

7 years agomon: change "mon_pg_warn_min_per_osd" to uint64_t
Kefu Chai [Tue, 17 Oct 2017 04:17:09 +0000 (12:17 +0800)]
mon: change "mon_pg_warn_min_per_osd" to uint64_t

* this silences a warning from -Wsign-compare.
* also switch all its user to the new-style option

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c3cd0c13a1585388df0f17ab53c170e39c711615)

7 years agoMerge pull request #18702 from liewegas/wip-pr-18670-luminous
Sage Weil [Wed, 8 Nov 2017 04:27:15 +0000 (22:27 -0600)]
Merge pull request #18702 from liewegas/wip-pr-18670-luminous

luminous: qa/tasks/thrashosds: set min_in default to 4

7 years agoMerge pull request #18596 from smithfarm/wip-21955-luminous
Abhishek L [Tue, 7 Nov 2017 21:28:58 +0000 (22:28 +0100)]
Merge pull request #18596 from smithfarm/wip-21955-luminous

luminous: tests: add EC data pool to testing

Reviewed-By: Patrick Donelly <pdonelly@redhat.com>
7 years agorgw_file: introduce rgw_mount2 with a bucket name parameter 18800/head
Gui Hecheng [Sat, 22 Jul 2017 05:12:06 +0000 (13:12 +0800)]
rgw_file: introduce rgw_mount2 with a bucket name parameter

This explicitly allow mount a specified bucket in librgw.
Originally, mounting a bucket is implemented like a sub-directory
mount in nfs-ganesha FSAL_RGW with lookup_path.
With this change, we allow root_fh to points exactly to the root
of a mounted fs instance, which is a bucket or "/" rather than
always let root_fh points to "/".

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>
(cherry picked from commit 860716e17e840df11a7e1d8b6ca0c1ee694b038b)

7 years agorgw_file: new fsid for fs instance
Gui Hecheng [Fri, 26 May 2017 09:45:29 +0000 (17:45 +0800)]
rgw_file: new fsid for fs instance

We use an in-memory fs_inst_counter for fsid currently, but it
inevitably cause nfs automount problem across a nfs-ganesha crash.
So here we relate the fsid to the s3 uid with a hash.
With this, fsid remains consistent across all fs instances and
across crashes.

We should be able to upgrade from old fsid to new fsid with
update_fh().

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>
(cherry picked from commit 178b547e83a2c0eeb024bbe4ca2ff8d8e7d50572)

7 years agoMerge pull request #18628 from batrick/i21953
Abhishek L [Tue, 7 Nov 2017 21:07:45 +0000 (22:07 +0100)]
Merge pull request #18628 from batrick/i21953

luminous: mds: sanitize mdsmap of removed pools

Reviewed-By: Sage Weil <sage@redhat.com>
7 years agoMerge pull request #18626 from dillaman/wip-21968
Abhishek L [Tue, 7 Nov 2017 17:33:21 +0000 (18:33 +0100)]
Merge pull request #18626 from dillaman/wip-21968

luminous: cls/journal: fixed possible infinite loop in expire_tags

Reviewed-By: Mykola Golub <trociny@gmail.com>
7 years agoMerge pull request #18688 from liewegas/wip-balancer-luminous
Abhishek L [Tue, 7 Nov 2017 17:32:20 +0000 (18:32 +0100)]
Merge pull request #18688 from liewegas/wip-balancer-luminous

luminous: mgr/balancer: backport crush-compat work from master

Reviewed-By: John Spray <jspray@redhat.com>
7 years agorgw: fix BZ 1500904, Stale bucket index entry remains after object deletion 18794/head
J. Eric Ivancich [Fri, 3 Nov 2017 13:15:13 +0000 (09:15 -0400)]
rgw: fix BZ 1500904, Stale bucket index entry remains after object deletion

We have a race condition:

 1. RGW client #1: requests an object be deleted.
 2. RGW client #1: sends a prepare op to bucket index OSD #1.
 3. OSD #1:        prepares the op, adding pending ops to the bucket dir entry
 4. RGW client #2: sends a list bucket to OSD #1
 5. RGW client #2: sees that there are pending operations on bucket
                   dir entry, and calls check_disk_state
 6. RGW client #2: check_disk_state sees that the object still exists, so it
                   sends CEPH_RGW_UPDATE to bucket index OSD (#1)
 7. RGW client #1: sends a delete object to object OSD (#2)
 8. OSD #2:        deletes the object
 9. RGW client #2: sends a complete op to bucket index OSD (#1)
10. OSD #1:        completes the op
11. OSD #1:        receives the CEPH_RGW_UPDATE and updates the bucket index
                   entry, thereby **RECREATING** it

Solution implemented:

At step #5 the object's dir entry exists. If we get to beginning of
step #11 and the object's dir entry no longer exists, we know that the
dir entry was just actively being modified, and ignore the
CEPH_RGW_UPDATE operation, thereby NOT recreating it.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit b33f529e79b74314a2030231e1308ee225717743)

7 years agoMerge pull request #18569 from kmroz/wip-21939-luminous
Abhishek L [Tue, 7 Nov 2017 17:21:41 +0000 (18:21 +0100)]
Merge pull request #18569 from kmroz/wip-21939-luminous

luminous: list bucket which enable versioning get wrong result when user marker

Reviewed-By: Casey Bodley <cbodley@redhat.com>
7 years agoMerge pull request #18591 from cbodley/wip-21938
Abhishek L [Tue, 7 Nov 2017 17:21:03 +0000 (18:21 +0100)]
Merge pull request #18591 from cbodley/wip-21938

luminous: rgw: fix error handling in ListBucketIndexesCR
(approved in irc & standups)

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
7 years agoMerge pull request #18539 from linuxbox2/luminous-baixuyeyu-refcnt
Abhishek L [Tue, 7 Nov 2017 17:17:57 +0000 (18:17 +0100)]
Merge pull request #18539 from linuxbox2/luminous-baixuyeyu-refcnt

luminous: baixuyeyu refcnt

Reviewed-By: Casey Bodley <cbodley@redhat.com>
7 years agoMerge pull request #18516 from ceph/wip-bp-pr18193-luminous
Abhishek L [Tue, 7 Nov 2017 17:15:38 +0000 (18:15 +0100)]
Merge pull request #18516 from ceph/wip-bp-pr18193-luminous

qa/rgw: ignore errors from 'pool application enable'

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>
7 years agoMerge pull request #18599 from linuxbox2/lum-rgwfile-21940
Abhishek L [Tue, 7 Nov 2017 17:14:21 +0000 (18:14 +0100)]
Merge pull request #18599 from linuxbox2/lum-rgwfile-21940

rgw_file:  set s->obj_size from bytes_written

Reviewed-By: Casey Bodley <cbodley@redhat.com>
7 years agoMerge pull request #18566 from kmroz/wip-21914-luminous
Abhishek L [Tue, 7 Nov 2017 12:07:09 +0000 (13:07 +0100)]
Merge pull request #18566 from kmroz/wip-21914-luminous

luminous: [rbd-mirror] peer cluster connections should filter out command line optionals

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>
7 years agoosd,mon: change "mon_max_pg_per_osd" to uint64_t
Kefu Chai [Tue, 17 Oct 2017 04:10:55 +0000 (12:10 +0800)]
osd,mon: change "mon_max_pg_per_osd" to uint64_t

as it should never be a negative number. if this option is disabled,
it's 0.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0d68197376b6cf48c6679d77ec9f28f279f3da68)

7 years agoosd: add max-pg-per-osd limit
Kefu Chai [Tue, 26 Sep 2017 07:54:14 +0000 (15:54 +0800)]
osd: add max-pg-per-osd limit

osd will refused to create new pgs, until its pg number is lower
than the max-pg-per-osd upper bound setting.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 4c7df944c7f28232873ba681eedce72cdb062ea5)

7 years agoqa/tasks/ceph: add "create_rbd_pool" option
Kefu Chai [Sun, 15 Oct 2017 12:46:28 +0000 (20:46 +0800)]
qa/tasks/ceph: add "create_rbd_pool" option

True by default, but we can opt not to create the "rbd" pool.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit ac1e13ce63a6f32ae9ab0db8196c8cc8100d94ed)

7 years agoosd: print osd as "osd.$osdid" in log message
Kefu Chai [Thu, 12 Oct 2017 12:07:46 +0000 (20:07 +0800)]
osd: print osd as "osd.$osdid" in log message

easier to read this way

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c6b71ebf6b7b7c54560f331d5f8e4e24f5a60633)

7 years agomon/OSDMonitor: bypass checks if pg_temp.forced
Kefu Chai [Tue, 26 Sep 2017 07:50:08 +0000 (15:50 +0800)]
mon/OSDMonitor: bypass checks if pg_temp.forced

originally, monitor ignores the pg_temp messages sent from replica osds,
but to re-trigger peering if replica osd's pg number drops down below
the setting, we need to bypass the checks.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit da6f7a810055a0dc9e364aca8e6efbfbc32881b3)

7 years agomessages/MOSDPGTemp: add `forced` field
Kefu Chai [Tue, 26 Sep 2017 07:49:11 +0000 (15:49 +0800)]
messages/MOSDPGTemp: add `forced` field

the monitor is supposed to bypass some checks if `forced` is true.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6335bf791003e9e33b2f738ab3da06f3c626537a)

7 years agoMerge pull request #18775 from tchaikov/wip-upgrade-luminous-x
Kefu Chai [Tue, 7 Nov 2017 09:48:34 +0000 (17:48 +0800)]
Merge pull request #18775 from tchaikov/wip-upgrade-luminous-x

qa: add 1-ceph-install to upgrade:luminous-x

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
7 years agoqa: add 1-ceph-install to upgrade:luminous-x 18775/head
Yuri Weinstein [Tue, 29 Aug 2017 16:50:51 +0000 (09:50 -0700)]
qa: add 1-ceph-install to upgrade:luminous-x

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit aff342627cf77e2faae6904e626d9fb2c897f155)

7 years agoqa/suites/rados/thrash/d-*/*balancer*: debug osd in mgr 18688/head
Sage Weil [Tue, 7 Nov 2017 03:17:22 +0000 (21:17 -0600)]
qa/suites/rados/thrash/d-*/*balancer*: debug osd in mgr

Signed-off-by: Sage Weil <sage@redhat.com>
7 years agorgw: remove placement_rule from cls_user_bucket_entry 18767/head
Casey Bodley [Tue, 31 Oct 2017 20:56:01 +0000 (16:56 -0400)]
rgw: remove placement_rule from cls_user_bucket_entry

placement_rule is no longer needed in cls_user_bucket_entry, because the
only time that it's needed, we can read it from the bucket instance in
RGWRados::update_containers_stats()

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8e62e3526643da67f5af7daa687120feed469785)

7 years agorgw: remove placement_rule from rgw_link_bucket()
Casey Bodley [Tue, 31 Oct 2017 20:29:31 +0000 (16:29 -0400)]
rgw: remove placement_rule from rgw_link_bucket()

with the fallback in RGWRados::update_containers_stats(), we no longer
need to pass placement_rule into the cls_user_bucket_entry

this removes the dependency between bucket entrypoint metadata and its
bucket instance metadata during multisite sync

Fixes: http://tracker.ceph.com/issues/21990
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit dd5d278a4f9b080234cbb77e448208803ec0cf93)

7 years agorgw: take placement_rule from bucket info in update_containers_stats
Casey Bodley [Tue, 31 Oct 2017 20:26:54 +0000 (16:26 -0400)]
rgw: take placement_rule from bucket info in update_containers_stats

in case the cls_user_bucket_entry doesn't contain a placement_rule, take
it from the bucket instance info

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 531195d80d5bac774d58b210307eb19b57d26345)

7 years agoqa/suites/rados/thrash: combine balancer with require-luminous
Sage Weil [Sat, 4 Nov 2017 14:32:51 +0000 (09:32 -0500)]
qa/suites/rados/thrash: combine balancer with require-luminous

We can only do the balancer stuff if we've done the mkfs-time
require luminous because otherwise the mgr commands aren't proxied via
the mon.

Only do balancer for thrash tests; not for basic, monthrash, verify.

Signed-off-by: Sage Weil <sage@redhat.com>
7 years agoqa: stop testing deprecated "ceph osd create" 18741/head
Kefu Chai [Wed, 1 Nov 2017 02:45:09 +0000 (10:45 +0800)]
qa: stop testing deprecated "ceph osd create"

"ceph osd create" is not idempotent, and is considered deprecated.

Fixes: http://tracker.ceph.com/issues/21993
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 371d3f40911917704f4992c48236b4c60905bf4e)

7 years agomgr: set explicit thread name 18738/head
John Spray [Fri, 15 Sep 2017 16:23:39 +0000 (12:23 -0400)]
mgr: set explicit thread name

This gets used as our process name in some situations
when respawning.  This is the same as what commit 4f177bb6b
did for the MDS.

Fixes: http://tracker.ceph.com/issues/21404
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 44dce0a5664639b2ae0949cc434f1be1176d872d)

7 years agopybind/mgr/dashboard: updated favicon (old one was 404) 18737/head
Nick Erdmann [Tue, 22 Aug 2017 16:55:11 +0000 (16:55 +0000)]
pybind/mgr/dashboard: updated favicon (old one was 404)

Signed-off-by: Nick Erdmann <n@nirf.de>
(cherry picked from commit 7a54520f5c3d4db27fe7e2a12760dcd838ba89aa)

7 years agopybind/mgr/dashboard: clean up
Nick Erdmann [Tue, 22 Aug 2017 08:08:45 +0000 (08:08 +0000)]
pybind/mgr/dashboard: clean up

Signed-off-by: Nick Erdmann <n@nirf.de>
(cherry picked from commit f512ac3d7712850f274cd01bb589aa9fbc8c8bef)

7 years agopybind/mgr/dashboard: add url_prefix
Nick Erdmann [Mon, 21 Aug 2017 17:21:10 +0000 (17:21 +0000)]
pybind/mgr/dashboard: add url_prefix

This adds a configuration variable url_prefix to the dashboard that
that is prepended to all URLs so you can access the dashboard at
http://$IP:$PORT/$PREFIX/. This is necessary if you wish to use a
reverse http proxy that forwards to the dashboard under a sub-path.

Fixes: http://tracker.ceph.com/issues/20568
Signed-off-by: Nick Erdmann <n@nirf.de>
(cherry picked from commit 6d285fff0df598d66f46d38bf0a6a2cc9c0dc62f)

7 years agomgr/dashboard: sort servers and OSDs in OSD list 18736/head
John Spray [Wed, 27 Sep 2017 14:13:48 +0000 (15:13 +0100)]
mgr/dashboard: sort servers and OSDs in OSD list

Fixes: http://tracker.ceph.com/issues/21572
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit f409099fa25b73fe580ee4662aff51636f118ed2)

7 years agoMerge pull request #18673 from dzafman/wip-21833
Sage Weil [Sat, 4 Nov 2017 15:52:27 +0000 (10:52 -0500)]
Merge pull request #18673 from dzafman/wip-21833

osd: build_past_intervals_parallel: Ignore new partially created PGs

7 years agomgr/dashboard: handle null in format_number 18728/head
John Spray [Wed, 27 Sep 2017 13:33:58 +0000 (14:33 +0100)]
mgr/dashboard: handle null in format_number

Fixes: http://tracker.ceph.com/issues/21570
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit f43859d4dca696ba4cac224e0c7e87e4b2aad40f)

7 years agomon: don't blow away bootstrap-mgr on upgrades 18727/head
John Spray [Thu, 19 Oct 2017 13:28:18 +0000 (09:28 -0400)]
mon: don't blow away bootstrap-mgr on upgrades

Fixes: http://tracker.ceph.com/issues/20950
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 57229ea2a4369518c7a16b7a09b045b7896f5a70)

7 years agomon: handle monitor lag when killing mgrs 18726/head
John Spray [Thu, 12 Oct 2017 10:57:50 +0000 (11:57 +0100)]
mon: handle monitor lag when killing mgrs

Fixes: http://tracker.ceph.com/issues/20629
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 3c3776b30a2da3f5d2ece15c57713c0ce191c778)

7 years agoosd: make stat_bytes and stat_bytes_used counters PRIO_USEFUL 18723/head
Yao Zongyou [Tue, 31 Oct 2017 05:23:49 +0000 (13:23 +0800)]
osd: make stat_bytes and stat_bytes_used counters PRIO_USEFUL

These two counters should be reported to mgr because the dashboard
plugin needing them to display osd detailed information.

Fixes: http://tracker.ceph.com/issues/21981
Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>
(cherry picked from commit 40bb3c5f05b6f2a917a7e400f88677f41a9629a3)

7 years agopybind/mgr/mgr_module: fix calc_pg_upmaps
Sage Weil [Thu, 2 Nov 2017 21:18:03 +0000 (16:18 -0500)]
pybind/mgr/mgr_module: fix calc_pg_upmaps

This was fixed in the object cleanup.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ed442164d52c803916ca43a5ed76047089d3da66)

7 years agomgr/balancer: enable module by default
Sage Weil [Thu, 2 Nov 2017 21:11:26 +0000 (16:11 -0500)]
mgr/balancer: enable module by default

It will still be "off".

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 26710f0a9b176289237a52e5fa5894342ad163dc)

7 years agoqa/tasks/thrashosds: set min_in default to 4 18702/head
Sage Weil [Wed, 1 Nov 2017 13:31:31 +0000 (08:31 -0500)]
qa/tasks/thrashosds: set min_in default to 4

We have EC tests with k=2,m=2, so we need a min of 4.

Fixes: http://tracker.ceph.com/issues/21997
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit d21809b14ea58dc1f44df844e407ebab5a315062)

7 years agoMerge pull request #18446 from theanalyst/wip-21854-luminous
Abhishek L [Fri, 3 Nov 2017 09:26:44 +0000 (10:26 +0100)]
Merge pull request #18446 from theanalyst/wip-21854-luminous

luminous: rgw_file: explicit NFSv3 open() emulation

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Matt Benjamin <mbenjamin@redhat.com>
7 years agoosd: upmap should respect osd reweights 18699/head
Theofilos Mouratidis [Mon, 25 Sep 2017 14:17:47 +0000 (16:17 +0200)]
osd: upmap should respect osd reweights

Modify OSDMap::calc_pg_upmaps to take the osd reweight into account when
computing the size of each OSD.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>
Fixes: http://tracker.ceph.com/issues/21538
(cherry picked from commit b8561cefa705d31954d47723125376a3c087f4f4)

7 years agotest/cli/osdmaptool: test upmap with an out osd
Theofilos Mouratidis [Mon, 25 Sep 2017 14:15:45 +0000 (16:15 +0200)]
test/cli/osdmaptool: test upmap with an out osd

Test upmap with one osd marked out.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>
(cherry picked from commit ff9e024a59d87eda6cbf3be9464ee7ae6f1140d7)

7 years agoosdmaptool: add --mark-out option
Theofilos Mouratidis [Mon, 25 Sep 2017 14:14:09 +0000 (16:14 +0200)]
osdmaptool: add --mark-out option

Add a --mark-down <id> option which sets the reweight for a given
osd id to 0.0.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>
(cherry picked from commit bd75a78187143f750f90fb937984eaa6333159d3)

7 years agoMerge pull request #18439 from theanalyst/wip-21655-luminous
Abhishek L [Thu, 2 Nov 2017 20:23:50 +0000 (21:23 +0100)]
Merge pull request #18439 from theanalyst/wip-21655-luminous

luminous: expose --sync-stats via admin api

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>
7 years agomgr/balancer: simplify pool_info tracking
Sage Weil [Fri, 27 Oct 2017 17:53:52 +0000 (12:53 -0500)]
mgr/balancer: simplify pool_info tracking

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a59a1df85382d559362c51bb45284b61caaca1d4)

7 years agomgr/balancer: less verbose on 'eval' by default; add 'eval-verbose'
Sage Weil [Thu, 26 Oct 2017 03:47:02 +0000 (22:47 -0500)]
mgr/balancer: less verbose on 'eval' by default; add 'eval-verbose'

The verbose output is helpful for debugging and understanding what is
being measured, but most of the time all that you care about is the score.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2c300bc8d7189b464f59c93deb5eb56915df62b8)

7 years agomgr/balancer: fix pg vs object terminology
Sage Weil [Sat, 21 Oct 2017 15:46:27 +0000 (10:46 -0500)]
mgr/balancer: fix pg vs object terminology

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6513e8b0fc2ebcbbbda937ec6d533ea084ef278f)

7 years agomgr/balancer: restrict to time of day
Sage Weil [Tue, 24 Oct 2017 22:12:54 +0000 (17:12 -0500)]
mgr/balancer: restrict to time of day

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3f0b50b50aeb34ed39767289b8475ec5b877cc27)

7 years agomgr/module: adjust osd_weight min step to .005
Sage Weil [Thu, 5 Oct 2017 22:31:06 +0000 (17:31 -0500)]
mgr/module: adjust osd_weight min step to .005

That should be ~1 PG or less on average; no real sense it taking a
step smaller than that!

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 297e2d65abcebe45c6be69a768cf3ea98e1b354d)

7 years agomgr/balancer: if score regresses, take a few more steps
Sage Weil [Thu, 5 Oct 2017 22:26:16 +0000 (17:26 -0500)]
mgr/balancer: if score regresses, take a few more steps

To avoid getting stuck in a local minima, take a few additional
steps if our score drops and see if it gets better.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 70b503495a0ceb7d1aa4f5fce4403a8eff9a03b5)

7 years agomgr/balancer: allow 5% misplaced
Sage Weil [Thu, 5 Oct 2017 22:25:19 +0000 (17:25 -0500)]
mgr/balancer: allow 5% misplaced

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1ad3101818d1855c8eea8386d7682fab2fe9c8b8)

7 years agomgr/balancer: more aggressive steps
Sage Weil [Thu, 5 Oct 2017 22:25:06 +0000 (17:25 -0500)]
mgr/balancer: more aggressive steps

We are smart enough to back off if we misplace too much.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 0dbcf737eaed02b5aa94bc6f27231b4b87f09ce5)

7 years agoqa/suites/rados/thrash/d-balancer: enable balancer in various modes
Sage Weil [Sun, 1 Oct 2017 21:00:06 +0000 (16:00 -0500)]
qa/suites/rados/thrash/d-balancer: enable balancer in various modes

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2c9c18d1ec3d33d385adf2a30dc1d3b8e987f9a5)

7 years agomgr/balancer: crush-compat: phase out osd_weights
Sage Weil [Wed, 27 Sep 2017 21:05:05 +0000 (17:05 -0400)]
mgr/balancer: crush-compat: phase out osd_weights

Phase out the osd_weight values as we optimize the crush compat
weight set.  Allow a small regression in overall score if it means
we are taking a step to remove the osd_weight value.

Fix the 'osd reweightn' invocation.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 861c37fd99a76d5f1665a610224778fcbb41e9e1)

7 years agomgr/balancer: crush_compat: cope with 'out' osds
Sage Weil [Wed, 27 Sep 2017 19:12:54 +0000 (15:12 -0400)]
mgr/balancer: crush_compat: cope with 'out' osds

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c2bba5820a0bf2225dfc9ba39cc67c78083aba11)

7 years agomgr/balancer: stop if we get a perfect score
Sage Weil [Wed, 27 Sep 2017 19:12:36 +0000 (15:12 -0400)]
mgr/balancer: stop if we get a perfect score

..and fix the 'worse' threshold (we're just working around floating
point imprecision; 1.01 was too coarse).

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f1f3014ffcc67200c20e6fb2b2e6e8f1141a0a1d)

7 years agomgr/balancer: more dead code
Sage Weil [Wed, 27 Sep 2017 15:40:36 +0000 (11:40 -0400)]
mgr/balancer: more dead code

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 37bd64405b8a578329b9a903c9a53322ea29caaf)

7 years agomgr/balancer: crush-compat: throttle changes based on max_misplaced
Sage Weil [Wed, 27 Sep 2017 15:27:42 +0000 (11:27 -0400)]
mgr/balancer: crush-compat: throttle changes based on max_misplaced

Take smaller steps if we overshoot max_misplaced.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit df2e008d3388b657d62ea4547e6b1b7a0251aa04)

7 years agomgr/balancer: remove dead code
Sage Weil [Wed, 27 Sep 2017 15:19:30 +0000 (11:19 -0400)]
mgr/balancer: remove dead code

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 020be37fd06a75ec86e05af966be318c2199da9f)

7 years agomgr/balancer: include pg up mapping in MappingState
Sage Weil [Wed, 27 Sep 2017 15:19:23 +0000 (11:19 -0400)]
mgr/balancer: include pg up mapping in MappingState

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f6f2d253c379a6235e810808e7ce3b83cc4e212c)

7 years agomgr/balancer: normalize weight-set weights to sum to target weight
Sage Weil [Tue, 26 Sep 2017 22:36:06 +0000 (18:36 -0400)]
mgr/balancer: normalize weight-set weights to sum to target weight

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 99fcd161bd222a4f0d2ddde0f1a4caecd1a78882)

7 years agomgr/balancer: note root id in Eval
Sage Weil [Tue, 26 Sep 2017 22:35:42 +0000 (18:35 -0400)]
mgr/balancer: note root id in Eval

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 26a7ec0c8a4c215791369d8650cd17d41317aefe)

7 years agomgr/balancer: make crush-compat mode work!
Sage Weil [Tue, 26 Sep 2017 22:00:08 +0000 (18:00 -0400)]
mgr/balancer: make crush-compat mode work!

- it does multiple iterations, like the upmap optimizer.
- it decreases the step size if it isn't improving, in the hope
that it is overshooting the minimum
- debug output is cleaned up a bit (the info level should be
genuinely useful)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit d9a31595ba15de5fda104a0154778e3200fc46a0)

7 years agoMerge pull request #18590 from liewegas/wip-prime-fixes-luminous
Sage Weil [Thu, 2 Nov 2017 19:54:50 +0000 (14:54 -0500)]
Merge pull request #18590 from liewegas/wip-prime-fixes-luminous

luminous: ceph-bluestore-tool: prime-osd-dir: update symlinks instead of bailing

7 years agoMerge pull request #18620 from tchaikov/wip-21527-luminous
Sage Weil [Thu, 2 Nov 2017 19:54:18 +0000 (14:54 -0500)]
Merge pull request #18620 from tchaikov/wip-21527-luminous

luminous: mon/mgr: sync "mgr_command_descs","osd_metadata" and "mgr_metadata" prefixes to new mons

Reviewed-by: Nathan Cutler <ncutler@suse.com>
7 years agoMerge pull request #18621 from tchaikov/wip-21534-luminous
Sage Weil [Thu, 2 Nov 2017 19:53:16 +0000 (14:53 -0500)]
Merge pull request #18621 from tchaikov/wip-21534-luminous

luminous: mon: update get_store_prefixes implementations

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
7 years agoMerge pull request #18623 from liewegas/wip-pr-18585-luminous
Sage Weil [Thu, 2 Nov 2017 19:52:32 +0000 (14:52 -0500)]
Merge pull request #18623 from liewegas/wip-pr-18585-luminous

luminous: ceph-bluestore-tool: prime-osd-dir: update symlinks instead of bailing

Reviewed-by: Kefu Chai <kchai@redhat.com>
7 years agoMerge pull request #18625 from tchaikov/wip-ceph-disk-unlock-dmcrypted-partitions...
Sage Weil [Thu, 2 Nov 2017 19:51:58 +0000 (14:51 -0500)]
Merge pull request #18625 from tchaikov/wip-ceph-disk-unlock-dmcrypted-partitions-luminous

luminous: ceph-disk: unlocks dmcrypted partitions when activating them

Reviewed-by: Nathan Cutler <ncutler@suse.com>
7 years agoMerge pull request #18650 from b-ranto/wip-allow-getattr-luminous
Sage Weil [Thu, 2 Nov 2017 19:51:32 +0000 (14:51 -0500)]
Merge pull request #18650 from b-ranto/wip-allow-getattr-luminous

luminous: selinux: Allow getattr on lnk sysfs files

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
7 years agoMerge pull request #18675 from jcsp/wip-luminous-mgr
Sage Weil [Thu, 2 Nov 2017 19:51:11 +0000 (14:51 -0500)]
Merge pull request #18675 from jcsp/wip-luminous-mgr

luminous: bulk backport of ceph-mgr improvements

7 years agoMerge pull request #18687 from ceph/backport-18656
Alfredo Deza [Thu, 2 Nov 2017 17:39:05 +0000 (13:39 -0400)]
Merge pull request #18687 from ceph/backport-18656

luminous: "ceph-volume: adds functional testing for bluestore"

Reviewed-by: Alfredo Deza <adeza@redhat.com>
7 years agoMerge pull request #18293 from smithfarm/wip-21795-luminous
Kefu Chai [Thu, 2 Nov 2017 17:02:26 +0000 (01:02 +0800)]
Merge pull request #18293 from smithfarm/wip-21795-luminous

luminous: Ubuntu amd64 client can not discover the ubuntu arm64 ceph cluster

Reviewed-by: Kefu Chai <kchai@redhat.com>
7 years agoceph-volume tests: adds objecstore as a testing factor 18687/head
Andrew Schoen [Tue, 31 Oct 2017 20:25:23 +0000 (15:25 -0500)]
ceph-volume tests: adds objecstore as a testing factor

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit f8b114be61d033c853a2e8295b966e30c6a5f721)

7 years agoceph-volume: enable the centos7-bluestore and xenial-bluestore tests
Andrew Schoen [Tue, 31 Oct 2017 20:10:28 +0000 (15:10 -0500)]
ceph-volume: enable the centos7-bluestore and xenial-bluestore tests

This also updates the ansible and testinfra version to match the current
ceph-ansible master branch.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit b49a19a9773599de295197979b498d06ca500cf8)

7 years agoceph-volume: adds the xenial-bluestore testing scenario
Andrew Schoen [Tue, 31 Oct 2017 20:09:52 +0000 (15:09 -0500)]
ceph-volume: adds the xenial-bluestore testing scenario

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 38372890c9c40e6f536026f61c5ef6e6d23d7111)

7 years agoceph-volume: adds the centos7-bluestore testing scenario
Andrew Schoen [Tue, 31 Oct 2017 20:05:46 +0000 (15:05 -0500)]
ceph-volume: adds the centos7-bluestore testing scenario

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 45e4b0c9d7d4fea1d7bd3481d46edf3b42c40d26)

7 years agoqa: fix mgr _load_module helper 18675/head
John Spray [Thu, 2 Nov 2017 11:42:56 +0000 (07:42 -0400)]
qa: fix mgr _load_module helper

I inadvertently broke this with the latest change
to the module ls output.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 4fb3025682d812f86c50fa36532290fc0f8857ae)

7 years agomgr: fix up make_unique usage for backport
John Spray [Wed, 1 Nov 2017 19:10:19 +0000 (19:10 +0000)]
mgr: fix up make_unique usage for backport

This was getting the definition some other way in master,
but in luminous we need to include the backport14 header.

Signed-off-by: John Spray <john.spray@redhat.com>
7 years agomon: fix up mgr_inactive_grace for backport
John Spray [Fri, 20 Oct 2017 12:54:29 +0000 (13:54 +0100)]
mon: fix up mgr_inactive_grace for backport

This was converted to options.cc only, but we cannot
backport another commit that removes this legacy health
reporting code, so this commit updates the legacy health
reporting code for the options.cc style config opt.

Signed-off-by: John Spray <john.spray@redhat.com>
7 years agomon: include disabled modules in `mgr module ls`
John Spray [Wed, 1 Nov 2017 10:35:14 +0000 (06:35 -0400)]
mon: include disabled modules in `mgr module ls`

Otherwise, when someone wants to see what's possible
to do with `mgr module enable` they have to trawl
through the whole mgr map dump.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 5861c4c022e825fe28347ba7e61ae86a6f260f24)

7 years agodoc: describe using `mgr module ...` commands
John Spray [Wed, 1 Nov 2017 10:34:36 +0000 (06:34 -0400)]
doc: describe using `mgr module ...` commands

...including the new "mgr services" command.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit d220e1adc337daeac418563dee125f161e85afdf)

7 years agodoc: describe how to implement standby modules
John Spray [Wed, 1 Nov 2017 10:28:30 +0000 (06:28 -0400)]
doc: describe how to implement standby modules

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit ceb2a91190441b19171e5b5726748e43ee4d1e92)

7 years agoqa: fix mgr caps
John Spray [Mon, 23 Oct 2017 09:37:49 +0000 (05:37 -0400)]
qa: fix mgr caps

This was still using Kraken era settings

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 9988ebed9530718b523a23d729d499dedab5eb6d)