]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
7 years agoosd: Change shard digests to hex like object info digests 21397/head
David Zafman [Fri, 13 Apr 2018 22:01:52 +0000 (15:01 -0700)]
osd: Change shard digests to hex like object info digests

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 22ddc6da5fd03e65a7baf96748a8893cc258d977)

7 years agotest: Luminous backport specific changes
David Zafman [Wed, 11 Apr 2018 15:43:56 +0000 (08:43 -0700)]
test: Luminous backport specific changes

osd-scrub-repair.sh:
    Remove legacy_snaps and redirect_target to keep test more in sync with master
    We don't have omap_digest bluestore handling
        Additional cases of omap_digest_mismatch_oi changes
        omap_digest still set
        ROBJ10 output still applies
osd-scrub-snaps.sh:
    Add head_exists in snapset
    Add snapset for object with head_mismatch

Signed-off-by: David Zafman <dzafman@redhat.com>
7 years agotest: Use jq in a compatible way and for easier diff analysis
David Zafman [Mon, 16 Apr 2018 04:49:48 +0000 (21:49 -0700)]
test: Use jq in a compatible way and for easier diff analysis

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 458babe7ee2e5fc0c4519a94249cbc1bd7cec9b6)

7 years agodoc: Add release notes for the output format changes
David Zafman [Tue, 10 Apr 2018 18:07:48 +0000 (11:07 -0700)]
doc: Add release notes for the output format changes

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit da0f5887a49363d2d7f2f0a5aac34f5992648b06)

Conflicts:
PendingReleaseNotes (trivial)

7 years agotest: Be smarter about when jsonschema can be used
David Zafman [Mon, 9 Apr 2018 22:09:38 +0000 (15:09 -0700)]
test: Be smarter about when jsonschema can be used

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 9c5ef19f93d999c50641b8bf464b462e0ef5e8f2)

7 years agoosd rados command: Show snapset in list-inconsistent-snapset
David Zafman [Wed, 4 Apr 2018 23:09:39 +0000 (16:09 -0700)]
osd rados command: Show snapset in list-inconsistent-snapset

Add SnapSet bufferlist to inconsistent_snapset_t

Partial fix for http://tracker.ceph.com/issues/23428

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 60ae2b8eb3ef73e9eb175340340681f48beae48d)

Conflicts:
src/common/scrub_types.cc (uses ::decode and ::encode)
src/osd/PrimaryLogPG.cc (trivial)

7 years agoosd: SnapSet::dump() handle snapid_t not in clone_overlap or clone_size
David Zafman [Wed, 4 Apr 2018 23:07:09 +0000 (16:07 -0700)]
osd: SnapSet::dump() handle snapid_t not in clone_overlap or clone_size

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 866a52e5673e296a66dfa9407dcf9375f80b7f73)

7 years agotest: Add getjson variable to save output
David Zafman [Wed, 4 Apr 2018 23:04:41 +0000 (16:04 -0700)]
test: Add getjson variable to save output

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 1b1d45bf517237b5221581c3dfb1a7e5fccf712f)

7 years agoosd rados command: Change error name snapset_mismatch to snapset_error
David Zafman [Wed, 4 Apr 2018 22:58:57 +0000 (15:58 -0700)]
osd rados command: Change error name snapset_mismatch to snapset_error

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 007cb45fe5c8fd1ffb8db2422a11d67d74bbf5b5)

Conflicts:
doc/rados/command/list-inconsistent-snap.json (trivial)
src/common/scrub_types.h (trivial)
src/include/rados/rados_types.hpp (trivial)
src/tools/rados/rados.cc (trivial)

7 years agotest: Clean-up test and use local values for number of objects and osds
David Zafman [Tue, 20 Mar 2018 23:07:49 +0000 (16:07 -0700)]
test: Clean-up test and use local values for number of objects and osds

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0c7ac9db3bbb07eaa781f2cea1206ea9a8724cc5)

7 years agoosd rados command: list-inconsistent-obj attribute improvements
David Zafman [Fri, 16 Mar 2018 06:59:52 +0000 (23:59 -0700)]
osd rados command: list-inconsistent-obj attribute improvements

System attributes shown as "object_info", "snapset" and "hashinfo"
Only output user attributes as "attrs"
Drop leading undescore "_" for user attribute keys
Improve logic as to when to show user attributes or specific system attributes

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 982509514c0c990dcdf7f35725dece61714bfa59)

Conflicts:
qa/standalone/scrub/osd-scrub-repair.sh (trivial)
src/osd/osd_types.cc (trivial)
src/tools/rados/rados.cc (uses ::decode())

7 years agoosd rados command: Change "oi" to "info" in scrub handling errors
David Zafman [Fri, 16 Mar 2018 04:50:30 +0000 (21:50 -0700)]
osd rados command: Change "oi" to "info" in scrub handling errors

data_digest_mismatch_oi -> data_digest_mismatch_info
omap_digest_mismatch_oi -> omap_digest_mismatch_info
size_mismatch_oi -> size_mismatch_info
obj_size_oi_mismatch -> obj_size_info_mismatch

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 01687b052fdeec6dd2a4798b1e637fc8d15702c7)

7 years agoosd rados command: Change "oi_attr" to "info" in scrub handling errors
David Zafman [Fri, 16 Mar 2018 00:09:43 +0000 (17:09 -0700)]
osd rados command: Change "oi_attr" to "info" in scrub handling errors

oi_attr_missing -> info_missing
oi_attr_corrupted -> info_corrupted

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 273f6213ea9f637de896f4244622d0e8e4847bad)

7 years agoosd rados command: Rename ss_attr_missing/ss_attr_corrupted to snapset_missing/snapse...
David Zafman [Thu, 15 Mar 2018 21:29:54 +0000 (14:29 -0700)]
osd rados command: Rename ss_attr_missing/ss_attr_corrupted to snapset_missing/snapset_corrupted

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit bec67e3d4095484970ad3376a9b2613d07a6606f)

Conflicts:
qa/standalone/scrub/osd-scrub-snaps.sh (still have head_mismatch due to snapdir)

7 years agoosd rados command: Improve scrub handling of HashInfo (hinfo_key xattr)
David Zafman [Wed, 14 Mar 2018 23:07:01 +0000 (16:07 -0700)]
osd rados command: Improve scrub handling of HashInfo (hinfo_key xattr)

Fixes: http://tracker.ceph.com/issues/23364
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit d713c7dad0adf8605c14cd8181c1b2c09b20fb8c)

7 years agoosd: Don't crash in get_hash_info() with a corrupt HashInfo
David Zafman [Wed, 14 Mar 2018 23:29:18 +0000 (16:29 -0700)]
osd: Don't crash in get_hash_info() with a corrupt HashInfo

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0d4acc55ffd7f0f45324d1c933a2a5d0d47b84ec)

Conflicts:
src/osd/ECBackend.cc (Uses ::decode())

7 years agotest: Remove check that masks differences (let diff fail)
David Zafman [Wed, 4 Apr 2018 23:05:08 +0000 (16:05 -0700)]
test: Remove check that masks differences (let diff fail)

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit be815f9b2b9ded0a0dad19f88488d79ce5efff07)

Conflicts:
qa/standalone/scrub/osd-scrub-snaps.sh (trivial)

7 years agoMerge pull request #21103 from dzafman/wip-23267-luminous
David Zafman [Wed, 11 Apr 2018 14:50:12 +0000 (07:50 -0700)]
Merge pull request #21103 from dzafman/wip-23267-luminous

luminous: scrub errors not cleared on replicas can cause inconsistent pg state when replica takes over primary

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
7 years agoMerge pull request #21335 from ceph/backport-rm23644
Andrew Schoen [Wed, 11 Apr 2018 14:47:44 +0000 (16:47 +0200)]
Merge pull request #21335 from ceph/backport-rm23644

luminous: ceph-volume lvm.activate error if no bluestore OSDs are found

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
7 years agoosd: Fix stale scrub stats when a primary takes over 21103/head
David Zafman [Wed, 28 Mar 2018 17:21:39 +0000 (10:21 -0700)]
osd: Fix stale scrub stats when a primary takes over

Fixes: http://tracker.ceph.com/issues/23267
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5cfb8241f482ed53c63bb97262425b2acb733d7d)

7 years agotest: Eliminate use of bc (use awk) in get_timeout_delays()
David Zafman [Tue, 27 Mar 2018 00:27:19 +0000 (17:27 -0700)]
test: Eliminate use of bc (use awk) in get_timeout_delays()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit ce9c029858d8deb249afe8effd5d80ae623f69a2)

7 years agoosd: Improve object_stat_sum_t handling
David Zafman [Fri, 23 Mar 2018 00:17:42 +0000 (17:17 -0700)]
osd: Improve object_stat_sum_t handling

Keep post split PGs as inconsistent from original
Keep num_scrub_errors as total of other 2

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 39adde2655ebb40ad5c010fa77509acc35b25b91)

Conflicts:
src/osd/osd_types.h (trivial)

7 years agoMerge pull request #21124 from smithfarm/wip-23507-luminous
Yuri Weinstein [Tue, 10 Apr 2018 19:56:37 +0000 (12:56 -0700)]
Merge pull request #21124 from smithfarm/wip-23507-luminous

luminous: test_admin_socket.sh may fail on wait_for_clean

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>
7 years agoMerge pull request #21192 from pdvian/wip-23524-luminous
Yuri Weinstein [Tue, 10 Apr 2018 19:55:56 +0000 (12:55 -0700)]
Merge pull request #21192 from pdvian/wip-23524-luminous

luminous: is_qemu_running in qemu_rebuild_object_map.sh and qemu_dynamic_features.sh may return false positive

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
7 years agoMerge pull request #21230 from pdvian/wip-23542-luminous
Yuri Weinstein [Tue, 10 Apr 2018 19:55:26 +0000 (12:55 -0700)]
Merge pull request #21230 from pdvian/wip-23542-luminous

luminous: rbd-nbd: fix ebusy when do map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
7 years agoceph-volume lvm.activate error if no bluestore OSDs are found 21335/head
Alfredo Deza [Tue, 10 Apr 2018 00:00:27 +0000 (20:00 -0400)]
ceph-volume lvm.activate error if no bluestore OSDs are found

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b1920d6e731f194c4829810d849dba6b087ce3a4)

7 years agoMerge pull request #20431 from pdvian/wip-22967-luminous
Yuri Weinstein [Mon, 9 Apr 2018 23:17:27 +0000 (16:17 -0700)]
Merge pull request #20431 from pdvian/wip-22967-luminous

luminous: osdc/Journaler: make sure flush() writes enough data

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
7 years agoMerge pull request #21173 from smithfarm/wip-22383-luminous
Yuri Weinstein [Mon, 9 Apr 2018 23:16:54 +0000 (16:16 -0700)]
Merge pull request #21173 from smithfarm/wip-22383-luminous

luminous: qa: src/test/libcephfs/test.cc:376: Expected: (len) > (0), actual: -34 vs 0

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
7 years agoMerge pull request #21176 from smithfarm/wip-23154-luminous
Yuri Weinstein [Mon, 9 Apr 2018 23:15:58 +0000 (16:15 -0700)]
Merge pull request #21176 from smithfarm/wip-23154-luminous

luminous: mds: FAILED assert (p != active_requests.end()) in MDRequestRef MDCache::request_get(metareqid_t)

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
7 years agoMerge PR #21267 into luminous
Patrick Donnelly [Mon, 9 Apr 2018 22:51:50 +0000 (15:51 -0700)]
Merge PR #21267 into luminous

* refs/pull/21267/head:
discard the mdsload clear after prep_rebalance in case we want to export it for debugging
make sure that MDBalancer uses heartbeat info from the same epoch

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
7 years agoMerge PR #21266 into luminous
Patrick Donnelly [Mon, 9 Apr 2018 22:51:44 +0000 (15:51 -0700)]
Merge PR #21266 into luminous

* refs/pull/21266/head:
make popular counter decay at proper rate

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
7 years agoMerge PR #21256 into luminous
Patrick Donnelly [Mon, 9 Apr 2018 22:51:38 +0000 (15:51 -0700)]
Merge PR #21256 into luminous

* refs/pull/21256/head:
mds: bump mds_log_max_segments for trim buffer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
7 years agoMerge pull request #20885 from mslovy/fix-bug-21811
Nathan Cutler [Sun, 8 Apr 2018 17:05:24 +0000 (19:05 +0200)]
Merge pull request #20885 from mslovy/fix-bug-21811

luminous: rgw: Admin API Support for bucket quota change

Reviewed-by: Casey Bodley <cbodley@redhat.com>
7 years agoMerge pull request #21048 from cbodley/wip-qa-swift-force-branch-luminous
Nathan Cutler [Sun, 8 Apr 2018 17:03:47 +0000 (19:03 +0200)]
Merge pull request #21048 from cbodley/wip-qa-swift-force-branch-luminous

luminous: tests: rgw: swift tests target ceph-luminous branch

Reviewed-by: Nathan Cutler <ncutler@suse.com>
7 years agoMerge pull request #20890 from pdvian/wip-23312-luminous
Yuri Weinstein [Fri, 6 Apr 2018 19:28:34 +0000 (12:28 -0700)]
Merge pull request #20890 from pdvian/wip-23312-luminous

luminous: invalid JSON returned when querying pool parameters

Reviewed-by: Sage Weil <sage@redhat.com>
7 years agoMerge pull request #20998 from pdvian/wip-23412-luminous
Yuri Weinstein [Fri, 6 Apr 2018 19:28:09 +0000 (12:28 -0700)]
Merge pull request #20998 from pdvian/wip-23412-luminous

luminous: log: Fix AddressSanitizer: new-delete-type-mismatch

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
7 years agoMerge pull request #21053 from pdvian/wip-23224-luminous
Yuri Weinstein [Fri, 6 Apr 2018 19:27:29 +0000 (12:27 -0700)]
Merge pull request #21053 from pdvian/wip-23224-luminous

luminous: mgr: quieten logging on missing OSD stats

Reviewed-by: Sage Weil <sage@redhat.com>
7 years agoMerge pull request #21059 from pdvian/wip-23226-luminous
Yuri Weinstein [Fri, 6 Apr 2018 19:26:59 +0000 (12:26 -0700)]
Merge pull request #21059 from pdvian/wip-23226-luminous

luminous: os/bluestore: trim cache every 50ms (instead of 200ms)

Reviewed-by: Sage Weil <sage@redhat.com>
7 years agoMerge pull request #21090 from tchaikov/wip-23477-luminous
Yuri Weinstein [Fri, 6 Apr 2018 19:26:18 +0000 (12:26 -0700)]
Merge pull request #21090 from tchaikov/wip-23477-luminous

luminous: common: should not check for VERSION_ID

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
7 years agoMerge pull request #20675 from rzarzynski/wip-bs-no-blobrefing-in-do_read-luminous
Yuri Weinstein [Fri, 6 Apr 2018 19:25:18 +0000 (12:25 -0700)]
Merge pull request #20675 from rzarzynski/wip-bs-no-blobrefing-in-do_read-luminous

luminous: os/bluestore: avoid unneeded BlobRefing in _do_read().

Reviewed-by: Sage Weil <sage@redhat.com>
7 years agoMerge pull request #20676 from rzarzynski/wip-bs-dump-onode-by-ref-luminous
Yuri Weinstein [Fri, 6 Apr 2018 19:24:58 +0000 (12:24 -0700)]
Merge pull request #20676 from rzarzynski/wip-bs-dump-onode-by-ref-luminous

luminous: os/bluestore: _dump_onode() don't prolongate Onode anymore.

Reviewed-by: Sage Weil <sage@redhat.com>
7 years agoMerge pull request #20712 from pdvian/wip-23175-luminous
Yuri Weinstein [Fri, 6 Apr 2018 19:24:25 +0000 (12:24 -0700)]
Merge pull request #20712 from pdvian/wip-23175-luminous

luminous: mgr: die on bind() failure

Reviewed-by: Sage Weil <sage@redhat.com>
7 years agoMerge pull request #20867 from pdvian/wip-23306-luminous
Yuri Weinstein [Fri, 6 Apr 2018 19:24:01 +0000 (12:24 -0700)]
Merge pull request #20867 from pdvian/wip-23306-luminous

luminous: msg/async: size of EventCenter::file_events should be greater than fd.

Reviewed-by: Sage Weil <sage@redhat.com>
7 years agoMerge PR #21198 into luminous
Patrick Donnelly [Fri, 6 Apr 2018 19:10:15 +0000 (12:10 -0700)]
Merge PR #21198 into luminous

* refs/pull/21198/head:
ceph_authtool: add mode option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
7 years agoMerge pull request #20642 from b-ranto/wip-mgr-prom-luminous
Yuri Weinstein [Fri, 6 Apr 2018 15:59:07 +0000 (08:59 -0700)]
Merge pull request #20642 from b-ranto/wip-mgr-prom-luminous

[luminous] Backport recent prometheus exporter changes

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
7 years agoMerge pull request #20668 from pdvian/wip-23160-luminous
Yuri Weinstein [Fri, 6 Apr 2018 15:57:28 +0000 (08:57 -0700)]
Merge pull request #20668 from pdvian/wip-23160-luminous

luminous: osd: Remove partially created pg known as DNE

Reviewed-by: David Zafman <dzafman@redhat.com>
7 years agoMerge pull request #20710 from pdvian/wip-23174-luminous
Yuri Weinstein [Fri, 6 Apr 2018 15:56:19 +0000 (08:56 -0700)]
Merge pull request #20710 from pdvian/wip-23174-luminous

luminous: common/dns_resolv.cc: Query for AAAA-record if ms_bind_ipv6 is True

Reviewed-by: Wido den Hollander <wido@widodh.nl>
7 years agoMerge pull request #20766 from t19903/wip-cache-tier-forward-hang
Yuri Weinstein [Fri, 6 Apr 2018 15:55:03 +0000 (08:55 -0700)]
Merge pull request #20766 from t19903/wip-cache-tier-forward-hang

luminous: Objecter: add ignore overlay flag if got redirect reply

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
7 years agoMerge pull request #21079 from tchaikov/wip-23471-luminous
Yuri Weinstein [Fri, 6 Apr 2018 15:52:59 +0000 (08:52 -0700)]
Merge pull request #21079 from tchaikov/wip-23471-luminous

luminous: add --add-bucket and --move options to crushtool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
7 years agoqa/suites/rados/singleton/diverget_priors*: broaden whitelist
Sage Weil [Wed, 30 Aug 2017 19:13:08 +0000 (15:13 -0400)]
qa/suites/rados/singleton/diverget_priors*: broaden whitelist

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 39e5efbad2c2834acf8e6463258b148321ea85a4)

7 years agodiscard the mdsload clear after prep_rebalance in case we want to export it for debugging 21267/head
Jianyu Li [Fri, 17 Nov 2017 07:08:35 +0000 (15:08 +0800)]
discard the mdsload clear after prep_rebalance in case we want to export it for debugging

Signed-off-by: Jianyu Li <joannyli@tencent.com>
(cherry picked from commit 513f2e57666bd08242fc639ebc5c65db12742a01)

7 years agomake sure that MDBalancer uses heartbeat info from the same epoch
Jianyu Li [Wed, 15 Nov 2017 07:43:44 +0000 (15:43 +0800)]
make sure that MDBalancer uses heartbeat info from the same epoch

Currently mds saves the heartbeat info from others in mds_load, once the mds_load.size( ) equals mds number, it considers that have received all heartbeats info and start the rebalance work. However, after prep_rebalance returns, it doesn't clear the mds_load immediately, but wait until receives the next round hearbeat from mds0. If there are mutiple mds(e.g. greater than 2), there is a chance for one mds receiving the first next round heartbeat other than mds0 due to the network delay.

Signed-off-by: Jianyu Li <joannyli@tencent.com>
(cherry picked from commit 591fd4970863ae51dad5e16be3cef24c1c6c92d6)

7 years agomake popular counter decay at proper rate 21266/head
Jianyu Li [Fri, 3 Nov 2017 08:48:10 +0000 (16:48 +0800)]
make popular counter decay at proper rate

Signed-off-by: Jianyu Li <joannyli@tencent.com>
(cherry picked from commit 6d4709e649beacb5f2d6197d1bea57c8590c5f71)

7 years agoMerge pull request #20762 from pdvian/wip-23180-luminous
Yuri Weinstein [Thu, 5 Apr 2018 20:21:53 +0000 (13:21 -0700)]
Merge pull request #20762 from pdvian/wip-23180-luminous

luminous: radosgw-admin data sync run crashes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
7 years agoMerge pull request #20953 from pdvian/wip-23347-luminous
Yuri Weinstein [Thu, 5 Apr 2018 20:20:36 +0000 (13:20 -0700)]
Merge pull request #20953 from pdvian/wip-23347-luminous

luminous: rgw: fix memory fragmentation problem reading data from client.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
7 years agoMerge pull request #21054 from pdvian/wip-23225-luminous
Yuri Weinstein [Thu, 5 Apr 2018 20:16:52 +0000 (13:16 -0700)]
Merge pull request #21054 from pdvian/wip-23225-luminous

luminous: cls/rgw: fix bi_log_iterate_entries return wrong truncated

Reviewed-by: Casey Bodley <cbodley@redhat.com>
7 years agoMerge pull request #20722 from theanalyst/wip-luminous-23221
Yuri Weinstein [Thu, 5 Apr 2018 20:09:32 +0000 (13:09 -0700)]
Merge pull request #20722 from theanalyst/wip-luminous-23221

luminous: fixes for openssl & libcurl

Reviewed-by: Casey Bodley <cbodley@redhat.com>
7 years agoMerge pull request #21214 from smithfarm/wip-23025-luminous
Yuri Weinstein [Thu, 5 Apr 2018 20:07:26 +0000 (13:07 -0700)]
Merge pull request #21214 from smithfarm/wip-23025-luminous

luminous: rgw: data sync of versioned objects, note updating bi marker

Reviewed-by: Casey Bodley <cbodley@redhat.com>
7 years agoMerge pull request #20896 from smithfarm/wip-23349-luminous
Abhishek L [Thu, 5 Apr 2018 17:21:01 +0000 (19:21 +0200)]
Merge pull request #20896 from smithfarm/wip-23349-luminous

luminous: Couldn't init storage provider (RADOS)

Reviewed-By: Sage Weil <sage@redhat.com>
7 years agoMerge pull request #20250 from smithfarm/wip-22856-luminous
Abhishek L [Thu, 5 Apr 2018 17:20:20 +0000 (19:20 +0200)]
Merge pull request #20250 from smithfarm/wip-22856-luminous

luminous: build Debian installation packages failure

Reviewed-By: Sage Weil <sage@redhat.com>
7 years agoMerge pull request #20737 from pdvian/wip-23178-luminous
Yuri Weinstein [Thu, 5 Apr 2018 17:00:44 +0000 (10:00 -0700)]
Merge pull request #20737 from pdvian/wip-23178-luminous

luminous: build/ops: run-make-check.sh: fix SUSE support

Reviewed-by: Nathan Cutler <ncutler@suse.com>
7 years agoMerge pull request #20748 from smithfarm/wip-jinja2-luminous
Yuri Weinstein [Thu, 5 Apr 2018 16:58:31 +0000 (09:58 -0700)]
Merge pull request #20748 from smithfarm/wip-jinja2-luminous

build/ops: deb: move python-jinja2 dependency to mgr

Reviewed-by: John Spray <john.spray@redhat.com>
7 years agoMerge pull request #21016 from pdvian/wip-23077-luminous
Yuri Weinstein [Thu, 5 Apr 2018 16:56:59 +0000 (09:56 -0700)]
Merge pull request #21016 from pdvian/wip-23077-luminous

luminous: mon: ops get stuck in "resend forwarded message to leader"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
7 years agoMerge pull request #21111 from wido/luminous-21813
Yuri Weinstein [Thu, 5 Apr 2018 16:55:41 +0000 (09:55 -0700)]
Merge pull request #21111 from wido/luminous-21813

luminous: common/ipaddr: Do not select link-local IPv6 addresses

Reviewed-by: Kefu Chai <kchai@redhat.com>
7 years agoMerge pull request #21118 from smithfarm/wip-23500-luminous
Yuri Weinstein [Thu, 5 Apr 2018 16:54:54 +0000 (09:54 -0700)]
Merge pull request #21118 from smithfarm/wip-23500-luminous

luminous: snapmapper inconsistency, crash on luminous

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
7 years agoMerge pull request #21134 from smithfarm/wip-23522-luminous
Yuri Weinstein [Thu, 5 Apr 2018 16:54:00 +0000 (09:54 -0700)]
Merge pull request #21134 from smithfarm/wip-23522-luminous

luminous: tests: unittest_pglog timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>
7 years agomds: bump mds_log_max_segments for trim buffer 21256/head
Patrick Donnelly [Wed, 4 Apr 2018 23:47:06 +0000 (16:47 -0700)]
mds: bump mds_log_max_segments for trim buffer

Under create heavy workloads, the MDS will sometimes get behind trimming but
catch up. This avoids unnecessary warnings.

Fixes: http://tracker.ceph.com/issues/23560
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 86e76d15f3b3c06e96d7b27bb01d5840d311cf5d)

Conflicts:
    src/common/options.cc

7 years agotests: swift.py: clone the ceph-luminous branch 21048/head
Nathan Cutler [Sun, 25 Jun 2017 07:27:47 +0000 (09:27 +0200)]
tests: swift.py: clone the ceph-luminous branch

The master branch of ceph/swift.git contains tests that are incompatible with
Jewel and Luminous. The ceph-luminous branch omits these tests.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit a86ce728954a765797ce634025d43650d990e480)

Conflicts: qa/tasks/swift.py (cherry-picked jewel version to luminous)

7 years agodoc: Add usage doc for individual bucket quota Admin API 20885/head
Jeegn Chen [Thu, 4 Jan 2018 07:59:00 +0000 (07:59 +0000)]
doc: Add usage doc for individual bucket quota Admin API

Add usage doc for a new Admin API for individual bucket quota management

Fixes: http://tracker.ceph.com/issues/21811
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>
(cherry picked from commit 0cbd26e88de3c9c71c1ca4532308502f3a8c7ff8)

7 years agorgw: Add URL format support for bucket quota modification
Jeegn Chen [Thu, 14 Dec 2017 08:08:49 +0000 (08:08 +0000)]
rgw: Add URL format support for bucket quota modification

Allow admin to change the quota of a individual bucket through Admin API
with params in URL

Fixes: http://tracker.ceph.com/issues/21811
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>
(cherry picked from commit 46b1186f144c0928394835301fc993d89598398b)

7 years agorbd-nbd: fix ebusy when do map 21230/head
Li Wang [Thu, 7 Dec 2017 14:03:45 +0000 (22:03 +0800)]
rbd-nbd: fix ebusy when do map

When doing rbd-nbd map, if the Ceph service is not available,
the codes will wait on rados.connect(), unless killing the process.
In that case, the close_nbd logic is skipped with NBD_CLEAR_SOCK ioctl
not called. On the CentOS 7 kernel, it leaves nbd->file not cleared, which
causes the subsequent map requests return EBUSY, this patch fixes it
by connecting Ceph first prior to calling NBD_SET_SOCK ioctl

Fixes: http://tracker.ceph.com/issues/23528
Signed-off-by: Li Wang <laurence.liwang@gmail.com>
(cherry picked from commit ab77dcc0170c0d63795fe0d50427cda630bfd593)

7 years agopybind/mgr/prometheus: extend daemon metadata with hostname and ceph version 20642/head
Jan Fajerski [Tue, 27 Feb 2018 09:19:39 +0000 (10:19 +0100)]
pybind/mgr/prometheus: extend daemon metadata with hostname and ceph version

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit d03c79676b1761b016ceaed82f7e3f03f99cda73)

7 years agopybind/mgr/prometheus: add fs and mds metadata metrics
Jan Fajerski [Tue, 20 Feb 2018 10:06:19 +0000 (11:06 +0100)]
pybind/mgr/prometheus: add fs and mds metadata metrics

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 0095e6a1ba0fb9473cfb60b6977498f2c1159079)

7 years agopybind/mgr/prometheus: new mon metrics
Jan Fajerski [Mon, 19 Feb 2018 11:25:02 +0000 (12:25 +0100)]
pybind/mgr/prometheus: new mon metrics

Remove mon_quorum_count and replace it with per-MON quorum status
(mon_quorum_status). Also add mon_metadata metrics.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 71a6408b1adfedab64208fe1a23231df43960924)

7 years agopybind/mgr/prometheus: add Metrics class to manage Metric instances
Jan Fajerski [Sat, 17 Feb 2018 11:07:46 +0000 (12:07 +0100)]
pybind/mgr/prometheus: add Metrics class to manage Metric instances

The central change of this commit is that per-daemon metrics are now
managed by first appending the metric (using Metrics.append) to a
staging area. Then the metrics for specific paths (metric names) are
overwritten by the staged metrics (by calling Metrics.reset). This gets
rid of metrics from daemon that are no longer in the cluster. I.e. when
ceph no longer reports metrics for one OSD daemon (because it was
removed from the cluster) the prometheus module will no longer export
metrics for that daemon.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit f88877ab0bd895d8b67222225c22a12d479b7548)

7 years agopybind/mgr/prometheus: don't crash on OSDs without metadata
Christopher Blum [Fri, 23 Feb 2018 17:48:49 +0000 (18:48 +0100)]
pybind/mgr/prometheus: don't crash on OSDs without metadata

Fix issue where the ceph_exporter crashes after a Ceph upgrade with a broken OSD - that OSD was never online with Luminous and thus we have no metadata for it

Signed-off-by: Christopher Blum zeichenanonym@web.de
(cherry picked from commit 3beee663f6fd6a16cef5a65ea5f9937cbea7a537)

7 years agopybing/mgr/prometheus: tidy up cherrypy engine start and stop
Jan Fajerski [Thu, 30 Nov 2017 10:27:04 +0000 (11:27 +0100)]
pybing/mgr/prometheus: tidy up cherrypy engine start and stop

Calling cherrypy.engine.block() in the stanby module results in a failing
mgr failover.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit ff471d485cd2068b6425b74a17875d8d6a823595)

7 years agopybind/mgr/prometheus: return 503 if MON cluster is down
Jan Fajerski [Wed, 29 Nov 2017 11:23:55 +0000 (12:23 +0100)]
pybind/mgr/prometheus: return 503 if MON cluster is down

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit eda9f15dd939de3e7205ceb0dfcc9fcfce7d89bb)

7 years agopybind/mgr/prometheus: add StandbyModule; return empty answer
Jan Fajerski [Wed, 29 Nov 2017 11:23:18 +0000 (12:23 +0100)]
pybind/mgr/prometheus: add StandbyModule; return empty answer

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 18ffd7529b0975897121d8a5c37d0d20560a1fc7)

7 years agomgr/prometheus: Expose OSD Flags
Boris Ranto [Sat, 17 Feb 2018 03:45:01 +0000 (04:45 +0100)]
mgr/prometheus: Expose OSD Flags

Signed-off-by: Boris Ranto <branto@redhat.com>
7 years agomgr/prometheus: Fix pg_* counts
Boris Ranto [Fri, 16 Feb 2018 17:45:58 +0000 (18:45 +0100)]
mgr/prometheus: Fix pg_* counts

Currently, the pg_* counts are not computed properly. We split the
current state by '+' sign but do not add the pg count to the already
found pg count. Instead, we overwrite any existing pg count with the new
count. This patch fixes it by adding all the pg counts together for all
the states.

It also introduces a new pg_total metric for pg_total that shows the
total count of PGs.

Signed-off-by: Boris Ranto <branto@redhat.com>
7 years agomgr/prometheus: Skip bogus entries
Boris Ranto [Fri, 16 Feb 2018 03:53:39 +0000 (04:53 +0100)]
mgr/prometheus: Skip bogus entries

The osd data can contain bogus '-' entries, skip these when populating
osd metadata and disk occupation.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 965aaad5273eac311e4535559a0e0a9df8a39bee)

7 years agorgw: log unlink_instance mtime as object's mtime 21214/head
Yehuda Sadeh [Wed, 15 Feb 2017 00:18:53 +0000 (16:18 -0800)]
rgw: log unlink_instance mtime as object's mtime

This is needed so that when squashing entries during data
sync we get a consistent mtime for that object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit bef22609de95843008b5d02eb075252bb97fbe90)

7 years agoceph_authtool: add mode option 21198/head
Sébastien Han [Fri, 23 Jun 2017 13:45:57 +0000 (15:45 +0200)]
ceph_authtool: add mode option

We now have the ability to set the keyring file mode with the help of
'--mode MODE'. The mode needs to be specified in octal using the format:
0600.

Closes: http://tracker.ceph.com/issues/23513
Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit b179cd275526613a3838cb52e6f1c35950b6f3a8)

7 years agocommon: FreeBSD wants the correct struct selection for ipv6 21111/head
Willem Jan Withagen [Fri, 30 Mar 2018 14:53:48 +0000 (16:53 +0200)]
common: FreeBSD wants the correct struct selection for ipv6

Lets see if this also works for Linux

Fixes: http://tracker.ceph.com/issues/21813
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
(cherry picked from commit 3806d8ec7c145d0239c94fc5b601d460b36757a5)

7 years agoMerge pull request #21177 from tchaikov/wip-luminous-pr-21159
Kefu Chai [Tue, 3 Apr 2018 04:15:57 +0000 (12:15 +0800)]
Merge pull request #21177 from tchaikov/wip-luminous-pr-21159

luminous: src: s/--use-wheel//

Reviewed-by: Nathan Cutler <ncutler@suse.com>
7 years agoqa/suites/rbd: set qemu task time_wait param 21192/head
Mykola Golub [Thu, 29 Mar 2018 11:10:58 +0000 (14:10 +0300)]
qa/suites/rbd: set qemu task time_wait param

so workloads qemu_dynamic_features.sh and qemu_rebuild_object_map.sh,
which check if qemu is finished with periodicity 60 sec, have enough
time to detect this before the rbd image is removed.

Fixes: https://tracker.ceph.com/issues/23502
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 6fe8d6470506a9947f685b05683e2e046457a9f5)

7 years agoqa/tasks/qemu: add a parameter to wait for workloads detect qemu finished
Mykola Golub [Thu, 29 Mar 2018 11:06:13 +0000 (14:06 +0300)]
qa/tasks/qemu: add a parameter to wait for workloads detect qemu finished

In the case when a workload needs to detect qemu finished by running a
check with a periodicity of N sec it needs to set time_wait to 2 * N
in order to avoid races on finish.

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit af15b2cef46c8d3ab1f9770601ae1abfe9c6b300)

7 years agosrc: s/--use-wheel// 21177/head
Kefu Chai [Sun, 1 Apr 2018 12:27:24 +0000 (20:27 +0800)]
src: s/--use-wheel//

--use-wheel was deprecated in favor of --only-binary in pip v7.0.0. and
--use-wheel was removed in a recent release of pip. but some packages
are source packages, so we cannot simply replace use-wheel with
only-binary. so a simpler approach is to drop --use-wheel option, as pip
respects --find-links, and will find the required package from the
wheelhouse.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a80cce4ba6afee921a7c6adce802c91e30f8c0a9)

Conflicts:
src/ceph-detect-init/CMakeLists.txt
src/ceph-disk/CMakeLists.txt: trivial resolution
src/pybind/mgr/dashboard/CMakeLists.txt: dashboard2 is not
in luminous, so drop this change.

7 years agomds: always handle SESSION_REQUEST_RENEWCAPS messages 21176/head
Yan, Zheng [Fri, 23 Feb 2018 04:17:04 +0000 (12:17 +0800)]
mds: always handle SESSION_REQUEST_RENEWCAPS messages

Fixes: http://tracker.ceph.com/issues/23059
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 543547827c9cee2626952fa645ec8b05ab9c1366)

7 years agomds: don't cleanup request that has pending remote authpin/wrlock/xlock
Yan, Zheng [Fri, 23 Feb 2018 03:33:41 +0000 (11:33 +0800)]
mds: don't cleanup request that has pending remote authpin/wrlock/xlock

Fixes: http://tracker.ceph.com/issues/23059
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit a197a89d65c76c51cbc24890474ad27f72ebe382)

7 years agotest: pass correct buffer length to ceph_getdents 21173/head
Patrick Donnelly [Wed, 22 Nov 2017 05:34:32 +0000 (21:34 -0800)]
test: pass correct buffer length to ceph_getdents

This test fails if the randvar r is 0.

Fixes: http://tracker.ceph.com/issues/22221
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit ef5699441e056133967ffbf29076eb79594bf5e5)

7 years agotests: remove TestPGLog ASSERT_DEATH test 21134/head
Nathan Cutler [Thu, 29 Mar 2018 09:40:09 +0000 (11:40 +0200)]
tests: remove TestPGLog ASSERT_DEATH test

Fixes: http://tracker.ceph.com/issues/23504
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit adcba97a41795c25de8b6dfe263a1014564a305f)

7 years agomon: mark mgr reports and osd_failure as no_reply 21016/head
Kefu Chai [Tue, 27 Mar 2018 09:45:30 +0000 (17:45 +0800)]
mon: mark mgr reports and osd_failure as no_reply

see also: #20517

Fixes: http://tracker.ceph.com/issues/22114
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0daccfb4951471ab3f70ae87870f23e14357d9c8)

7 years agoMerge pull request #20254 from k0ste/luminous_backports
Yuri Weinstein [Thu, 29 Mar 2018 13:24:07 +0000 (06:24 -0700)]
Merge pull request #20254 from k0ste/luminous_backports

luminous: mgr: prometheus: set metadata metrics value to '1' (#22717).

Reviewed-by: John Spray <john.spray@redhat.com>
7 years agoMerge pull request #20334 from tchaikov/wip-pr-20255-luminous
Yuri Weinstein [Thu, 29 Mar 2018 13:23:31 +0000 (06:23 -0700)]
Merge pull request #20334 from tchaikov/wip-pr-20255-luminous

luminous: config: Change bluestore_cache_kv_max to type INT64

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
7 years agoMerge pull request #20358 from pdvian/wip-22940-luminous
Yuri Weinstein [Thu, 29 Mar 2018 13:22:48 +0000 (06:22 -0700)]
Merge pull request #20358 from pdvian/wip-22940-luminous

luminous: Double free in rados_getxattrs_next

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
7 years agoMerge pull request #20359 from xiexingguo/wip-luminous-balancer-fixes
Yuri Weinstein [Thu, 29 Mar 2018 13:22:10 +0000 (06:22 -0700)]
Merge pull request #20359 from xiexingguo/wip-luminous-balancer-fixes

luminous: mgr/balancer: pool-specific optimization support and bug fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
7 years agoMerge pull request #20399 from smithfarm/wip-22942-luminous
Yuri Weinstein [Thu, 29 Mar 2018 13:21:27 +0000 (06:21 -0700)]
Merge pull request #20399 from smithfarm/wip-22942-luminous

luminous: ceph osd force-create-pg cause all ceph-mon to crash and unable to come up again

Reviewed-by: Nathan Cutler <ncutler@suse.com>
7 years agoMerge pull request #20433 from pdvian/wip-22971-luminous
Yuri Weinstein [Thu, 29 Mar 2018 13:20:49 +0000 (06:20 -0700)]
Merge pull request #20433 from pdvian/wip-22971-luminous

mon: allow removal of tier of ec overwritable pool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>