ceph.git
5 years ago12.2.12 v12.2.12
Jenkins Build Slave User [Thu, 11 Apr 2019 12:33:50 +0000 (12:33 +0000)]
12.2.12

5 years agoMerge pull request #26989 from ceph/backport-luminous-26957
Yuri Weinstein [Mon, 8 Apr 2019 19:44:57 +0000 (12:44 -0700)]
Merge pull request #26989 from ceph/backport-luminous-26957

luminous: ceph-volume: look for rotational data in lsblk

Reviewed-by: Alfredo Deza <adeza@redhat.com>
5 years agoMerge pull request #27312 from ifed01/wip-ifed-fix-bmap-alloc-luminous 27364/head
Yuri Weinstein [Wed, 3 Apr 2019 20:22:15 +0000 (13:22 -0700)]
Merge pull request #27312 from ifed01/wip-ifed-fix-bmap-alloc-luminous

luminous: os/bluestore: bitmap allocator might fail to return contiguous chunk despite having enough space

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #26475 from pdvian/wip-38338-luminous
Yuri Weinstein [Wed, 3 Apr 2019 20:18:40 +0000 (13:18 -0700)]
Merge pull request #26475 from pdvian/wip-38338-luminous

luminous: tests: run-standalone.sh set local library location so mgr can find li…

Reviewed-by: David Zafman <dzafman@redhat.com>
5 years agoMerge pull request #27322 from liewegas/wip-valgrind-luminous
Sage Weil [Wed, 3 Apr 2019 14:57:12 +0000 (09:57 -0500)]
Merge pull request #27322 from liewegas/wip-valgrind-luminous

luminous: qa/valgrind.supp: backport valgrind suppressions

5 years agoMerge pull request #26191 from cbodley/wip-38081-luminous
Yuri Weinstein [Tue, 2 Apr 2019 20:04:41 +0000 (13:04 -0700)]
Merge pull request #26191 from cbodley/wip-38081-luminous

luminous: rgw: fetch_remote_obj filters out olh attrs

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #26549 from cbodley/wip-38412-luminous
Yuri Weinstein [Tue, 2 Apr 2019 20:03:40 +0000 (13:03 -0700)]
Merge pull request #26549 from cbodley/wip-38412-luminous

luminous: rgw: fix rgw_data_sync_info::json_decode()

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #27295 from theanalyst/luminous-sse-c-backport
Yuri Weinstein [Tue, 2 Apr 2019 20:02:38 +0000 (13:02 -0700)]
Merge pull request #27295 from theanalyst/luminous-sse-c-backport

luminous: sse-c-fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #27299 from cbodley/wip-39073
Yuri Weinstein [Tue, 2 Apr 2019 20:01:44 +0000 (13:01 -0700)]
Merge pull request #27299 from cbodley/wip-39073

luminous: rgw multisite: data sync checks empty next_marker for datalog

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #26064 from pdvian/wip-37987-luminous
Yuri Weinstein [Tue, 2 Apr 2019 15:59:28 +0000 (08:59 -0700)]
Merge pull request #26064 from pdvian/wip-37987-luminous

luminous: core: Objecter::calc_op_budget: Fix invalid access to extent union member

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
5 years agoqa/valgrind.supp: backport valgrind suppressions 27322/head
Sage Weil [Tue, 2 Apr 2019 15:57:31 +0000 (10:57 -0500)]
qa/valgrind.supp: backport valgrind suppressions

in the correction location

Signed-off-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #27173 from pdvian/wip-38755-luminous
Yuri Weinstein [Tue, 2 Apr 2019 15:44:44 +0000 (08:44 -0700)]
Merge pull request #27173 from pdvian/wip-38755-luminous

luminous: rgw: ldap: fix LDAPAuthEngine::init() when uri !empty()

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #27180 from pdvian/wip-38771-luminous
Yuri Weinstein [Tue, 2 Apr 2019 15:44:05 +0000 (08:44 -0700)]
Merge pull request #27180 from pdvian/wip-38771-luminous

luminous : rgw: nfs: skip empty (non-POSIX) path segments

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoos/bluestore: unconditionally cap chunks returned by allocator to 2^31 27312/head
Igor Fedotov [Wed, 20 Mar 2019 18:10:04 +0000 (21:10 +0300)]
os/bluestore: unconditionally cap chunks returned by allocator to 2^31

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 1337443e37ffbfcaf494bdb2bb80db4cf41b97f9)

5 years agoos/bluestore: start using 64-bit intervals for bitmap allocator
Igor Fedotov [Wed, 13 Mar 2019 16:09:40 +0000 (19:09 +0300)]
os/bluestore: start using 64-bit intervals for bitmap allocator

This helps to avoid 4Gb+ values wrapping which resulted in
allocation failures.

Fixes: https://tracker.ceph.com/issues/38761
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit d1466963a60ed2345a6fc159071ec09ce422b656)

5 years agoos/bluestore: make bluestore interval base template.
Igor Fedotov [Wed, 13 Mar 2019 15:51:46 +0000 (18:51 +0300)]
os/bluestore: make bluestore interval base template.

We might need intervals at BlueStore with both 32-bit and 64-bit
lengths. Having base template eliminates the need for copy/paste

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f775d4227fb9432840c795115669766565c5e718)

5 years agotests/fastbmap_alloc: UT to reproduce 4G allocation bug
Igor Fedotov [Wed, 13 Mar 2019 14:02:13 +0000 (17:02 +0300)]
tests/fastbmap_alloc: UT to reproduce 4G allocation bug

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit ed8a93ba387458961444913a8695f538595efacf)

5 years agoos/bluestore: os/bluestore: implement dump for bitmap allocator
Igor Fedotov [Tue, 12 Mar 2019 11:04:17 +0000 (14:04 +0300)]
os/bluestore: os/bluestore: implement dump for bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 80310d98a7456afdb2002eaf2891853ea21eaf68)

5 years agorgw: fixup_range() clamps offsets to valid part range 27295/head
Casey Bodley [Wed, 27 Mar 2019 17:52:59 +0000 (13:52 -0400)]
rgw: fixup_range() clamps offsets to valid part range

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4b3bac20d550cae274c07a1d555751483dde463b)

5 years agotest_rgw_crypto: add invalid ranges beyond obj boundary
Abhishek Lekshmanan [Wed, 27 Mar 2019 12:45:27 +0000 (13:45 +0100)]
test_rgw_crypto: add invalid ranges beyond obj boundary

This is just to assert we're not doing an invalid memory access

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit b34a00de0f6ecea6d97fd80f65d2a2dca0876e6e)

5 years agoMerge pull request #26387 from smithfarm/wip-38188-luminous
Yuri Weinstein [Mon, 1 Apr 2019 21:47:10 +0000 (14:47 -0700)]
Merge pull request #26387 from smithfarm/wip-38188-luminous

luminous: osd/bluestore: deep fsck fails on inspecting very large onodes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
5 years agoMerge pull request #26535 from k0ste/luminous_backports3
Yuri Weinstein [Mon, 1 Apr 2019 21:46:42 +0000 (14:46 -0700)]
Merge pull request #26535 from k0ste/luminous_backports3

luminous: mgr/prometheus: add interface and objectstore to osd metadata

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
5 years agoMerge pull request #26616 from pdvian/wip-38442-luminous
Yuri Weinstein [Mon, 1 Apr 2019 21:21:57 +0000 (14:21 -0700)]
Merge pull request #26616 from pdvian/wip-38442-luminous

luminous: qa/standalone/osd/osd-markdown: disable CLI command dups

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
5 years agoMerge pull request #26751 from noahdesu/cherrypy-ipv6-luminous
Yuri Weinstein [Mon, 1 Apr 2019 21:08:49 +0000 (14:08 -0700)]
Merge pull request #26751 from noahdesu/cherrypy-ipv6-luminous

luminous: mgr/dashboard: fix for using '::' on hosts without ipv6

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
5 years agoMerge pull request #26979 from ifed01/wip-ifed-bmap-alloc-luminous
Yuri Weinstein [Mon, 1 Apr 2019 20:40:55 +0000 (13:40 -0700)]
Merge pull request #26979 from ifed01/wip-ifed-bmap-alloc-luminous

luminous: os/bluestore: backport new bitmap allocator

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agorgw: data sync checks empty next_marker for datalog 27299/head
Casey Bodley [Fri, 29 Mar 2019 14:27:56 +0000 (10:27 -0400)]
rgw: data sync checks empty next_marker for datalog

RGWReadRemoteDataLogShardCR tracks the marker and next_marker
separately, because next_marker will be empty when it reaches the end

this allows RGWDataSyncShardCR to avoid clearing its sync_marker and
restarting datalog listing from the beginning

Fixes: http://tracker.ceph.com/issues/39033
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2ff63778342282ab5c676529f6e7d90f30e6da1a)

Conflicts:
src/rgw/rgw_data_sync.cc: no sync tracing

5 years agotests: rgw crypto tests with partlen vector
Abhishek Lekshmanan [Tue, 26 Mar 2019 18:12:15 +0000 (19:12 +0100)]
tests: rgw crypto tests with partlen vector

4 tests are added when either obj_size for an aligned object, and when an object
is not aligned to a boundary (1 B last part), or part_size is not aligned to a
boundary. Comments on boundary case explains the logic on choosing whether to
align to a 4095B boundary or not

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 1bf59f68db608084a1e7a05aca4cca8a8ab76062)
luminous changes: std::make_unique -> ceph::make_unique

5 years agotest_rgw_crypt: BlockCryptNone takes a block_size param
Abhishek Lekshmanan [Tue, 26 Mar 2019 18:11:28 +0000 (19:11 +0100)]
test_rgw_crypt: BlockCryptNone takes a block_size param

So that block sizes like 4096 can be tested

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 9e13a42daaba26751f56f2454f9ec314d7d0d460)

5 years agorgw_crypt: make decrypt's parts_len protected
Abhishek Lekshmanan [Tue, 26 Mar 2019 18:10:04 +0000 (19:10 +0100)]
rgw_crypt: make decrypt's parts_len protected

This is so that testing classes can set this and use that to test the branches
in fixup_range tests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 78b37a5ccd67c464f3c6e61568e9e827c327ad6d)

5 years agorgw_crypt: take care of start & end boundaries when parsing range
Abhishek Lekshmanan [Mon, 25 Mar 2019 14:24:15 +0000 (15:24 +0100)]
rgw_crypt: take care of start & end boundaries when parsing range

As http range header is inclusive of start and end ranges, this otherwise breaks
at ranges close to multipart boundaries

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 9c92b56d77932283ccc0336645dc3b76bd0fffa2)

5 years agorgw: decrypt filter does not cross multipart boundaries
Casey Bodley [Fri, 22 Mar 2019 13:57:33 +0000 (09:57 -0400)]
rgw: decrypt filter does not cross multipart boundaries

multipart uploads with sse encrypts each part separately, using an
initialization vector based on the part offset

decryption must respect the same part boundaries, and start each part
with a fresh initialization vector. this means that the decrypt filter
must flush data up to part boundaries before starting the next

Fixes: http://tracker.ceph.com/issues/38700
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b782902954abd20e49c0ed1c14cebee4c8ab6f88)
luminous modifications: std::make_unique -> ceph::make_unique

5 years agoMerge pull request #26642 from pdvian/wip-38449-luminous
Yuri Weinstein [Mon, 1 Apr 2019 13:59:14 +0000 (06:59 -0700)]
Merge pull request #26642 from pdvian/wip-38449-luminous

luminous: MDSMonitor: do not assign standby-replay when degraded

Reviewed-by: Venky Shankar <vshankar@redhat.com>
5 years agoMerge pull request #26776 from pdvian/wip-38541-luminous
Yuri Weinstein [Mon, 1 Apr 2019 13:58:49 +0000 (06:58 -0700)]
Merge pull request #26776 from pdvian/wip-38541-luminous

luminous : qa: bump fsstress timeout to 6h

Reviewed-by: Venky Shankar <vshankar@redhat.com>
5 years agoMerge pull request #26805 from pdvian/wip-38543-luminous
Yuri Weinstein [Mon, 1 Apr 2019 13:58:18 +0000 (06:58 -0700)]
Merge pull request #26805 from pdvian/wip-38543-luminous

luminous: qa: unmount clients before deleting fs

Reviewed-by: Venky Shankar <vshankar@redhat.com>
5 years agoMerge pull request #26820 from smithfarm/wip-38545-luminous
Yuri Weinstein [Mon, 1 Apr 2019 13:57:38 +0000 (06:57 -0700)]
Merge pull request #26820 from smithfarm/wip-38545-luminous

luminous: qa: Loading libcephfs-jni: Failure!

Reviewed-by: Venky Shankar <vshankar@redhat.com>
5 years agoMerge pull request #26964 from ashishkumsingh/wip-38735-luminous
Yuri Weinstein [Mon, 1 Apr 2019 13:57:14 +0000 (06:57 -0700)]
Merge pull request #26964 from ashishkumsingh/wip-38735-luminous

luminous: qa: extend MDS heartbeat grace for valgrind

Reviewed-by: Venky Shankar <vshankar@redhat.com>
5 years agoMerge pull request #27024 from pdvian/wip-38669-luminous
Yuri Weinstein [Mon, 1 Apr 2019 13:56:42 +0000 (06:56 -0700)]
Merge pull request #27024 from pdvian/wip-38669-luminous

luminous: mds: wait for client to release shared cap when re-acquiring xlock

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>
5 years agoMerge pull request #27207 from pdvian/wip-38854-luminous
Yuri Weinstein [Fri, 29 Mar 2019 19:45:31 +0000 (12:45 -0700)]
Merge pull request #27207 from pdvian/wip-38854-luminous

luminous: mon/MgrStatMonitor: ensure only one copy of initial service map

Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoMerge pull request #27224 from xiexingguo/wip-luminous-upmap-backports
Yuri Weinstein [Fri, 29 Mar 2019 19:44:55 +0000 (12:44 -0700)]
Merge pull request #27224 from xiexingguo/wip-luminous-upmap-backports

luminous: osd: backport recent upmap fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #27226 from pdvian/wip-38857-luminous
Yuri Weinstein [Fri, 29 Mar 2019 19:44:25 +0000 (12:44 -0700)]
Merge pull request #27226 from pdvian/wip-38857-luminous

luminous: should set EPOLLET flag on del_event()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoqa/suites: fix bluestore links 26979/head
Sage Weil [Wed, 5 Dec 2018 16:27:53 +0000 (10:27 -0600)]
qa/suites: fix bluestore links

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 269910fc8b2bdf64b98534ce525318e108aa9409)

5 years agoMerge pull request #26358 from smithfarm/wip-37557-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:45:34 +0000 (08:45 -0700)]
Merge pull request #26358 from smithfarm/wip-37557-luminous

luminous: multisite: es sync null versioned object failed because of olh info

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #26530 from pdvian/wip-38354-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:44:28 +0000 (08:44 -0700)]
Merge pull request #26530 from pdvian/wip-38354-luminous

luminous: rgw: check for non-existent bucket in RGWGetACLs

Reviewed-by: Adam Emerson <aemerson@redhat.com>
5 years agoMerge pull request #26619 from pdvian/wip-38446-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:43:58 +0000 (08:43 -0700)]
Merge pull request #26619 from pdvian/wip-38446-luminous

luminous: rgw: only update last_trim marker on ENODATA

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #26761 from pdvian/wip-38529-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:43:36 +0000 (08:43 -0700)]
Merge pull request #26761 from pdvian/wip-38529-luminous

luminous: rgw: data sync drains lease stack on lease failure

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #26946 from ashishkumsingh/wip-38727-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:43:09 +0000 (08:43 -0700)]
Merge pull request #26946 from ashishkumsingh/wip-38727-luminous

luminous: rgw: bucket limit check misbehaves for > max-entries buckets (usually 1000)

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #27030 from pdvian/wip-38671-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:42:46 +0000 (08:42 -0700)]
Merge pull request #27030 from pdvian/wip-38671-luminous

luminous: rgw: sync module: avoid printing attrs of objects in log

Reviewed-by: Casey Bodley <cbodley@redhat.com>
5 years agoMerge pull request #26516 from yehudasa/wip-rgw-es-fixes-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:14:03 +0000 (08:14 -0700)]
Merge pull request #26516 from yehudasa/wip-rgw-es-fixes-luminous

 luminous: rgw: multiple es related fixes and improvements

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
5 years agoosd/OSDMap: add log for better debugging 27224/head
xie xingguo [Mon, 25 Mar 2019 10:24:16 +0000 (18:24 +0800)]
osd/OSDMap: add log for better debugging

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a89281ffbb50a4dfc700398e864138b5faaf00f5)

5 years agoosd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only
xie xingguo [Sat, 23 Mar 2019 01:50:27 +0000 (09:50 +0800)]
osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only

The current implementation will try to cancel any pg_upmaps that
would otherwise re-map a PG out from an underfull osd, which is wrong,
e.g., because it could reliably fire the following assert:

src/osd/OSDMap.cc: 4405: FAILED assert(osd_weight.count(i.first))

Also it would not match the expectation if automatic balancing
has been strictly restricted to some specific pools by admin.

Fix by excluding any wild PG that does not belong to the origin pools
passed in when trying to do upmap/unmap.

Fixes: http://tracker.ceph.com/issues/38897
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 01e8e9482ce7194d347e02ef41acfa6d8d14f614)

5 years agoosd/OSDMap: drop local pool filter in calc_pg_upmaps
xie xingguo [Sat, 23 Feb 2019 00:33:40 +0000 (08:33 +0800)]
osd/OSDMap: drop local pool filter in calc_pg_upmaps

The local pre-loaded pool filter is completely redundant since
the below check:

if (!only_pools.empty() && !only_pools.count(i.first))

could reliably catch both cases - either optimization should be
restricted to specific pools feeded, or all existing pools.

Let's clean it up.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 3e6bad9821b5fb3e780d970666fbdfbf217d905e)

5 years agocrush: add root_bucket to identify underfull buckets
huangjun [Wed, 20 Mar 2019 08:44:02 +0000 (16:44 +0800)]
crush: add root_bucket to identify underfull buckets

All underfull buckets under root_buckets will be taken as target

For the crule rule:
    step take datacenter0
    step chooseleaf firstn 2 type host
    step emit
    step take datacenter1
    step chooseleaf firstn 2 type host
    step emit

If one host contains overfull osd but no underfull osd,
it will use other underfull buckets as target, which
maybe not in the same datacenter, that will
broke the rule.

Fixes: http://tracker.ceph.com/issues/38826
Signed-off-by: huangjun <huangjun@xsky.com>
(cherry picked from commit 3d5678d3561d90a10d9de3cb6e7e0405dbe8fdfe)

5 years agoMerge pull request #27104 from rzarzynski/wip-nss-importsymkey-in-fips-luminous
Neha Ojha [Thu, 28 Mar 2019 01:20:36 +0000 (18:20 -0700)]
Merge pull request #27104 from rzarzynski/wip-nss-importsymkey-in-fips-luminous

luminous: crypto: don't use PK11_ImportSymKey() in FIPS mode

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoEventEpoll: refactor del_event() a bit 27226/head
Roman Penyaev [Tue, 19 Mar 2019 10:55:52 +0000 (11:55 +0100)]
EventEpoll: refactor del_event() a bit

The main purpose of the patch is to set 'ee' members only on EPOLL_CTL_MOD
path, deletion path does not requie any of the members set.

This will make code a bit better.

Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
(cherry picked from commit 2292120ffd2730a20fd2750a605952982f031c83)

5 years agoEventEpoll: set EPOLLET flag on del_event()
Roman Penyaev [Tue, 5 Mar 2019 12:22:59 +0000 (13:22 +0100)]
EventEpoll: set EPOLLET flag on del_event()

When new fd is being added to epoll fd EPOLLET (edge triggered) flag
is set, i.e. we assume that in and out works in edge triggered mode.

But on del_event(), when eventually epoll_ctl(EPOLL_CTL_MOD) is
invoked, EPOLLET flag is cleared, i.e. level triggered mode is
enabled.

Do not lose EPOLLET.

Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
(cherry picked from commit c415b9fed4fbecd2fb0709fa0c0c783102f60e01)

5 years agoauth, rgw, common: switch to PK11_ImportSymKey_FIPS wrapper. 27104/head
Radoslaw Zarzynski [Tue, 12 Mar 2019 14:17:43 +0000 (15:17 +0100)]
auth, rgw, common: switch to PK11_ImportSymKey_FIPS wrapper.

PK11_ImportSymKey() is a part of NSS API that becomes unavailable
in the FIPS mode. Apparently NSS targets stricter restrictions
than those coming from Level 1 of FIPS 140-2. In the consequence,
loading a symmetric key from plain keyring or key db (which Ceph
needs to do due to architectural reasons) fails.

The same issue affected corosync and this patchset deals with it
the same way like already followed by corosync [1]:
a raw crypto key is in-memory wrapped with fresh, random wrapping
key just before being imported via PK11_UnwrapSymKey(). Of course,
this effectively lowers to FIPS level 1. Still, this would be no
different from what OpenSSL (to which we are currently migrating
in master) gives in the matter.

The patch can be *roughly* verified in following steps:
  1. mkdir ./nssdb
  2. certutil -N -d ./nssdb --empty-password
  3. modutil -dbdir ./nssdb -fips true
  4. ../src/vstart.sh -l -n -b -o "nss_db_path=/work/ceph-3/build/nssdb"

This fix is dedicated to Luminous. In master we're switching to OpenSSL.

[1] 5dadebd21862074deaeb9a337fc9e49f5e9f692a in corosync's public repo.

Fixes: http://tracker.ceph.com/issues/38843
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
5 years agocrypto: bring PK11_ImportSymKey_FIPS doing PK11_UnwrapSymKey.
Radoslaw Zarzynski [Tue, 12 Mar 2019 01:23:53 +0000 (02:23 +0100)]
crypto: bring PK11_ImportSymKey_FIPS doing PK11_UnwrapSymKey.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
5 years agomon/MgrStatMonitor: ensure only one copy of initial service map 27207/head
Sage Weil [Thu, 21 Mar 2019 16:58:29 +0000 (11:58 -0500)]
mon/MgrStatMonitor: ensure only one copy of initial service map

It's possible for create_pending to be called multiple times.

Fixes: http://tracker.ceph.com/issues/38839
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 204a8deacd2068989bd52eb130f4aac434b45f35)

Conflicts:
src/mon/MgrStatMonitor.cc : Resolved in create_initial

5 years agoMerge pull request #26758 from pdvian/wip-38510-luminous
Yuri Weinstein [Tue, 26 Mar 2019 16:56:26 +0000 (09:56 -0700)]
Merge pull request #26758 from pdvian/wip-38510-luminous

luminous: CLI: ability to change file ownership

5 years agoMerge pull request #26830 from pdvian/wip-38562-luminous
Yuri Weinstein [Tue, 26 Mar 2019 16:49:48 +0000 (09:49 -0700)]
Merge pull request #26830 from pdvian/wip-38562-luminous

luminous: mgr/BaseMgrModule: drop GIL for ceph_send_command

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoMerge pull request #26900 from pdvian/wip-38586-luminous
Yuri Weinstein [Tue, 26 Mar 2019 16:48:50 +0000 (09:48 -0700)]
Merge pull request #26900 from pdvian/wip-38586-luminous

luminous: OSD crashes in get_str_map while creating with ceph-volume

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #26962 from pdvian/wip-38665-luminous
Yuri Weinstein [Tue, 26 Mar 2019 16:48:21 +0000 (09:48 -0700)]
Merge pull request #26962 from pdvian/wip-38665-luminous

luminous: qa: ignore slow metadata io wrn during osd thrash

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agorgw: nfs: skip empty (non-POSIX) path segments 27180/head
Matt Benjamin [Thu, 14 Mar 2019 17:58:17 +0000 (13:58 -0400)]
rgw: nfs: skip empty (non-POSIX) path segments

Such a path could be created through S3 upload,
e.g., s3://my_files//data/file.pdf.

Previously we asserted for this case, but it's harmless--such a
path segment should just be ignored.

Fixes: http://tracker.ceph.com/issues/38744
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit a53e6c1d5b785a2a46007292041d0a8cb552e3e6)

Conflicts:
src/rgw/rgw_file.h : Resolved in operator()

5 years agorgw: ldap: fix LDAPAuthEngine::init() when uri !empty() 27173/head
Matt Benjamin [Tue, 12 Mar 2019 12:58:53 +0000 (08:58 -0400)]
rgw: ldap: fix LDAPAuthEngine::init() when uri !empty()

Fixes: https://tracker.ceph.com/issues/38699
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 6ef98c6e0fcf4f9b6e431b3409975e0966c5c21a)

5 years agoMerge pull request #27033 from cbodley/wip-38796
Nathan Cutler [Fri, 22 Mar 2019 13:22:02 +0000 (14:22 +0100)]
Merge pull request #27033 from cbodley/wip-38796

luminous: doc/rgw: document placement target configuration

Reviewed-by: Nathan Cutler <ncutler@suse.com>
5 years agoMerge pull request #26557 from dzafman/wip-38244
David Zafman [Tue, 19 Mar 2019 23:59:39 +0000 (16:59 -0700)]
Merge pull request #26557 from dzafman/wip-38244

luminous: scrub warning check incorrectly uses mon scrub interval

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
5 years agoMerge pull request #26793 from dzafman/wip-38274
David Zafman [Tue, 19 Mar 2019 23:48:17 +0000 (16:48 -0700)]
Merge pull request #26793 from dzafman/wip-38274

luminous: Fix recovery and backfill priority handling

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #26544 from ifed01/wip-ifed-fix-compress-luminous
Yuri Weinstein [Mon, 18 Mar 2019 18:59:14 +0000 (11:59 -0700)]
Merge pull request #26544 from ifed01/wip-ifed-fix-compress-luminous

luminous: os/bluestore: do not assert on non-zero err codes from compress() call

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agodoc/rgw: document placement target configuration 27033/head
Casey Bodley [Fri, 15 Mar 2019 18:35:13 +0000 (14:35 -0400)]
doc/rgw: document placement target configuration

Fixes: http://tracker.ceph.com/issues/24508
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c95585b1984e60708addee436a774d0c92ce57a6)

5 years agorgw: sync module: avoid printing attrs of objects in log 27030/head
Abhishek Lekshmanan [Thu, 13 Dec 2018 14:57:40 +0000 (15:57 +0100)]
rgw: sync module: avoid printing attrs of objects in log

Since attrs are verbose and not necessarily needed as a debug info (as they can
be obtained from the required object itself) avoid logging them. Also
additionally removed another place where such a log is done by the ES sync
module

Fixes: http://tracker.ceph.com/issues/37646
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 8f37b70630ceb62aff4019e9ad2508977590a873)

Conflicts:
src/rgw/rgw_sync_module.cc : Resolved in RGWCallStatRemoteObjCR::operate

5 years agomds: wait for client to release shared cap when re-acquiring xlock 27024/head
Yan, Zheng [Thu, 28 Feb 2019 13:36:28 +0000 (21:36 +0800)]
mds: wait for client to release shared cap when re-acquiring xlock

MDS may revoke shared cap from xlocker client during 'LOCK_XLOCKDONE ->
LOCK_LOCK_XLOCK' state transition.

Fixes: http://tracker.ceph.com/issues/38491
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 218e784f2eb1eec6bbb7910b5432f7f36263dbe7)

Conflicts:
src/mds/SimpleLock.h : Resovled in put_xlock

5 years agoqa/objectstore: bluestore -> bluestore-{bitmap,stupid}
Sage Weil [Fri, 30 Nov 2018 15:55:38 +0000 (09:55 -0600)]
qa/objectstore: bluestore -> bluestore-{bitmap,stupid}

The new default is bitmap, so we were testing bitmap twice.  Instead,
explicitly call out stupid and bitmap cases so a future default change
won't break coverage.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f390704b4616c837352e7d6a8df91526cb047f0d)

5 years agoos/bluestore: fix minor issues in bitmap logging output
Igor Fedotov [Mon, 25 Jun 2018 14:49:49 +0000 (17:49 +0300)]
os/bluestore: fix minor issues in bitmap logging output

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dc4b5fc5afcea4be9e3b4c91dcc9dcce38d4e906)

5 years agoos/bluestore: fix overlappedd allocation returned by bitmap allocator
Igor Fedotov [Mon, 25 Jun 2018 14:42:41 +0000 (17:42 +0300)]
os/bluestore: fix overlappedd allocation returned by bitmap allocator

Fixes: https://tracker.ceph.com/issues/24598
Signed-off-by: Igor Fedotov <ifedotov@suse.som>
(cherry picked from commit 539af4f27e773cb21d187b75086c7c9498dd7749)

5 years agoos/bluestore: provide a rough estimate for bitmap allocator
Igor Fedotov [Fri, 15 Jun 2018 14:04:57 +0000 (17:04 +0300)]
os/bluestore: provide a rough estimate for bitmap allocator
fragmentation.
The approach is counting 'partial' and 'free' slots at L1 on the fly and
use partial / (partial+free) value as rough fragmentation estimate.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dbbe12fe6f543fc6942f41a108effc7eb1632f67)

5 years agoos/bluestore: rename new bitmap allocator class to BitmapAllocator.
Igor Fedotov [Tue, 12 Jun 2018 11:50:30 +0000 (14:50 +0300)]
os/bluestore: rename new bitmap allocator class to BitmapAllocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit a8e8c192acb48ce93553b04d1953ce8bee1b16e4)

5 years agoos/bluestore: perform allocations aligned with min_length in new bitmap allocator
Igor Fedotov [Fri, 1 Jun 2018 18:59:15 +0000 (21:59 +0300)]
os/bluestore: perform allocations aligned with min_length  in new bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 51974c48e38e7080eb12c6028f95f4da666fbe76)

5 years agotest/objectstore/unitetest_fastbmap_allocator: replace ASSERT_TRUE with
Igor Fedotov [Fri, 1 Jun 2018 13:35:04 +0000 (16:35 +0300)]
test/objectstore/unitetest_fastbmap_allocator: replace ASSERT_TRUE with
ASSERT_EQ

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 69ccb4367dcdcc51fbcfdaaea308c761291621e1)

5 years agoos/bluestore: respect min_length as allocation granularity for new
Igor Fedotov [Thu, 31 May 2018 13:53:31 +0000 (16:53 +0300)]
os/bluestore: respect min_length as allocation granularity for new
bitmap allocator.
It was used a real minimum threshold before this fix which allowed e.g.
allocated extent length to be equal to min_length + 1.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 8a188b2aa3892b7f5a6c85042ffae8c2847ca869)

5 years agoos/bluestore: cosmetic new allocator internal method rename.
Igor Fedotov [Thu, 31 May 2018 11:28:28 +0000 (14:28 +0300)]
os/bluestore: cosmetic new allocator internal method rename.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit fa06ff234f11f266bb10d7fa9ea28847e5ce72fe)

5 years agoos/bluestore: properly respect min_length when analysing partially free
Igor Fedotov [Thu, 31 May 2018 10:53:37 +0000 (13:53 +0300)]
os/bluestore: properly respect min_length when analysing partially free
slots in bitmap allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 288bbd42531bebd8f6c619c848e75bfd9b8f93fd)

5 years agoos/bluestore: cosmetic cleanup in new bitmap allocator.
Igor Fedotov [Thu, 31 May 2018 10:34:21 +0000 (13:34 +0300)]
os/bluestore: cosmetic cleanup in new bitmap allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 3db9d9a66cbd62c543cdd88ff4ab636fba2c2a2b)

5 years agoos/bluestore: more verbose logging in new bitmap allocator
Igor Fedotov [Wed, 30 May 2018 12:24:05 +0000 (15:24 +0300)]
os/bluestore: more verbose logging in new bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 160c73032db2e122f5e643090c86eed85e679eb4)

5 years agoMerge pull request #26597 from smithfarm/wip-release-notes-12_2_12-luminous
Nathan Cutler [Fri, 15 Mar 2019 16:18:32 +0000 (17:18 +0100)]
Merge pull request #26597 from smithfarm/wip-release-notes-12_2_12-luminous

luminous: doc: move keyring caps RN to correct point release

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
5 years agoMerge pull request #26861 from ashishkumsingh/wip-38647-luminous
Nathan Cutler [Fri, 15 Mar 2019 15:43:41 +0000 (16:43 +0100)]
Merge pull request #26861 from ashishkumsingh/wip-38647-luminous

luminous: doc: Fix incorrect mention of 'osd_deep_mon_scrub_interval'

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoceph-volume: look for rotational data in lsblk 26989/head
Andrew Schoen [Thu, 14 Mar 2019 20:14:40 +0000 (15:14 -0500)]
ceph-volume: look for rotational data in lsblk

Also ask lsblk if a device is rotational if no information
is found in /sys/block, default to True if nothing is found.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit de748f9aef894b247622ee5dc49c1fbc31097ad0)

5 years agoMerge pull request #26372 from pdvian/wip-38095-luminous
Nathan Cutler [Fri, 15 Mar 2019 15:19:41 +0000 (16:19 +0100)]
Merge pull request #26372 from pdvian/wip-38095-luminous

luminous: doc: rados/configuration: refresh osdmap section

Reviewed-by: Nathan Cutler <ncutler@suse.com>
5 years agoos/bluestore: remove original bitmap allocator
Igor Fedotov [Mon, 14 May 2018 13:09:22 +0000 (16:09 +0300)]
os/bluestore: remove original bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 95bd72fc22aa313112de096ac3ef98e45c71bce5)

5 years agoos/bluestore: align BitMap allocator's init_rm_free/init_add_free parameters with...
Igor Fedotov [Fri, 11 May 2018 21:26:49 +0000 (00:26 +0300)]
os/bluestore: align BitMap allocator's init_rm_free/init_add_free parameters with min_alloc_size

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 1b6a56e80066decf8c3727090a2cd51ba9ded9bc)

5 years agoos/bluestore: fix improper access to a BitmapFastAllocator::last_pos
Igor Fedotov [Thu, 10 May 2018 15:54:36 +0000 (18:54 +0300)]
os/bluestore: fix improper access to a BitmapFastAllocator::last_pos
from multiple threads.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 0c69672fce55362a7e6eb8ed6955f163afe85482)

5 years agotest/allocator: move bluestore allocator's benchmarks to a standalone UT
Igor Fedotov [Fri, 4 May 2018 17:03:37 +0000 (20:03 +0300)]
test/allocator: move bluestore allocator's benchmarks to a standalone UT

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit ac2ec80cddc51f1642849c7bf226e9bed116c0d6)

5 years agoos/bluestore: add new bitmap allocator
Igor Fedotov [Thu, 3 May 2018 17:22:15 +0000 (20:22 +0300)]
os/bluestore: add new bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 631041a7b788eaaf010f1e39f58b5546ab5580fc)

5 years agoMerge pull request #26969 from pdvian/wip-38667-luminous
Nathan Cutler [Fri, 15 Mar 2019 11:31:40 +0000 (12:31 +0100)]
Merge pull request #26969 from pdvian/wip-38667-luminous

luminous: doc/radosgw: Document mappings of S3 Operations to ACL grants

Reviewed-by: Nathan Cutler <ncutler@suse.com>
5 years agoqa: extend MDS heartbeat grace for valgrind 26964/head
Patrick Donnelly [Wed, 13 Mar 2019 16:13:02 +0000 (09:13 -0700)]
qa: extend MDS heartbeat grace for valgrind

Valgrind makes the MDS slowwwww. The newish mds_heartbeat_grace config allows
us to keep sending beacons to the mons even if the internal heartbeat is slow.
This avoids the laggy messages which are useful to grep for unrelated messaging
issues.

Fixes: http://tracker.ceph.com/issues/38723
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 7b520755ce62461060b6dc75fb7a39c464d2bdcb)

5 years agodoc/radosgw: Document mappings of S3 Operations to ACL grants 26969/head
Adam C. Emerson [Thu, 7 Mar 2019 20:18:42 +0000 (15:18 -0500)]
doc/radosgw: Document mappings of S3 Operations to ACL grants

Since the ACL documentation is a bit sparse, let people know exactly
what operations they can expect to enable with a given ACL grant.

Fixes: https://tracker.ceph.com/issues/38523
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit bd94c93fada952537c7015743db55bff7619afa8)

5 years agoqa: ignore slow metadata io wrn during osd thrash 26962/head
Patrick Donnelly [Fri, 8 Mar 2019 18:49:10 +0000 (10:49 -0800)]
qa: ignore slow metadata io wrn during osd thrash

Fixes: http://tracker.ceph.com/issues/38651
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5abcc32ff642b03c4b53e565e995ebb94802579a)

5 years agoMerge pull request #26798 from dzafman/wip-slow-luminous
Yuri Weinstein [Thu, 14 Mar 2019 21:34:23 +0000 (14:34 -0700)]
Merge pull request #26798 from dzafman/wip-slow-luminous

luminous: qa: Add "slow request" where missing based on REQUEST_SLOW

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agotest/allocator: get rid off bitmap allocator specifics and introduce new
Igor Fedotov [Thu, 3 May 2018 16:25:01 +0000 (19:25 +0300)]
test/allocator: get rid off bitmap allocator specifics and introduce new
performance test cases

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dcd30610fb1c87711c9f43f7d3d6636888792345)

5 years agoos/bluestore: add release(PExtentVector) helper to Allocator class to
Igor Fedotov [Mon, 7 May 2018 13:58:43 +0000 (16:58 +0300)]
os/bluestore: add release(PExtentVector) helper to Allocator class to
free PExtentVector explicitly.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 328ea72a84edef4a6c4e0558dff52d0e85f05122)

5 years agoos/bluestore: get rid off allocator's reserve method
Igor Fedotov [Mon, 7 May 2018 13:26:03 +0000 (16:26 +0300)]
os/bluestore: get rid off allocator's reserve method

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 88bea46242d4eb6bdfd8309ddf8843c9a14e5f25)