Conflicts:
src/tools/ceph_objectstore_tool.cc (in jewel, ::encode() takes only two
arguments, while in luminous/master it takes a third which we omit
here)
In cls_timeindex_list() though `to_index` has expired for a timespan, the marker is set for a subsequent index during the time boundary check.
This marker is further returned to RGWObjectExpirer::process_single_shard(), where this out_marker is trimmed from the respective shard,
resulting in a lost removal hint and a leaked object.
Adam C. Emerson [Wed, 20 Dec 2017 22:06:32 +0000 (17:06 -0500)]
rgw: Plumb refresh logic into object cache
Now when we force a refetch of bucket info it will actually go to the
OSD rather than simply using the objects in the object cache.
Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit d997f657750faf920170843e62deacab70008d8b) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Adam C. Emerson [Tue, 19 Dec 2017 21:47:09 +0000 (16:47 -0500)]
rgw: Add expiration in the object cache
We had it in the chained caches, but it doesn't do much good if
they just fetch objects out of the object cache.
Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 82a7e6ca31b416a7f0e41b5fda4c403d1d6be947) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Adam C. Emerson [Tue, 19 Dec 2017 17:53:05 +0000 (12:53 -0500)]
rgw: retry CORS put/delete operations on ECANCELLED
Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit bff7e61ca5a66b301ec49c1cf9054d1b74535832) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Adam C. Emerson [Fri, 17 Nov 2017 22:15:26 +0000 (17:15 -0500)]
rgw: Expire entries in bucket info cache
To bound the degree to which an RGW instance can go out to lunch if
the watch/notify breaks down, force refresh of any cache entry over a
certain age.
Fifteen minutes by default, and expiration can be turned off entirely.
This is separate from the LRU. The LRU removes entries based on the
last time of access. This expiration patch forces refresh based on the
last time they were updated.
Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 4489cb58a15647a31ac0546d70400af5668404cb) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Adam C. Emerson [Fri, 17 Nov 2017 21:05:06 +0000 (16:05 -0500)]
rgw: Handle stale bucket info in RGWDeleteBucketWebsite
Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit f4d274248e43cb38ff2b27782c010b2c35b12b2b)
Adam C. Emerson [Fri, 17 Nov 2017 21:03:13 +0000 (16:03 -0500)]
rgw: Handle stale bucket info in RGWSetBucketWebsite
Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit b2b7385f194def1025a8947bab876c9856b06400) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Adam C. Emerson [Fri, 17 Nov 2017 20:59:44 +0000 (15:59 -0500)]
rgw: Handle stale bucket info in RGWSetBucketVersioning
Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit a0a1e7c2ef992b8758bcfb20d893730c1b202475) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Adam C. Emerson [Fri, 17 Nov 2017 20:53:05 +0000 (15:53 -0500)]
rgw: Handle stale bucket info in RGWPutMetadataBucket
Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit ebb86301b20098e15824f469001f6153b27965f5) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Adam C. Emerson [Fri, 17 Nov 2017 20:51:42 +0000 (15:51 -0500)]
rgw: Add retry_raced_bucket_write
If the OSD informs us that our bucket info is out of date when we need
to write, we should have a way to update it.
This template function allows us to wrap relevant sections of code so
they'll be retried against new bucket info on -ECANCELED.
Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 1a3fcc70c0747791aa423cd0aa7d2596eaf3d73c)
Adam C. Emerson [Thu, 16 Nov 2017 19:42:58 +0000 (14:42 -0500)]
rgw: Add try_refresh_bucket_info function
Sometimes operations fail with -ECANCELED. This means we got raced. If
this happens we should update our bucket info from cache and try again.
Some user reports suggest that our cache may be getting and staying
out of sync. This is a bug and should be fixed, but it would also be
nice if we were robust enough to notice the problem and refresh.
So in that case, we invalidate the cache and fetch direct from the
OSD, putting a warning in the log.
Fixes: http://tracker.ceph.com/issues/22517 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 9114e5e50995f0c7d2be5c24aa4712d89cd89f48) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Marcus Watts [Tue, 2 Jan 2018 22:24:38 +0000 (17:24 -0500)]
radosgw: fix doubled underscore with s3/swift server-side copy
A name is almost an oid. Except that if a name starts
with a _, the corresponding oid starts with a double _.
The copy logic passes the oid straight in as a name,
which results in a triple _ on the oid, resulting in
an object name with a double __.
Fix: remove one _ when converting an oid back into
a name, so that the final oid only has a double _.
Fixes: http://tracker.ceph.com/issues/22529
This change is not needed on master or Luminous, due to
already-merged #18662.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Kefu Chai [Fri, 22 Dec 2017 14:42:16 +0000 (22:42 +0800)]
install-deps.sh: update g++ symlink also
we need to update g++ symlink also, if it points to the wrong version
http://tracker.ceph.com/issues/22220 Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts: the libboost issue does not affect master. as master builds
boost from source. so, it's not cherry-picked from master.
Kefu Chai [Wed, 13 Dec 2017 05:36:54 +0000 (13:36 +0800)]
install-deps.sh: point gcc to the one shipped by distro
to define a struct in a method is legal in C++11, but it causes internal
compiler error due to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82155
if we are using GCC-7. so we need to either workaround in our source
code by moving the struct definition out of the member method or revert
to a GCC without this bug. but if we go with the first route, the jewel
build still fails, because GCC-7 starts to use the new CXX11 ABI, which
is not compatible with the libboost we use in jewel. the libboost was
still built with the old ABI for backward compatibility. so let's just
fix the install-deps.sh to point gcc to the origin one.
See: http://tracker.ceph.com/issues/22220 Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts: the libboost issue does not affect master. as master builds
boost from source. so, it's not cherry-picked from master.
This is needed for jewel-x point to point upgrade because earlier point
releases can't handle our ec profiles with ruleset-* (later ones can) and
the test races with the mon upgrades.
Nathan Cutler [Tue, 21 Nov 2017 10:36:02 +0000 (11:36 +0100)]
tests: ceph-disk: ignore E722 in flake8 test
Very old, and very new, versions of flake8 treat E722 as an error:
flake8 runtests: commands[0] | flake8 --ignore=H105,H405,E127 ceph_disk tests
ceph_disk/main.py:1575:9: E722 do not use bare except'
ceph_disk/main.py:1582:9: E722 do not use bare except'
ceph_disk/main.py:3252:5: E722 do not use bare except'
ceph_disk/main.py:3288:21: E722 do not use bare except'
ceph_disk/main.py:3296:17: E722 do not use bare except'
ceph_disk/main.py:4358:5: E722 do not use bare except'
tests/test_main.py:26:1: E722 do not use bare except'
ERROR: InvocationError: '/opt/j/ws/mkck/src/ceph-disk/.tox/flake8/bin/flake8 --ignore=H105,H405,E127 ceph_disk tests'
David Zafman [Wed, 13 Sep 2017 00:17:13 +0000 (17:17 -0700)]
osd: Only scan for omap corruption once
Before
state 2: Can have complete tables (some may be bad)
state 3: Never had complete tables
After
state 2: Can have complete tables (some may be bad)
state 3 with legacy: Can have complete tables (bad ones are cleared)
state 3: Never had complete tables
Once OSDs boot with this change you can't downgrade to a previous release.
If someone does downgrade they could have unstable OSDs that hit assert(state.v < 3).
The following command run after shutting down the cluster but before downgrading
ceph packages would be a way to fix this.
Conflicts: remove bluestore.yaml as jewel does not support it. and
remove links to objectstore from where the tests do not exist in jewel
yet, for instance, qa/suites/mgr/basic.
liuchang0812 [Fri, 30 Jun 2017 12:56:04 +0000 (20:56 +0800)]
osd: new command compact via tell/daemon
user could manual compact OSD's omap as following:
1. ceph tell osd.id compact
2. ceph daemon osd.id compact
user's requests will be impacted during compaction.
Fixes: http://tracker.ceph.com/issues/19592 Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
(cherry picked from commit b4ad4297652df2f6ebfadcdededc7a47607ab534)
Conflicts:
src/osd/OSD.cc
Removed all admin socket register and unregister commands
which are not part of this backport
Changed admin_command to command variable because in jewel
we use command variable.
Conflicts:
src/os/bluestore/BlueStore.h
Removed declarations which are not part of this backport
inject_data_error()
inject_mdata_error()
debug_data_eio()
debug_mdata_eio()
debug_oj_on_delete()
src/os/filestore/FileStore.h
Removed declarations which are not part of this backport
set<ghobject_t> data_error_set
set<ghobject_t> mdata_error_set
inject_data_error() override
inject_mdata_error() override
Kefu Chai [Sat, 7 Oct 2017 14:15:11 +0000 (22:15 +0800)]
ceph-disk: retry on OSError
we are likely to
1) create partition, for instance, sdc1
2) partprobe sdc
3) udevadm settle
4) check the device by its path: /dev/sdc1
but there is chance that the uevent sent from kernel fails to reach udev
before we call "udevadm", hence "/dev/sdc1" does not exist even after
"udevadm settle" returns. so we retry in case of OSError here.
Conflicts:
src/ceph-disk/ceph_disk/main.py: jewel does not have PROCDIR,
so resolve it by using '/proc'. also, in jewel, unmount() does not
have `do_rm` parameter, so do not handle it.
Conflicts:
qa/objectstore/filestore-btrfs.yaml: we add some notes in this
file in master, but didn't backport the commit(s) adding these notes to
jewel. we are removing this file anyway. so who cares!
Conflicts:
qa/suites/fs/recovery/xfs.yaml: in master, this file is factored
into a facet: a/suites/fs/recovery/fs/xfs.yaml, but in jewel, it still a
plain xfs.yaml. but it's good enough for us, as what we need is just
xfs.
Matt Benjamin [Mon, 2 Oct 2017 15:49:05 +0000 (11:49 -0400)]
radosgw: fix awsv4 header line sort order.
The awsv4 signature calculation includes a list of header lines, which
are supposed to be sorted. The existing code sorts by header name, but
it appears that in fact it is necessary to sort the whole header *line*,
not just the field name. Sorting by just the field name usually works,
but not always. The s3-tests teuthology suite includes
s3tests.functional.test_s3.test_object_header_acl_grants
s3tests.functional.test_s3.test_bucket_header_acl_grants
which include the following header lines,
projects / ceph.git / log