qa/tasks/workunit.py: use "overrides" as the default settings of workunit
otherwise the settings in "workunit" tasks are always overridden by the
settings in template config. so we'd better follow the way of how
"install" task updates itself with the "overrides" settings: it uses the
"overrides" as the *defaults*.
Kefu Chai [Thu, 30 Mar 2017 04:37:01 +0000 (12:37 +0800)]
tasks/workunit.py: specify the branch name when cloning a branch
c1309fb failed to specify a branch when cloning using --depth=1, which
by default clones the HEAD. and we can not "git checkout" a specific
sha1 if it is not HEAD, after cloning using '--depth=1', so in this
change, we dispatch "tag", "branch", "HEAD" using three Refspec classes.
Dan Mick [Wed, 29 Mar 2017 03:08:13 +0000 (20:08 -0700)]
tasks/workunit.py: when cloning, use --depth=1
Help avoid killing git.ceph.com. A depth 1 clone takes about
7 seconds, whereas a full one takes about 3:40 (much of it
waiting for the server to create a huge compressed pack)
Matt Benjamin [Tue, 28 Feb 2017 20:49:06 +0000 (15:49 -0500)]
rgw_file: use fh_hook::is_linked() to check residence
Previously we assumed that !deleted handles were resident--there
is an observed case where a !deleted handle is !linked. Since
we currently use safe_link mode, an is_linked() check is
available, and exhaustive.
Fixes: http://tracker.ceph.com/issues/19111 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit c0aa515f8d8c57ec5ee09e3b60df3cac60453c40)
Matt Benjamin [Wed, 1 Mar 2017 01:24:12 +0000 (20:24 -0500)]
rgw_file: RGWFileHandle dtor must also cond-unlink from FHCache
Formerly masked in part by the reclaim() action, direct-delete now
substitutes for reclaim() iff its LRU lane is over its high-water
mark, and in particular, like reclaim() the destructor is certain
to see handles still interned on the FHcache when nfs-ganesha is
recycling objects from its own LRU.
Fixes: http://tracker.ceph.com/issues/19112 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit d51a3b1224ba62bb53c6c2c7751fcf7853c35a4b)
Matt Benjamin [Tue, 17 Jan 2017 16:23:45 +0000 (11:23 -0500)]
rgw_file: split last argv on ws, if provided
This is intended to allow an "extra" unparsed argument string
containing various cmdline options to be passed as the last argument
in the argv array of librgw_create(), which nfs-ganesha is
expecting to happen.
Matt Benjamin [Mon, 13 Feb 2017 01:18:26 +0000 (20:18 -0500)]
rgw_file: fix hiwat behavior
Removed logic to skip reclaim processing conditionally on hiwat,
this probably meant to be related to a lowat value, which does
not exist.
Having exercised the hiwat reclaim behavior, noticed that the
path which moves unreachable objects to LRU, could and probably
should remove them altogether when q.size exceeds hiwat. Now
the max unreachable float is lane hiwat, for all lanes.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit b8791b2217e9ca87b2d17b51f283fa14bd68b581) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Sun, 12 Feb 2017 23:20:43 +0000 (18:20 -0500)]
rgw_file: refcnt bugfixes
This change includes 3 related changes:
1. add required lock flags for FHCache updates--this is a crash
bug under concurrent update/lookup
2. omit to inc/dec refcnt on root filehandles in 2 places--the
root handle current is not on the lru list, so it's not
valid to do so
3. based on observation of LRU behavior during creates/deletes,
update (cohort) LRU unref to move objects to LRU when their
refcount falls to SENTINEL_REFCNT--this cheaply primes the
current reclaim() mechanism, so very significanty improves
space use (e.g., after deletes) in the absence of scans
(which is common due to nfs-ganesha caching)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit beaeff059375b44188160dbde8a81dd4f4f8c6eb) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Sat, 11 Feb 2017 21:38:05 +0000 (16:38 -0500)]
rgw_file: add refcount dout traces at debuglevel 17
These are helpful for checking RGWFileHandle refcnt invariants.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 462034e17f919fb783ee33e2c9fa8089f93fd97d) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Fri, 10 Feb 2017 22:14:16 +0000 (17:14 -0500)]
rgw_file: add pretty-print for RGWFileHandle
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit ef330f385d3407af5f470b5093145f59cc4dcc79) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Mon, 20 Feb 2017 20:05:18 +0000 (15:05 -0500)]
rgw_file: fix marker computation
Fixes: http://tracker.ceph.com/issues/19018 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 4454765e7dd08535c50d24205858e18dba4b454c)
Matt Benjamin [Mon, 20 Feb 2017 01:34:31 +0000 (20:34 -0500)]
rgw_file: rgw_readdir can't list multi-segment dirs
This issue has one root cause in librgw, namely that the marker
argument to these requests was incorrectly formatted (though the
marker cache was working as intended).
Secondarily, for nfs-ganesha users, there is a compounding issue
that the RGW fsal was required by "temporary" convention to
populate the entire dirent cache for a directory on a single
readdir() invocation--the cache_inode/mdcache implementations
invariantly pass (before future 2.5 changesets, currently in
progress) a null pointer for the start cookie offset, intended
to convey this.
Fixes: http://tracker.ceph.com/issues/18991 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 2cd60ee9712291b906123aca1704288b18a9742b)
Matt Benjamin [Sun, 19 Feb 2017 23:21:06 +0000 (18:21 -0500)]
rgw_file: allow setattr on placeholder directories
When a POSIX path <bucket>/foo/ is known only as an implicit path
segment from other objects (e.g., <bucket>/foo/bar.txt), a case
that would usually arise from S3 upload of such an object, an
RGWFileHandle object representing "<bucket>/foo/" will be constructed
as needed, with no backing in RGW.
This is by design, but subsequently, if a setattr is performed on
such a handle, we must be ready to create the object inline with
storing the attributes.
Fixes: http://tracker.ceph.com/issues/18989 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 55eec1c0a0e136736961423b7b6244d0f3693c1a)
Gui Hecheng [Thu, 2 Mar 2017 09:21:57 +0000 (17:21 +0800)]
rgw_file: posix style atime,ctime,mtime
As an ganesha FSAL backend, rgw_file should properly maintain
the atime,ctime,mtime properly against operations such as:
(read,write) for file
(create,unlink,mkdir,rmdir,rename) for dir
(setattr) for file and dir
Matt Benjamin [Mon, 20 Feb 2017 20:05:18 +0000 (15:05 -0500)]
rgw_file: fix marker computation
Fixes: http://tracker.ceph.com/issues/19018 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 4454765e7dd08535c50d24205858e18dba4b454c) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Mon, 20 Feb 2017 01:34:31 +0000 (20:34 -0500)]
rgw_file: rgw_readdir can't list multi-segment dirs
This issue has one root cause in librgw, namely that the marker
argument to these requests was incorrectly formatted (though the
marker cache was working as intended).
Secondarily, for nfs-ganesha users, there is a compounding issue
that the RGW fsal was required by "temporary" convention to
populate the entire dirent cache for a directory on a single
readdir() invocation--the cache_inode/mdcache implementations
invariantly pass (before future 2.5 changesets, currently in
progress) a null pointer for the start cookie offset, intended
to convey this.
Fixes: http://tracker.ceph.com/issues/18991 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 2cd60ee9712291b906123aca1704288b18a9742b) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Sun, 19 Feb 2017 23:21:06 +0000 (18:21 -0500)]
rgw_file: allow setattr on placeholder directories
When a POSIX path <bucket>/foo/ is known only as an implicit path
segment from other objects (e.g., <bucket>/foo/bar.txt), a case
that would usually arise from S3 upload of such an object, an
RGWFileHandle object representing "<bucket>/foo/" will be constructed
as needed, with no backing in RGW.
This is by design, but subsequently, if a setattr is performed on
such a handle, we must be ready to create the object inline with
storing the attributes.
Fixes: http://tracker.ceph.com/issues/18989 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 55eec1c0a0e136736961423b7b6244d0f3693c1a) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Javier M. Mellid [Tue, 26 Jul 2016 12:53:44 +0000 (14:53 +0200)]
rgw: multipart uploads copy part support
Add multipart uploads copy part feature.
Fixes: http://tracker.ceph.com/issues/12790 Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>
(cherry picked from commit 949480c2e9760855ed6a0501d364d5e766c8207d)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
rgw: make sending Content-Length in 204 and 304 controllable
This commit introduces a new configurable "rgw print prohibited
content length" to let operator decide whether RadosGW complies
to RFC 7230 (a part of the HTTP specification) or violates it
but follows the Swift's behavior.
Casey Bodley [Thu, 9 Mar 2017 20:24:08 +0000 (15:24 -0500)]
rgw: clear old zone short ids on period update
the short ids of old, removed zones were being kept in the period to
guard against hash collisions with new zones
but for a hash collision to cause a wrong object to sync, that object
would have to be uploaded simultaneously to two different zones that had
the same short id
to avoid this, we just have to prevent the period from containing two
colliding zones at the same time - we don't have to remember old zone
short ids forever
Nathan Cutler [Thu, 23 Mar 2017 08:46:53 +0000 (09:46 +0100)]
Merge branch 'jewel-next' into jewel
Branch "jewel-next" was merged to jewel on Dec 22 (5b402f8a7b), and then PR
https://github.com/ceph/ceph/pull/13143 merged to jewel-next much later (Feb
1st), and then jewel-next was never merged to jewel again before we tagged
v10.2.6.
This merge brings in a single commit, i.e. the one from
https://github.com/ceph/ceph/pull/13143
Reported-by: Ken Dreyer <kdreyer@redhat.com> Signed-off-by: Nathan Cutler <ncutler@suse.com>
jewel: osd/PGLog: reindex properly on pg log split
When PGLog::IndexedLog::split_into runs it builds the list, which means
the old indices are wrong (point to bad memory), but index() will not
rebuild them because ever since b858e86 we won't rebuild them if they
are already built. Fix that by calling unindex() before the split.
Based on 643ae42cf27f16dd6ed4e1402acc0483bb9fca74. Notice that both
the child and the parent log are re-indexed in Jewel, so the only
problem is missing unindex().
Conflicts:
qa/workunits/rbd/rbd-nbd.sh: the original commit removes
TOO_LARGE_IMAGE test, do the same thing
src/tools/rbd_nbd/rbd-nbd.cc: help git to add
"include/stringify.h"
Other changes:
src/tools/rbd_nbd/rbd-nbd.cc: #include <fstream> so
the code compiles
Kefu Chai [Tue, 7 Mar 2017 10:49:46 +0000 (18:49 +0800)]
osd/ReplicatedPG: try with pool's use-gmt setting if hitset archive not found
due to http://tracker.ceph.com/issues/19185, a hammer OSD could store
pg_hit_set_info_t with "use-gmt = true" even the pool setting is false.
so we use the pool setting as a fallback if the hitset archive is not
found locally and the pool.use_gmt does not match with hitset.use_gmt.
Fixes: http://tracker.ceph.com/issues/19185 Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts:
osd/ReplicatedPG.cc: This cannot be cherry-picked from master
because: hammer should upgrade to jewel first before moving to a
higher version. so there is no necessary to include this workaround
in releases later than jewel.
Matt Benjamin [Thu, 23 Feb 2017 15:21:38 +0000 (10:21 -0500)]
rgw_file: return of RGWFileHandle::FLAG_EXACT_MATCH
Allow callers of rgw_lookup() on objects attested in an
rgw_readdir() callback the ability to bypass exact match in
RGWLibFS::stat_leaf() case 2, but restore exact match enforcement
for general lookups.
This preserves required common_prefix namespace behavior, but
prevents clients from eerily permitting things like "cd sara0" via
partial name match on "sara01."
Fixes: http://tracker.ceph.com/issues/19059 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 70ef7d45e0abf2661bd4e23161d4e70cf5178079)
Matt Benjamin [Sun, 19 Feb 2017 22:43:17 +0000 (17:43 -0500)]
rgw_file: invalid use of RGWFileHandle::FLAG_EXACT_MATCH
The change which introduced this flag also caused it to be
given as the flags argument to RGWLibFS::stat_leaf() when called
from rgw_lookup().
This was incorrect: in particular, when a directory is known only
as a common prefix of other objects, the AWS namespace mapping
convention requires lookup("foo") to match a non-materialized
instance of "foo/" (case 2 in RGWLibFS::stat_leaf's stat loop).
Fixes: http://tracker.ceph.com/issues/18992 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit e31e9eb980f958640150e8d7f17de1b9e5478b1e)
Matt Benjamin [Wed, 22 Feb 2017 19:57:59 +0000 (14:57 -0500)]
rgw_file: rgw_lookup: don't ref for "/" or ".."
These refs won't be returned by nfs-ganesha, and are sufficiently
magical that other consumers should be persuaded to understand
their specialness.
Fixes: http://tracker.ceph.com/issues/19060 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit dea8d1ee373399a21851690a9753388b659b8ede)
Matt Benjamin [Wed, 22 Feb 2017 15:24:29 +0000 (10:24 -0500)]
rgw_file: avoid stranding invalid-name bucket handles in fhcache
To avoid a string copy in the common mkdir path, handles for
proposed buckets currently are staged in the handle table, before
being rejected. They need to be destaged, not just marked deleted
(because deleted objects are now assumed not to be linked, as of beaeff059375b44188160dbde8a81dd4f4f8c6eb).
This triggered an unhandled Boost assert when deleting staged
handles, as current safe_link mode requires first removing from
the FHCache.
Fixes: http://tracker.ceph.com/issues/19036 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 6cde812c92e5bba9f85fbf8486ebe69b55952370)
ff0e521 resolved the issue for the other daemons but not for rgw since
it calls setuid (via civetweb) after the new code sets PR_SET_DUMPABLE.
Add another prctl call before wait_shutdown.
projects / ceph.git / log