]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
12 years agov0.56.4 v0.56.4
Gary Lowell [Mon, 25 Mar 2013 18:02:31 +0000 (11:02 -0700)]
v0.56.4

12 years agorgw: bucket index ops on system buckets shouldn't do anything
Yehuda Sadeh [Mon, 25 Mar 2013 16:50:33 +0000 (09:50 -0700)]
rgw: bucket index ops on system buckets shouldn't do anything

Fixes: #4508
Backport: bobtail
On certain bucket index operations we didn't check whether
the bucket was a system bucket, which caused the operations
to fail. This triggered an error message on bucket removal
operations.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 70e0ee8ba955322832f0c366537ddf7a0288761e)

12 years agosystest: restrict list error acceptance
Josh Durgin [Mon, 25 Feb 2013 23:02:50 +0000 (15:02 -0800)]
systest: restrict list error acceptance

Only ignore errors after the midway point if the midway_sem_post is
defined.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit 5b24a68b6e7d57bac688021b822fb2f73494c3e9)

12 years agosystest: fix race with pool deletion
Josh Durgin [Mon, 25 Feb 2013 22:55:34 +0000 (14:55 -0800)]
systest: fix race with pool deletion

The second test have pool deletion and object listing wait on the same
semaphore to connect and start. This led to errors sometimes when the
pool was deleted before it could be opened by the listing process. Add
another semaphore so the pool deletion happens only after the listing
has begun.

Fixes: #4147
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit b0271e390564119e998e18189282252d54f75eb6)

12 years agoos/FileJournal: fix aio self-throttling deadlock
Sage Weil [Tue, 19 Mar 2013 21:26:16 +0000 (14:26 -0700)]
os/FileJournal: fix aio self-throttling deadlock

This block of code tries to limit the number of aios in flight by waiting
for the amount of data to be written to grow relative to a function of the
number of aios.  Strictly speaking, the condition we are waiting for is a
function of both aio_num and the write queue, but we are only woken by
changes in aio_num, and were (in rare cases) waiting when aio_num == 0 and
there was no possibility of being woken.

Fix this by verifying that aio_num > 0, and restructuring the loop to
recheck that condition on each wakeup.

Fixes: #4079
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit e5940da9a534821d0d8f872c13f9ac26fb05a0f5)

12 years agocommon/MemoryModel: remove logging to /tmp/memlog
Sage Weil [Fri, 22 Mar 2013 20:25:49 +0000 (13:25 -0700)]
common/MemoryModel: remove logging to /tmp/memlog

This was a hack for dev purposes ages ago; remove it.  The predictable
filename is a security issue.

CVE-2013-1882

Reported-by: Michael Scherer <misc@zarb.org>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit c524e2e01da41ab5b6362c117939ea1efbd98095)

12 years agoinit-ceph: clean up temp ceph.conf filename on exit
Sage Weil [Fri, 22 Mar 2013 20:25:43 +0000 (13:25 -0700)]
init-ceph: clean up temp ceph.conf filename on exit

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 6a7ad2eac1db6abca3d7edb23ca9b80751400a23)

12 years agoinit-ceph: push temp conf file to a unique location on remote host
Sage Weil [Fri, 22 Mar 2013 20:25:33 +0000 (13:25 -0700)]
init-ceph: push temp conf file to a unique location on remote host

The predictable file name is a security problem.

CVE-2013-1882

Reported-by: Michael Scherer <misc@zarb.org>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 051734522fea92878dd8139f28ec4e6b01371ede)

12 years agomkcephfs: make remote temp directory name unique
Sage Weil [Fri, 22 Mar 2013 20:25:23 +0000 (13:25 -0700)]
mkcephfs: make remote temp directory name unique

The predictable file name is a security problem.

CVE-2013-1882

Reported-by: Michael Scherer <misc@zarb.org>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit f463ef78d77b11b5ad78b31e9a3a88d0a6e62bca)

12 years agoPG::GetMissing: need to check need_up_thru in MLogRec handler
Samuel Just [Fri, 22 Mar 2013 20:51:14 +0000 (13:51 -0700)]
PG::GetMissing: need to check need_up_thru in MLogRec handler

Backport: bobtail
Fixes: #4534
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 4fe4deafbe1758a6b3570048aca57485bd562440)

12 years agoPG,osd_types: improve check_new_interval debugging
Samuel Just [Fri, 22 Mar 2013 20:48:49 +0000 (13:48 -0700)]
PG,osd_types: improve check_new_interval debugging

Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit d611eba9caf45f2d977c627b123462a073f523a4)

12 years agoFileStore: fix reversed collection_empty return value
Samuel Just [Wed, 6 Mar 2013 00:06:20 +0000 (16:06 -0800)]
FileStore: fix reversed collection_empty return value

Backport: bobtail
Fixes: #4380
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 7a434d10da0f77e1b329de0b06b6645cd73cc81b)

Conflicts:
src/os/FileStore.cc

12 years agoFileStore: set replay guard on create_collection
Samuel Just [Mon, 11 Feb 2013 20:52:07 +0000 (12:52 -0800)]
FileStore: set replay guard on create_collection

This should prevent sequences like:

rmcoll a
mkcoll a
touch a foo
<crash>

from causing trouble by preventing the rmcoll
and mkcoll from being replayed.

Fixes: 4064
Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 411770c45734c9827745ddc4018d86c14f2858a6)

12 years agoFileStore: _split_collection should not create the collection
Samuel Just [Mon, 11 Feb 2013 20:24:14 +0000 (12:24 -0800)]
FileStore: _split_collection should not create the collection

This will simplify adding a replay guard to create_collection.

Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit b184ff581a08c9e6ce5b858f06ccbe9d0e2a170b)

12 years agoclient: use 4MB f_bsize and f_frsize for statfs
Sage Weil [Fri, 22 Feb 2013 23:15:27 +0000 (15:15 -0800)]
client: use 4MB f_bsize and f_frsize for statfs

Old stat(1) reports:

  Block size: 1048576    Fundamental block size: 1048576

and the df(1) arithmetic works out.  New stat(1) reports:

  Block size: 1048576    Fundamental block size: 4096

which is what we are shoving into statvfs, but we have the b_size and
fr_size arithmetic swapped.  However, doing the *correct* reporting would
then break the old stat by making both sizes appear to be 4KB (or
whatever).

Sidestep the issue by making *both* values 4MB.. which is both large enough
to report large FS sizes, and also the default stripe size and thus a
"reasonable" value to report for a block size.

Perhaps in the future, when we no longer care about old userland, we can
report the page size for f_bsize, which is probably the "most correct"
thing to do.

Fixes: #3794. See also #3793.
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 7c94083643891c9d66a117352f312b268bdb1135)

12 years agoos/FileStore: check replay guard on src for collection rename
Sage Weil [Tue, 19 Feb 2013 01:39:46 +0000 (17:39 -0800)]
os/FileStore: check replay guard on src for collection rename

This avoids a problematic sequence like:

     - rename A/ -> B/
     - remove B/1...100
     - destroy B/
     - create A/
     - write A/101...
     <crash>
     - replay A/ -> B/
     - remove B/1...100  (fails but tolerated)
     - destroy B/        (fails with ENOTEMPTY)

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 5fc83c8d9887d2a916af11436ccc94fcbfe59b7a)

12 years agoPG::proc_replica_log: oinfo.last_complete must be *before* first entry in omissing
Samuel Just [Fri, 22 Feb 2013 22:12:28 +0000 (14:12 -0800)]
PG::proc_replica_log: oinfo.last_complete must be *before* first entry in omissing

Fixes: #4189
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 2dae6a68ee85a20220ee940dbe33a2144d43457b)

12 years agoosd/PG: fix typo, missing -> omissing
Sage Weil [Fri, 22 Feb 2013 01:55:21 +0000 (17:55 -0800)]
osd/PG: fix typo, missing -> omissing

From ce7ffc34408bf32c66dc07e6f42d54b7ec489d41.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit dc181224abf6fb8fc583730ae3d90acdf0b80f39)

12 years agoPG::proc_replica_log: adjust oinfo.last_complete based on omissing
Samuel Just [Thu, 21 Feb 2013 23:31:36 +0000 (15:31 -0800)]
PG::proc_replica_log: adjust oinfo.last_complete based on omissing

Otherwise, search_for_missing may neglect to check the missing
set for some objects assuming that if the need version is
prior to last_complete, the replica must have it.

Fixes: #4994
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit ce7ffc34408bf32c66dc07e6f42d54b7ec489d41)

12 years agoosd: fix load_pgs collection handling
Sage Weil [Sat, 9 Feb 2013 08:05:33 +0000 (00:05 -0800)]
osd: fix load_pgs collection handling

On a _TEMP pg, is_pg() would succeed, which meant we weren't actually
hitting the cleanup checks.  Instead, restructure this loop as positive
checks and handle each type of collection we understand.

This fixes _TEMP cleanup.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit b19b6dced85617d594c15631571202aab2f94ae8)

12 years agoosd: fix load_pgs handling of pg dirs without a head
Sage Weil [Sat, 9 Feb 2013 08:04:29 +0000 (00:04 -0800)]
osd: fix load_pgs handling of pg dirs without a head

If there is a pgid that passes coll_t::is_pg() but there is no head, we
will populate the pgs map but then fail later when we try to do
read_state.  This is a side-effect of 55f8579.

Take explicit note of _head collections we see, and then warn when we
find stray snap collections.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 1f80a0b576c0af1931f743ad988b6293cbf2d6d9)

12 years agoOSD::load_pgs: first scan colls before initing PGs
Samuel Just [Thu, 7 Feb 2013 21:34:47 +0000 (13:34 -0800)]
OSD::load_pgs: first scan colls before initing PGs

Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 073f58ede2e473af91f76d01679631c169274af7)

12 years agoosd: Add digest of omap for deep-scrub
David Zafman [Wed, 9 Jan 2013 03:24:13 +0000 (19:24 -0800)]
osd: Add digest of omap for deep-scrub

Add ScrubMap encode/decode v4 message with omap digest
Compute digest of header and key/value.  Use bufferlist
to reflect structure and compute as we go, clearing
bufferlist to reduce memory usage.

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 509a93e89f04d7e9393090563cf7be8e0ea53891)

12 years agoOSD: split temp collection as well
Samuel Just [Fri, 15 Mar 2013 22:13:46 +0000 (15:13 -0700)]
OSD: split temp collection as well

Otherwise, when we eventually remove the temp collection, there might be
objects in the temp collection which were independently pulled into the child
pg collection.  Thus, removing the old stale parent link from its temp
collection also blasts the omap entries and snap mappings for the real child
object.

Backport: bobtail
Fixes: #4452
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit f8d66e87a5c155b027cc6249006b83b4ac9b6c9b)

12 years agoPG: ignore non MISSING pg query in ReplicaActive
Samuel Just [Fri, 15 Mar 2013 02:59:36 +0000 (19:59 -0700)]
PG: ignore non MISSING pg query in ReplicaActive

1) Replica sends notify
2) Prior to processing notify, primary queues query to replica
3) Primary processes notify and activates sending MOSDPGLog
to replica.
4) Primary does do_notifies at end of process_peering_events
and sends to Query.
5) Replica sees MOSDPGLog and activates
6) Replica sees Query and asserts.

In the above case, the Replica should simply ignore the old
Query.

Fixes: #4050
Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 8222cbc8f35c359a35f8381ad90ff0eed5615dac)

12 years agoFileJournal: queue_pos \in [get_top(), header.max_size)
Samuel Just [Wed, 13 Mar 2013 23:04:23 +0000 (16:04 -0700)]
FileJournal: queue_pos \in [get_top(), header.max_size)

If queue_pos == header.max_size when we create the entry
header magic, the entry will be rejected at get_top() on
replay.

Fixes: #4436
Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit de8edb732e3a5ce4471670e43cfe6357ae6a2758)

12 years agoOSD: expand_pg_num after pg removes
Samuel Just [Fri, 15 Mar 2013 01:52:02 +0000 (18:52 -0700)]
OSD: expand_pg_num after pg removes

Otherwise:
1) expand_pg_num removes a splitting pg entry
2) peering thread grabs pg lock and starts split
3) OSD::consume_map grabs pg lock and starts removal

At step 2), we run afoul of the assert(is_splitting)
check in split_pgs.  This way, the would be splitting
pg is marked as removed prior to the splitting state
being updated.

Backport: bobtail
Fixes: #4449
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit f1b031b3cf195cf6df3d3c47c7d606fba63ed4c4)

12 years agoosd: update snap collections for sub_op_modify log records conditionaly
Sage Weil [Mon, 11 Feb 2013 14:23:54 +0000 (06:23 -0800)]
osd: update snap collections for sub_op_modify log records conditionaly

The only remaining caller is sub_op_modify().  If we do have a non-empty
op transaction, we want to do this update, regardless of what we think
last_backfill is (our notion may be not completely in sync with the
primary).  In particular, our last_backfill may be the same object but
a different snapid, but the primary disagrees and is pushing an op
transaction through.

Instead, update the collections if we have a non-empty transaction.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 31e911b63d326bdd06981ec4029ad71b7479ed70)

12 years agoosd: include snaps in pg_log_entry_t::dump()
Sage Weil [Mon, 11 Feb 2013 01:02:45 +0000 (17:02 -0800)]
osd: include snaps in pg_log_entry_t::dump()

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 715d8717a0e8a08fbe97a3e7d3ffd33aa9529d90)

12 years agoosd: unconditionally encode snaps buffer
Sage Weil [Mon, 11 Feb 2013 00:59:48 +0000 (16:59 -0800)]
osd: unconditionally encode snaps buffer

Previously we would only encode the updated snaps vector for CLONE ops.
This doesn't work for MODIFY ops generated by the snap trimmer, which
may also adjust the clone collections.  It is also possible that other
operations may need to populate this field in the future (e.g.,
LOST_REVERT may, although it currently does not).

Fixes: #4071, and possibly #4051.
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 54b6dd924fea3af982f3d729150b6449f318daf2)

12 years agoosd: improve debug output on snap collections
Sage Weil [Sun, 10 Feb 2013 18:57:12 +0000 (10:57 -0800)]
osd: improve debug output on snap collections

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 8b05492ca5f1479589bb19c1ce058b0d0988b74f)

12 years agoPG: check_recovery_sources must happen even if not active
Samuel Just [Thu, 7 Mar 2013 20:53:51 +0000 (12:53 -0800)]
PG: check_recovery_sources must happen even if not active

missing_loc/missing_loc_sources also must be cleaned up
if a peer goes down during peering:

1) pg is in GetInfo, acting is [3,1]
2) we find object A on osd [0] in GetInfo
3) 0 goes down, no new peering interval since it is neither up nor
acting, but peer_missing[0] is removed.
4) pg goes active and try to pull A from 0 since missing_loc did not get
cleaned up.

Backport: bobtail
Fixes: #4371
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit de22b186c497ce151217aecf17a8d35cdbf549bb)

12 years agoHashIndex: _collection_list_partial must tolerate NULL next
Samuel Just [Tue, 5 Mar 2013 23:49:26 +0000 (15:49 -0800)]
HashIndex: _collection_list_partial must tolerate NULL next

Backport: bobtail
Fixes: #4379
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit ce4432adc67dc2fc06dd21ea08e59d179496bcc6)

12 years agoOSD: lock not needed in ~DeletingState()
Samuel Just [Tue, 5 Mar 2013 22:35:39 +0000 (14:35 -0800)]
OSD: lock not needed in ~DeletingState()

No further refs to the object can remain at this point.
Furthermore, the callbacks might lock mutexes of their
own.

Backport: bobtail
Fixes: #4378
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit e4bf1bcab159d7c5b720f5da01877c0f67c16d16)

12 years agoReplicatedPG: don't leak reservation on removal
Samuel Just [Sun, 10 Mar 2013 19:50:01 +0000 (12:50 -0700)]
ReplicatedPG: don't leak reservation on removal

Fixes: 4431
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 32bf131e0141faf407b5ff993f75f97516b27c12)

Conflicts:

src/osd/ReplicatedPG.cc

12 years agorgw: set up curl with CURL_NOSIGNAL
Yehuda Sadeh [Tue, 12 Mar 2013 19:56:01 +0000 (12:56 -0700)]
rgw: set up curl with CURL_NOSIGNAL

Fixes: #4425
Backport: bobtail
Apparently, libcurl needs that in order to be thread safe. Side
effect is that if libcurl is not compiled with c-ares support,
domain name lookups are not going to time out.
Issue affected keystone.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 88725316ddcfa02ff110e659f7a8131dc1ea2cfc)

12 years agoosd: mark down connections from old peers
Sage Weil [Fri, 8 Mar 2013 16:56:44 +0000 (08:56 -0800)]
osd: mark down connections from old peers

Close out any connection with an old peer.  This avoids a race like:

- peer marked down
- we get map, mark down the con
- they reconnect and try to send us some stuff
- we share our map to tell them they are old and dead, but leave the con
  open
...
- peer marks itself up a few times, eventually reuses the same port
- sends messages on their fresh con
- we discard because of our old con

This could cause a tight reconnect loop, but it is better than wrong
behavior.

Other possible fixes:
 - make addr nonce truly unique (augment pid in nonce)
 - make a smarter 'disposable' msgr state (bleh)

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 881e9d850c6762290f8be24da9e74b9dc112f1c9)

12 years agoosd/PG: rename require_same_or_newer_map -> is_same_or_newer_map
Sage Weil [Fri, 8 Mar 2013 16:53:40 +0000 (08:53 -0800)]
osd/PG: rename require_same_or_newer_map -> is_same_or_newer_map

This avoids confusion with the OSD method of the same name, and better
matches what the function tests (and does not do).

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit ba7e815a18cad110525f228db1b3fe39e011409e)

Conflicts:

src/osd/ReplicatedPG.cc

12 years agolog: drop default 'log max recent' from 100k -> 10k
Sage Weil [Mon, 11 Mar 2013 23:25:16 +0000 (16:25 -0700)]
log: drop default 'log max recent' from 100k -> 10k

Use less memory.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit c021c5ccf0c063cccd7314964420405cea6406de)

12 years agoFix radosgw actually reloading after rotating logs.
Jan Harkes [Fri, 8 Mar 2013 17:45:57 +0000 (12:45 -0500)]
Fix radosgw actually reloading after rotating logs.

The --signal argument to Debian's start-stop-daemon doesn't
make it send a signal, but defines which signal should be send
when --stop is specified.

Signed-off-by: Jan Harkes <jaharkes@cs.cmu.edu>
(cherry picked from commit 44f1cc5bc42f9bb6d5a386037408d2de17dc5413)

12 years agocommon: reduce default in-memory logs for non-daemons
Josh Durgin [Thu, 7 Mar 2013 01:42:03 +0000 (17:42 -0800)]
common: reduce default in-memory logs for non-daemons

The default of 100000 can result in hundreds of MBs of extra memory
used. This was most obvious when using librbd with caching enabled,
since there was a dout(0) accidentally left in the ObjectCacher.

refs: #4352
backport: bobtail
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit 7c208d2f8e3f28f4055a4ae51eceae892dcef1dc)

12 years agoosd: allow (some) log trim when degraded, but not during recovery
Sage Weil [Sat, 23 Feb 2013 01:01:53 +0000 (17:01 -0800)]
osd: allow (some) log trim when degraded, but not during recovery

We allow some trim during degraded, although we keep more entries around to
improve our chances of a restarting OSD of doing log-based recovery.

Still disallow during recovery...

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 6d89b34e5608c71b49ef33ab58340e90bd8da6e4)

12 years agoosd: restructure calc_trim
Sage Weil [Mon, 25 Feb 2013 23:33:35 +0000 (15:33 -0800)]
osd: restructure calc_trim

No functional change, except that we log more debug, yay!

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 86df164d04f6e31a0f20bbb94dbce0599c0e8b3d)

12 years agoosd: allow pg log trim during (non-classic) scrub
Sage Weil [Sat, 23 Feb 2013 00:48:02 +0000 (16:48 -0800)]
osd: allow pg log trim during (non-classic) scrub

Chunky (and deep) scrub do not care about PG log trimming.  Classic scrub
still does.

Deep scrub can take a long time, so not trimming the log during that period
may eat lots of RAM; avoid that!

Might fix: #4179
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 0ba8db6b664205348d5499937759916eac0997bf)

12 years agomsgr: drop messages on cons with CLOSED Pipes
Sage Weil [Thu, 28 Feb 2013 20:46:00 +0000 (12:46 -0800)]
msgr: drop messages on cons with CLOSED Pipes

Back in commit 6339c5d43974f4b495f15d199e01a141e74235f5, we tried to make
this deal with a race between a faulting pipe and new messages being
queued.  The sequence is

- fault starts on pipe
- fault drops pipe_lock to unregister the pipe
- user (objecter) queues new message on the con
- submit_message reopens a Pipe (due to this bug)
- the message managed to make it out over the wire
- fault finishes faulting, calls ms_reset
- user (objecter) closes the con
- user (objecter) resends everything

It appears as though the previous patch *meant* to drop *m on the floor in
this case, which is what this patch does.  And that fixes the crash I am
hitting; see #4271.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 0f42eddef5da6c1babe9ed51ceaa3212a42c2ec4)

12 years agoFix output of 'ceph osd tree --format=json'
Concubidated [Fri, 8 Mar 2013 21:44:39 +0000 (13:44 -0800)]
Fix output of 'ceph osd tree --format=json'

Signed-off-by: Tyler Brekke <tyler.brekke@inktank.com>
(cherry picked from commit 9bcba944c6586ad5f007c0a30e69c6b5a886510b)

12 years agodeb: Add ceph-coverage to ceph-test deb package
Sam Lang [Tue, 12 Feb 2013 17:32:29 +0000 (11:32 -0600)]
deb:  Add ceph-coverage to ceph-test deb package

Teuthology uses the ceph-coverage script extensively
and expects it to be installed by the ceph task.  Add
the script to the ceph-test debian package so that it
gets installed for that use case.

Signed-off-by: Sam Lang <sam.lang@inktank.com>
(cherry picked from commit 376cca2d4d4f548ce6b00b4fc2928d2e6d41038f)

12 years agorgw: set attrs on various list bucket xml results (swift)
Yehuda Sadeh [Fri, 22 Feb 2013 23:04:37 +0000 (15:04 -0800)]
rgw: set attrs on various list bucket xml results (swift)

Fixes: #4247
The list buckets operation was missing some attrs on the different
xml result entities. This fixes it.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 4384e59ad046afc9ec53a2d2f1fff6a86e645505)

12 years agoformatter: add the ability to dump attrs in xml entities
Yehuda Sadeh [Fri, 22 Feb 2013 23:02:02 +0000 (15:02 -0800)]
formatter: add the ability to dump attrs in xml entities

xml entities may have attrs assigned to them. Add the ability
to set them. A usage example:

formatter->open_array_section_with_attrs("container",
     FormatterAttrs("name", "foo", NULL));

This will generate the following xml entity:
<container name="foo">

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 7cb6ee28073824591d8132a87ea09a11c44efd66)

Conflicts:
src/common/Formatter.cc

12 years agorgw: don't iterate through all objects when in namespace
Yehuda Sadeh [Thu, 7 Mar 2013 03:32:21 +0000 (19:32 -0800)]
rgw: don't iterate through all objects when in namespace

Fixes: #4363
Backport: argonaut, bobtail
When listing objects in namespace don't iterate through all the
objects, only go though the ones that starts with the namespace
prefix

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 6669e73fa50e3908ec825ee030c31a6dbede6ac0)

12 years agoObjectCacher: fix debug log level in split
Josh Durgin [Thu, 28 Feb 2013 20:13:45 +0000 (12:13 -0800)]
ObjectCacher: fix debug log level in split

Level 0 should never be used for this kind of debugging.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit cb3ee33532fb60665f39f6ccb1d69d67279fd5e1)

12 years agorados: remove unused "check_stdio" parameter
Dan Mick [Thu, 24 Jan 2013 21:38:25 +0000 (13:38 -0800)]
rados: remove unused "check_stdio" parameter

Signed-off-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit bb860e49a7faeaf552538a9492ef0ba738c99760)

12 years agorados: obey op_size for 'get'
Sage Weil [Thu, 24 Jan 2013 05:31:11 +0000 (21:31 -0800)]
rados: obey op_size for 'get'

Otherwise we try to read the whole object in one go, which doesn't bode
well for large objects (either non-optimal or simply broken).

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 234becd3447a679a919af458440bc31c8bd6b84f)

12 years agoFileJournal::wrap_read_bl: adjust pos before returning
Samuel Just [Thu, 28 Feb 2013 00:58:45 +0000 (16:58 -0800)]
FileJournal::wrap_read_bl: adjust pos before returning

Otherwise, we may feed an offset past the end of the journal to
check_header in read_entry and incorrectly determine that the entry is
corrupt.

Fixes: 4296
Backport: bobtail
Backport: argonaut
Reviewed-by: Sage Weil <sage@inktank.com>
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 5d54ab154ca790688a6a1a2ad5f869c17a23980a)

12 years agoosd: leave osd_lock locked in shutdown()
Sage Weil [Wed, 16 Jan 2013 21:14:00 +0000 (13:14 -0800)]
osd: leave osd_lock locked in shutdown()

No callers expect the lock to be dropped.

Fixes: #3816
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 98a763123240803741ac9f67846b8f405f1b005b)

12 years agomsg: fix entity_addr_t::is_same_host() for IPv6
Sage Weil [Tue, 26 Feb 2013 22:07:12 +0000 (14:07 -0800)]
msg: fix entity_addr_t::is_same_host() for IPv6

We weren't checking the memcmp return value properly!  Aie...

Backport: bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit c8dd2b67b39a8c70e48441ecd1a5cc3c6200ae97)

12 years agoosd: requeue pg waiters at the front of the finished queue
Sage Weil [Mon, 18 Feb 2013 06:35:50 +0000 (22:35 -0800)]
osd: requeue pg waiters at the front of the finished queue

We could have a sequence like:

- op1
- notify
- op2

in the finished queue.  Op1 gets put on waiting_for_pg, the notify
creates the pg and requeues op1 (and the end), op2 is handled, and
finally op1 is handled.  That breaks ordering; see #2947.

Instead, when we wake up a pg, queue the waiting messages at the front
of the dispatch queue.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 56c5a07708d52de1699585c9560cff8b4e993d0a)

12 years agoosd: pull requeued requests off one at a time
Sage Weil [Mon, 18 Feb 2013 04:49:52 +0000 (20:49 -0800)]
osd: pull requeued requests off one at a time

Pull items off the finished queue on at a time.  In certain cases, an
event may result in new items betting added to the finished queue that
will be put at the *front* instead of the back.  See latest incarnation
of #2947.

Note that this is a significant changed in behavior in that we can
theoretically starve if an event keeps resulting in new events getting
generated.  Beware!

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit f1841e4189fce70ef5722d508289e516faa9af6a)

12 years agomds: open mydir after replay
Sage Weil [Fri, 18 Jan 2013 06:00:42 +0000 (22:00 -0800)]
mds: open mydir after replay

In certain cases, we may replay the journal and not end up with the
dirfrag for mydir open.  This is fine--we just need to open it up and
fetch it below.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit e51299fbce6bdc3d6ec736e949ba8643afc965ec)

12 years agomds: use inode_t::layout for dir layout policy
Greg Farnum [Thu, 21 Feb 2013 17:21:01 +0000 (09:21 -0800)]
mds: use inode_t::layout for dir layout policy

Remove the default_file_layout struct, which was just a ceph_file_layout,
and store it in the inode_t.  Rip out all the annoying code that put this
on the heap.

To aid in this usage, add a clear_layout() function to inode_t.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
12 years agomds: parse ceph.*.layout vxattr key/value content
Sage Weil [Mon, 21 Jan 2013 05:53:37 +0000 (21:53 -0800)]
mds: parse ceph.*.layout vxattr key/value content

Use qi to parse a strictly formatted set of key/value pairs.  Be picky
about whitespace.  Any subset of recognized keys is allowed.  Parse the
same set of keys as the ceph.*.layout.* vxattrs.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 5551aa5b3b5c2e9e7006476b9cd8cc181d2c9a04)

12 years agorgw: fix multipart uploads listing
Yehuda Sadeh [Mon, 18 Feb 2013 17:10:43 +0000 (09:10 -0800)]
rgw: fix multipart uploads listing

Fixes: #4177
Backport: bobtail
Listing multipart uploads had a typo, and was requiring the
wrong resource (uploadId instead of uploads).

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit db99fb4417b87301a69cb37b00c35c838b77197e)

12 years agorgw: don't copy object when it's copied into itself
Yehuda Sadeh [Fri, 15 Feb 2013 18:22:54 +0000 (10:22 -0800)]
rgw: don't copy object when it's copied into itself

Fixes: #4150
Backport: bobtail

When object copied into itself, object will not be fully copied: tail
reference count stays the same, head part is rewritten.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 34f885be536d0ac89c10fd29b1518751d2ffc547)

12 years agoPG: remove weirdness log for last_complete < log.tail
Samuel Just [Tue, 19 Feb 2013 18:49:33 +0000 (10:49 -0800)]
PG: remove weirdness log for last_complete < log.tail

In the case of a divergent object prior to log.tail,
last_complete may end up before log.tail.

Backport: bobtail
Fixes #4174
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit dbadb3e2921297882c5836c67ca32bb8ecdc75db)

Conflicts:

src/osd/PG.cc

12 years agoStrip any trailing whitespace from rbd showmapped
James Page [Mon, 18 Feb 2013 16:24:54 +0000 (16:24 +0000)]
Strip any trailing whitespace from rbd showmapped

More recent versions of ceph append a bit of whitespace to the line
after the name of the /dev/rbdX device; this causes the monitor check
to fail as it can't find the device name due to the whitespace.

This fix excludes any characters after the /dev/rbdN match.
(cherry picked from commit ad84ea07cac5096de38b51b8fc452c99f016b8d8)

12 years agoMerge pull request #64 from dalgaaf/wip-bobtail-memleaks
Sage Weil [Mon, 18 Feb 2013 05:52:55 +0000 (21:52 -0800)]
Merge pull request #64 from dalgaaf/wip-bobtail-memleaks

cherry-pick some memleak fixes from master to bobtail

12 years agorgw/rgw_rest.cc: fix 4K memory leak 64/head
Danny Al-Gaaf [Fri, 8 Feb 2013 16:17:59 +0000 (17:17 +0100)]
rgw/rgw_rest.cc: fix 4K memory leak

Fix 4K memory leak in case RGWClientIO::read() fails in
read_all_chunked_input().

Error from cppcheck was:
Checking src/rgw/rgw_rest.cc...
[src/rgw/rgw_rest.cc:688]: (error) Memory leak: data

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 89df090e04ef9fc5aae29122df106b0347786fab)

12 years agoSyntheticClient.cc: fix some memory leaks in the error handling
Danny Al-Gaaf [Fri, 8 Feb 2013 16:14:19 +0000 (17:14 +0100)]
SyntheticClient.cc: fix some memory leaks in the error handling

Fix some memory leaks in case of error handling due to failed
client->open() calls.

Error from cppcheck was:
[src/client/SyntheticClient.cc:1980]: (error) Memory leak: buf
[src/client/SyntheticClient.cc:2040]: (error) Memory leak: buf
[src/client/SyntheticClient.cc:2090]: (error) Memory leak: buf
(cherry picked from commit f0ba80756d1c3c313014ad7be18191981fb545be)

12 years agorgw/rgw_xml.cc: fix realloc memory leak in error case
Danny Al-Gaaf [Fri, 8 Feb 2013 15:57:20 +0000 (16:57 +0100)]
rgw/rgw_xml.cc: fix realloc memory leak in error case

Fix error from cppcheck:

[src/rgw/rgw_xml.cc:212]: (error) Common realloc mistake: 'buf'
  nulled but not freed upon failure

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit d48cc789ea075ba2745754035640ada4131b2119)

12 years agoos/FileStore.cc: fix realloc memory leak in error case
Danny Al-Gaaf [Fri, 8 Feb 2013 15:54:33 +0000 (16:54 +0100)]
os/FileStore.cc: fix realloc memory leak in error case

Fix error from cppcheck:

[src/os/FileStore.cc:512]: (error) Common realloc mistake: 'fiemap'
  nulled but not freed upon failure

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit c92a0f552587a232f66620170660d6b2ab6fb3a5)

12 years agocommon/fiemap.cc: fix realloc memory leak
Danny Al-Gaaf [Fri, 8 Feb 2013 15:49:36 +0000 (16:49 +0100)]
common/fiemap.cc: fix realloc memory leak

Fix error from cppcheck:

[src/common/fiemap.cc:73]: (error) Common realloc mistake: 'fiemap'
  nulled but not freed upon failure

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit f26f1470e7af36fa1eb8dc59c8a7c62c3c3a22ba)

12 years agoosd/OSDCap: add unit test for parsing pools/objects with _ and -
Sage Weil [Thu, 14 Feb 2013 19:37:57 +0000 (11:37 -0800)]
osd/OSDCap: add unit test for parsing pools/objects with _ and -

Hunting #4122, where a user saw

2013-02-13 19:39:25.467916 7f766fdb4700 10 osd.0 10  session 0x2c8cc60 client.libvirt has caps osdcap[grant(object_prefix rbd^@children  class-read),grant(pool libvirt^@pool^@test rwx)] 'allow class-read object_prefix rbd_children, allow pool libvirt-pool-test rwx'

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 2ce28ef1d7f95e71e1043912dfa269ea3b0d1599)
(cherry picked from commit a6534bc8a0247418d5263b765772d5266f99229c)

12 years agoosd/OSDCap: tweak unquoted_word parsing in osd caps
Sage Weil [Thu, 14 Feb 2013 23:39:43 +0000 (15:39 -0800)]
osd/OSDCap: tweak unquoted_word parsing in osd caps

Newer versions of spirit (1.49.0-3.1ubuntu1.1 in quantal, in particular)
dislike the construct with alnum and replace the - and _ with '\0' in the
resulting string.

Fixes: #4122
Backport: bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit 6c504d96c1e4fbb67578fba0666ca453b939c218)

12 years agov0.56.3 v0.56.3
Gary Lowell [Wed, 13 Feb 2013 18:10:20 +0000 (10:10 -0800)]
v0.56.3

12 years agorgw: change json formatting for swift list container
Yehuda Sadeh [Fri, 8 Feb 2013 21:14:49 +0000 (13:14 -0800)]
rgw: change json formatting for swift list container

Fixes: #4048
There is some difference in the way swift formats the
xml output and the json output for list container. In
xml the entity is named 'name' and in json it is named
'subdir'.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 3e4d79fe42dfc3ca70dc4d5d2aff5223f62eb34b)

12 years agolibrbd: unprotect any non-unprotected snapshot
Josh Durgin [Tue, 12 Feb 2013 01:08:55 +0000 (17:08 -0800)]
librbd: unprotect any non-unprotected snapshot

Include snapshots in the UNPROTECTING state as well, which can occur
after an unprotect is interrupted.

Fixes: #4100
Backport: bobtail
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit fe283813b44a7c45def6768ea0788a3a0635957e)

12 years agojava: make CephMountTest use user.* xattr names
Sage Weil [Sat, 9 Feb 2013 05:36:13 +0000 (21:36 -0800)]
java: make CephMountTest use user.* xattr names

Changes to the xattr code in Ceph require
a few tweaks to existing test cases.
Specifically, there is now a ceph.file.layout
xattr by default and user defined xattrs
are prepended with "user."

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Noah Watkins <noahwatkins@gmail.com>
12 years agomon: fix typo in C_Stats
Sage Weil [Fri, 8 Feb 2013 17:59:25 +0000 (09:59 -0800)]
mon: fix typo in C_Stats

Broken by previous commit.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 3cf3710be0b4cccc8de152a97be50d983c35116d)

12 years agomon: retry PGStats message on EAGAIN
Sage Weil [Fri, 8 Feb 2013 07:13:11 +0000 (23:13 -0800)]
mon: retry PGStats message on EAGAIN

If we get EAGAIN from a paxos restart/election/whatever, we should
restart the message instead of just blindly acking it.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Luis <joao.luis@inktank.com>
(cherry picked from commit 4837063d447afb45554f55bb6fde1c97559acd4b)

12 years agomon: handle -EAGAIN in completion contexts
Sage Weil [Fri, 8 Feb 2013 06:06:14 +0000 (22:06 -0800)]
mon: handle -EAGAIN in completion contexts

We can get ECANCELED, EAGAIN, or success out of the completion contexts,
but in the EAGAIN case (meaning there was an election) we were sending
a success to the client.  This resulted in client hangs and all-around
confusion when the monitor cluster was thrashing.

Backport: bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Luis <joao.luis@inktank.com>
(cherry picked from commit 17827769f1fe6d7c4838253fcec3b3a4ad288f41)

12 years agoosd: only share maps on hb connection of OSD_HBMSGS feature is set
Sage Weil [Tue, 12 Feb 2013 22:11:09 +0000 (14:11 -0800)]
osd: only share maps on hb connection of OSD_HBMSGS feature is set

Back in 1bc419a7affb056540ba8f9b332b6ff9380b37af we started sharing maps
with dead osds via the heartbeat connection, but old code will crash on an
unexpected message.  Only do this if the OSD_HBMSGS feature is present.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 302b26ff70ee5539da3dcb2e5614e2b7e83b9dcd)

12 years agoosd: tolerate unexpected messages on the heartbeat interface
Sage Weil [Tue, 12 Feb 2013 22:10:51 +0000 (14:10 -0800)]
osd: tolerate unexpected messages on the heartbeat interface

We should note but not crash on unexpected messages.  Announce this awesome
new "capability" via a feature bit.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit afda30aeaae0a65f83c6886658354ad2b57c4c43)

Conflicts:

src/include/ceph_features.h

12 years agoMerge remote-tracking branch 'gh/wip-bobtail-osd-msgr' into bobtail
Sage Weil [Tue, 12 Feb 2013 21:39:52 +0000 (13:39 -0800)]
Merge remote-tracking branch 'gh/wip-bobtail-osd-msgr' into bobtail

12 years agotest_libcephfs: fix xattr test
Sage Weil [Wed, 30 Jan 2013 19:32:23 +0000 (11:32 -0800)]
test_libcephfs: fix xattr test

Ignore the ceph.*.layout xattrs.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit b0d4dd21c7be86eb47728a4702a3c67ca44424ac)

12 years agoradosgw-admin: fix cli test
Sage Weil [Fri, 8 Feb 2013 06:51:29 +0000 (22:51 -0800)]
radosgw-admin: fix cli test

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 1b05b0edbac09d1d7cf0da2e536829df05e48573)

12 years agoMerge remote-tracking branch 'gh/wip-bobtail-vxattrs' into bobtail
Sage Weil [Fri, 8 Feb 2013 07:25:30 +0000 (23:25 -0800)]
Merge remote-tracking branch 'gh/wip-bobtail-vxattrs' into bobtail

12 years agomon: enforce reweight be between 0..1
Sage Weil [Mon, 4 Feb 2013 17:14:39 +0000 (09:14 -0800)]
mon: enforce reweight be between 0..1

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Luis <joao.luis@inktank.com>
(cherry picked from commit 4e29c95d6f61daa838888840cef0cceedc0fcfdd)

12 years agoPG: dirty_info on handle_activate_map
Samuel Just [Thu, 7 Feb 2013 18:38:00 +0000 (10:38 -0800)]
PG: dirty_info on handle_activate_map

We need to make sure the pg epoch is persisted during
activate_map.

Backport: bobtail
Reviewed-by: Sage Weil <sage@inktank.com>
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit dbce1d0dc919e221523bd44e1d0834711da1577d)

12 years agoosd: flush peering queue (consume maps) prior to boot
Sage Weil [Thu, 7 Feb 2013 18:21:49 +0000 (10:21 -0800)]
osd: flush peering queue (consume maps) prior to boot

If the osd itself is behind on many maps during boot, it will get more and
(as part of that) flush the peering wq to ensure the pgs consume them.
However, it is possible for OSD to have latest/recnet maps, but pgs to be
behind, and to jump directly to boot and join.  The OSD is then laggy and
unresponsive because the peering wq is way behind.

To avoid this, call consume_map() (kick the peering wq) at the end of
init and flush it to ensure we are *internally* all caught up before we
consider joining the cluster.

I'm pretty sure this is the root cause of #3905 and possibly #3995.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit af95d934b039d65d3667fc022e2ecaebba107b01)

12 years agorgw: a tool to fix clobbered bucket info in user's bucket list
Yehuda Sadeh [Thu, 7 Feb 2013 01:10:00 +0000 (17:10 -0800)]
rgw: a tool to fix clobbered bucket info in user's bucket list

This fixes bad entries in user's bucket list that may have occured
due to issue #4039. Syntax:

 $ radosgw-admin user check --uid=<uid> [--fix]

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 9cb6c33f0e2281b66cc690a28e08459f2e62ca13)

Conflicts:
src/rgw/rgw_admin.cc

12 years agorgw: bucket recreation should not clobber bucket info
Yehuda Sadeh [Thu, 7 Feb 2013 00:43:48 +0000 (16:43 -0800)]
rgw: bucket recreation should not clobber bucket info

Fixes: #4039
User's list of buckets is getting modified even if bucket already
exists. This fix removes the newly created directory object, and
makes sure that user info's data points at the correct bucket.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 9d006ec40ced9d97b590ee07ca9171f0c9bec6e9)

Conflicts:
src/rgw/rgw_op.cc
src/rgw/rgw_rados.cc

12 years agorgw: a tool to fix buckets with leaked multipart references
Yehuda Sadeh [Tue, 5 Feb 2013 22:50:54 +0000 (14:50 -0800)]
rgw: a tool to fix buckets with leaked multipart references

Checks specified bucket for the #4011 symptoms, optionally fix
the issue.

sytax:
  radosgw-admin bucket check --bucket=<bucket> [--fix]

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 2d8faf8e5f15e833e6b556b0f3c4ac92e4a4151e)

Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rados.h

12 years agorgw: radosgw-admin object unlink
Yehuda Sadeh [Tue, 5 Feb 2013 21:54:11 +0000 (13:54 -0800)]
rgw: radosgw-admin object unlink

Add a radosgw-admin option to remove object from bucket index

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 16235a7acb9543d60470170bb2a09956364626cd)

Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rados.h
src/test/cli/radosgw-admin/help.t

12 years agoosd: kill unused addr-based send_map()
Sage Weil [Fri, 25 Jan 2013 17:30:00 +0000 (09:30 -0800)]
osd: kill unused addr-based send_map()

Not used, old API, bad.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit e359a862199c8a94cb238f7271ba1b0edcc0863c)

12 years agoosd: share incoming maps via Connection*, not addrs
Sage Weil [Fri, 25 Jan 2013 17:29:37 +0000 (09:29 -0800)]
osd: share incoming maps via Connection*, not addrs

Kill a set of parallel methods that are using the old addr/inst-based
msgr APIs, and instead use Connection handles.  This is much safer and gets
us closer to killing the old msgr API.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 5e2fab54a4fdf2f59e2b635cbddef8a5909acb7c)

12 years agoosd: pass new maps to dead osds via existing Connection
Sage Weil [Fri, 25 Jan 2013 17:27:00 +0000 (09:27 -0800)]
osd: pass new maps to dead osds via existing Connection

Previously we were sending these maps to dead osds via their old addrs
using a new outgoing connection and setting the flags so that the msgr
would clean up.  That mechanism is possibly buggy and fragile, and we can
avoid it entirely if we just reuse the existing heartbeat Connection.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 1bc419a7affb056540ba8f9b332b6ff9380b37af)

12 years agoosd: requeue osdmaps on heartbeat connections for cluster connection
Sage Weil [Fri, 25 Jan 2013 17:25:28 +0000 (09:25 -0800)]
osd: requeue osdmaps on heartbeat connections for cluster connection

If we receive an OSDMap on the cluster connection, requeue it for the
cluster messenger, and process it there where we normally do.  This avoids
any concerns about locking and ordering rules.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 76705ace2e9767939aa9acf5d9257c800f838854)

12 years agomsgr: add get_loopback_connection() method
Sage Weil [Fri, 25 Jan 2013 17:23:23 +0000 (09:23 -0800)]
msgr: add get_loopback_connection() method

Return the Connection* for ourselves, so we can queue messages for
ourselves.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit a7059eb3f3922cf08c1e5bb5958acc2d45952482)

12 years agoqa: add layout_vxattrs.sh test script
Sage Weil [Sat, 19 Jan 2013 19:33:04 +0000 (11:33 -0800)]
qa: add layout_vxattrs.sh test script

Test virtual xattrs for file and directory layouts.

TODO: create a data pool, add it to the fs, and make sure we can use it.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 61fbe27a52d12ecd98ddeb5fc0965c4f8ee7841a)

12 years agomds: allow dir layout/policy to be removed via removexattr on ceph.dir.layout
Sage Weil [Sat, 19 Jan 2013 18:11:18 +0000 (10:11 -0800)]
mds: allow dir layout/policy to be removed via removexattr on ceph.dir.layout

This lets a user remove a policy that was previously set on a dir.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit db31a1f9f27416e4d531fda716e32d42a275e84f)