]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
12 years agotest: test_store_tool: global init before using LevelDBStore
Joao Eduardo Luis [Mon, 17 Jun 2013 13:43:36 +0000 (14:43 +0100)]
test: test_store_tool: global init before using LevelDBStore

Fixes a segfault

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit a7a7d3fc8a2ba4a30ef136a32f2903d157b3e19a)

12 years agomon: OSDMonitor: fix a bug introduced on 97462a32
Joao Eduardo Luis [Wed, 24 Jul 2013 11:00:28 +0000 (12:00 +0100)]
mon: OSDMonitor: fix a bug introduced on 97462a32

Fixes: #5737
Backport: cuttlefish

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 76cd7ac1c2094b34ad36bea89b2246fa90eb2f6d)

12 years agomon/Paxos: fix pn for uncommitted value during collect/last phase
Sage Weil [Sun, 21 Jul 2013 15:48:18 +0000 (08:48 -0700)]
mon/Paxos: fix pn for uncommitted value during collect/last phase

During the collect/last exchange, peers share any uncommitted values
with the leader.  They are supposed to also share the pn under which
that value was accepted, but were instead using the just-accepted pn
value.  This effectively meant that we *always* took the uncommitted
value; if there were multiples, which one we accepted depended on what
order the LAST messages arrived, not which pn the values were generated
under.

The specific failure sequence I observed:

 - collect
  - learned uncommitted value for 262 from myself
  - send collect with pn 901
 - got last with pn 901 (incorrect) for 200 (old) from peer
  - discard our own value, remember the other
 - finish collect phase
  - ignore old uncommitted value

Fix this by storing a pending_v and pending_pn value whenever we accept
a value.  Use this to send an appropriate pn value in the LAST reply
so that the leader can make it's decision about which uncommitted value
to accept based on accurate information.  Also use it when we learn
the uncommitted value from ourselves.

We could probably be more clever about storing less information here,
for example by omitting pending_v and clearing pending_pn at the
appropriate point, but that would be more fragile.  Similarly, we could
store a pn for *every* commit if we wanted to lay some groundwork for
having multiple uncommitted proposals in flight, but I don't want to
speculate about what is necessary or sufficient for a correct solution
there.

Fixes: #5698
Backport: cuttlefish, bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 20baf662112dd5f560bc3a2d2114b469444c3de8)

12 years agomon/Paxos: debug ignored uncommitted values
Sage Weil [Sun, 21 Jul 2013 15:12:46 +0000 (08:12 -0700)]
mon/Paxos: debug ignored uncommitted values

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 19b29788966eb80ed847630090a16a3d1b810969)

12 years agomon/Paxos: only learn uncommitted value if it is in the future
Sage Weil [Sun, 21 Jul 2013 15:11:22 +0000 (08:11 -0700)]
mon/Paxos: only learn uncommitted value if it is in the future

If an older peer sends an uncommitted value, make sure we only take it
if it is in the future, and at least as new as any current uncommitted
value.

(Prior to the previous patch, peers could send values from long-past
rounds.  The pn values are also bogus.)

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit b3253a453c057914753846c77499f98d3845c58e)

12 years agomon/Paxos: only share uncommitted value if it is next
Sage Weil [Mon, 22 Jul 2013 21:13:23 +0000 (14:13 -0700)]
mon/Paxos: only share uncommitted value if it is next

We may have an uncommitted value from our perspective (it is our lc + 1)
when the collector has a much larger lc (because we have been out for
the last few rounds).  Only share an uncommitted value if it is in fact
the next value.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit b26b7f6e5e02ac6beb66e3e34e177e6448cf91cf)

12 years agov0.61.6 v0.61.6
Gary Lowell [Tue, 23 Jul 2013 20:52:19 +0000 (13:52 -0700)]
v0.61.6

12 years agomon/OSDMonitor: fix base case for 7fb3804fb workaround
Sage Weil [Tue, 23 Jul 2013 20:32:12 +0000 (13:32 -0700)]
mon/OSDMonitor: fix base case for 7fb3804fb workaround

After cluster creation, we have no full map stored and first_committed ==
1.  In that case, there is no need for a full map, since we can get there
from OSDMap() and the incrementals.

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao@inktank.com>
(cherry picked from commit e807770784175b05130bba938113fdbf874f152e)

12 years agomon: OSDMonitor: work around a full version bug introduced in 7fb3804fb
Joao Eduardo Luis [Tue, 23 Jul 2013 16:25:13 +0000 (17:25 +0100)]
mon: OSDMonitor: work around a full version bug introduced in 7fb3804fb

In 7fb3804fb860dcd0340dd3f7c39eec4315f8e4b6 we moved the full version
stashing logic to the encode_trim_extra() function.  However, we forgot
to update the osdmap's 'latest_full' key that should always point to
the latest osdmap full version.  This eventually degenerated in a missing
full version after a trim.  This patch works around this bug by looking
for the latest available full osdmap version in the store and updating
'latest_full' to its proper value.

Related-to: #5704
Backport: cuttlefish

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 97462a3213e5e15812c79afc0f54d697b6c498b1)

12 years agomon: OSDMonitor: update the osdmap's latest_full with the new full version
Joao Eduardo Luis [Tue, 23 Jul 2013 15:36:52 +0000 (16:36 +0100)]
mon: OSDMonitor: update the osdmap's latest_full with the new full version

We used to do this on encode_full(), but since [1] we no longer rely on
PaxosService to manage the full maps for us.  And we forgot to write down
the latest_full version to the store, leaving it in a truly outdated state.

[1] - 7fb3804fb860dcd0340dd3f7c39eec4315f8e4b6

Fixes: #5704
Backport: cuttlefish
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit a815547ed3e5ffdbbb96c8c0c1b8d6dd8c62bfba)

12 years agomon: decline to scrub when paxos is not active
Sage Weil [Thu, 18 Jul 2013 21:35:19 +0000 (14:35 -0700)]
mon: decline to scrub when paxos is not active

In f1ce8d7c955a2443111bf7d9e16b4c563d445712 we close a race between scrub
and paxos commit completion on the leader.  The fix is nontrivial to
backport and probably not worthwhile; just avoid scrubbing at that time
for now.

Note that the actual fix for this is in commit
f1ce8d7c955a2443111bf7d9e16b4c563d445712.

Signed-off-by: Sage Weil <sage@inktank.com>
12 years agov0.61.5 v0.61.5
Gary Lowell [Wed, 17 Jul 2013 23:39:08 +0000 (16:39 -0700)]
v0.61.5

12 years agoceph-disk: rely on /dev/disk/by-partuuid instead of special-casing journal symlinks
Sage Weil [Tue, 16 Jul 2013 20:14:50 +0000 (13:14 -0700)]
ceph-disk: rely on /dev/disk/by-partuuid instead of special-casing journal symlinks

This was necessary when ceph-disk-udev didn't create the by-partuuid (and
other) symlinks for us, but now it is fragile and error-prone.  (It also
appears to be broken on a certain customer RHEL VM.)  See
d7f7d613512fe39ec883e11d201793c75ee05db1.

Instead, just use the by-partuuid symlinks that we spent all that ugly
effort generating.

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 64379e701b3ed862c05f156539506d3382f77aa8)

12 years agomon: Monitor: StoreConverter: clearer debug message on 'needs_conversion()'
Joao Eduardo Luis [Tue, 16 Jul 2013 15:49:48 +0000 (16:49 +0100)]
mon: Monitor: StoreConverter: clearer debug message on 'needs_conversion()'

The previous debug message outputted the function's name, as often our
functions do.  This was however a source of bewilderment, as users would
see those in logs and think their stores would need conversion.  Changing
this message is trivial enough and it will make ceph users happier log
readers.

Backport: cuttlefish
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit ad1392f68170b391d11df0ce5523c2d1fb57f60e)

12 years agomon: Monitor: do not reopen MonitorDBStore during conversion
Joao Eduardo Luis [Tue, 16 Jul 2013 15:45:39 +0000 (16:45 +0100)]
mon: Monitor: do not reopen MonitorDBStore during conversion

We already open the store on ceph_mon.cc, before we start the conversion.
Given we are unable to reproduce this every time a conversion is triggered,
we are led to believe that this causes a race in leveldb that will lead
to 'store.db/LOCK' being locked upon the open this patch removes.

Regardless, reopening the db here is pointless as we already did it when
we reach Monitor::StoreConverter::convert().

Fixes: #5640
Backport: cuttlefish

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 036e6739a4e873863bae3d7d00f310c015dfcdb3)

12 years agomessages/MClientReconnect: clear data when encoding
Sage Weil [Sun, 14 Jul 2013 22:54:29 +0000 (15:54 -0700)]
messages/MClientReconnect: clear data when encoding

The MClientReconnect puts everything in the data payload portion of
the message and nothing in the front portion.  That means that if the
message is resent (socket failure or something), the messenger thinks it
hasn't been encoded yet (front empty) and reencodes, which means
everything gets added (again) to the data portion.

Decoding keep decoding until it runs out of data, so the second copy
means we decode garbage snap realms, leading to the crash in bug

Clearing data each time around resolves the problem, although it does
mean we do the encoding work multiple times.  We could alternatively
(or also) stick some data in the front portion of the payload
(ignored), but that changes the wire protocol and I would rather not
do that.

Fixes: #4565
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 314cf046b0b787ca69665e8751eab6fe7adb4037)

12 years agomon: once sync full is chosen, make sure we don't change our mind
Sage Weil [Mon, 15 Jul 2013 17:05:37 +0000 (10:05 -0700)]
mon: once sync full is chosen, make sure we don't change our mind

It is possible for a sequence like:

 - probe
 - first probe reply has paxos trim that indicates a full sync is
   needed
 - start sync
 - clear store
 - something happens that makes us abort and bootstrap (e.g., the
   provider mon restarts
 - probe
 - first probe reply has older paxos trim bound and we call an election
 - on election completion, we crash because we have no data.

Non-determinism of the probe decision aside, we need to ensure that
the info we share during probe (fc, lc) is accurate, and that once we
clear the store we know we *must* do a full sync.

This is a backport of aa60f940ec1994a61624345586dc70d261688456.

Fixes: #5621
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
12 years agomon: do not scrub if scrub is in progress
Sage Weil [Tue, 9 Jul 2013 21:12:15 +0000 (14:12 -0700)]
mon: do not scrub if scrub is in progress

This prevents an assert from unexpected scrub results from the previous
scrub on the leader.

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 00ae543b3e32f89d906a0e934792cc5309f57696)

12 years agomessages/MPGStats: do not set paxos version to osdmap epoch
Sage Weil [Wed, 10 Jul 2013 17:06:20 +0000 (10:06 -0700)]
messages/MPGStats: do not set paxos version to osdmap epoch

The PaxosServiceMessage version field is meant for client-coordinated
ordering of messages when switching between monitors (and is rarely
used).  Do not fill it with the osdmap epoch lest it be compared to a
pgmap version, which may cause the mon to (near) indefinitely put it on
a wait queue until the pgmap version catches up.

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit b36338be43f43b6dd4ee87c97f2eaa23b467c386)

12 years agoosd/OSDmap: fix OSDMap::Incremental::dump() for new pool names
Sage Weil [Fri, 12 Jul 2013 01:43:24 +0000 (18:43 -0700)]
osd/OSDmap: fix OSDMap::Incremental::dump() for new pool names

The name is always present when pools are created, but not when they are
modified.  Also, a name may be present with a new_pools entry if the pool
is just renamed.  Separate it out completely in the dump.

Backport: cuttlefish, bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 3e4a29111e89588385e63f8d92ce3d67739dd679)

12 years agomon/PaxosService: prevent reads until initial service commit is done
Sage Weil [Mon, 8 Jul 2013 17:49:28 +0000 (10:49 -0700)]
mon/PaxosService: prevent reads until initial service commit is done

Do not process reads (or, by PaxosService::dispatch() implication, writes)
until we have committed the initial service state.  This avoids things like
EPERM due to missing keys when we race with mon creation, triggered by
teuthology tests doing their health check after startup.

Fixes: #5515
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
(cherry picked from commit d08b6d6df7dba06dad73bdec2c945f24afc02717)

12 years agoclient: send all request put's through put_request()
Sage Weil [Fri, 28 Jun 2013 19:21:58 +0000 (12:21 -0700)]
client: send all request put's through put_request()

Make sure all MetaRequest reference put's go through the same path that
releases inode references, including all of the error paths.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 87217e1e3cb2785b79d0dec49bd3f23a827551f5)

12 years agoclient: fix remaining Inode::put() caller, and make method psuedo-private
Sage Weil [Fri, 28 Jun 2013 18:50:11 +0000 (11:50 -0700)]
client: fix remaining Inode::put() caller, and make method psuedo-private

Not sure I can make this actually private and make Client::put_inode() a
friend method (making all of Client a friend would defeat the purpose).
This works well enough, though!

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 9af3b86b25574e4d2cdfd43e61028cffa19bdeb1)

12 years agoclient: use put_inode on MetaRequest inode refs
Sage Weil [Fri, 28 Jun 2013 04:39:35 +0000 (21:39 -0700)]
client: use put_inode on MetaRequest inode refs

When we drop the request inode refs, we need to use put_inode() to ensure
they get cleaned up properly (removed from inode_map, caps released, etc.).
Do this explicitly here (as we do with all other inode put() paths that
matter).

Fixes: #5381
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 81bee6487fb1ce9e090b030d61bda128a3cf4982)

12 years agomon: be smarter about calculating last_epoch_clean lower bound
Sage Weil [Mon, 8 Jul 2013 22:57:48 +0000 (15:57 -0700)]
mon: be smarter about calculating last_epoch_clean lower bound

We need to take PGs whose mapping has not changed in a long time into
account.  For them, the pg state will indicate it was clean at the time of
the report, in which case we can use that as a lower-bound on their actual
latest epoch clean.  If they are not currently clean (at report time), use
the last_epoch_clean value.

Fixes: #5519
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit cc0006deee3153e06ddd220bf8a40358ba830135)

12 years agoosd: report pg stats to mon at least every N (=500) epochs
Sage Weil [Mon, 8 Jul 2013 20:27:58 +0000 (13:27 -0700)]
osd: report pg stats to mon at least every N (=500) epochs

The mon needs a moderately accurate last_epoch_clean value in order to trim
old osdmaps.  To prevent a PG that hasn't peered or received IO in forever
from preventing this, send pg stats at some minimum frequency.  This will
increase the pg stat report workload for the mon over an idle pool, but
should be no worse that a cluster that is getting actual IO and sees these
updates from normal stat updates.

This makes the reported update a bit more aggressive/useful in that the epoch
is the last map epoch processed by this PG and not just one that is >= the
currenting interval.  Note that the semantics of this field are pretty useless
at this point.

See #5519

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit da81228cc73c95737f26c630e5c3eccf6ae1aaec)

12 years agoosd: fix warning
Sage Weil [Wed, 10 Jul 2013 18:32:34 +0000 (11:32 -0700)]
osd: fix warning

From 653e04a79430317e275dd77a46c2b17c788b860b

Backport: cuttlefish, bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit bc291d3fc3fc1cac838565cbe0f25f71d855a6e3)

12 years agoMerge remote-tracking branch 'gh/wip-mon-sync-2' into cuttlefish
Sage Weil [Fri, 12 Jul 2013 22:21:20 +0000 (15:21 -0700)]
Merge remote-tracking branch 'gh/wip-mon-sync-2' into cuttlefish

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
12 years agoGet device-by-path by looking for it instead of assuming 3rd entry.
Sandon Van Ness [Wed, 10 Jul 2013 21:55:52 +0000 (14:55 -0700)]
Get device-by-path by looking for it instead of assuming 3rd entry.

On some systems (virtual machines so far) the device-by-path entry
from udevadm is not always in the same spot so instead actually
look for the right output instead of blindy assuming that its a
specific field in the output.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
Reviewed-by: Gary Lowell <gary.lowell@inktank.com>
12 years agoMerge remote-tracking branch 'gh/cuttlefish' into wip-mon-sync-2 399/head
Sage Weil [Wed, 10 Jul 2013 18:40:37 +0000 (11:40 -0700)]
Merge remote-tracking branch 'gh/cuttlefish' into wip-mon-sync-2

12 years agoosd: limit number of inc osdmaps send to peers, clients
Sage Weil [Wed, 10 Jul 2013 18:02:08 +0000 (11:02 -0700)]
osd: limit number of inc osdmaps send to peers, clients

We should not send an unbounded number of inc maps to our peers or clients.
In particular, if a peer is not contacted for a while, we may think they
have a very old map (say, 10000 epochs ago) and send thousands of inc maps
when the distribution shifts and we need to peer.

Note that if we do not send enough maps, the peers will make do by
requesting the map from somewhere else (currently the mon).  Regardless
of the source, however, we must limit the amount that we speculatively
share as it usually is not needed.

Backport: cuttlefish, bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 653e04a79430317e275dd77a46c2b17c788b860b)

12 years agorgw: Fix return value for swift user not found
Christophe Courtaut [Mon, 1 Jul 2013 12:57:17 +0000 (14:57 +0200)]
rgw: Fix return value for swift user not found

http://tracker.ceph.com/issues/1779 fixes #1779

Adjust the return value from rgw_get_user_info_by_swift call
in RGW_SWIFT_Auth_Get::execute() to have the correct
return code in response.
(cherry picked from commit 4089001de1f22d6acd0b9f09996b71c716235551)

12 years agomon/OSDMonitor: make 'osd crush rm ...' slightly more idempotent
Sage Weil [Wed, 10 Jul 2013 04:55:51 +0000 (21:55 -0700)]
mon/OSDMonitor: make 'osd crush rm ...' slightly more idempotent

This is a manual backport of 18a624fd8b90d9959de51f07622cf0839e6bd9aa.
Do not return immediately if we are looking at uncommitted state.t

Signed-off-by: Sage Weil <sage@inktank.com>
12 years agomon/OSDMonitor: fix base case for loading full osdmap
Sage Weil [Tue, 9 Jul 2013 00:46:40 +0000 (17:46 -0700)]
mon/OSDMonitor: fix base case for loading full osdmap

Right after cluster creation, first_committed is 1 and latest stashed in 0,
but we don't have the initial full map yet.  Thereafter, we do (because we
write it with trim).  Fixes afd6c7d8247075003e5be439ad59976c3d123218.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
(cherry picked from commit 43fa7aabf1f7e5deb844c1f52d451bab9e7d1006)

12 years agomon: fix osdmap stash, trim to retain complete history of full maps
Sage Weil [Mon, 8 Jul 2013 22:04:59 +0000 (15:04 -0700)]
mon: fix osdmap stash, trim to retain complete history of full maps

The current interaction between sync and stashing full osdmaps only on
active mons means that a sync can result in an incomplete osdmap_full
history:

 - mon.c starts a full sync
 - during sync, active osdmap service should_stash_full() is true and
   includes a full in the txn
 - mon.c sync finishes
 - mon.c update_from_paxos gets "latest" stashed that it got from the
   paxos txn
 - mon.c does *not* walk to previous inc maps to complete it's collection
   of full maps.

To fix this, we disable the periodic/random stash of full maps by the
osdmap service.

This introduces a new problem: we must have at least one full map (the first
one) in order for a mon that just synced to build it's full collection.
Extend the encode_trim() process to allow the osdmap service to include
the oldest full map with the trim txn.  This is more complex than just
writing the full maps in the txn, but cheaper--we only write the full
map at trim time.

This *might* be related to previous bugs where the full osdmap was
missing, or case where leveldb keys seemed to 'disappear'.

Fixes: #5512
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit afd6c7d8247075003e5be439ad59976c3d123218)

12 years agomon: implement simple 'scrub' command
Sage Weil [Mon, 8 Jul 2013 22:07:57 +0000 (15:07 -0700)]
mon: implement simple 'scrub' command

Compare all keys within the sync'ed prefixes across members of the quorum
and compare the key counts and CRC for inconsistencies.

Currently this is a one-shot inefficient hammer.  We'll want to make this
work in chunks before it is usable in production environments.

Protect with a feature bit to avoid sending MMonScrub to mons who can't
decode it.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit a9906641a1dce150203b72682da05651e4d68ff5)

Conflicts:

src/mon/MonCommands.h
src/mon/Monitor.cc

12 years agoElector.h: features are 64 bit
Samuel Just [Wed, 3 Jul 2013 18:18:33 +0000 (11:18 -0700)]
Elector.h: features are 64 bit

Fixes: #5497
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Luis <joao.luis@inktank.com>
(cherry picked from commit 3564e304e3f50642e4d9ff25e529d5fc60629093)

12 years agoceph_features.h: declare all features as ULL
Samuel Just [Wed, 3 Jul 2013 18:18:19 +0000 (11:18 -0700)]
ceph_features.h: declare all features as ULL

Otherwise, the first 32 get |'d together as ints.  Then, the result
((int)-1) is sign extended to ((long long int)-1) before being |'d
with the 1LL entries.  This results in ~((uint64_t)0).

Fixes: #5497
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Luis <joao.luis@inktank.com>
(cherry picked from commit 4255b5c2fb54ae40c53284b3ab700fdfc7e61748)

12 years agoPipe: use uint64_t not unsigned when setting features
Samuel Just [Wed, 3 Jul 2013 04:09:36 +0000 (21:09 -0700)]
Pipe: use uint64_t not unsigned when setting features

Fixes: #5497
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Luis <joao.luis@inktank.com>
(cherry picked from commit bc3e2f09f8860555d8b3b49b2eea164b4118d817)

12 years agoclient: remove O_LAZY
Sage Weil [Mon, 8 Jul 2013 18:24:48 +0000 (11:24 -0700)]
client: remove O_LAZY

The once-upon-a-time unique O_LAZY value I chose forever ago is now
O_NOATIME, which means that some clients are choosing relaxed
consistency without meaning to.

It is highly unlikely that a real O_LAZY will ever exist, and we can
select it in the ceph case with the ioctl or libcephfs call, so drop
any support for doing this via open(2) flags.

Update doc/lazy_posix.txt file re: lazy io.

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 94afedf02d07ad4678222aa66289a74b87768810)

12 years agoosd/osd_types: fix pg_stat_t::dump for last_epoch_clean
Sage Weil [Mon, 8 Jul 2013 19:55:20 +0000 (12:55 -0700)]
osd/osd_types: fix pg_stat_t::dump for last_epoch_clean

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 69a55445439fce0dd6a3d32ff4bf436da42f1b11)

12 years agomon: remove bad assert about monmap version
Sage Weil [Fri, 5 Jul 2013 23:03:49 +0000 (16:03 -0700)]
mon: remove bad assert about monmap version

It is possible to start a sync when our newest monmap is 0.  Usually we see
e0 from probe, but that isn't always published as part of the very first
paxos transaction due to the way PaxosService::_active generates it's
first initial commit.

In any case, having e0 here is harmless.

Fixes: #5509
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
(cherry picked from commit 85a1d6cc5d3852c94d1287b566656c5b5024fa13)

12 years agomon/Paxos: fix sync restart
Sage Weil [Fri, 5 Jul 2013 02:33:06 +0000 (19:33 -0700)]
mon/Paxos: fix sync restart

If we have a sync going, and an election intervenes, the client will
try to continue by sending a new start_chunks request.  In order to
ensure that we get all of the paxos commits from our original starting
point (and thus properly update the keys from which they started),
only pay attention if they *also* send their current last_committed
version.  Otherwise, start them at the beginning.

Signed-off-by: Sage Weil <sage@inktank.com>
12 years agomon: uninline _trim_enable and Paxos::trim_{enable,disable} so we can debug them
Sage Weil [Thu, 4 Jul 2013 21:57:06 +0000 (14:57 -0700)]
mon: uninline _trim_enable and Paxos::trim_{enable,disable} so we can debug them

Signed-off-by: Sage Weil <sage@inktank.com>
12 years agomon/Paxos: increase paxos max join drift
Sage Weil [Thu, 4 Jul 2013 21:55:34 +0000 (14:55 -0700)]
mon/Paxos: increase paxos max join drift

A value of 10 is too aggressive for large, long-running syncs. 100 is
about 2 minutes of activity at most, which should be a more forgiving
buffer.

Signed-off-by: Sage Weil <sage@inktank.com>
12 years agomon/Paxos: configure minimum paxos txns separately
Sage Weil [Thu, 4 Jul 2013 21:21:04 +0000 (14:21 -0700)]
mon/Paxos: configure minimum paxos txns separately

We were using paxos_max_join_drift to control the minimum number of
paxos transactions to keep around.  Instead, make this explicit, and
separate from the join drift.

Signed-off-by: Sage Weil <sage@inktank.com>
12 years agomon: include any new paxos commits in each sync CHUNK message
Sage Weil [Fri, 5 Jul 2013 00:09:07 +0000 (17:09 -0700)]
mon: include any new paxos commits in each sync CHUNK message

We already take note of the paxos version when we begin the sync.  As
sync progresses and there are new paxos commits/txns, include those
and update last_committed, so that when sync completes we will have
a full view of everything that happened during sync.

Note that this does not introduce any compatibility change.  This change
*only* affects the provider.  The key difference is that at the end
of the sync, the provide will set version to the latest version, and
not the version from the start of the sync (as was done previously).

Signed-off-by: Sage Weil <sage@inktank.com>
12 years agomon/MonitorDBStore: expose get_chunk_tx()
Sage Weil [Thu, 4 Jul 2013 19:17:28 +0000 (12:17 -0700)]
mon/MonitorDBStore: expose get_chunk_tx()

Allow users get the transaction unencoded.

Signed-off-by: Sage Weil <sage@inktank.com>
12 years agomon: enable leveldb cache by default
Sage Weil [Thu, 4 Jul 2013 00:15:56 +0000 (17:15 -0700)]
mon: enable leveldb cache by default

256 is not as large as the upstream 512 MB, but will help signficiantly and
be less disruptive for existing cuttlefish clusters.

Sort-of backport of e93730b7ffa48b53c8da2f439a60cb6805facf5a.

Signed-off-by: Sage Weil <sage@inktank.com>
12 years agomon/Paxos: make 'paxos trim disabled max versions' much much larger
Sage Weil [Wed, 3 Jul 2013 23:56:06 +0000 (16:56 -0700)]
mon/Paxos: make 'paxos trim disabled max versions' much much larger

108000 is about 3 hours if paxos is going full-bore (1 proposal/second).
That ought to be pretty safe.  Otherwise, we start trimming to soon and a
slow sync will just have to restart when it finishes.

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
(cherry picked from commit 71ebfe7e1abe4795b46cf00dfe1b03d1893368b0)

Conflicts:

src/common/config_opts.h

12 years agomon: do not reopen MonitorDBStore during startup
Sage Weil [Wed, 26 Jun 2013 13:01:40 +0000 (06:01 -0700)]
mon: do not reopen MonitorDBStore during startup

level doesn't seem to like this when it races with an internal compaction
attempt (see below).  Instead, let the store get opened by the ceph_mon
caller, and pull a bit of the logic into the caller to make the flow a
little easier to follow.

    -2> 2013-06-25 17:49:25.184490 7f4d439f8780 10 needs_conversion
    -1> 2013-06-25 17:49:25.184495 7f4d4065c700  5 asok(0x13b1460) entry start
     0> 2013-06-25 17:49:25.316908 7f4d3fe5b700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f4d3fe5b700

 ceph version 0.64-667-g089cba8 (089cba8fc0e8ae8aef9a3111cba7342ecd0f8314)
 1: ceph-mon() [0x649f0a]
 2: (()+0xfcb0) [0x7f4d435dccb0]
 3: (leveldb::Table::BlockReader(void*, leveldb::ReadOptions const&, leveldb::Slice const&)+0x154) [0x806e54]
 4: ceph-mon() [0x808840]
 5: ceph-mon() [0x808b39]
 6: ceph-mon() [0x806540]
 7: (leveldb::DBImpl::DoCompactionWork(leveldb::DBImpl::CompactionState*)+0xdd) [0x7f363d]
 8: (leveldb::DBImpl::BackgroundCompaction()+0x2c0) [0x7f4210]
 9: (leveldb::DBImpl::BackgroundCall()+0x68) [0x7f4cc8]
 10: ceph-mon() [0x80b3af]
 11: (()+0x7e9a) [0x7f4d435d4e9a]
 12: (clone()+0x6d) [0x7f4d4196bccd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit ea1f316e5de21487ae034a1aa929068ba23ac525)

12 years agosysvinit, upstart: handle symlinks to dirs in /var/lib/ceph/*
Sage Weil [Tue, 2 Jul 2013 21:43:17 +0000 (14:43 -0700)]
sysvinit, upstart: handle symlinks to dirs in /var/lib/ceph/*

Match a symlink to a dir, not just dirs.  This fixes the osd case of e.g.,
creating an osd in /data/osd$id in which ceph-disk makes a symlink from
/var/lib/ceph/osd/ceph-$id.

Fix proposed by Matt Thompson <matt.thompson@mandiant.com>; extended to
include the upstart users too.

Fixes: #5490
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 87c98e92d1375c8bc76196bbbf06f677bef95e64)

12 years agorgw: add RGWFormatter_Plain allocation to sidestep cranky strlen()
Sage Weil [Tue, 2 Jul 2013 00:33:11 +0000 (17:33 -0700)]
rgw: add RGWFormatter_Plain allocation to sidestep cranky strlen()

Valgrind complains about an invalid read when we don't pad the allocation,
and because it is inlined we can't whitelist it for valgrind.  Workaround
the warning by just padding our allocations a bit.

Fixes: #5346
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 49ff63b1750789070a8c6fef830c9526ae0f6d9f)

12 years agomds: warn on unconnected snap realms
Yan, Zheng [Wed, 15 May 2013 03:24:36 +0000 (11:24 +0800)]
mds: warn on unconnected snap realms

When there are more than one active MDS, restarting MDS triggers
assertion "reconnected_snaprealms.empty()" quite often. If there
is no snapshot in the FS, the items left in reconnected_snaprealms
should be other MDS' mdsdir. I think it's harmless.

If there are snapshots in the FS, the assertion probably can catch
real bugs. But at present, snapshot feature is broken, fixing it is
non-trivial. So replace the assertion with a warning.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
(cherry picked from commit 26effc0e583b0a3dade6ec81ef26dec1c94ac8b2)

12 years agomon/PGMonitor: use post_paxos_update, not init, to refresh from osdmap
Sage Weil [Wed, 26 Jun 2013 13:53:08 +0000 (06:53 -0700)]
mon/PGMonitor: use post_paxos_update, not init, to refresh from osdmap

We do two things here:
 - make init an one-time unconditional init method, which is what the
   health service expects/needs.
 - switch PGMonitor::init to be post_paxos_update() which is called after
   the other services update, which is what PGMonitor really needs.

This is a new version of the fix originally in commit
a2fe0137946541e7b3b537698e1865fbce974ca6 (and those around it).  That is,
this re-fixes a problem where osds do not see pg creates from their
subscribe due to map_pg_creates() not getting called.

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit e635c47851d185eda557e36bdc4bf3775f7b87a2)

Conflicts:
src/mon/PGMonitor.cc
src/mon/PGMonitor.h

12 years agomon/PaxosService: add post_paxos_update() hook
Sage Weil [Wed, 26 Jun 2013 13:52:01 +0000 (06:52 -0700)]
mon/PaxosService: add post_paxos_update() hook

Some services need to update internal state based on other service's
state, and thus need to be run after everyone has pulled their info out of
paxos.

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 131686980f0a930d5de7cbce8234fead5bd438b6)

12 years agoceph-disk: s/else if/elif/
Greg Farnum [Thu, 27 Jun 2013 21:58:14 +0000 (14:58 -0700)]
ceph-disk: s/else if/elif/

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Joao Luis <joao.luis@inktank.com>
(cherry picked from commit bd8255a750de08c1b8ee5e9c9a0a1b9b16171462)
(cherry picked from commit 9e604ee6943fdb131978afbec51321050faddfc6)

12 years agorgw: fix radosgw-admin buckets list
Yehuda Sadeh [Wed, 26 Jun 2013 18:28:57 +0000 (11:28 -0700)]
rgw: fix radosgw-admin buckets list

Fixes: #5455
Backport: cuttlefish
This commit fixes a regression, where radosgw-admin buckets list
operation wasn't returning any data.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit e1f9fe58d2860fcbb18c92d3eb3946236b49a6ce)

12 years agoceph-disk: use unix lock instead of lockfile class
Sage Weil [Thu, 20 Jun 2013 00:27:49 +0000 (17:27 -0700)]
ceph-disk: use unix lock instead of lockfile class

The lockfile class relies on file system trickery to get safe mutual
exclusion.  However, the unix syscalls do this for us.  More
importantly, the unix locks go away when the owning process dies, which
is behavior that we want here.

Fixes: #5387
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 2a4953b697a3464862fd3913336edfd7eede2487)

12 years agoceph-disk: do not mount over an osd directly in /var/lib/ceph/osd/$cluster-$id
Sage Weil [Thu, 27 Jun 2013 01:27:49 +0000 (18:27 -0700)]
ceph-disk: do not mount over an osd directly in /var/lib/ceph/osd/$cluster-$id

If we see a 'ready' file in the target OSD dir, do not mount our device
on top of it.

Among other things, this prevents ceph-disk activate on stray disks from
stepping on teuthology osds.

Fixes: #5445
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 8a17f33b14d858235dfeaa42be1f4842dcfd66d2)

12 years agomds: fix underwater dentry cleanup
Yan, Zheng [Tue, 2 Apr 2013 07:46:51 +0000 (15:46 +0800)]
mds: fix underwater dentry cleanup

If the underwater dentry is a remove link, we shouldn't mark the
inode clean

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
(cherry picked from commit 81d073fecb58e2294df12b71351321e6d2e69652)

12 years agomon/Elector: cancel election timer if we bootstrap
Sage Weil [Tue, 25 Jun 2013 01:51:07 +0000 (18:51 -0700)]
mon/Elector: cancel election timer if we bootstrap

If we short-circuit and bootstrap, cancel our timer.  Otherwise it will
go off some time later when we are in who knows what state.

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
(cherry picked from commit 9ae0ec83dabe37ac15e5165559debdfef7a5f91d)

12 years agomon: cancel probe timeout on reset
Sage Weil [Tue, 25 Jun 2013 01:12:11 +0000 (18:12 -0700)]
mon: cancel probe timeout on reset

If we are probing and get (say) an election timeout that calls reset(),
cancel the timer.  Otherwise, we assert later with a splat like

2013-06-24 01:09:33.675882 7fb9627e7700  4 mon.b@0(leader) e1 probe_timeout 0x307a520
2013-06-24 01:09:33.676956 7fb9627e7700 -1 mon/Monitor.cc: In function 'void Monitor::probe_timeout(int)' thread 7fb9627e7700 time 2013-06-24 01:09:43.675904
mon/Monitor.cc: 1888: FAILED assert(is_probing() || is_synchronizing())

 ceph version 0.64-613-g134d08a (134d08a9654f66634b893d493e4a92f38acc63cf)
 1: (Monitor::probe_timeout(int)+0x161) [0x56f5c1]
 2: (Context::complete(int)+0xa) [0x574a2a]
 3: (SafeTimer::timer_thread()+0x425) [0x7059a5]
 4: (SafeTimerThread::entry()+0xd) [0x7065dd]
 5: (()+0x7e9a) [0x7fb966f62e9a]
 6: (clone()+0x6d) [0x7fb9652f9ccd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Fixes: #5438
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
(cherry picked from commit 03d3be3eaa96a8e72754c36abd6f355c68d52d59)

12 years agoceph-disk: make list_partition behave with unusual device names
Alexandre Maragone [Tue, 18 Jun 2013 23:18:01 +0000 (16:18 -0700)]
ceph-disk: make list_partition behave with unusual device names

When you get device names like sdaa you do not want to mistakenly conclude that
sdaa is a partition of sda.  Use /sys/block/$device/$partition existence
instead.

Fixes: #5211
Backport: cuttlefish
Signed-off-by: Alexandre Maragone <alexandre.maragone@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 8c0daafe003935881c5192e0b6b59b949269e5ae)

12 years agoclient: fix warning
Sage Weil [Tue, 18 Jun 2013 03:28:24 +0000 (20:28 -0700)]
client: fix warning

client/Client.cc: In member function 'virtual void Client::ms_handle_remote_reset(Connection*)':
warning: client/Client.cc:7892:9: enumeration value 'STATE_NEW' not handled in switch [-Wswitch]
warning: client/Client.cc:7892:9: enumeration value 'STATE_OPEN' not handled in switch [-Wswitch]
warning: client/Client.cc:7892:9: enumeration value 'STATE_CLOSED' not handled in switch [-Wswitch]

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 8bd936f077530dfeb2e699164e4492b1c0973088)

12 years agomon/AuthMonitor: ensure initial rotating keys get encoded when create_initial called 2x
Sage Weil [Tue, 25 Jun 2013 00:58:48 +0000 (17:58 -0700)]
mon/AuthMonitor: ensure initial rotating keys get encoded when create_initial called 2x

The create_initial() method may get called multiple times; make sure it
will unconditionally generate new/initial rotating keys.  Move the block
up so that we can easily assert as much.

Broken by commit cd98eb0c651d9ee62e19c2cc92eadae9bed678cd.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 521fdc2a4e65559b3da83283e6ca607b6e55406f)

12 years agoinit-radosgw.sysv: remove -x debug mode
Sage Weil [Tue, 25 Jun 2013 00:42:04 +0000 (17:42 -0700)]
init-radosgw.sysv: remove -x debug mode

Fixes: #5443
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 31d6062076fdbcd2691c07a23b381b26abc59f65)

12 years agocommon/pick_addresses: behave even after internal_safe_to_start_threads
Sage Weil [Mon, 24 Jun 2013 19:52:44 +0000 (12:52 -0700)]
common/pick_addresses: behave even after internal_safe_to_start_threads

ceph-mon recently started using Preforker to working around forking issues.
As a result, internal_safe_to_start_threads got set sooner and calls to
pick_addresses() which try to set string config values now fail because
there are no config observers for them.

Work around this by observing the change while we adjust the value.  We
assume pick_addresses() callers are smart enough to realize that their
result will be reflected by cct->_conf and not magically handled elsewhere.

Fixes: #5195, #5205
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit eb86eebe1ba42f04b46f7c3e3419b83eb6fe7f9a)

12 years agomon/PaxosService: allow paxos service writes while paxos is updating
Sage Weil [Thu, 20 Jun 2013 22:39:23 +0000 (15:39 -0700)]
mon/PaxosService: allow paxos service writes while paxos is updating

In commit f985de28f86675e974ac7842a49922a35fe24c6c I mistakenly made
is_writeable() false while paxos was updating due to a misread of
Paxos::propose_new_value() (I didn't see that it would queue).
This is problematic because it narrows the window during which each service
is writeable for no reason.

Allow service to be writeable both when paxos is active and updating.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 11169693d086e67dcf168ce65ef6e13eebd1a1ab)

12 years agomon/PaxosService: not active during paxos UPDATING_PREVIOUS
Sage Weil [Fri, 7 Jun 2013 18:41:21 +0000 (11:41 -0700)]
mon/PaxosService: not active during paxos UPDATING_PREVIOUS

Treat this as an extension of the recovery process, e.g.

 RECOVERING -> ACTIVE
or
 RECOVERING -> UPDATING_PREVIOUS -> ACTIVE

and we are not active until we get to "the end" in both cases.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 392a8e21f8571b410c85be2129ef62dd6fc52b54)

12 years agomon: simplify states
Sage Weil [Fri, 7 Jun 2013 18:40:22 +0000 (11:40 -0700)]
mon: simplify states

- make states mutually exclusive (an enum)
- rename locked -> updating_previous
- set state prior to begin() to simplify things a bit

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit ee34a219605d1943740fdae0d84cfb9020302dd6)

12 years agomon/Paxos: not readable when LOCKED
Sage Weil [Fri, 7 Jun 2013 18:14:58 +0000 (11:14 -0700)]
mon/Paxos: not readable when LOCKED

If we are re-proposing a previously accepted value from a previous quorum,
we should not consider it readable, because it is possible it was exposed
to clients as committed (2/3 accepted) but not recored to be committed, and
we do not want to expose old state as readable when new state was
previously readable.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit ec2ea86ed55e00265c2cc5ad0c94460b4c92865c)

12 years agomon/Paxos: cleanup: drop unused PREPARING state bit
Sage Weil [Fri, 7 Jun 2013 18:07:38 +0000 (11:07 -0700)]
mon/Paxos: cleanup: drop unused PREPARING state bit

This is never set when we block, and nobody looks at it.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 7b7ea8e30e20704caad9a841332ecb2e39819a41)

12 years agomon/PaxosService: simplify is_writeable
Sage Weil [Thu, 6 Jun 2013 22:20:05 +0000 (15:20 -0700)]
mon/PaxosService: simplify is_writeable

Recast this in terms of paxos check + our conditions, and make it
match wait_for_writeable().

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit f985de28f86675e974ac7842a49922a35fe24c6c)

12 years agomon/PaxosService: simplify readable check
Sage Weil [Wed, 5 Jun 2013 00:03:15 +0000 (17:03 -0700)]
mon/PaxosService: simplify readable check

Recast this in terms of the paxos check and our additional conditions,
which match wait_for_readable().

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 3aa61a0beb540e48bf61ceded766d6ff52c95eb2)

12 years agomon: simplify Monitor::init_paxos()
Sage Weil [Fri, 31 May 2013 23:45:08 +0000 (16:45 -0700)]
mon: simplify Monitor::init_paxos()

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit e832e76a4af04b091c806ad412bcfd0326d75a2d)

12 years agomon/Paxos: go active *after* refreshing
Sage Weil [Fri, 31 May 2013 23:39:37 +0000 (16:39 -0700)]
mon/Paxos: go active *after* refreshing

The update_from_paxos() methods occasionally like to trigger new activity.
As long as they check is_readable() and is_writeable(), they will defer
until we go active and that activity will happen in the normal callbacks.

This fixes the problem where we active but is_writeable() is still false,
triggered by PGMonitor::check_osd_map().

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit e68b1bd36ed285e38a558899f83cf224d3aa60ed)

12 years agomon: safely signal bootstrap from MonmapMonitor::update_from_paxos()
Sage Weil [Fri, 31 May 2013 22:32:06 +0000 (15:32 -0700)]
mon: safely signal bootstrap from MonmapMonitor::update_from_paxos()

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit dc83430124a5fd37573202a4cc0986c3c03739ef)

12 years agomon/Paxos: do paxos refresh in finish_proposal; and refactor
Sage Weil [Sun, 2 Jun 2013 23:57:11 +0000 (16:57 -0700)]
mon/Paxos: do paxos refresh in finish_proposal; and refactor

Do the paxos refresh inside finish_proposal, ordered *after* the leader
assertion so that MonmapMonitor::update_from_paxos() calling bootstrap()
does not kill us.

Also, remove unnecessary finish_queued_proposal() and move the logic inline
where the bad leader assertion is obvious.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit a42d7582f816b45f5d19c393fd45447555e78fdd)

12 years agomon/PaxosService: cache {first,last}_committed
Joao Eduardo Luis [Sun, 2 Jun 2013 23:15:02 +0000 (16:15 -0700)]
mon/PaxosService: cache {first,last}_committed

Refresh the in-memory values when we are told the on-disk paxos state
may have changed.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
(cherry picked from commit 2fccb300bdf6ffd44db3462eb05115da11322ed4)

12 years agomon: no need to refresh from _active
Sage Weil [Fri, 31 May 2013 21:30:48 +0000 (14:30 -0700)]
mon: no need to refresh from _active

The refresh is done explicitly by the monitor, independent of the more
fragile PaxosService callbacks.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit d941363d6e4249e97b64faff0e573f75e918ac0c)

12 years agomon: remove unnecessary update_from_paxos calls
Sage Weil [Sun, 2 Jun 2013 23:10:57 +0000 (16:10 -0700)]
mon: remove unnecessary update_from_paxos calls

The refresh() will do this when the state changes; no need to
opportunistically call this method all of the time.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 03014a4ecc06cde420fad0c6c2a0177ebd7b839d)

12 years agomon: explicitly refresh_from_paxos() when leveldb state changes
Sage Weil [Sun, 2 Jun 2013 23:14:01 +0000 (16:14 -0700)]
mon: explicitly refresh_from_paxos() when leveldb state changes

Instead of opportunistically calling each service's update_from_paxos(),
instead explicitly refresh all in-memory state whenever we know the
paxos state may have changed.  This is simpler and less fragile.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit cc339c07312006e65854207523f50542d00ecf87)

12 years agomon/AuthMonitor: make initial auth include rotating keys
Sage Weil [Sun, 23 Jun 2013 16:25:55 +0000 (09:25 -0700)]
mon/AuthMonitor: make initial auth include rotating keys

This closes a very narrow race during mon creation where there are no
service keys.

Fixes: #5427
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit cd98eb0c651d9ee62e19c2cc92eadae9bed678cd)

12 years agomds: fix iterator invalidation for backtrace removal
Sage Weil [Fri, 21 Jun 2013 18:53:29 +0000 (11:53 -0700)]
mds: fix iterator invalidation for backtrace removal

- Don't increment before we dereference!
- We need to update the iterator before we delete the item.

This code is changed in master, so this fix is for cuttlefish only.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
12 years agoosd: init test_ops_hook
Sage Weil [Thu, 9 May 2013 16:44:20 +0000 (09:44 -0700)]
osd: init test_ops_hook

CID 1019628 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
2. uninit_member: Non-static class member "test_ops_hook" is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit e30a03210c3efb768b1653df5ae58917ef26e579)

12 years agoosd: initialize OSDService::next_notif_id
Sage Weil [Thu, 9 May 2013 16:45:51 +0000 (09:45 -0700)]
osd: initialize OSDService::next_notif_id

CID 1019627 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member "next_notif_id" is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 499edd8bfc355c2d590f5fa1ef197d1ea5680351)

12 years agomon: more fix dout use in sync_requester_abort()
Sage Weil [Thu, 20 Jun 2013 16:46:42 +0000 (09:46 -0700)]
mon: more fix dout use in sync_requester_abort()

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit d60534b8f59798feaeeaa17adba2a417d7777cbf)

12 years agomon: fix raw use of *_dout in sync_requester_abort()
Sage Weil [Mon, 10 Jun 2013 18:48:25 +0000 (11:48 -0700)]
mon: fix raw use of *_dout in sync_requester_abort()

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 8a4ed58e39b287fd8667c62b45848487515bdc80)

12 years agov0.61.4 v0.61.4
Gary Lowell [Wed, 19 Jun 2013 20:51:38 +0000 (13:51 -0700)]
v0.61.4

12 years agomessages/MOSDMarkMeDown: fix uninit field
Sage Weil [Wed, 22 May 2013 21:29:37 +0000 (14:29 -0700)]
messages/MOSDMarkMeDown: fix uninit field

Fixes valgrind warning:
==14803== Use of uninitialised value of size 8
==14803==    at 0x12E7614: sctp_crc32c_sb8_64_bit (sctp_crc32.c:567)
==14803==    by 0x12E76F8: update_crc32 (sctp_crc32.c:609)
==14803==    by 0x12E7720: ceph_crc32c_le (sctp_crc32.c:733)
==14803==    by 0x105085F: ceph::buffer::list::crc32c(unsigned int) (buffer.h:427)
==14803==    by 0x115D7B2: Message::calc_front_crc() (Message.h:441)
==14803==    by 0x1159BB0: Message::encode(unsigned long, bool) (Message.cc:170)
==14803==    by 0x1323934: Pipe::writer() (Pipe.cc:1524)
==14803==    by 0x13293D9: Pipe::Writer::entry() (Pipe.h:59)
==14803==    by 0x120A398: Thread::_entry_func(void*) (Thread.cc:41)
==14803==    by 0x503BE99: start_thread (pthread_create.c:308)
==14803==    by 0x6C6E4BC: clone (clone.S:112)

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit eb91f41042fa31df2bef9140affa6eac726f6187)

12 years agoMerge remote-tracking branch 'gh/wip-4976-cuttlefish' into cuttlefish
Sage Weil [Wed, 19 Jun 2013 17:56:51 +0000 (10:56 -0700)]
Merge remote-tracking branch 'gh/wip-4976-cuttlefish' into cuttlefish

Reviewed-by: Samuel Just <sam.just@inktank.com>
12 years agocommon/Preforker: fix warning
Sage Weil [Tue, 18 Jun 2013 03:32:15 +0000 (20:32 -0700)]
common/Preforker: fix warning

common/Preforker.h: In member function ‘int Preforker::signal_exit(int)’:
warning: common/Preforker.h:82:45: ignoring return value of ‘ssize_t safe_write(int, const void*, size_t)’, declared with attribute warn_unused_result [-Wunused-result]

This is harder than it should be to fix.  :(
  http://stackoverflow.com/questions/3614691/casting-to-void-doesnt-remove-warn-unused-result-error

Whatever, I guess we can do something useful with this return value.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit ce7b5ea7d5c30be32e4448ab0e7e6bb6147af548)

12 years agomon: Monitor: make sure we backup a monmap during sync start
Joao Eduardo Luis [Wed, 19 Jun 2013 01:50:45 +0000 (02:50 +0100)]
mon: Monitor: make sure we backup a monmap during sync start

First of all, we must find a monmap to backup.  The newest version.

Secondly, we must make sure we back it up before clearing the store.

Finally, we must make sure that we don't remove said backup while
clearing the store; otherwise, we would be out of a backup monmap if the
sync happened to fail (and if the monitor happened to be killed before a
new sync had finished).

This patch makes sure these conditions are met.

Fixes: #5256 (partially)
Backport: cuttlefish

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 5e6dc4ea21b452e34599678792cd36ce1ba3edb3)

12 years agomon: Monitor: obtain latest monmap on sync store init
Joao Eduardo Luis [Wed, 19 Jun 2013 01:36:44 +0000 (02:36 +0100)]
mon: Monitor: obtain latest monmap on sync store init

Always use the highest version amongst all the typically available
monmaps: whatever we have in memory, whatever we have under the
MonmapMonitor's store, and whatever we have backed up from a previous
sync.  This ensures we always use the newest version we came across
with.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 6284fdce794b73adcc757fee910e975b6b4bd054)

12 years agomon: Monitor: don't remove 'mon_sync' when clearing the store during abort
Joao Eduardo Luis [Wed, 19 Jun 2013 01:21:58 +0000 (02:21 +0100)]
mon: Monitor: don't remove 'mon_sync' when clearing the store during abort

Otherwise, we will end up losing the monmap we backed up when we started
the sync, and the monitor may be unable to start if it is killed or
crashes in-between the sync abort and finishing a new sync.

Fixes: #5256 (partially)
Backport: cuttlefish

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit af5a9861d7c6b4527b0d2312d0efa792910bafd9)

12 years agoos/FileStore: drop posix_fadvise(...DONTNEED)
Sage Weil [Wed, 19 Jun 2013 04:31:23 +0000 (21:31 -0700)]
os/FileStore: drop posix_fadvise(...DONTNEED)

On XFS this call is problematic because it directly calls the filemap
writeback without vectoring through xfs.  This can break the delicate
ordering of writeback and range zeroing; see #4976 and this thread

      http://oss.sgi.com/archives/xfs/2013-06/msg00066.html

Drop this behavior for now to avoid subtle data corruption.

Signed-off-by: Sage Weil <sage@inktank.com>
12 years agoos/FileStore: use fdatasync(2) instead of sync_file_range(2)
Sage Weil [Wed, 19 Jun 2013 04:24:16 +0000 (21:24 -0700)]
os/FileStore: use fdatasync(2) instead of sync_file_range(2)

The use of sync_file_range(2) on XFS screws up XFS' delicate ordering
of writeback and range zeroing; see #4976 and this thread:

  http://oss.sgi.com/archives/xfs/2013-06/msg00066.html

Instead, replace all sync_file_range(2) calls with fdatasync(2), which
*does* do ordered writeback and should not leak unzeroed blocks.

Signed-off-by: Sage Weil <sage@inktank.com>
12 years agoconfig: fix run_dir typo
Sage Weil [Thu, 13 Jun 2013 04:47:09 +0000 (21:47 -0700)]
config: fix run_dir typo

From 654299108bfb11e7dce45f54946d1505f71d2de8.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit e9689ac6f5f50b077a6ac874f811d204ef996c96)

12 years agoceph.spec: create /var/run on package install
Sage Weil [Tue, 18 Jun 2013 21:51:08 +0000 (14:51 -0700)]
ceph.spec: create /var/run on package install

The %ghost %dir ... line will make this get cleaned up but won't install
it.

Reported-by: Derek Yarnell <derek@umiacs.umd.edu>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Gary Lowell <gary.lowell@inktank.com>
(cherry picked from commit 64ee0148a5b7324c7df7de2d5f869b880529d452)