page: redefine PAGE_* macros

Saw this on sid i386:

msg/SimpleMessenger.cc: In function 'void alloc_aligned_buffer(ceph::bufferlist&
, int, int)':
msg/SimpleMessenger.cc:1782:14: error: '__sysconf' was not declared in this scop
e
msg/SimpleMessenger.cc:1789:23: error: '__sysconf' was not declared in this scop

Some header is clobbering out PAGE_* macros.  Make our header more
forceful.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: add --mkkey mkfs option

Optionally generate a new key as part of the mkfs step.  This makes life
a bit easier for the admin.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: 'auth caps <name> [svc value [svc2 value2 [...]]]'

Avoid having to futz with cauthtool if possible.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

PG: update same_acting_since when acting or up changes

This is a hack since we currently use same_up_since to denote the beginning of an interval.
We should probably change this usaged or rename it to same_interval since.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

crush: allow - and _ in crushmap type/item names

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crushtool: strip leading spaces from identifiers

No idea where these are coming from!  Weird.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: avoid clearing connection_state on pipe replacement

read_message and write_message both dereference connection-state, so avoid
clearing it when replacing a pipe.

read_message still uses it to find rx_buffers in ways that may interfere
when two Pipes reference the connection, but currently that is only used
for lossy pipes.  We could still take pipe_lock in that case, but it is
only an optimization (we copy the data if the buffers don't get used
directly) and probably not worth bothering with.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: add obsync

Signed-off-by: Sage Weil <sage@newdream.net>

debian: obsync

Signed-off-by: Sage Weil <sage@newdream.net>

obsync: no .py

Signed-off-by: Sage Weil <sage@newdream.net>

PG: Replicas send Notifies in response to queries

Replicas only send Infos during activate.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: choose_log_location, fix error when scanning up set

++up.begin() does not skip the primary. Primary might not be up[0].

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: PG can receive a log in WaitActingChange requested in GetLog

Discard logs requested during GetLog once we are in WaitActingChange.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: Don't use exit to call proc_master_log

exit is also invoked when transitioning to Reset...

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

mon: health WARN if monitor quorum is incomplete

Signed-off-by: Sage Weil <sage@newdream.net>

fix segfault introduced by commit de640d85fa3e0e5e5a31704eab5a8714a1ffe867

That commit introduces the line 'cur_con->put()' which has the possibility
of being called while cur_con is not initialized.

osd: lazily close connections to down peers

If we hear from a peer that should be dead, tell them, but mark our
connection so that it will close after that message is delivered or if
it encounters any errors.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: mark_down_on_empty and mark_disposable

Mark a connection to close when messages are sent, and to close on any
error.  We can use this to tell people who should be dead that they should
be dead, but not waste resources reconnecting to them.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

PG: Remove downed osds from peer_missing and peer_info

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

PG: Only pull the master log from a member of the prior_set

There must be a member of the prior_set such that no other
osd has a more recent last_update.  This way, prior_set_affected
will ensure that we reset peering if the master log source
goes down.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: search_for_missing takes the other osd's missing set

Previously, search_for_missing was erroneously passed the
primary's missing in a few places.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

radosgw_admin: fix clitest

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

radosgw_admin: dump log by object

instead of only by date+bucket.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: in S3 PUT, don't crash on Content-Length == 0

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

expand testceph to check xattrs

Signed-off-by: Brian Chrisman <brchrisman@gmail.com>

client: support security. namespace

Signed-off-by: Brian Chrisman <brchrisman@gmail.com>

support for xattrs in libceph

Signed-off-by: Brian Chrisman <brchrisman@gmail.com>

monclient: fix crash on shutdown

cur_con may be null on shutdown.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

monclient: maintain explicit session connection; ignore stray messages

Maintain an explicit Connection handle to send messages and mark_down old
monitor connections.  Ignore any incoming message that is not part of that
session.  This fixes problems with incoming messages that race with
session restarts.

Fixes: #1033
Reported-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Sage Weil <sage@newdream.net>

librados: don't crash if we call connect twice

Fixes: #1034
Reported-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'origin/stable'

rados_sync: implement --delete-after, fix bugs

Implement --delete-after for both export and import.

Fix DIR* leaks.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Allow dashes in ceph_argparse, etc.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test_rados_sync: test --force

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_sync: support --force

Support --force, which re-copies all objects all the time.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: initialize oi.oloc if on-disk value is bogus

If the on-disk locator is undefined (upgrade of an old cluster?) initialize
the oloc fields based on the PG::Info.

Reported-by: ar Fred <ar.fred@yahoo.com>
Tested-by: ar Fred <ar.fred@yahoo.com>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: switch bucket creation operations

First we create the pool, then we create the bucket object. This
should have the effect of making the bucket creation sync.

osd: rename Pending -> WaitActingChange

We only use the Pending state while waiting for the acting set to change.
Rename the state and log it appropriately so we can see how much time the
OSD is spending doing this.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: log debug output for Crashed state

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: wait for up_thru updates

Before the primary can go active we need to wait for the up_thru in the
osdmap to reflect that we were alive during the current interval.  This got
lost in the translation to the new state machine.

Add a new state WaitUpThru between GetMissing and Active, so that we can
continue doing the work of peering and finding unfound objects while we
are waiting for the osdmap to update.  Request said update early (in
GetInfo).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix pollution of peer_info

The ++ postfix has no effect here!  We really want +1.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix osd$foo typos

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: simplify build_might_have_unfound

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: reset last_complete on mark_all_unfound_as_lost if no more missing

If we marked _all_ missing as lost, reset last_complete, since missing is
now empty!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: drop bad warning

The stats won't match reality if there are any missing or if there are any
snapped objects.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rados tool: remove import/export

rados_sync replaces rados import / rados export

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

.gitignore: rados_sync

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: reassert our assert definition after including boostchart

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

assert: make our assert clobber any others too

Two can play this game, /usr/include/assert.h!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix compilation for some g++ versions

wasn't compiling on lenny, g++ ver 4.3.2. Might be that
it's also due to differebt boost version.

rgw: minor cleanup

rgw: opening bucket io ctx stats bucket info if failed

this should trigger request of a new osdmap if we were racing
with bucket creation.

Merge branch 'wip-rados-sync'

test_rados_sync: check that second sync does nada

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_sync: more fixes

* separate BackedUpObject::from_path and BackedUpObject::from_file.

* librados functions return negative values on error, but most may
return positive values on success, even things like setxattr.

* implement read_xattrs_from_rados

* change rados_sync printout a bit

* fix up test_rados_sync xattr tests

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: warn when obc locator doesn't match ops

This will hopefully help us track down #1028.

Signed-off-by: Sage Weil <sage@newdream.net>

rados_sync: fix memory leak, other misc fixes

* Store BackedUpObject in auto_ptr to avoid memory leaks

* better comment for BackedUpObject::get_fs_path()

* xattrs: handle only_in_a list separately from diff list, since
librados complains if you call setxattr(foo) and there is already an
xattr foo.

* xattrs: add xattrs_to_str for debugging purposes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osdmap: refactor mapping; don't assert on missing pool

We were asserting on a pg with a missing pool fed to pg_to_acting_osds
because raw_to_temp_osds called raw_pg_to_pg.  Refactor to let us look up
the pool only once, and return gracefully instead of crashing.

Signed-off-by: Sage Weil <sage@newdream.net>

test_rados_sync: test xattr preservation

test xattr preservation across import/export

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test_rados_sync: add import/export test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_sync: complain if import src is inaccessible

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_sync: fix --create for import

Fix import's pool creation faculty.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

add test_rados_sync.sh

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

client: map file stripes to acting osds

Old result was just wrong if any osds were down.

Signed-off-by: Sage Weil <sage@newdream.net>

rados_sync: fix import a bit

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_sync: nicer error formatting

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: be a little less scary in our startup warning

- be less scary
- be helpful and tell them where to go

Signed-off-by: Sage Weil <sage@newdream.net>

rados_sync: misc bugfixes

Avoid using a temporary c_str pointer when it goes out of scope.

Complain if XATTR_FULLNAME is empty, as well as missing.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_sync: nicer printouts

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

man: mkcephfs: small fixes

Signed-off-by: Sage Weil <sage@newdream.net>

rados_sync: do_export: fix diff'ing

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_sync: more improvements to xattr_test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mkcephfs.man: update man page

Not beautiful, but at least it's accurate.

Signed-off-by: Sage Weil <sage@newdream.net>

rados_sync: Add xattr_test

Add xattr_test to test if the underlying filesystem supports xattrs

Fix BackedUpObject::get_fs_path a bit.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'stable'

osd: used fixed size types for fiemap/mapext/sparseread encoding

The client expects <uint64_t,uint64_t>, so this breaks on any 32-bit osd.

Signed-off-by: Sage Weil <sage@newdream.net>

cfuse: don't -d

Signed-off-by: Sage Weil <sage@newdream.net>

mds: assert auth on journal_cow_dentry

We have no business doing cow (or journaling) non-auth dentries.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: don't journal src when not auth on rename

If we aren't auth for the srcdn, don't put it in the journal, or else we'll
barf on journal replay.

This fixes a crash reproduced with

sudo mkdir mnt/foo
sudo touch mnt/foo/a
sudo ln mnt/foo/a mnt/foo/b
sudo rm mnt/foo/a

sudo mkdir mnt/bar
sudo touch mnt/bar/nothing

./ceph mds tell 0 export_dir /bar 1

sudo mv mnt/foo/b mnt/bar/b

+ mds restart.

Signed-off-by: Sage Weil <sage@newdream.net>

PG: choose_acting needs the value of the osd, not its index

This caused two osds to keep flipping the acting set between [2] and
[0,2] when osd.0 was far behind and needed a backlog. This is visible
as toggling between peering and peering+degraded.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Create rados_sync tool to back up rados pools

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

stop.sh: Avoid bashisms.

I have a habit of running "sh -x stop.sh" whenever it seems
to fail, and that runs it with dash, not bash. Since it
doesn't actually need the bashisms, remove them.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

stop.sh: Whitespace.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

osd: fix include in cosd.cc too

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: rearrange #includes to get our assert

Make sure we include boostchart headers before our common/assert.h so that
ours clobbers theirs.  Otherwise the generic one will clobber ours and our
assert output won't get logged or be as pretty.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: fix multi-osd 'ceph in|out N..' operations

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph_argparse: ignore dashes after equals sign

When parsing argv, ignore dashes after equals signs. This is so that
things like --log-file=/tmp/foo-file will work correctly.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'wip_pg_refactor'

PG: strays don't peer (only the primary does)

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

PG: clear peering flag when we transition from Peering to Reset

This is safe because the only other transitions are to Pending (still
peering) and Active, which clears the state itself.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge branch 'stable'

osdmap: fix temp osd pg mapping

If you feed in a raw pg (full precision) you should get the same mapping
out as when you plug in the effective/reduced precision pg.  The
raw_to_temp_osds() wasn't doing that, which gave you results like

flak:src 04:01 PM $ ./ceph pg map 0.4
2011-05-05 16:01:18.524051 mon <- [pg,map,0.4]
2011-05-05 16:01:18.524987 mon2 -> 'osdmap e11 pg 0.4 (0.4) -> up [1,0] acting [0]' (0)
flak:src 04:01 PM $ ./ceph pg map 0.7ed4
2011-05-05 16:01:21.755490 mon <- [pg,map,0.7ed4]
2011-05-05 16:01:21.755996 mon1 -> 'osdmap e11 pg 0.7ed4 (0.4) -> up [1,0] acting [1,0]' (0)

The objecter was feeding in raw pgs, so this was sending requests to the
wrong nodes.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: do not stop mds0 unless all other nodes have also stopped

If we are the root node or the tableserver, we have to shut down last.
(And even then, if we have client sessions, we can't fully shut down, we
can only kill ourselves!)

Fixes: #1048 (sorta)
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'stable'

Conflicts:
src/mds/Resetter.cc

PG: handle MOSDPGLog messages in Active state.

We previously didn't, but we definitely need to as we can go
active while waiting on strays to send in logs to recover missing
objects.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ceph_crypto: Fix ceph::crypto::init mutex for NSS.

Even after ceph::crypto::shutdown, the library looked
already initialized; this broke the ForkDeathTest.MD5
in "make check", and NSS-using daemons.

See 921d4b3d8b79485a6786f8ca75b2a96ea377da84 for more.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

v0.27.1

ceph::crypto::init: add PTHREAD_MUTEX_INITIALIZER

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: fix --reset-journal

Don't fork.  (Already fixed in master branch by the start_with_nonce
refactor, so this is just for 0.27.1.)

Signed-off-by: Sage Weil <sage@newdream.net>

obsync: remove misguided strip_owner

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>