msgr: Document recv_stamp and add a dispatch_stamp and throttle_wait.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Revert "common/Throttle: Remove unused return type on Throttle::get()"

This reverts commit 4549501c9b0968ce4243e06ff7e9ef03b19de667.
We're about to use it to avoid a time lookup if possible.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: remove unused PG::block_if_wrlocked declaration

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: remove the unused require_current_map

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: fix warning

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

Merge remote branch 'gh/wip-op-data-mux'

Reviewed-by: Greg Farnum <greg.farnum@dreamhost.com>
Reviewed-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

Convert mount.ceph to use KEY_SPEC_PROCESS_KEYRING

having mount.ceph use KEY_SPEC_USER_KEYRING to pass keys to the kernel has
several disadvantages:

1) It leaves the key setting in the uid_keyring, which is reachable from the
session keyring via a link (see keyctl list <root session keyring ref>).  This
means its accessible to other processes in the same session that don't need
access to it, even after the kernel is done with it.

2) The user keyring has some very counter-intuitive semantics as far as keyring
permissions goes.  The user keyring is access via a link from the session
keyring, which a process may not have permission to access in some situations.
For instance if mount.ceph is executed via su without having started a new
session, mount.ceph will not have access to the uid keyring unless the calling
proces (in this case su) has granted access permission.  The result is a -EPERM
error when executing mount.ceph to a cephx enabled server.  If the same command
is attempted in a new root session (e.g. su - or su -l), the mount command will
work fine

Switching the mount.ceph command to use the KEY_SPEC_PROCESS_KEYRING solves both
of these problems.  By using this keyring, accessibility is guaranteed because
its added and accessed in the same process context both in user space and the
kernel, assuring aceesability, despite the session specifics.  It also ensures
that the key will get cleaned up after the mount.ceph process exits
automatically, since there is no longer a need for it (the kernel clones the key
during the mount process and releases it on unmount).

I've tested this here on my local ceph cluster, and it works properly under both
su and su -l .

Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
CC: Josh Durgin <josh.durgin@dreamhost.com>

Merge branch 'wip-rgw-simplelog'

rgw: adjust high level debug level

setting it to 2 instead of 1

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge remote branch 'gh/wip-rgw-simplelog'

* gh/wip-rgw-simplelog:
  rgw: add timestamp to high level log
  rgw: log host_bucket, http status
  rgw: simple request logging

Reviewed-by: Sage Weil <sage@newdream.net>

rgw: fix intent log processing

Intent log processing was completely broken. First, it wasn't
parsing the date correctly (due to failure to initalize strptime).
Second, it was trying to load the entire log to memory in one
piece (and in a racy way). This fixed bug #1948.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: initialize tm before calling strptime

strptime assumes tm is already initialized.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

objecter: some helpful multiop result debug output

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: make getxattrs set rval on decode error

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: add stat ops to op vector!

They work better that way.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: gift reply data to outbl _after_ demuxing

Divvy up the result bl first, then gift the whole shebang to outbl.  If
we gift it first, there's nothing to demux (since we move intead of copy
the bufferlist ptrs).

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/master' into wip-op-data-mux

osd: make in/outdata split/merge helpers static OSDOp methods

Avoid defining new global functions.

Also add basic doxygen descriptions.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: log_show_next() fix reading of the next buffer

Bug #1939. Failed reading large logs.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge branch 'master' of ssh://github.com/NewDreamNetwork/ceph

Merge remote branch 'gh/wip-backfill'

Reviewed-by: Samuel Just <samuel.just@dreamhost.com>
Conflicts:
src/ceph_mds.cc
src/ceph_osd.cc

filestore: overwrite fsid during --mkfs

This mainly matters because read_fsid() now looks at the file size to
determine if it's an old- or new-style fsid, and not overwriting mean a
downgrade confuses things.  Not that anyone would do that, but...

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: reset timestamp when processing starts

otherwise we'd count also the time waiting for the request.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

hadoop: fix unix timestamp calculation in hadoop lib

Hadoop always see wrong dates due of wrong timestamp calculation. Properly
convert nanoseconds to millis when adding.
Possibly fixes #1666.

Signed-off-by: Andrey Stepachev <octo@yandex-team.ru>
Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

hadoop: check for valid filehandler, before using in next calls

In case of nonexistent file, calling Client::replication()
triggers assert.

Signed-off-by: Andrey Stepachev <octo@yandex-team.ru>
Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

doc: update control file for setting pg num on pool create

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

TestRados: fix {min,max}_stride_size initialization

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'master' of ssh://ceph.newdream.net/git/ceph

osd: fix bind error checks

Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: fix testkeys non-tcmalloc linkage

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: add timestamp to high level log

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: log host_bucket, http status

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: simple request logging

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

mds: abort startup if we fail to bind

Signed-off-by: Sage Weil <sage@newdream.net>

osd: abort on startup if we fail to bind to a port

Signed-off-by: Sage Weil <sage@newdream.net>

ceph: fix "run_uml.sh" script

Last-minute cleverness prior to checkin broke the "run-uml.sh" script.
Rearange where a few definitions are done to make it work again.

Signed-off-by: Alex Elder <elder@dreamhost.com>

TestRados: implement max_seconds, reimplement argument parsing

Signed-off-by: Sage Weil <sage@newdream.net>

RadosModel: prefix line with m_op

So we can guage progress...

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix uninitialized value in MClientLease::h

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'gh/master' into wip-op-data-mux

msgr: move operator<< for sockaddr_storage to msg_types.cc

tcp.{cc,h} aren't built/linked cleanly.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/workunits/rados/load-gen-mix.sh

10k objects, not 100k!

Signed-off-by: Sage Weil <sage@newdream.net>

qa: rados load-gen: use rbd pool

No replay interval.

rados: load-gen: wake up on reply

So we can send requests more than once per second.

Signed-off-by: Sage Weil <sage@newdream.net>

rados: fix load-gen 'max-ops'

This was mixed up with min/max_op_len.  And max_ops wasn't being used
the initial object creation stage, flooding the OSDs.  Or during run().

Signed-off-by: Sage Weil <sage@newdream.net>

librados: allow ObjectReadOperation::stat() to get time_t mtime

We can't use the internal utime_t type here.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'gh/master' into wip-op-data-mux

Conflicts:
src/librados.cc
src/objclass/class_api.cc
src/rgw/rgw_rados.cc

osd: recover_primary_got() -> recover_got()

This is called on primary and replicas alike.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: clear missing set on replica when restarting backfill

The primary does the same in PG::activate().

Signed-off-by: Sage Weil <sage@newdream.net>

msgr: don't assert on socket(2) failure

This can happen if we're connecting to an invalid address.  Generate an
error message instead of crashing.

See #1942
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: uninline operator<< on sockaddr_storage

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: rev osd internal cluster protocol

Prevent backfill code from talking to pre-backfill code.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable'

mds: require OSDREPLYMUX feature bit

We use ObjectOperations now and need a new server to decompose replies
into their constituent components.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librados: require OSDREPLYMUX feature

We need this since we now rely on the server telling us rvals and
payload_lens for each OSDOp.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

define new OSDREPLYMUX feature bit

This corresponds to the OSDs ability to pass payload_len hints and
return values for each OSDOp in the MSDOOpReply messages.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objecter: fix add_*() calls to use proper helper

The helper resizes the other vectors; need that everywhere.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ReplicatedPG: munge truncate_seq 1/truncate_size -1 to seq 0/size 0

Truncate with seq 1 and size -1 is a noop.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Sage Weil <sage@newdream.net>

ReplicatedPG: munge truncate_seq 1/truncate_size -1 to seq 0/size 0

Truncate with seq 1 and size -1 is a noop.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Sage Weil <sage@newdream.net>

rgw: limit object PUT size

objecter: fix up stat, getxattrs handlers

- try/catch
- stat mtime

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

v0.40

Merge remote branch 'gh/master' into wip-backfill

librados: make new ObjectReadOperations arguments non-optional

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: use new librados ObjectReadOperation method arguments

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ReplicatedPG: Update stat accounting for truncate during write

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: wrap cls_cxx_map_* with try/catch around decoding

rgw: bucket index creation and init in a single operation

librados: add ObjectOperation::exec

secret: move null check before strlen(key_name) deref

Coverity cid: 98
Signed-off-by: Sage Weil <sage@newdream.net>

osd: stat op, don't compare in memory state to object

might be that object is being created by the current compound request.

librados: specify read return value pointers in ObjectReadOperation methods

This lets librados do the work of parsing the reply from compound
operations, instead of requiring callers to have knowledge of the internal
encodings.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objecter: specify read return values pointers in ObjectOperatio methods

This let's Objecter do the demuxing work for compount read operations.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: put result data in OSDOp.outdata

The removes an argument from do_osd_ops() and cleans up the surrounding
code a bit.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: mux/demux OSDOp::outdata in MOSDOpReply

Bump encoding, so that we don't try to demux old encoded messages, which
will likely have OSDOp::payload_len == indata.length().

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: include return code in OSDOp

This will expose the per-operation return values to the caller.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fill in empty item in peer_missing for strays

If we search_for_missing() on a host, make a corresponding entry in our
peer_missing map (if it isn't already there).  This ensure we get (empty)
entries for strays, which makes all_unfound_are_queried_or_lost() happy.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: don't crash when copying a zero sized object

ReplicatedPG: Do a write even for 0 length operation

Otherwise, a 0 length write to an offset past the end of the file will
cause the internal accounting to reflect the full size of the file, but
not the file on disk.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: fix stat accounting error in CEPH_OSD_OP_WRITEFULL

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

qa/client/gen-1774.sh

Capture Alexandre's script for reproducing #1774 here for posterity, until
we write a properly harnessed test for this.  Currently, workunits can't
mount/unmount, and we don't have a way to make ceph-fuse drop it's cache.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix PG::Log::copy_up_to() tail

The tail needs to refer to the entry preceeding the first entry in the
log.  This updates copy_up_to() to match the basic structure of the other
copy_*() methods.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: reset last_complete on backfill restart

Since last_backfill is hobject_t(), we can set this equal to last_update.
This fixes a problem where last_complete preceeds the abbreviated log we
send to the replica below.

Signed-off-by: Sage Weil <sage@newdream.net>

client: avoid taking inode ref in case of nonexistent dir

Signed-off-by: Andrey Stepachev <octo@yandex-team.ru>
Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-makefile'

COPYING: note licenses for all files, not just the default

This (mostly) copies debian/copyright for now, but there are format
restrictions for that file.  Suggestions for a cleaner way to handle this
are welcome.  In the meantime, this is better...

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

debian/copyright: note acx_pthread.m4 license

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: Add headers that were omitted in make dist and prevented tests from building

Signed-off-by: Kacper Kowalik (Xarthisius) <xarthisius@gentoo.org>

Makefile: Handle corner case of crypto++ correctly

i.e. use c++ while compiling, append to CRYPTO_LIBS instead of LIBS

Signed-off-by: Kacper Kowalik (Xarthisius) <xarthisius@gentoo.org>

Makefile: Use ACX_PTHREAD in configure.ac and resulting flags in src/Makefile.am

instead of hardcoded flags

Signed-off-by: Kacper Kowalik (Xarthisius) <xarthisius@gentoo.org>

Makefile: Add recent acx_pthread.m4 that has a fix for nostdlib issue.

See http://code.google.com/p/protobuf/issues/detail?id=188 for details

Signed-off-by: Kacper Kowalik (Xarthisius) <xarthisius@gentoo.org>

PG: gen_prefix should grab a map reference atomically

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw-admin: add pool rm and pools list

rgw-admin: clean up unused commands

osd: bound log we send when restarting backfill

Use the new tunable from b1da5115aa0756aefa4f0aad36395911e82fce28.

Signed-off-by: Sage Weil <sage@newdream.net>

rados.py: avoid getting return value of void function

rados_ioctx_locator_set_key is void. The return value seems to have
been uninitialized, so the tests failed rarely.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

pg: remove unnecessary guard from calc_trim_to()

The num_objects check doesn't make sense, and could only make trimming
happen more often than it should. Sage did not remember why it was
added.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

pg: add a configurable lower bound on log size

This helps prevent problems with retrying requests being detected as
duplicates. The problem occurs when the log is trimmed too
aggressively, and an earlier tid is removed from the log, while a
later one is not. The later request will be detected as a duplicate
and responded to immediately, possibly violating the ordering of the
requests.

Partially fixes #1490.
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge remote branch 'gh/master' into wip-backfill

osd: limit size of log sent to reset backfill targets

Need to replace magic number with new tunable, once that is merged.

Signed-off-by: Sage Weil <sage@newdream.net>