rgw: fix logging env variable name

osync: fix hash compares

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: fix s3store iterator

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: rados_ioctx_lookup -> rados_pool_lookup

rados_pool_lookup has nothing to do with io contexts!

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: bump minor version number

rados_create_internal -> rados_create_with_config

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: remove some debug printout

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: dump content length for certain operations

rgw: handle swift auth requests

still doesn't really do anything

osync: s3 fixes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: first version

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados tool: fix fd leak in import

rgw: fix plain formatting

rgw: fix openstack list buckets with plain formatting

rgw_admin: show '-' if bucket owner doesn't exist (in log)

librados: always call keyring_init in connect

Formerly only the C interface was calling keyring_init in connect!

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: don't call keyring_init in init_internal

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_create_internal calls keyring_init

Merge commit 'origin/rgw_librados_priv' into rgw

rgw: use rados::init_internal

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: add rados_create_internal

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: merge crypto changes

Merge commit 'origin/master' into rgw

Conflicts:

src/Makefile.am
src/rados.cc
src/rgw/rgw_admin.cc
src/rgw/rgw_main.cc
src/rgw/rgw_rest.cc

rgw: explicit case mtime to uint64 before encoding/decoding

rados tool: export bucket

logging: don't add --debug

--debug is already taken to change the global debug level.
Just offer -d for now.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

testlibrbd, testradospp: read default conf file

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

logging: --foreground options reorganization

-f now just means stay in the foreground.
-d now means stay in the foreground and log to foreground.
Both options now disable pid-files.

Update man pages.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

filestore: instrument filestore, journal throughput and throttling

Signed-off-by: Sage Weil <sage@newdream.net>
Conflicts:

src/os/FileJournal.cc
src/os/FileStore.cc
src/os/FileStore.h
src/os/JournalingObjectStore.cc

filestore: adjust op_queue throttle max during fs commit

The underlying FS (btrfs at least) will block writes for a period while it
is doing a commit.  If an OSD workload is write limited, we should raise
the op_queue max (operations that are queued to be applied to disk) during
the commit period.

For example, for a normally journal throughput limited (writeahead mode)
workload:

 - journal queue throttle normally limits things.
 - sync starts
 - journaled items getting moved to op_queue soon fills up op_queue max
 - all writes stop
 - sync completes
 - op_queue drains, new writes come in again
 - journal queue throttle fills up, again starts limiting tput

For an fs throughput limited workload (writeahead):

 - kernel buffer cache hits dirty limit
 - op_queue throttle limits tput
 - sync starts
 - opq stalls, new writes stall on throttler
 - sync completes
 - opq drains (quickly: kernel has no dirty pages)
 - new writes flood in
 - etc.
(Actually this isn't super realistic, because hitting the kernel dirty
limit will do all sorts of other weird things with userland memory
allocations.)

In both cases, the commit phase blocks up the op queue, and raising the
limit temporarily will keep things flowing.  This should be ok because the
disks are still busy during this period; they're just flushing dirty
data and metadata.  Once the sync completes the opq will quickly dump dirty
data into the kernel page cache and "catch up".

Signed-off-by: Sage Weil <sage@newdream.net>

testrados: test default conf file location

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: add default to rados_conf_read_file

In rados_conf_read_file, read from the default configuration file
locations if the library user passes NULL as the location of the
configuration file.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

objecter: fix leak of bufferlist from MPoolOpReply

bufferlist->claim already clears the source bufferlist,
but setting it to NULL prevented it from being destroyed.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge branch 'stable'

Conflicts:
configure.ac
debian/changelog
src/cfuse.cc
src/rgw/rgw_rest.cc

rados tool: close dir after reading through

rados tool: recursively import from dir to pool

cfuse: set proper defaults

Since cfuse usually runs as a nonprivileged user, its defaults must be a
little different from those of the other daemons. Add a flag to
common_init which can be used to set unprivileged daemon defaults.

SimpleMessenger::start() now just takes a boolean telling it whether to
daemonize. It doesn't need to check global variables or other arguments;
it just daemonizes if you tell it to; otherwise not.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

v0.25.1

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

cfuse: always daemonize hack

Always daemonize, until the next round of common_init fixes lands.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: small pull recovery adjustments

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ReplicatedPG,OSD: Track which osds we are pulling from

Currently, a PG waiting on a pull from a dead OSD cannot continue
recovery.  ReplicatedPG::pull now tracks open pulls by peer in
rec_from_peer (map<int, set<sobject_t> >).

OSD::advance_map now calls check_recovery_op_pulls to allow the PG to
reset pulls from failed peers.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

vstart.sh: escape " "

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

radosgw: first authorize user then try to read the acls

can't read acls if user wasn't authorized yet.

cpp_strerror: handle negative error code

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librbd: delete block completion in aio_read callback

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: initialize done in C_NotifyComplete

Caught by valgrind, this was uninitialized in the caller.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

monclient: fix leak of AuthClientHandler

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

testlibrbdpp: read conf file and don't double-delete AioCompletion

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

vstart: put logging options in daemon sections, not [global]

We don't want random command line utils to have non-default logging
behavior.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: journal fragment rollback events

If we rollback a prepared but uncommitted dir refragment, journal it!
This prevents a pattern like

2011-03-14 09:53:41.402212 7ff5edc00710 mds1.log _replay 4332021~624 / 4350736 2011-03-14 09:53:22.348896: EFragment prepare 610 10* by 1 [metablob 610.100*, 2 dirs]
2011-03-14 09:53:41.404433 7ff5edc00710 mds1.log _replay 4336070~116 / 4350736 2011-03-14 09:53:22.350024: EFragment commit 610 01* by 1 [metablob]
2011-03-14 09:53:48.740230 7ff5f0109710 mds1.log submit_entry 4408799~1264 : EFragment prepare 610 10* by 1 [metablob 101, 3 dirs]

where the second (apparently dup!) prepare crashes the next replay attempt.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'stable'

Conflicts:
src/include/rados/librados.h
src/include/rados/librados.hpp
src/librados.cc

librbd: size_t -> uint64_t on internal resize

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librados: use uint64_t for extent len in extmap

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librados: fixup rados_trunc

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

cpp_strerror: handle negative error code

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_rest: fix printf format specifier

32-bit

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rados: fix map type

32-bit build issue

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librados: fix size_t -> uint64_t

Only comes up on 32-bit builds.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librbd: more off_t and size_t -> uint64_t changes

These only pop on up on 32-bit builds.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librados: rename statfs to cluster_stat

librados: add statfs to C interface

also rename the corresponding C++ method

librbd: uint64_t for image sizes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados, rbd: off_t -> uint64_t

off_t is 32-bit on 32-bit archs when compiled without _FILE_OFFSET_BITS=64.
Avoid any such ambiguity in the library interface.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Revert "librados,librbd: use 64-bit offsets and lengths"

This reverts commit f294b26746a30960971757c42d679208a99995a7.

logging: disable log_sym_history for non-daemons

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: make AioCompletions delete themselves when released

AioCompletion->release() already accounted for the implementation, but the wrapper was kept around.

mds: fix stray anchortable debug spam

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

buffer: disable buffer alloc tracking by default

This can have high overhead on SMP systems.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librbd: delete block completion after it's finished

librbd: don't leak rados_completion

The implementation is already referenced by librados callbacks, so it
will stick around.

librados: Remove OperationOp from WatchCtx

It was unused and doesn't need to be on the heap.

librados: cleanup RadosClient's objecter instance when destroyed

librbd: free RBD::AioCompletion when it's released

rgw: avoid duplicate openstack users

debian: Conflict on earlier -dev packages with sonames.

This ensures a smooth, automatic transition, by telling apt/dpkg that
it's ok to replace the old package with the new one.

Continuation of 95db4c5cb868c26567e3f5095108481aa7e76a80.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw_admin: typo

rgw: remove bucket metadata object on bucket delete

rgw: configurable prefix for openstack urls

utime: fix operator double() to use the right number of zeros.

We should probably use a const (or even a #define would be better)
instead of typing out the constant in all these places!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

librados,librbd: use 64-bit offsets and lengths

size_t is usually 32-bit on 32-bit architectures and 64 on 64-bit ones.
On the other hand, we want our offsets and lengths for librados and
librbd to be 64 bit everywhere. So we need to use uint64_t for offsets
and lengths.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mkcephfs: some fixes

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'origin/stable'

Conflicts:
debian/control

autoconf: Fix typo that made --without-gtk2 not work.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

autoconf: Handle --without-debug, --with-debug=junk properly.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

autoconf: Handle --without-profiler, --with-profiler=junk properly.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: add udev rules

Add /lib/udev/rules.d/50-rbd.rules to debian package.
crbdnamer generates the dev path:
/dev/rbd/<pool>/<name>:<number>

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Conflicts:

debian/rules
src/crbdnamer

osd: fix osdmap scanning on pg creation

On PG creation we were scanning the complete history of all osdmaps ever.
Fix initialization of PG::Info::History epoch_created and same_*_since
fields in the base (creation) case to make this work the way it was
supposed to.

Reported-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Destructors for virtual classes should be virtual

Add -Wnon-virtual-dtor to the makefile. Fix the warnings that were
generated.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

autoconf: If given --with-gtk2, not finding it is fatal.

Use --with-gtk2=check (default) if you're ok with both cases.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

atomic.h: fix "crappy slow implementation"

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

auth: Let common_init (or CephCrypto::init) init auth crypto.

This avoids doing it in a global constructor.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

common: Initialize ceph_crypto in common_init.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ceph_crypto: Implement HMAC-SHA1 wrapper for NSS.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ceph_crypto: Add unittests for the HMAC-SHA1 compatibility shim.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: Make rgw_main use HMAC-SHA1 via compatibility shim.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ceph_crypto: Implement MD5 wrapper for NSS.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ceph_crypto: Add unittests for the MD5 compatibility shim.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: Make rgw_op use MD5 via compatibility shim.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

auth: Change header file guard define, to make room for non-auth crypto.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: use random byte generation in auth/Crypto, instead of CryptoPP.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

auth: Add a microbenchmark for AES encrypt/decrypt.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>