calc_hmac_sha: fix access-past-end-of-buffer

Fix a place where we access a buffer past the end of its length. Clean
up the function a bit.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Use macro for digest size, part 2: tests

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mkcephfs: minor fixes

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

do_autogen: create radosgw

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

RGW: validate bucket names and object names

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

FileStore: replace op_queue_throttle with op_queue_reserve_throttle

Previously, queue_op would call op_queue_throttle while holding the
journal_lock.  op_queue_throttle, however, can sleep.

We fix the problem by:
1) Factor build_op out of queue_op
2) op_queue_throttle is now op_queue_reserve_throttle and takes an op as
an argument.  op_queue_reserve_throttle can be called before the journal
lock is taken.  This also avoids the race between calling throttle and
incrementing op_queue_bytes and op_queue_len.
3) queue_op now takes the op generated using build_op as an argument.
4) _journaled_ahead no longer needs to call throttle as
queue_transactions has already reserved space.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ceph_crypto: fix undefined references

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

boto_del.py -> boto_tool.py

boto_tool.py has some additional functionality.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obsync: remove obsolete cmdline options

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'master' of ssh://ceph.newdream.net/git/ceph

rgw: fix compilation warning

rgw: change the swift auth config a bit

context: conditional context debugging

Mainly C_Gather, which is a reasonably frequent source of pain.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Rename objsync -> obsync

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add boto_del.py

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add boto_del.py

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: save roundtrip when reading user info

Merge branch 'rgw' of ssh://ceph.newdream.net/git/ceph into rgw

rgw: self signed tokens

rgw: keep user info copy in the indexes

We index user by email and by openstack id, keeping a copy of the
user info structure there saves a round trip.

rgw: store openstack secret as part of the user info

librbd: remove RBD_MAX_SEG_NAME_SIZE

The only use of this was incorrect - block_name has size RBD_MAX_BLOCK_NAME_SIZE.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

common: Add utf8 validation functions, test

Add utility functions for validating a buffer as valid UTF-8.

Add a unit test to check the functions.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: remove strerror references

We have cpp_strerror, let's use it and be threadsafe!

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: Add utf8 validation functions, test

Add utility functions for validating a buffer as valid UTF-8.

Add a unit test to check the functions.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: fix mdsmap dump formatting

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Rename osync to objsync

Rename osync to objsync to avoid confusion with O_SYNC.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Rename osync to objsync

Rename osync to objsync to avoid confusion with O_SYNC.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'rgw'

Merge branch 'rgw'

CDir: make check_rstats friendlier.

Only fail asserts if mds_debug_scatterstat is enabled.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

MDCache: make linkunlink rstat propagation work properly.

We could be in a lock state (ie, gather) where we can't take new locks.
But if we're in this function for linkunlink we have to already have
a lock, so in that case make sure the function succeeds and assert
that we do have a lock.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

MDCache: set up rstats properly in create_system_inode.

Fix the previous users to deal with this, including
create_mydir_hierarchy which was completely broken.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

MDCache: properly update and project the stats when removing inodes.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: add check_rstats function and a few users.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osync: implement delete-before and delete-after

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: add osync-test.sh

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'master' into rgw

Conflicts:

debian/control

rgw: fix logging env variable name

cosd: Close TEXT_RED with TEXT_NORMAL in a few places.

Guess these got missed or typoed on creation or something.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osync-test: test --delete in bucket-to-bucket xfer

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync-test.py: test --dry-run, --delete stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: implement --dry-run option

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rpm: package python bindings for rados

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync-test.sh: test bucket-to-bucket transfers

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: add AKEY and SKEY

Add AKEY and SKEY, for when you want to keys / secret keys to be the
same for both source and destination.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: fix automatic bucket creation

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: add alternate syntax for file:// transfers

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync-test: test file-to-bucket transfers

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: add SRC_AKEY, etc. environment variables

Add SRC_AKEY, SRC_SKEY, DST_AKEY, and DST_SKEY. This allows us to
synchronize between buckets with different authentication
configurations.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Fix manpage typos

Close escape names correctly.

Remove unused cdbs build dependency

Debian packaging uses debhelper and not cdbs; remove that build dependency.

Change wording of Debian package descriptions

Be more general in naming users per policy recommendation.

Make Ceph Debian packaging Linux only

Build Ceph on all Linux archs, but only build-depend google-perftools on
x86 and x64 archs only.

Disable libatomic-ops on armel archs

armel supports ARMv4t or above instructions sets.
libatomic-ops is only usable with Ceph for ARMv6 or above.

Signed-off-by: Hector Oron <zumbi@debian.org>

Add cross compilation support

Taken from Debian bugreport #618939 , courtesy by Hector Oron.

Signed-off-by: Hector Oron <zumbi@debian.org>

Add parallel building support for multi{core,processor} systems.

CephxProtocol.cc: invalid authorizer data should not crash the osd

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ceph.spec.in: some CentOS fixes

BuildRequires: cryptopp-devel has been replaced by nss-devel.  Skip
google-perftools-devel because that package is not available for x86-64.
Add python.

Don't install libcls_rbd.so.1.0.0.debug.

Package crbdnamer and librados-config.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: fix hash compares

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: fix s3store iterator

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind: convert to new API

Fix the python bindings to use the new librados API.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

backtrace: user the proper version header

Signed-off-by: Sage Weil <sage@newdream.net>

libceph: use the proper version header

Signed-off-by: Sage Weil <sage@newdream.net>

libceph: pull version from new version define

Signed-off-by: Sage Weil <sage@newdream.net>

configure: no ~

This confuses fedora and isn't really necessary.

Signed-off-by: Sage Weil <sage@newdream.net>

use 'git describe' version

librados: rados_ioctx_lookup -> rados_pool_lookup

rados_pool_lookup has nothing to do with io contexts!

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: bump minor version number

rados_create_internal -> rados_create_with_config

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: bump minor version number

rados_create_internal -> rados_create_with_config

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: rados_ioctx_lookup -> rados_pool_lookup

rados_pool_lookup has nothing to do with io contexts!

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

direct_io_test: use mkstmp instead of mkostemps

mkostemps isn't present in older glibc versions, like the ones in CentOS
5.5. We don't really use any of the extra functionality of mkostemps in
this test.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Makefile: check for new enough version of gtkmm

Versions older than 2.13 don't build, so check for that with automake.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: remove some debug printout

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: dump content length for certain operations

rgw: handle swift auth requests

still doesn't really do anything

osync: s3 fixes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osync: first version

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

msgr: move test binaries to updated msgr bind/start interface

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: fix start() return value

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

PG,OSD: activate pg during replay

Replay PGs already accept and queue transactions.  PGs will now go to
active during replay in order to simplify the state reported to the user
and to allow recovery to being.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

blobhash: Avoid size_t in templatized hash functions.

On S/390, the earlier rjhash<size_t> failed with
"no match for call to '(rjhash<long unsigned int>) (size_t&)'".
It seems the rjhash<size_T> logic was only enabled
on some architectures, and relied on some pretty deep
internals of the bit layout (__LP64__).

Use an explicitly 32-bit type as early as possible, and
convert back to size_t only when really needed. This
should work, and simplifies the code. In theory, we might
have a narrower output (size_t might be 64-bit, max value
we now output is 32-bit), but this doesn't matter as this
is only ever used for picking a slot in an in-memory hash
table, hash(key) modulo num_of_buckets, there won't be >4G
buckets.

Closes: #837
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

msgr: let user explicitly set nonce

There will be problems if two messengers use the same entity_addr_t because
they are on the same ip and choose the same nonce (e.g., because they are
in the same process).  Let the caller sort this out in whatever way it
finds most appropriate.

For libceph, librados, and csyn, all N million to the pid.

Fixes: #877
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

config: whitespace fix

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: fix get_val, set_val

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: check whether objecter is initialized before shutting it down

Fixes failing unit test Librados.CreateShutdown

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados tool: fix fd leak in import

objecter: close all sessions when shutdown

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: fix plain formatting

rgw: fix openstack list buckets with plain formatting

mds: fix replay of fragment ROLLBACK

In the rollback event the bits are negative.  Replay accordingly.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

common: disable log_per_instance for non-daemons

Turn off the logging and symlink rotation, not just symlink rotation.

This is a somewhat arbitrary distinction (log per instance only for
daemons), but its only used by vstart and only really useful for
development/debugging, so who cares.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: drop libradosgw_a LDFLAGS

Fixes the warning

src/Makefile.am:299: variable `libradosgw_a_LDFLAGS' is defined but no program or
src/Makefile.am:299: library has `libradosgw_a' as canonic name (possible typo)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: resync fragmentation during cache rejoin

During rejoin we may find that different MDSs have different fragmentation
for directories.  When that happens we should refragment as needed on the
replicas to match what's on the primary.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw_admin: show '-' if bucket owner doesn't exist (in log)

rados_create: correctly handle null id

Passing a null id to rados_create means "use the default id."

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: always call keyring_init in connect

Formerly only the C interface was calling keyring_init in connect!

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: don't call keyring_init in init_internal

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

objecter: make response_data bufferlist static

Putting it on the heap unnecessary additional complexity.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rados_create_internal calls keyring_init