rgw: more sane defaults

Notably, this gives us log_to_stderr = 1.

Signed-off-by: Sage Weil <sage@newdream.net>

common: CINIT_FLAG_NO_BANNER

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: rgw_log -> debug_rgw

Normalize with the rest of the code base.

Also, 'rgw log' might be confused with 'log file'.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: remove useless RGWConf::log_level

Not used for anything.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: set dout condvar to rgw_log

We would normally set DOUT_SUBSYS and get debug_$foo, but the setting in
this case is rgw_log.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: RGW_LOG -> dout

Signed-off-by: Sage Weil <sage@newdream.net>

rados-api/test: don't srand(time())

This will give duplicate results if called more than once per second, which
leads to things like -EEXIST (#1558).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: make MPoolOp*::print() readable

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-oldmaps'

radosgw: run as 'user' field in ceph.conf

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: set xmlns on ListBucketResult element

Certain S3 clients (e.g., s3fs) require the xmlns to parse the
results correctly.

Signed-off-by: Henry C Chang <henry.cy.chang@gmail.com>

osd: get latest osdmaps before booting

- get the latest osdmaps before adding/marking ourselves up
- behave if there is a discontinuity in the osdmap history

This lets us behave sanely if an osd has been down for a very long time,
or if we replace (wipe) an osd, or otherwise take a fresh new osd and add
it to an aged cluster (with lots of old maps or, more likely, an oldest
map that has a large epoch).

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: include addrs in json dump

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: show weight for out osds

This info is reduncant (weight is 0 for out osds), but makes the output
cleaner.

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: show old addrs for down osds

This is useful for admins to find which ip/node to connect to.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: limit maps send on onetime osdmap subscribe

This throttles the load put on the monitor by making the client request
osdmaps in discrete chunks.  Use a feature bit to control this, since the
old kernel clients will only send a single onetime request if they think
they are behind.

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: include oldest, newest known epoch in message

If non-zero, these are filled in with the oldest and newest map epochs
available.  This helps the client decide whether to ask for newer or older
maps.

Some minor refactoring in the monitor.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: use pointers for handle_advance_map

So that lastmap can be optional.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: fix locking on is_latest_map checks

We need to take the lock protecting these structures in the completion
context.

Signed-off-by: Sage Weil <sage@newdream.net>

monclient: use an async finisher

Otherwise we can deadlock if any completions call back into MonClient.

Signed-off-by: Sage Weil <sage@newdream.net>

monclient: clean up old version_requests on session reset

This throws out the old contexts.  We assume that the caller will not wait
for them.

Signed-off-by: Sage Weil <sage@newdream.net>

monclient: implement get_version() to get newest _and_ oldest map versions

Reimplement is_latest_map() in terms of more general helper.  Add field
to reply message.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: is_active() instead of !is_booting() for map sharing checks

Signed-off-by: Sage Weil <sage@newdream.net>

osd: do not request osdmap when loner if not booted

Signed-off-by: Sage Weil <sage@newdream.net>

mon: trim old osdmaps

Trakc the oldest clean pg, and trim osdmaps that are older than that.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: add last_epoch_clean to pg_stat_t

Signed-off-by: Sage Weil <sage@newdream.net>

mon: implicitly mark lost_at when a fresh osd rejoins the cluster

If an osd is wiped an rejoins the cluster, record that by implicitly
setting lost_at.

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: run as 'user' field in ceph.conf

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: make 'log list' behave when .log pool dne

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

debian: don't recommend gceph

Not until it's worth installing.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'stable'

radosgw: make stop succeed when not running

This fixes apt-get errors like

No /usr/bin/radosgw found running; none killed.
invoke-rc.d: initscript radosgw, action "stop" failed.
dpkg: warning: subprocess old pre-removal script returned error exit status 1

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: fix init-radosgw provides

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: set xmlns on ListBucketResult element

Certain S3 clients (e.g., s3fs) require the xmlns to parse the
results correctly.

Signed-off-by: Henry C Chang <henry.cy.chang@gmail.com>

debian: add non-stripped exception for libcls_rgw rados class

Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: more radosgw_admin -> radosgw-admin

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix possible deadlock in multi-mds setup

This should fix the file_excl case on a file_max update when there
is more than one mds.

If we dont issue caps here its possible when we eval_gather on
the returning lockack that we don't push the lock forward which
will leave the inode auth pinned. If an export occurs that tries
to freeze a tree this inode is in, we'll have a freezing tree
waiting for an auth_pin that can no longer be removed.

Signed-off-by: Brandon Seibel <brandon@seibelnet.ca>

workunit: clean up temporary files

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ReplicatedPG: reset return code after find_object_context

This way the object is actually deleted when it has no snapshots,
since the transaction is not aborted. This makes
test_rbd:TestImage.test_rollback_to_snap_sparse pass.

Fixes: #1554
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

more radosgw_admin -> radosgw-admin

Signed-off-by: Sage Weil <sage@newdream.net>

osd: trim ondisk log using Log::Entry::offset

The block_map is a waste of memory.  It is simpler just to keep track of
what offset a Log::Entry was read/written/appended to, and trim based on
that.

Motivated by alldown-to-cleanboth.pdf on #1583.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: combine log_op into append_log

They're both short, and log_op only ends up hiding information that
append_log will soon need (the raw Log::Entry's).

Signed-off-by: Sage Weil <sage@newdream.net>

mds: make jouranl writeable in MDLog::append()

When restarting a stopped MDS, we need to mark the Journaler read/write
before we use it.  Do this in MDLog::append(), when we position the write
pointer at the end of the journal.

Signed-off-by: Sage Weil <sage@newdream.net>

mdcache: tolerate no subtrees in create_subtree_map()

We don't really need mydir here.  It is normally opened up in the
subsequent call to open_root().  That will be identical on journal replay,
so no need to have it beforehand (and in the ESubtreeMap entry).

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/stable'

ceph.spec.in still packages libceph.so in 0.36

Package libcephfs.so instead of libceph.so.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>

v0.36

Merge branch 'wip-pool-sharing'

doc: add documentation for librbd python bindings

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: add class and method documentation

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: use a dict to translate error codes

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: allow setting an image to no snapshot

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: turn EROFS into an exception

This is returned when trying to write to a snapshot.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: don't remove on-disk bucket object in rgw_user::rgw_remove_bucket

This is the same object that rgw_rados removes from the root pool now.
And since nobody calls rgw_remove_bucket_info, kill it.
(Hopefully we can kill all of the pi_buckets functions soon.)

While we're at it, rename rgw_remove_bucket to rgw_remove_user_bucket_info,
since that's what it does.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: revert part of commit:30b814e2677659ddda4109a1421f33a47d83a05b

It's more consistent for RGWRados to delete the bucket object, rather
than the rgw_user functions.
And commenting functions out is not a happy practice.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: don't remove bucket object twice

Or else the second attempt will faile with -ENOENT.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: fix bucket-id assignment

Get the version from the same io_ctx that you write with.

Also, use a private one, so that we don't race with other threads.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: separate bucket vs object owner tracking

- track bucket_owner in session
- separte bucket and object owner fields in log entries

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: fix acl vs content-length check

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: rename clitest dir

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-pool-sharing' of github.com:NewDreamNetwork/ceph into wip-pool-sharing

Conflicts:
src/rgw/rgw_bucket.h

radosgw-admin: fix help clitest

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: set log entry owner to user if s->acl is not set

This is probably wrong, but it fixes the immediate problem of the first
entry in the log not having the owner field set.  The 'log show' looks at
that first entry to spit out key bucket metadata.

A better approach is probably to put that bucket/log metadata (bucket_id,
owner) that doesn't change as an xattr on the object instead of dupilcating
it in every record.  Or something like that...

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: simplify bucket info storage

Use a single method to store bucket metadata.

Store it in both locations, so it never diverges.  (This can still be
cleaned up.. probably one object should just point to the other, so the
data is only in one place.)

Disable to 'pool create' thing for now, which is still confusing me...

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: pull 'log show' owner+bucket from first entry

Don't rely on the bucketinfo object for this, as it may be deleted.  Having
it there most of the time but not always is going to mean breakage for
users.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: fix RGWBucketInfo encoding of owner

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: fix log retry after bucket creation

We were clobbering the object name for some reason.. probably cruft from
some old change.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: store bucket information in the .rgw pool instead of .buckets

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

radosgw_admin: we do want to set markers on these buckets

Commit 25499b644b1c5a7b6f4c8876261334dda10fbec3 introduced the initial
work for pool sharing and markers, but didn't set radosgw_admin to
use that infrastructure. We definitely want to be doing so, though,
since radosgw_admin isn't creating system buckets. Do so.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Cond.h: adjust nlock during waits

Previously, is_locked could erroneously return true if there were any
threads waiting on a cond on that lock.  In particular, any lock
associated with a timer would always return true for is_locked if any
events were scheduled on the timer.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: dump_watchers in do_osd_op_effects

dump_watchers requires the osd watch_lock.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

librados: Make notify a linger op

Notifies must be refreshed when the mapping changes.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test_async_snap: loop a finite number of times.

Signed-off-by: Sage Weil <sage@newdream.net>

qa: btrfs: test_async_snap

Hammer on the btrfs async sync/snap create/remove ioctls.

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: 'log list' and 'log rm'

The normal object listing API seemed like a real poor match here so I
hacked in something that just lists the raw rados objects.  It also looks
like that may need some fixing.

This patch at least establishes a usable interface that is purely
radosgw-admin based for listing, fetching, and removing logs, so users can
avoid looking at rados pools directly.

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: dump ints, not strings

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: fix bucket stats

- 'usage' object instead of 'categories' array.
- dump ints, not strings.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: protect the id-based accesses from namespace collisions

Users can create buckets named "8", but they can't name them ".8". So
prepend the actual on-disk objects with a period.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove get_bucket_info and convert users.

This just lives in struct rgw_bucket now.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

radosgw-admin: fix 'bucket stats --uid='

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: include 'suspended' in user info output

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: clean up some cout usage

Errors to stderr.  Avoid cout for commands with structured output.

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: use formatter for 'bucket list'

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: 'bucket stats --uid'

List stats for all buckets on for the given user.

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: clean up 'bucket list' output

Signed-off-by: Sage Weil <sage@newdream.net>

rgw_admin: rip out format.empty() special cases

If we want a plain output, we should implement a PlainFormatter of some
sort.  For now, pretty json is good enough.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Conflicts:

src/rgw/rgw_admin.cc

rgw_admin: fix --pretty-format and --purge-data flag parsing

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Conflicts:

src/rgw/rgw_admin.cc

rgw_admin: default to json output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

radosgw-admin: use formatter for 'bucket list'

Signed-off-by: Sage Weil <sage@newdream.net>

rdosgw_admin -> radosgw-admin

Be consistent with other binary tool naming.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: remove now-unused struct RGWPoolInfo

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ceph: make -o - the default

Finally.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph: make --concise hide 'N byte written' message

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: remove rgw_[retrieve|store]_pool_info and ".pool" pool

Convert these to rgw_[get|store]_bucket_info, add the owner field
to bucket info, and store both lookup formats in the ".bucket" pool.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: fix leak of src_obc references

We never moved the src_obc map from the local var into the RepGather
member!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librbd: fix end_block calculation for read_iterate

All end block calculations match now.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: don't zero negative lengths

They are past the end of the buffer (and wrap around).

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: remove references to pool_id and replace with new bucket_id.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Use LIBEDIT_LIBS in Makefile.am

The editline library might require some additional libs, so
instead of hardcoding '-ledit' we should be using LIBEDIT_LIBS
in Makefile.am

Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-dotnames'