global_init: do not count threads before daemonize()

We were verifying that there was only 1 thread (the presumably main()) when
we call daemonize.  However, with the new logging code, we stop a thread
right before the check, and /proc apparently updates asynchronously such
that our attempt to count running threads gives us a bad answer.

Just remove this kludgey check; we'll have to catch this class of bugs
the hard way.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

OpRequest: only show a small set of the oldest messages, instead of all.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: update cache interface for put_obj_meta

This fixes issue #2381.
The method interface was different than the one needed in order
to override the one in RGWRados.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

doc: fix some underscores

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-doc-rebase-2'

Fixed link to blog.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

Fixed another link to the blog.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

Fixed link.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

Clean up. Changed ceph.newdream.net to ceph.com.
Removed {ARCH} references.
Added link to Source.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

doc: more fonts

Signed-off-by: Sage Weil <sage@newdream.net>

doc: new theme

Signed-off-by: Ross Turk <ross.turk@inktank.com>

doc/install/debian: simplify more

Signed-off-by: Sage Weil <sage@newdream.net>

doc/install: reorg, simplify

Signed-off-by: Sage Weil <sage@newdream.net>

doc: clarify path names for mkcephfs

Signed-off-by: Sage Weil <sage@newdream.net>

doc: simplify quick start

Remove dup items

Signed-off-by: Sage Weil <sage@newdream.net>

doc: Whitespace cleanup.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Rename to use dashes not underscores in URLs.

This makes the-separate-words in the url match as separate words in
searches, where this_way only matches an explicit "this_way" search.
http://www.mattcutts.com/blog/dashes-vs-underscores/

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Remove leading whitespace, that syntax means it's a definition list.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Updated URLs for the new ceph.com site.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

objecter: ping connections with lingering ops every tick()

Also, make sure that we initialize linger_op.session once
we have it.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Removed "Ceph Development Status" per Bryan
Modified title syntax per Tommi
Modified paragraph width to 80-chars per Dan
Moved "Build from Source" out of Install
Renamed create_cluster to config-cluster
Added config-ref with configuration reference tables
Added a toc ref for man/1/obsync per Dan
Removed redundant sections from Ops
Deleted "Why use Ceph" and "Introduction to Storage Clusters"

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

Merge branch 'stable'

Conflicts:
src/common/config_opts.h

config: $cluster-whatever for admin_socket, log files

This is more consistent with /var/lib/ceph/$type/$cluster-$id.

Signed-off-by: Sage Weil <sage@newdream.net>

config: Admin socket path had literal "name" in it instead of $name.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

config: Admin socket path had literal "name" in it instead of $name.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mon: make 'osd map <poolname> <objectname>' return pg and osd mapping

This keeps coming up and is annoying to do manually.

Fixes: #2141
Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

rbd.py: add new error for deleting an image with snapshots

Forgot to update the python bindings in
88dda3be5aed38c8f587af9b27cb54a0fcca7530

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

config: no -C shorthand for --cluster

ceph-authtool uses it!

Fixes: #2376
Signed-off-by: Sage Weil <sage@newdream.net>

Edits.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Corrections.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Significant updates to introduction, getting packages, building from source, installing packages, and creating a cluster.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: document NOIN, NOOUT, NOUP, NODOWN flags and flapping

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Removed this. As part of restructuring of install to include admin host.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Minor edits. Still WIP.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

The new files contain S3 APIs and a build from source doc.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Minor cleanup.

Signed off by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Added a new landing page graphic, and made some minor edits on the landing page.

Submitted by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Removed some files for reorg.

Submitted by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Added a plug for commercial support. ;)

Submitted by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Building out information architecture. Modified getting involved, why use ceph, etc.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Remove reference to Introduction to RADOS OSDs

Submitted by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Added introduction to clustered storage and deleted older files that have been moved.

Submitted by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Restructuring documentation.

Submitted by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Initial cut of introduction, getting started, and installing. More to do on installation. RADOS gateway to follow.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Put change to rbd manpage in .rst instead, update man/ from output

doc: move documentation build instructions to doc/dev section

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Treat rename across pools as an error
Fixes: #2370
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

doc: add warning about multiple monitors on one machine.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: normalize bucket/obj before updating cache

Fixes bug #2369. The problem was that sometimes we send the
notification with the un-normalized bucket/obj pair. We
should make sure that we use the caonical name before doing
any cache update.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

PG: Add probe set to HB peers during GetInfo

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: check_new_interval now handles adding new maps to past intervals

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

filestore: allow flusher+sync_flush enable/disable via injectargs

This only affects the decision to queue or do things inline, so it is safe
to change while the filestore is up and running.

Also adjust the #ifdef so that there we share a single path through the
code when sync_file_range() is missing.

Fixes: #2368
Signed-off-by: Sage Weil <sage@newdream.net>

filestore: fix op_queue_{len,bytes} instrumentation

(re)set these in logger when they actually change.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-throttle'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

throttle: count get_or_fail success/failure explicitly

Signed-off-by: Sage Weil <sage@newdream.net>

osd: pg creation calc_priors_during() should count primary as up

If only want to include down osds if *all* of the prior acting osds are
down.  If osd->whoami is one of them, then we're okay.

For example, if osd.13 is down, then the below should be satisfied that
osd.14 (osd->whoami) is alive:

2012-04-27 10:46:38.746681 7f5258a63700 15 osd.14 27 calc_priors_during 6.5 [9,25)
2012-04-27 10:46:38.746688 7f5258a63700 20 osd.14 27   6.5 in epoch 9 was [13,14]
2012-04-27 10:46:38.746695 7f5258a63700 20 osd.14 27   6.5 in epoch 10 was [13,14]
2012-04-27 10:46:38.746701 7f5258a63700 20 osd.14 27   6.5 in epoch 11 was [13,14]
2012-04-27 10:46:38.746709 7f5258a63700 20 osd.14 27   6.5 in epoch 12 was [13,14]
2012-04-27 10:46:38.746715 7f5258a63700 20 osd.14 27   6.5 in epoch 13 was [13,14]
2012-04-27 10:46:38.746722 7f5258a63700 20 osd.14 27   6.5 in epoch 14 was [13,14]
2012-04-27 10:46:38.746729 7f5258a63700 20 osd.14 27   6.5 in epoch 15 was [14]
2012-04-27 10:46:38.746735 7f5258a63700 20 osd.14 27   6.5 in epoch 16 was [14]
2012-04-27 10:46:38.746742 7f5258a63700 20 osd.14 27   6.5 in epoch 17 was [14]
2012-04-27 10:46:38.746748 7f5258a63700 20 osd.14 27   6.5 in epoch 18 was [13,14]
2012-04-27 10:46:38.746755 7f5258a63700 20 osd.14 27   6.5 in epoch 19 was [13,14]
2012-04-27 10:46:38.746762 7f5258a63700 20 osd.14 27   6.5 in epoch 20 was [13,14]
2012-04-27 10:46:38.746768 7f5258a63700 20 osd.14 27   6.5 in epoch 21 was [13,14]
2012-04-27 10:46:38.746775 7f5258a63700 20 osd.14 27   6.5 in epoch 22 was [14]
2012-04-27 10:46:38.746781 7f5258a63700 20 osd.14 27   6.5 in epoch 23 was [14]
2012-04-27 10:46:38.746788 7f5258a63700 20 osd.14 27   6.5 in epoch 24 was [14]
2012-04-27 10:46:38.746790 7f5258a63700 10 osd.14 27 calc_priors_during 6.5 [9,25) = 13

In that case, it wasn't, and the pg creation was blocked.

Fixes: #2355
Signed-off-by: Sage Weil <sage@newdream.net>

throttle: note current value and max in perfcounters

This exposes a snapshot of the current Throttle value and limit.

Signed-off-by: Sage Weil <sage@newdream.net>

.gitignore: ceph-kdump-copy

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote-tracking branch 'gh/wip-ceph-kdump-copy'

osd: add is_unmanaged_snaps_mode() to pg_pool_t; use more consistently

Create an is_unmanaged_snaps_mode() function to parallel
is_pool_snaps_mode(), and replace all the checks directly referencing
removed_snaps or snaps with calls to these functions.
Fixes #2345.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

pick_address: don't bother checking struct ifaddrs which have a null ifa_addr

I assume that's the localhost interface or similar.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote-tracking branch 'gh/wip-2352'

Reviewed-by: Sage Weil <sage@newdream.net>

msgr: include msgr name in dispatch_throttler name

Signed-off-by: Sage Weil <sage@newdream.net>

msgr: name messengers

Give each Messenger a logical name describing its role.  For instance, the
OSD will have client, cluster, and heartbeat messengers.

Signed-off-by: Sage Weil <sage@newdream.net>

throttle: report stats via perfcounter

Fixes: #2358
Signed-off-by: Sage Weil <sage@newdream.net>

global_init: don't fail out if there is no default config.

There are plenty of scenarios where the user doesn't need a config file.
Instead, just print a warning and let things move on.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

global: fix incorrect CINIT flag.

There is nobody responding to CLOSE_STDERR, but this block sure looks
like it should be doing so. Fix that!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

throttle: feed cct, name, and add logging

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge tag 'v0.46'

v0.46

osdmap: do no dereference NULL entity_addr_t pointer in addr accessors

These may be NULL if we expand the addr vectors but haven't ever stored an
address yet.  Check for NULL and return a reference to a blank
entity_addr_t as needed.

Signed-off-by: Sage Weil <sage@newdream.net>

OSD: add different config options for map bl caches

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

mon: fix nion -> noin typo

Thanks Greg!

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-pi'

Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

v0.46

librbd: use unique error code for image removal failures

This allows the rbd tool to provide a useful error message, instead of
compounding more possible causes into one error code.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

run_xfstests.sh: drop #62

Until #2359 is resolved.

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: fix identify_osd() and find_osd_on_ip()

In 313c1566d3b649ef81fcdc722678d77dccfa888f we switched to using the
get_addr() accessor methods, which assert that the osd exists.  Check that
before calling.

Fixes: #2361
Signed-off-by: Sage Weil <sage@newdream.net>

osd: keep pgs locked during handle_osd_map dance

Currently we drop and retake locks during handle_osd_map calls to
advance_map and activate_map.  Instead, take them all once, and hold them.
This avoids leaving dirty in-core state in the PG without the lock held.

This will clearly go away as soon as the map threading stuff is redone.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: drop obsolete osd/PG.h #includes

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: set dirty flags on rewind_divergent_log

Make sure we record any rewind_divergent_log.  In the activate case, this
will happen anyway, but mark it dirty here for correctness/completeness.

The merge_log case might be a bug.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: use dirty flags in activate(), merge_log()

These are all called from within the state machine, so we can simply set
the dirty flags.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix nested transaction in all_activated_and_committed()

all_activated_and_committed() is called from _activate_committed(), called
from a objectstore completion, and also from the state machine, which is
part of a larger transaction.

Instead, set dirty_info, and build/apply a transaction in the caller
(the completion) as needed.  Fixes part of #2360.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: use PG::write_if_dirty() helper

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: do not merge history on query

We shouldn't modify the local notion of the history without recording it to
disk.  And we (probably) also don't need to do that at all on query.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: dirty_info if history.merge updated anything

In proc_replica_info and proc_primary_info, we may or may not update
the pg_info_t.  If we do, set dirty_info, so that it will be recorded.
Same goes for when the primary pushes out updated stats to us.

Also, do not write a purged_snaps() update directory; rely on the caller
to write out dirty info.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: write dirty info on handle info, notify, log

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: skip scrub scheduling if we aren't up

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix dirty_info check for advance/activate paths

Previously we would check and write dirty_info *without the pg lock* after
doing the advance and activate map calls.  This was unlikely to race with
anything because the queues were drained, but definitely not right.

Instead, do the write in activate_map, or explicitly if activate_map is
not called (so that we record our progress after handling maps when we are
not up).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'next'

run_seed_to.sh: clean out merge cruft

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

log: do not set on_exit() callback for libraries

Set this up in either global_init() or common_init_finish(), both opportune
times that occur after config parsing has happened and the user has the
option to modify this behavior.  The exception would be libraries like
librados, which can't use rados_conf_* to enable this.  Arguably flush
functionality should be exposed through the librados API directly, instead
of futzing with on_exit().

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'gh/wip-filestore-misc'

Conflicts:
src/test/filestore/run_seed_to.sh

Merge remote branch 'gh/wip-2353'

Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

osd: always share past_intervals

Share past intervals when starting up new replicas.  This can happen via
an MOSDPGInfo or an MOSDPGLog message.

Fix up get_or_create_pg() so the past_intervals arg is required (and a ref,
like the other args). Fix doxygen comment.

Now the only time generate_past_intervals() should do any work is when
upgrading old clusters, during pg creation, and (possibly) during pg
split (when that is fully implemented).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-osdmap'

Conflicts:
src/mon/PGMonitor.cc
src/osd/OSDMap.h

fix file_layout.sh layouts test

preferred_osd is not gone.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-mon'

Reviewed-by: Gregory Farnum <gregory.farnum@dreamhost.com>

mon: 'osd [un]set noin'

Missed this one.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'next'

osd: set dirty_info in generate_past_intervals

This ensures that we save our work.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>