Merge remote branch 'upstream/wip-backfill-ordering' into wip-backfill

PG: add some documentation

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: delay op while snapdir is missing/degraded

We cannot get/update a snapcontext if snapdir is missing/degraded.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: don't manage waiting_on_backfill in start/finish_recovery_op

Set waiting_on_backfill in recover_backfill and clear in do_scan.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: apply_repop: apply local_t before op_t

We create snap_collections in local_t and clone into them in op_t.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: Do not update_snap_collections for log entries > last_backfill

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: Fix stat debug output

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

calc_acting: Prefer up[0] as primary if possible

Previously, we could get into a state where although up[0] has been
fully backfilled, acting[0] could be selected as a primary if it is able
to pull another peer into the acting set.  This also collects the logic
of choosing the best info into a helper function.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: fix backfill mismatch error output

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: calc_clone_subsets fix other clone_overlap case

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

OSD: use disk_tp.pause() without osd_lock

Previously, we called disk_tp.pause_new().  This can cause a race
where snap_trimmer queues more transactions after we flush the
store.  Calling disk_tp.pause() under the osd_lock causes a
deadlock with pg removal.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd: wait for src_oid if it on other side of last_backfill from oid

If the target object is before last_backfill, then the backfill_target
will be asked to apply the operation.  If one of the src objects is past
last_backfill, that will fail, so we need to wait for the src object to
be not degraded.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: EINVAL on mismatched locator without waiting for degraded

No reason to recover before returning an error.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: preserve write order when waiting on src_oids

We need to preserve the order of write operations on each object.  If we
have a write on X that needs to read from Y, and Y is degraded, then we
need to wait for Y to repair.  Doing so blindly will allow other writes
to X to proceed while our clone op is still waiting, violating the
ordering.

Fix this by adding blocked_by and blocking vars to the ObjectContext.  If
we wait on a src_oid, the oid is "blocked" by that object, and any
subsequent writes should also wait on the same queue.

Use a helper to do the cleanup when we complete recovery, or when the
pg resets.

Signed-off-by: Sage Weil <sage@newdream.net>

ReplicatedPG: add_object_context_to_pg_stat, obc->ssc may be null

obc->ssc is not necessarily filled in by get_object_context.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: clear waiting_on_backfill during clear_recovery_state

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: list snapid 0 on collection_list_partial for backfill

0 will list all objects, CEPH_NO_SNAP will list only head objects.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge remote branch 'upstream/master' into wip_backfill_merged

ReplicatedPG: collection_list_partial also takes a snapid

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: Ask for digest at most once at a time

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd: track backfill target pg stats

Maintain backfill target pg stats to be the summation over objects to
the left of last_backfill.  Reflect this in the degraded stats we report
to the monitor.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: object_stat_sum_t::clear()

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

PG: activate, backfill replica can have last_complete < log_tail

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: calc_*_subsets must consider last_backfill

Objects yet to be backfilled do not show up in the missing set.  Thus,
we cannot use an object past last_backfill to clone into the object we
are pushing/pulling.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: A backfill osd can have last_complete < log_tail

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: there may now be backfill entries in the acting set

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: fix phantom entry in peer_info

In GetLog, do not call pg->peer_info[newest_update_osd] if
newest_update_osd is osd->whoami.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd: remove bad !is_incomplete() assert

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix signed/unsigned comp

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix calc_acting()

Look at usable, not want.size(), so we don't count backfill targets.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: some recover_backfill() comments

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: add incomplete, backfill states; simplify calculation

Set/clear states in peering state machine state ctor/dtors where possible.

Set degraded if the number of non-backfilling replicas is lower than the
target replication factor.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

hobject_t: fix hobject(sobject_t) constructor

Initialize max

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: more backfill changes

Always ship log for updates to backfill targets to preserve the repgather
ordering.

Fix up recover_backfill() bounds.  Re-scan the local collect on every pass
in case there were concurrent modifications.  (This could be optimized.)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: observe last_backfill in merge_log() and helpers

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objectstore: make list by hash *next > instead of >=

This means we should set it to a hash boundary or the last item of our
result set (not the next item we didn't include).

It means that during backfill we can set our last_backfill to the last
object we did recover and be sure that any new files locally will be
included in the next result set, and we can bound that result set by that
last object recovered and not include it in the resulting range.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: initialize repop gather set in issue_repop instead of new_repop

Simpler.  It will also make the last_backfill correction live in one
place.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: implement PG::copy_range()

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Revert "osd: don't keep push state on replicas"

This reverts commit 69c77e33f8530993dbc280525bd21218ea6f9ddb.

sub_op_pull() calls send_push_op directly, does not pass push_start().

osd: make backfill (basically) work again

Still need to handle concurrent updates, log recovery vs backfill, etc.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: keep backfill targets in acting set

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: advance last_backfill by keys only

This ensures that transactions are never split by last_backfill.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

hobject_t: fix sorting by hash key

Use get_effective_key() to return key (if explicit) or object name.  Sort
by that within each hash value.

Clean up operator<< so that it prints things in sort order.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: osd_kill_backfill_at

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: track backfill with last_backfill, not interval_set<>

We always fill from the bottom up anyway.  Using an hobject_t also gives us
a precise bound.  It also makes things conceptually simpler: last_complete
and last_backfill bounding each of the two dimensions of updatedness.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: pg whitespace

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: kill unused PG::trim_write_ahead

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: kill unused PG::Log::copy_after_unless_divergent

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: cleanup lingering backlog refs

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: cleanup

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: kill unused PG_STATE_SCANNING

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: remove recovery-from-backlog kludge last_update

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: kill PG::Info::backlog

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: remove backlog case from clean_up_local

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: remove log_backlog from PG::Info

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: state machine whitespace

I feel better now

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: strip backlog logic out of PG::activate()

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: kill backlog_requested

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: strip backlog case out of merge_log

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: strip out some backlog logic

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: kill PG::Log::copy_non_backlog

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: remove Query::BACKLOG processing

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: simplify replica queries for finding divergent objects

No need to request backlog here, clearly, since those don't exist anymore.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: remove backlog generation code

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: do not read backlog off disk

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: add Incomplete peering state

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test_backfill.sh

osd: recover discontiguous peers using backfill instead of backlog

Instead of generating a huge list of objects to recover, and then pushing
them, iterate over the collection and copy objects as we go.

Disable various bits of backlog code; it will all get ripped out shortly.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: add MOSDPGBackfill message

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

types: operator<< for multimaps

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: implement do_scan

Handle MOSDPGScan messages to request or send a digest of a range of
objects in a collection, sorted in hobject_t (hash) order.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: implement ReplicatedPG::scan_range

Scan a range of the local collection.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: implement ReplicatedPG::_lookup_object_context

Look up an existing ObjectContext without taking a reference.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: add PG::BackfillInterval type

Describe a range of objects for the purposes of backfilling a PG.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: MOSDPGScan

Message to query hash ranges of a PG.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: rewrite choose_acting process

Consolidate callers, eliminate obsolete backlog ones.

New process:
 - pick best log, with preferences for those that might end up primary
 - pick best primary that is log-contiguous with best log, with preference
   for longer tails that will result in more acting osds.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: don't keep push state on replicas

Primaries need this, but replicas don't: the primary will explicitly pull
the pieces of the object that it wants.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix push_to_replica typo

We are always pushing soid.  If we are missing snapdir locally, that means
we can't do an informed efficient clone, and should push the whole
object... NOT that we should push snapdir!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

swift: auth response returns X-Auth-Token instead of X-Storage-Token

osd: fix build_incremental_map_msg

We keep both the inc and the full for our oldest osdmap.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: trivial_libceph test

This currently fails... see #1827

Signed-off-by: Sage Weil <sage@newdream.net>

libceph: add missing #includes

Signed-off-by: Sage Weil <sage@newdream.net>

libceph: catch errors from Client::init()

And clean up error paths.

Signed-off-by: Sage Weil <sage@newdream.net>

client: return errors from init

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: dump transaction to log if we hit an error

This will let us see which operation in the transaction failed.

Signed-off-by: Sage Weil <sage@newdream.net>

objectstore: implement Transaction::dump()

Signed-off-by: Sage Weil <sage@newdream.net>

objectstore: create Transaction::iterator class

Remove iterator state from Transaction itself.

Signed-off-by: Sage Weil <sage@newdream.net>

qa: rados load-gen workunits

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/rados: add asynchronous write,append,read,write_full operations

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

doc: Clarify documentation of reweight command.

This caused some discussions on the mailing list, so let's try to be clear
about the meaning of an OSD weight.

Signed-Off-By: Andre Noll <maan@systemlinux.org>

doc: Add missing documentation for osd pool get.

"osd pool set" was already documented, but the corresponding "get"
command was not. This patch adds the list of valid fields for this
command, together with short descriptions.

Thanks to Gregory Farnum for pointing out a flaw in the description
of lpl_num and lpgp_num in an earlier version of the patch.

Signed-Off-By: Andre Noll <maan@systemlinux.org>

doc: Update the list of fields for the pool set command.

This list was lacking a few fields: crash_replay_interval, pg_num,
pgp_num and crush_ruleset. Include these fields and add add short
descriptions.

Signed-Off-By: Andre Noll <maan@systemlinux.org>

doc: Document pause and unpause osd commands.

These two commands were undocumented so far. This patch adds a short
description.

Signed-Off-By: Andre Noll <maan@systemlinux.org>

doc: Add documentation of missing osd commands.

The set of OSD commands which added by the previous commit is
incomplete. This patch adds documentation for the following
OSD commands which were previously missing: dump, tree, crush,
cluster_snap, lost, create, rm.

Signed-Off-By: Andre Noll <maan@systemlinux.org>

doc: Import the list of ceph subcommands from wiki.

This adds the content of the wiki page at

http://ceph.newdream.net/wiki/Monitor_commands

to doc/control.rst in order to make it available at the new official
location for the ceph documentation. This first patch is just the
result of a cut-and-paste operation. There are no changes in content,
but the text was converted to rst format.

Signed-Off-By: Andre Noll <maan@systemlinux.org>

Merge branch 'wip-mon-stats'

mon: add mds, mon info to cluster_logger

Signed-off-by: Sage Weil <sage@newdream.net>

mdsmap: rename get_num_*_mds() methods

Signed-off-by: Sage Weil <sage@newdream.net>

mon: report basic cluster stats via perfcounters

These are basic point-in-time cluster stats.

Signed-off-by: Sage Weil <sage@newdream.net>

crush: drop unused label

Signed-off-by: Sage Weil <sage@newdream.net>