doc: Added some detail. Calculating PGs, maps; reorganized a bit.

fixes: #2968

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #234 from ceph/wip-4758

Fixes #4758.

Reviewed-by: Greg Farnum <greg@inktank.com>

mon: commit LogSummary on every message

This moves our version pointer up so that we don't re-log (by re-consuming)
log messages to /var/log/ceph/ceph.log on ceph-mon restart.  OTOH, it means
we rewrite the summary of the last 50 messages, but we consider that to be
relatively cheap (and something we *always* did prior for bobtail and
earlier anyway).

Signed-off-by: Sage Weil <sage@inktank.com>

mon: set threshold to periodically stash_full

Set an interval to periodically write a full copy of the map that is lower
than the trim point (which is generally a very large number of commits).

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #230 from ceph/wip-mon-paxos-fixes

Wip mon paxos fixes

Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #225 from ceph/wip-4543

Fixes #4543

Reviewed-by: Greg Farnum <greg@inktank.com>

ceph-mon: Attempt to obtain monmap from several possible sources

In order of interest/priority:

  - our latest monmap version
  - a backup monmap version created during sync start, if the store
    appears to be in a post-aborted sync state
  - a mkfs monmap version

If none of these are found, we should go ahead and try to build a
monmap from ceph.conf to join an existing cluster.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: backup monmap prior to starting a store sync

If by fate we end up attempting a store sync after failing at
least one before, we might not have a monmap to read from the
store to backup.  Therefore, in that case, we shall backup the
current monmap being used by the monitor.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

rgw: don't send tail to gc if copying object to itself

Fixes: #4776
Backport: bobtail
Need to make sure that when copying an object into itself we don't
send the tail to the garbage collection.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #232 from ceph/wip-4710

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #233 from ceph/wip-mon-idempotent

Wip mon idempotent

Reviewed-by: Dan Mick <dan.mick@inktank.com>

mon: make 'osd pool rmsnap ...' idempotent

Signed-off-by: Sage Weil <sage@inktank.com>

mon: make 'osd pool mksnap ...' idempotent

Signed-off-by: Sage Weil <sage@inktank.com>

mon: make 'osd blacklist rm ...' idempotent

Signed-off-by: Sage Weil <sage@inktank.com>

rbd: only set STRIPINGV2 feature when needed

Only set the STRIPINGV2 feature if the striping parameters are non-default.
Specifically, fix the case where the passed-in size and count are == 0.

Fixes: #4710
Signed-off-by: Sage Weil <sage@inktank.com>

rbd: fix feature display for --info

Only include the feature if it is set!

Backport: bobtail
Signed-off-by: Sage Weil <sage@inktank.com>

rbd: avoid clobbering return value with udevadm settle

Fixes: #4707
Signed-off-by: Sage Weil <sage@inktank.com>

FileJournal: a valid entry after invalid entry =/=> corrupt

Out of order journal entry writes using aio may cause entry
n+2 to be written prior to n.  This does not indicate
corruption.

Fixes: #4736
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

radosgw: Fix duplicate 'Content-Type' when using 'response-content-type'

Signed-off-by: Sylvain Munaut <s.munaut@whatever-company.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

mon: MonmapMonitor: add function to obtain latest monmap

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: PaxosService: add 'exists_key/version' helper functions

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

ceph-create-keys:  Don't wait if permission denied

If get or create keys returns permssion denied, exit
gracefully instead of retrying.

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

doc: Aesthetic improvements. Removed unnecessary graphic and overrode margin for h3 tag.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added a scenario to PG troubleshooting.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Changed usage to "bucket-name". Description was okay.

fixes: #4102

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge branch 'wip-4201' into next

Reviewed-by: Samuel Just <sam.just@inktank.com>

tools/ceph-filestore-dump: Implement remove, export and import

Change local names to be clearer
Break real_log() into common function get_log()
Move infos_oid, biginfo_oid and log_oid to globals for general use

Feature: #4201 (osd: data loss: pg export/import/remove)

Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge branch 'wip_4662_clean' into next

Reviewed-by: Greg Farnum <greg@inktank.com>

ReplicatedPG::_finish_mark_all_unfound_lost: only requeue if !deleting

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::_applied_recovered_object*: don't queue scrub if deleting

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: check for pg change in ~FlushState

Fixes: #4662
Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: bail if deleting in _finish_recovery

Signed-off-by: Samuel Just <sam.just@inktank.com>

AsyncReserver: delete context in cancel_reservation

Fixes: #4662
Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>

tools/ceph-filestore-dump: Error messages lost because stderr is closed

Use cout instead of cerr for command errors
Use cerr for debug mode because stderr is avail
Output map_epoch in debug mode
Fix a message and only for debug mode

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Make clear_temp() public for use by remove

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Add flag to force version write in _write_info()

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Create static PG::_write_log() function

Signed-off-by: David Zafman <david.zafman@inktank.com>

OSDMonitor: pg split is no longer experimental

Fixes: #4711
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #228 from alram/next

Fix journal partition creation

Reviewed-by: Sage Weil <sage@inktank.com>

Fix journal partition creation

With OSD sharing data and journal, the previous code created the
journal partiton from the end of the device. A uint32_t is
used in sgdisk to get the last sector, with large HD, uint32_t
is too small.
The journal partition will be created backwards from the
a sector in the midlle of the disk leaving space before
and after it. The data partition will use whichever of
these spaces is greater. The remaining will not be used.

This patch creates the journal partition from the start as a workaround.

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

rbd: fix qa tests to use --allow-shrink

Fixes: #4763
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #227 from ceph/wip-4574

Reviewed-by: Greg Farnum <greg@inktank.com>

init-ceph: do not stop start on first failure

When starting we often loop over many daemon instances.  Currently we stop
on the first error and do not try to start other daemons.

Instead, try them all, but return a failure if anything did not start.

Fixes: #2545
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Gary Lowell <gary.lowell@inktank.com>

mon: Monitor: fix timechecks get_health clobbering overall status

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: fix health monitor calls

- unconditionally call get_health, regardless of formatter *
- return a meaningful health status code

Signed-off-by: Sage Weil <sage@inktank.com>

global: call observers (and start logging) in global_init

Call observers so that the logging infrastructure gets initailized and we
start logging.  Otherwise, unless a default log setting has been modified,
we won't start logging until we daemonize, and we won't get the nice
version banner in the log file.

Unlike the previous attempt to fix this (a3091774), we do this after all
of the lockdep initialization has completed.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: Create new static function PG::_write_info() for use by PG import

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Add OSD::make_infos_oid() as common function to create oid

Signed-off-by: David Zafman <david.zafman@inktank.com>

filestore, osd: Fixes to comform to programming guidelines

Signed-off-by: David Zafman <david.zafman@inktank.com>

mon: QuorumService: return health status on get_health()

This allows us to return the appropriate overall health status on
Monitor::get_health().

Fixes: 4574
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

OpRequest: don't maintain history if the OSD is shutting down

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: optionally track every pg ref

This involves three pieces:

For intrusive_ptr type references, we use TrackedIntPtr instead.  This
uses get_with_id and put_with_id to associate an id and backtrace with
each particular ref instance.

For refs taken via direct calls to get() and put(), get and put now
require a tag string.  The PG tracks individual ref counts for each tag
as well as the total.

Finally, PGs register/unregister themselves on construction/destruction
with OSDService.

As a result, on shutdown, we can check for live pgs and determine where
the references are held.

This behavior is compiled out by default, but can be included with the
--enable-pgrefdebugging flag.

Signed-off-by: Samuel Just <sam.just@inktank.com>

common/: add tracked_int_ptr.hpp

TrackedIntPtr acts like intrusive_ptr, but is able to
track a ref id.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use the ReplicatedPGRef typedef

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use ReplicatedPGRef for C_PG_MarkUnfoundLost

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use ReplicatedPGRef for C_OSD_OpCommit

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use ReplicatedPGRef for C_OSD_OpApplied

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: use PGRef in handle_pg_remove

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: use PGRef in handle_pg_stats_ack

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: use PGRef in QueuePeeringEvt

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: use PGRef in consume_map

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: use PGRef for FlushState

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: use PGRef for C_PG_FinishRecovery

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: use PGRef in C_PG_ActivateCommitted

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: do not put() in scrub() if pg is deleting

scrub() no longer handles the put, this call
must have been missed.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG,ReplicatedPG: move intrusive_ptr declarations to top

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: add ReplicatedPGRef

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::_do_copy_range: read(2) might return EINTR

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::_do_clone_range: _do_copy_range encodes error in return, not errno

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #224 from ceph/wip-mon-crush

Wip mon crush

Reviewed-by: Dan Mick <dan.mick@inktank.com>

config: clarify 'mon osd down out subtree limit'

Clarify the description; this is the subtree type that we won't mark out
if it is all down, but anything less than it will be.

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Trimmed toc depth for nicer visual appearance.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added new PG troubleshooting use case.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated title.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added PG troubleshooting to toctree.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Bifurcating OSD and PG Troubleshooting. Updated hyperlink.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Bifurcating OSD and PG Troubleshooting. Added PG troubleshooting doc.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Bifurcating OSD and PG Troubleshooting. Removed PG section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

mon: ensure 'osd crush rule ...' commands are idempotent

Ensure that we return 0 for these cases.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: make 'osd crush link ...' idempotent

We fixed move in f5ba0fbbe73e11418634bc95e1fc36d17edccf37 but missed this
one.

Signed-off-by: Sage Weil <sage@inktank.com>

rgw_bucket: Fix dump_index_check.

Signed-off-by caleb miles <caleb.miles@inktank.com>

Merge branch 'wip-max_size-3637' into next

Reviewed-by: Sage Weil <sage@inktank.com>

mds: journal the projected root xattrs in add_root()

In EMetaBlob::add_root(), we should log the projected root xattrs
instead of original ones to reflect xattr changes.

Signed-off-by: Kuan Kai Chiu <big.chiu@bigtera.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: fix setting/removing xattrs on root

MDS crashes while journaling dirty root inode in handle_client_setxattr
and handle_client_removexattr. We should use journal_dirty_inode to
safely log root inode here.

Signed-off-by: Kuan Kai Chiu <big.chiu@bigtera.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

debian/control:  Fix typo in libboost version number

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

build:  Add new package dependencies

Add libboost-system-dev (bug #4725).

Add hdparm to rpm installation requirements.  The hdparm
command is used to determin if write-caching is enabled on
the journal device.

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

mon: PaxosService: fix trim criteria so to avoid constantly trimming

Say a service establishes it will only keep 500 versions once a given
condition X is true.  Now say that said condition X only becomes true
after said service committing some 800 versions.

Once we decide to trim, this service would trim all 300 surplus versions
in one go.  After that, each committed version would also trim the
previous version.

Trimming an unbounded number of versions is not a good practice
as it will generate bigger transactions (thus a greater workload on
leveldb) and therefore bigger messages too.

Constantly trimming versions implies more frequent accesses to leveldb,
and keeping around a couple more versions won't hurt us in any significant
way, so let us put off trimming unless we go over a predefined minimum.

This patch adds two new options:

 paxos service trim min - minimum amount of versions to trigger a trim
                          (default: 30, 0 disables it)
 paxos service trim max - maximum amount of versions to trim during a
                          single proposal
                          (default: 50, 0 disables it)

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Paxos: increase debug levels for proposal listing

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

doc: Removed legacy man page index. Generates warning otherwise.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Clarified that admin-socket is accessed from same host.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated hyperlinks to new tshooting section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed this doc. Nothing referenced it, and parent directory echoes content.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Revised top-level ops page.

Consolidated authentication into high-level operations. Added a
troubleshooting section. Collapsed toc trees to make the appearance
cleaner.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed link to nowhere. Otherwise generates a warning.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed top-level tshoot page, and created new index.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Excised community from OSD tshoot, made it stand alone.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Moved monitor troubleshooting to troubleshooting section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Moved troubleshooting OSD to troubleshooting section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added extraneous rgw settings to rgw conf.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Moved memory profiling from operations to troubleshooting.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>