ceph-create-keys:  Don't wait if permission denied

If get or create keys returns permssion denied, exit
gracefully instead of retrying.

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

doc: Aesthetic improvements. Removed unnecessary graphic and overrode margin for h3 tag.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added a scenario to PG troubleshooting.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Changed usage to "bucket-name". Description was okay.

fixes: #4102

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge branch 'wip-4201' into next

Reviewed-by: Samuel Just <sam.just@inktank.com>

tools/ceph-filestore-dump: Implement remove, export and import

Change local names to be clearer
Break real_log() into common function get_log()
Move infos_oid, biginfo_oid and log_oid to globals for general use

Feature: #4201 (osd: data loss: pg export/import/remove)

Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge branch 'wip_4662_clean' into next

Reviewed-by: Greg Farnum <greg@inktank.com>

ReplicatedPG::_finish_mark_all_unfound_lost: only requeue if !deleting

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::_applied_recovered_object*: don't queue scrub if deleting

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: check for pg change in ~FlushState

Fixes: #4662
Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: bail if deleting in _finish_recovery

Signed-off-by: Samuel Just <sam.just@inktank.com>

AsyncReserver: delete context in cancel_reservation

Fixes: #4662
Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>

tools/ceph-filestore-dump: Error messages lost because stderr is closed

Use cout instead of cerr for command errors
Use cerr for debug mode because stderr is avail
Output map_epoch in debug mode
Fix a message and only for debug mode

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Make clear_temp() public for use by remove

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Add flag to force version write in _write_info()

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Create static PG::_write_log() function

Signed-off-by: David Zafman <david.zafman@inktank.com>

OSDMonitor: pg split is no longer experimental

Fixes: #4711
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #228 from alram/next

Fix journal partition creation

Reviewed-by: Sage Weil <sage@inktank.com>

Fix journal partition creation

With OSD sharing data and journal, the previous code created the
journal partiton from the end of the device. A uint32_t is
used in sgdisk to get the last sector, with large HD, uint32_t
is too small.
The journal partition will be created backwards from the
a sector in the midlle of the disk leaving space before
and after it. The data partition will use whichever of
these spaces is greater. The remaining will not be used.

This patch creates the journal partition from the start as a workaround.

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

rbd: fix qa tests to use --allow-shrink

Fixes: #4763
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #227 from ceph/wip-4574

Reviewed-by: Greg Farnum <greg@inktank.com>

init-ceph: do not stop start on first failure

When starting we often loop over many daemon instances.  Currently we stop
on the first error and do not try to start other daemons.

Instead, try them all, but return a failure if anything did not start.

Fixes: #2545
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Gary Lowell <gary.lowell@inktank.com>

mon: Monitor: fix timechecks get_health clobbering overall status

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: fix health monitor calls

- unconditionally call get_health, regardless of formatter *
- return a meaningful health status code

Signed-off-by: Sage Weil <sage@inktank.com>

global: call observers (and start logging) in global_init

Call observers so that the logging infrastructure gets initailized and we
start logging.  Otherwise, unless a default log setting has been modified,
we won't start logging until we daemonize, and we won't get the nice
version banner in the log file.

Unlike the previous attempt to fix this (a3091774), we do this after all
of the lockdep initialization has completed.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: Create new static function PG::_write_info() for use by PG import

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: Add OSD::make_infos_oid() as common function to create oid

Signed-off-by: David Zafman <david.zafman@inktank.com>

filestore, osd: Fixes to comform to programming guidelines

Signed-off-by: David Zafman <david.zafman@inktank.com>

mon: QuorumService: return health status on get_health()

This allows us to return the appropriate overall health status on
Monitor::get_health().

Fixes: 4574
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

OpRequest: don't maintain history if the OSD is shutting down

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: optionally track every pg ref

This involves three pieces:

For intrusive_ptr type references, we use TrackedIntPtr instead.  This
uses get_with_id and put_with_id to associate an id and backtrace with
each particular ref instance.

For refs taken via direct calls to get() and put(), get and put now
require a tag string.  The PG tracks individual ref counts for each tag
as well as the total.

Finally, PGs register/unregister themselves on construction/destruction
with OSDService.

As a result, on shutdown, we can check for live pgs and determine where
the references are held.

This behavior is compiled out by default, but can be included with the
--enable-pgrefdebugging flag.

Signed-off-by: Samuel Just <sam.just@inktank.com>

common/: add tracked_int_ptr.hpp

TrackedIntPtr acts like intrusive_ptr, but is able to
track a ref id.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use the ReplicatedPGRef typedef

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use ReplicatedPGRef for C_PG_MarkUnfoundLost

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use ReplicatedPGRef for C_OSD_OpCommit

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: use ReplicatedPGRef for C_OSD_OpApplied

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: use PGRef in handle_pg_remove

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: use PGRef in handle_pg_stats_ack

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: use PGRef in QueuePeeringEvt

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: use PGRef in consume_map

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: use PGRef for FlushState

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: use PGRef for C_PG_FinishRecovery

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: use PGRef in C_PG_ActivateCommitted

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: do not put() in scrub() if pg is deleting

scrub() no longer handles the put, this call
must have been missed.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG,ReplicatedPG: move intrusive_ptr declarations to top

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: add ReplicatedPGRef

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::_do_copy_range: read(2) might return EINTR

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::_do_clone_range: _do_copy_range encodes error in return, not errno

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #224 from ceph/wip-mon-crush

Wip mon crush

Reviewed-by: Dan Mick <dan.mick@inktank.com>

config: clarify 'mon osd down out subtree limit'

Clarify the description; this is the subtree type that we won't mark out
if it is all down, but anything less than it will be.

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Trimmed toc depth for nicer visual appearance.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added new PG troubleshooting use case.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated title.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added PG troubleshooting to toctree.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Bifurcating OSD and PG Troubleshooting. Updated hyperlink.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Bifurcating OSD and PG Troubleshooting. Added PG troubleshooting doc.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Bifurcating OSD and PG Troubleshooting. Removed PG section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

mon: ensure 'osd crush rule ...' commands are idempotent

Ensure that we return 0 for these cases.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: make 'osd crush link ...' idempotent

We fixed move in f5ba0fbbe73e11418634bc95e1fc36d17edccf37 but missed this
one.

Signed-off-by: Sage Weil <sage@inktank.com>

rgw_bucket: Fix dump_index_check.

Signed-off-by caleb miles <caleb.miles@inktank.com>

Merge branch 'wip-max_size-3637' into next

Reviewed-by: Sage Weil <sage@inktank.com>

mds: journal the projected root xattrs in add_root()

In EMetaBlob::add_root(), we should log the projected root xattrs
instead of original ones to reflect xattr changes.

Signed-off-by: Kuan Kai Chiu <big.chiu@bigtera.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: fix setting/removing xattrs on root

MDS crashes while journaling dirty root inode in handle_client_setxattr
and handle_client_removexattr. We should use journal_dirty_inode to
safely log root inode here.

Signed-off-by: Kuan Kai Chiu <big.chiu@bigtera.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

debian/control:  Fix typo in libboost version number

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

build:  Add new package dependencies

Add libboost-system-dev (bug #4725).

Add hdparm to rpm installation requirements.  The hdparm
command is used to determin if write-caching is enabled on
the journal device.

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

doc: Removed legacy man page index. Generates warning otherwise.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Clarified that admin-socket is accessed from same host.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated hyperlinks to new tshooting section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed this doc. Nothing referenced it, and parent directory echoes content.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Revised top-level ops page.

Consolidated authentication into high-level operations. Added a
troubleshooting section. Collapsed toc trees to make the appearance
cleaner.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed link to nowhere. Otherwise generates a warning.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed top-level tshoot page, and created new index.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Excised community from OSD tshoot, made it stand alone.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Moved monitor troubleshooting to troubleshooting section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Moved troubleshooting OSD to troubleshooting section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added extraneous rgw settings to rgw conf.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Moved memory profiling from operations to troubleshooting.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Moved CPU profiling from operations to troubleshooting.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Set toc depth to 1 level, and added troubleshooting so it appears in sidebar.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Moved journal discussion to OSD ref from Ceph config.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Reordered deployment tools in toc.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed logging from config index. Set depth to 1 for clean appearance.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed logging. Added references. Reorganized and edited.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed. Not in toc, and otherwise generates a warning.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated hyperlink.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed fragmented logging info. Consolidated into one doc.

Logging was variously described in the ceph configuration document,
a configuration reference, and a section in operations. Since
logging and debugging are generally used with troubleshooting,
I consolidated the docs and placed them in the troubleshooting
section. Also fixed the example and provided additional detail.

fixes: #3804

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

rbd: Only allow shrinking an image when --allow-shrink flag is passed

Signed-off-by: Wido den Hollander <wido@widodh.nl>
Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

client: disable invalidate callbacks :(

See #4746; it deadlocks right now.

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #219 from ceph/wip-rbd-progress

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

rbd: add --no-progress switch

Disable progress output to stderr.t

Signed-off-by: Sage Weil <sage@inktank.com>

leveldbstore: handle old versions of leveldb

The filter_policy (bloom filter) stuff is fairly new in LevelDB's life,
and it turns out that precise's version is too old for it. Add conditional
compilation for those members in order to build and work properly.

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge remote-tracking branch 'gh/wip-4521-fix' into next

Reviewed-by: Sage Weil <sage@inktank.com>

mds: change XLOCK/XLOCKDONE's next state to LOCK

For simplelock and filelock, XLOCK/XLOCKDONE's next state is SYNC.
But filelock in XLOCK/XLOCKDONE state allow Fb caps, filelock in
SYNC state does not. So filelock can be stuck in XLOCK/XLOCKDONE
state forever if there are Fb caps issued.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Sage Weil <sage@inktank.com>

mds: pass proper mask to CInode::get_caps_issued

There is a total of 22 cap bits and file lock uses 8 cap bits.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mon: Monitor: convert osdmap_full as well

Store conversion wasn't converting the osdmap_full/ versions, only the
incrementals under osdmap/ and the latest full version stashed.  This
would lead to some serious problems during OSDMonitor's update_from_paxos
when the latest stashed didn't correspond to the first available
incremental.

Fixes: #4521
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: PaxosService: add helper function to check if a given version exists

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

osd/PG.cc: initialize PG::flushed in constructor

Initialize PG::flushed in constructor with false as
described in doc/dev/osd_internals/pg.rst .

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit fb840c8ff75b0c66dfeed48e8558542fe3da4c24)

Merge pull request #215 from ceph/wip-leveldb-config

os: bring leveldbstore options up to date

Reviewed-by: Sage Weil <sage@inktank.com>

Fix policy handling for RESTful admin api.

Signed-off-by caleb miles <caleb.miles@inktank.com>

qa: pull qemu-iotests from ceph.com mirror

Signed-off-by: Sage Weil <sage@inktank.com>