os/FileStore: use new helpers from collection_{get,set,rm}attr[s]

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileStore: use _fgetattr() from getattr()

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileStore: remove unused _getattrs() methods

We are now using the _fgetattrs() variant everywhere instead.

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileStore: use _fgetattrs() from collection_getattr()

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileStore: use _fgetattrs() from _rmattrs()

Note that we subtly change failure handling to be more strict: if we
fail to open the file, we error out; before we ignored errors from
_getattr().

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileStore: use _fgetattrs() for _setattrs()

If we're using omap for spillover, we need to load the xattrs so that
they can be removed if necessary.

Note that we subtly change the failure behavior here to be more
strict: if we can't open the file, we fail, whereas before we did not.

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileStore: use _fgetattrs() for CLONE

We have the src fd.

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileStore: use _fgetattrs() for GETATTRS op

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileStore: add _fgetattr(), _fgetattrs() helpers

Introduce new fd-based getattr[s] helpers.

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileStore: use f*xattr() methods to test xattr support

Signed-off-by: Sage Weil <sage@inktank.com>

os/chain_xattr: move chained xattr helpers into separate file/module

* Rename do_* -> chain_*.
* Move #defines to a header.
* Fix *Index users
* Implement both *xattr() and f*xattr() variants for get, set, list,
  remove.

Signed-off-by: Sage Weil <sage@inktank.com>

doc: describe Hadoop configuration options

Start a document describing Ceph integration with Hadoop, and adds a
description of configuration options that the CephFileSystem shim will
accept.

Merge branch 'wip-reorg'

Merge branch 'wip-msgr-ports'

Reviewed-by: Greg Farnum <greg@inktank.com>

msg/Accepter: use derr

Signed-off-by: Sage Weil <sage@inktank.com>

msgr: change port range from 6800-6900 -> 6800-7100

This allows more OSDs per host.

Signed-off-by: Sage Weil <sage@inktank.com>

msgr: make bind port range configurable

Signed-off-by: Sage Weil <sage@inktank.com>

librados: move misc tests into test/librados/ dir

Signed-off-by: Sage Weil <sage@inktank.com>

librados: test/rados-api/* -> test/librados/*

Signed-off-by: Sage Weil <sage@inktank.com>

client: Support for fchown

Implement fchown in libcephfs and client,
which is needed by the samba vfs module.

Signed-off-by: Sam Lang <sam.lang@inktank.com>

cls_lock: reorg test_cls_lock

Signed-off-by: Sage Weil <sage@inktank.com>

cls_rgw: reorg test_cls_rgw

Signed-off-by: Sage Weil <sage@inktank.com>

fix cls_rbd reorg

librbd: reorg rbd tests

Signed-off-by: Sage Weil <sage@inktank.com>

cls_rbd: move into new location

Be consistent!  Make an example of ourselves!

Signed-off-by: Sage Weil <sage@inktank.com>

msg/Pipe: fix leak of AuthSessionHandler

Init ptr in ctor, delete in dtor.

Reported-by: Yan, Zheng <yan.zheng@intel.com>
Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'wip-mds'

Merge branch 'master' of https://github.com/ceph/ceph

doc: Added qemu caching section.

fixes: #3332

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added Commandline Usage and a few fixes.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

test script for s3 tests
Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

doc: Changed term "kernel object" to "kernel module". Added hyperlink to cephx.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Improved the landing page for RBD.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Clarified text and added reference back to main RBD pages. Added sidebar.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Changed RBD to librbd cache for accuracy.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: changed --user to --id, and fixed a typo.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

run_xfstests.sh: add optional iteration count

This adds a "-c <count>" option to the run_xfstests.sh script so
the full set of tests can be repeated more than once without having
to go through the setup process each time.

Signed-off-by: Alex Elder <elder@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

doc: minor edits to the index.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added layering to the snapshot section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: update notes on new rpm-based platforms

Signed-off-by: Sage Weil <sage@inktank.com>

doc: fix os-recommendations table

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Index entry for OS Recommendations

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: update os-recommendations

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'next'

ceph-disk-activate: avoid duplicating mounts if already activated

If the given device is already mounted at the target location, do not
mount --move it again and create a bunch of dup entries in the /etc/mtab
and kernel mount table.

Signed-off-by: Sage Weil <sage@inktank.com>

PG: requeue snap_trimmer after scrub finishes

Previously the snap_trimmer would continuously requeue itself until the
end of scrub. This degrades performance and fills up logs for No Good
Reason.

Signed-off-by: Mike Ryan <mike.ryan@inktank.com>

PG: requeue snap_trimmer after scrub finishes

Previously the snap_trimmer would continuously requeue itself until the
end of scrub. This degrades performance and fills up logs for No Good
Reason.

Signed-off-by: Mike Ryan <mike.ryan@inktank.com>

doc: tiny syntax fix.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added internal anchor references.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: using remote copy

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge remote-tracking branch 'upstream/wip_dep_fix'

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

README: add libboost-program-options-dev

Signed-off-by: Samuel Just <sam.just@inktank.com>

configure.ac: add program_options header check

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge branch 'wip_journal_perf'

ReplicatedPG: actually delay op for backfill_pos

3f952afe5da644b30015fead8e3d42a129b59989 neglected to
actually delay the op in ReplicatedPG::do_op.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Finisher: add perf counter for queue len

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileJournal: rename queue_lock to finisher_lock

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileJournal: write_cond is not used

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileJournal: break writeq locking from queue_lock

This prevents the relatively long process of queueing
finishers from preventing op submission.

In submit_entry, we no longer check for full before placing
the write in the writeq, committed_thru should work anyway,
and we don't want to grab the required lock.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Throttle: reduce lock hold periods

Previously, we tended to dump a lot of log output under
the Throttle lock.  The log level for most log statements
has been reduced to 10.

Additionally, count and max are now atomic_t and can be
read without the Throttle lock.

Finally, most of the perf counter manipulations have been
moved outside of the lock.

Signed-off-by: Samuel Just <sam.just@inktank.com>

os: instrument submit lock, apply lock, queue_lock, write_lock

Adds Mutex perfcounter tracking to mutexes of interest.

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: add op_throttle_lock

Avoid using op_tp lock for the op throttle.

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: don't lock op_tp in queue_op

Neither caller of queue_op can race.
1) in queue_transactions, already under submit lock
2) in _journaled_ahead, journal finisher is single threaded

Signed-off-by: Samuel Just <sam.just@inktank.com>

perf_counters: add dec()

Signed-off-by: Samuel Just <sam.just@inktank.com>

JournalingFileStore: move apply/commit sequencing to apply_manager

syncing the filestore requires a stable commit point (i.e., all ops
up to applied_seq must have been applied).  Previously, we used
journal_lock to atomically block new applies while waiting for
the remaining ones to finish.  This creates unnecessary contention.
We now use apply_manager to manage that state atomically with its
own lock.

Signed-off-by: Samuel Just <sam.just@inktank.com>

JournalingFileStore: create submit_manager to order op submission

Previously, we ensured op ordering by queueing for journal and
the op queue under the journal lock.  All that is required is
that obtaining an op sequence, queueing for journal, and
(for parallel) queueing for application to the fs are done
atomically.  To that end, submit_manager now handles op submission.

Signed-off-by: Samuel Just <sam.just@inktank.com>

JournalingObjectStore: remove force_commit, no longer needed

Signed-off-by: Samuel Just <sam.just@inktank.com>

JournalingObjectStore: whitespace fix

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: remove trigger_commit

This is no longer used.

Signed-off-by: Samuel Just <sam.just@inktank.com>

JournalingFileStore: pass -1 as the alignment if unimportant

Previously, data_align began at 0 and remained that way if no
transaction contained a large data segment.  This 0 was propagated
to prepare_single_write, which padded out most of a page to ensure
that the bl started with 0 alignment.  Passing -1 will ensure that
we don't prepad these small segments.

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: next_finish is not used

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/bench: add tp bench

Signed-off-by: Samuel Just <sam.just@inktank.com>

test/bench: small io benchmarker

Precreates objects and does writes to random offsets within
random objects.

Includes rados, filestore, and vanilla fs variants

Signed-off-by: Samuel Just <sam.just@inktank.com>

Mutex: Instrument Mutex with perfcouter for Lock() wait

Signed-off-by: Samuel Just <sam.just@inktank.com>

msg/SimpleMessenger: start accepter in ready()

Start the accepter thread when the first dispatcher is ready.  This ensures
that there will be someone around to verify authorizers for incoming
connections, and means we have a bit less failure noise on the monitors
as a result.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: separate pre- and post-fork init

Do most init pre-fork, then do the last little bit (start up messenger,
bootstrap) post-fork.

Signed-off-by: Sage Weil <sage@inktank.com>

msg/Pipe: fix seq # fix

02f6262f47f72178a78d410f4facab7bbc97b098 got this all wrong (though it
worked by accident).

Signed-off-by: Sage Weil <sage@inktank.com>

osd: verify authorizers for heartbeat dispatcher

This was broken with the fixed messenger behavior with missing
verify_authorizer methods in 100fcca3cb54c97c4332328aad67d4b796f33ec2.

Signed-off-by: Sage Weil <sage@inktank.com>

doc: fix typo in cinder upstart config name

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

doc: Added syntax fixes to Peter's session authentication doc.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

ceph-disk-prepare: poke kernel into refreshing partition tables

Prod the kernel to refresh the partition table after we create one.  The
partprobe program is packaged with parted, which we already use, so this
introduces no new dependency.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-disk-prepare: fix journal partition creation

The end value needs to have + to indicate it is relative to wherever the
start is.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-disk-prepare: assume parted failure means no partition table

If the disk has no valid label we get an error like

  Error: /dev/sdi: unrecognised disk label

Assume any error we get is that and go with an id label of 1.

Signed-off-by: Sage Weil <sage@inktank.com>

msg/Pipe: whitespace cleanup

Signed-off-by: Sage Weil <sage@inktank.com>

msg/Pipe: only randomize start seq #'s if MSG_AUTH feature is present

The kernel client expects seq #'s to start at 1 or else it is unhappy.
So, only randomize these values if the MSG_AUTH feature is present--that is
the only time it matters anyway.

Signed-off-by: Sage Weil <sage@inktank.com>

doc: update fs recommendations

More forceful about recommending XFS.  More warning about using btrfs in
production deployments.

Signed-off-by: Sage Weil <sage@inktank.com>

cephx: don't check signature if MSG_AUTH feature isn't present

Signed-off-by: Sage Weil <sage@inktank.com>

auth: include features in cephx SessionHandler

Signed-off-by: Sage Weil <sage@inktank.com>

Fixed problem with checking authorizer in accept().

Signed-off-by: Peter Reiher <reiher@inktank.com>

librbd: Fix 32-bit compilation errors

Switch size_t in clip_io to uint64_t; it's just easier, and the
alternative would be to limit 32-bit builds to sizes <= 4GB

Signed-off-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge branch 'master' of github.com:ceph/ceph

Temporary patch to a problem in Pipe related to monitor initialization.

Signed-off-by: Peter Reiher <reiher@inktank.com>

Merge branch 'wip-oc-neg'

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

osd: make pool_snap_info_t encoding backward compatible

Way back in fc869dee1e8a1c90c93cb7e678563772fb1c51fb (v0.42) when we redid
the osd type encoding we forgot to make this conditionally encode the old
format for old clients.  In particular, this means that kernel clients
will fail to decode the osdmap if there is a rados pool with a pool-level
snapshot defined.

Fixes: #3290
Signed-off-by: Sage Weil <sage@inktank.com>

dep-report.sh:  ceph package dependency report.

This script searches the ceph build area for dependent header files and
and libraries to attempt to identify ceph package dependecies.

client: Fix ref counting double free with hardlink

Peforming a hard link through the libcephfs interface causes
a double free on shutdown, due to the Client::link call decrementing
the parent (of the target) directory's inode.  This fix removes the
put_inode(dir) call, to match the behavior of Client::ll_link.

Signed-off-by: Sam Lang <sam.lang@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

test: Functional test for hardlink/unmount pattern

This test currently breaks on libcephfs as reported
in #3367.

Signed-off-by: Sam Lang <sam.lang@inktank.com>

osdc/ObjectCacher: remove dead locking code

This is unused, and mostly broken in that there is no cleanup when there
is a failure.  Also, the support in the OSD has been largely removed.

Signed-off-by: Sage Weil <sage@inktank.com>

librbd: clip requests past end-of-image.

Rename check_io to clip_io, which can modify the passed-in length
to clamp it to the device size.  This is expected behavior for
block-device emulation.

Call clip_io in rbd_write(); need to return clipped length there,
even though aio_write() is calling clip_io() as well (for the
direct path).

Signed-off-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

librbd: size max objects based on actual image object order size

This has to happen after we open the image.

Signed-off-by: Sage Weil <sage@inktank.com>