tools/MDSUtility: fix crash on bad config

Was calling messenger destructor before
calling shutdown in case where config
was bad, e.g. in vstart without "-c" flag.

Signed-off-by: John Spray <john.spray@inktank.com>

tools/JournalTool: update usage()

Signed-off-by: John Spray <john.spray@inktank.com>

qa: Add smoke test for cephfs-journal-tool

Signed-off-by: John Spray <john.spray@inktank.com>

tools/JournalTool: Fix 'header set' corner case

If the write_pos was behind the other pointers, we
would flag the header as invalid.  However, that
would prevent doing a "header set" to correct it.

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Create JournalPointer in MDLog::create

Otherwise it is created at next startup when
we see that it is missing: this is cleaner.

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Make EMetaBlob::dirlump::_decode_bits const

This requires making the late-decoded attributes mutable.
The motivation is to allow get_paths and all the code downstream
of that to be const too.

Signed-off-by: John Spray <john.spray@inktank.com>

osdc: new style encoding for Journal::Header

Switch to the ENCODE_START, DECODE_START macros
for Journal::Header.  As well as being nice generally,
this has the important side effect of making
journal headers written since JOURNAL_FORMAT_RESILIENT
unreadable to older MDSs, as they will fail their check
on Header.magic.

Signed-off-by: John Spray <john.spray@inktank.com>

osdc: Simplify JournalStream::read

It was doing an unnecessary series of splices() after
reading header, replace with a single pass through
the data and one splice at the end.

Signed-off-by: John Spray <john.spray@inktank.com>

osdc: make JournalStream::readable const

Signed-off-by: John Spray <john.spray@inktank.com>

osdc/JournalStream: DRY envelope size calc

Replace repetitive sizeof() calculation for the
envelope overhead per log event with #defines.

Signed-off-by: John Spray <john.spray@inktank.com>

doc: Update ceph-mds manpage

Add missing --hot-standby and --journal-check
arguments.

Fixes: #8258

osdc/Journaler: use pointers for out params

Fixes a style violation in the new JournalStream class.

Signed-off-by: John Spray <john.spray@inktank.com>

tools/cephfs: Update for JournalPointer

Signed-off-by: John Spray <john.spray@inktank.com>

Fix JournalStream::read length assertion

This was checking for new-format length even
when reading from an old-format journal.

Signed-off-by: John Spray <john.spray@inktank.com>

osdc/Journaler: include stream_format in dump()

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Introduce mds_journal_format config setting

This is useful for testing, so that we can create an
old-style journal and then test the version migration
by changing the config setting.

Signed-off-by: John Spray <john.spray@inktank.com>

tools: update Dumper to use JournalPointer

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Generalize JournalPointer functionality

...so that we can use it places like Resetter too.  And use it
there.

Signed-off-by: John Spray <john.spray@inktank.com>

dencoder: add JournalPointer to types.h

Signed-off-by: John Spray <john.spray@inktank.com>

mds: add atomic log rewrite on format change

Two main pieces to this:
 * A new JournalPointer object that stores two journal
   inodes so that we can do a double-buffered update,
   followed by an atomic swap.
 * An extended recovery process in MDLog that dereferences
   the JournalPointer and conditionally rewrites the
   journal to accomodate format updates.

The JournalPointer indirection should also be useful for
making cephfs-journal-tool do updates more safely.

Signed-off-by: John Spray <john.spray@inktank.com>

osdc: Add Journaler.erase

This is used in subsequent commits to delete journals
that are no longer needed, such as after rewriting
a journal in a different format.

Signed-off-by: John Spray <john.spray@inktank.com>

osdc/Journaler: privatize members

Many members were unnecessarily public.

Signed-off-by: John Spray <john.spray@inktank.com>

osdc: fix redundant branch

Signed-off-by: John Spray <john.spray@inktank.com>

osdc: Clean up journalstream readable check

Fix redundant (and subtly incorrect) calculation of
the number of bytes needed.  It worked because waiting
for a few more bytes before reading the entry size
of an old-format entry was harmless.

Signed-off-by: John Spray <john.spray@inktank.com>

mds/CDir: remove redundant inode enc/dec

Use the new CInode::encode_bare/decode_bare fns
in CDir, so that we only have one implementation
of the inode encode/decode code.

Signed-off-by: John Spray <john.spray@inktank.com>

docs: Add cephfs-journal-tool

Signed-off-by: John Spray <john.spray@inktank.com>

gitignore: Add cephfs-journal-tool

Signed-off-by: John Spray <john.spray@inktank.com>

debian: add cephfs-journal-tool to ceph-mds

Signed-off-by: John Spray <john.spray@inktank.com>

rpm: add cephfs-journal-tool to RPM build

Signed-off-by: John Spray <john.spray@inktank.com>

tools: Create cephfs-journal-tool

This is for debugging/repairing CephFS journals.

Signed-off-by: John Spray <john.spray@inktank.com>

mds: add getter for ESession.client_inst

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Add getter for EMetablob.client_name

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Add EMetaBlob::get_dentries

For tools that would like to know which dentries are
touched by a metablob, without understanding its
internal format.

Signed-off-by: John Spray <john.spray@inktank.com>

mds: switch __u32 with LogEvent::EventType typedef

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Add LogEvent::str_to_type

For cephfs-journal-tool to resolve user input to a EVENT_* constant

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Fix Dumper::undump (missing lock)

Two problems were causing undump to fail:
* Objecter lock was not being taken around call to
  .write() and .write_full() calls, causing assertion.
* Once that is fixed, it is necessary to use a separate,
  local lock to protect the completion condition for
  write operations

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Add EMetaBlob::get_inodes

Used for filtering in journal tool

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Publicize some EMetaBlob members

This makes them accessible for JournalTool.

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Refactor CINode encoding into CInodeStore

CInode itself combined the on-disk format and
encode/decode logic with lots of other complex
behaviours.  This separates the simple parts
out so that they can be used by other tools that
are interested in looking at inodes outside of
a running MDS.

There is a small overhead because CInodeStore
can't decode a SnapRealm inline, so it keeps
a temporary copy of the encoded bufferlist.

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Add get_metablob to LogEvent

Previously the only way to get at the payload
of things like EUpdate and EOpen was to replay() them
(required a full running MDS) or to use downcasting
(yuck).

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Add ENoOp for padding journals

This allows us to implement journal splicing
without moving blocks around, and without modifying
the outer journal syntax.

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Add get_paths method to EMetaBlob

Used by cephfs-journal-tool to filter events
by path fragment.

Signed-off-by: John Spray <john.spray@inktank.com>

osdc: Revise Journaler format

* Separate journal encoding/envelope format
  code (JournalStream) from I/O code (Journaler)
* Add new sentinel and start_ptr fields to
  prefix and suffix of log events.
* Add journal encoding version to journal header

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Fix typo 'Sesion'

Signed-off-by: John Spray <john.spray@inktank.com>

objecter: Don't warn on multiple admin sockets

Suppress messages about failure to register admin sockets
if they are EEXIST, because this is a case that can occur
naturally if multiple objecter/librados clients are instantiated
within the same process.

Signed-off-by: John Spray <john.spray@inktank.com>

common: Add write_stream(ostream) to bufferlist

Enable C++ style file I/O when writing binary output
from bufferlists.

Signed-off-by: John Spray <john.spray@inktank.com>

mds: Add LogEvent::get_type_str()

Signed-off-by: John Spray <john.spray@inktank.com>

Merge pull request #1821 from ceph/mds-optracker-fixes

mds: fix Server::submit_mdlog_entry()

mds: fix Server::submit_mdlog_entry()

mdr can be NULL when rolling back slave update

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge pull request #1809 from ceph/wip-4354-mds-optracker

MDS OpTracker

Reviewed-by: Sage Weil <sage@inktank.com>

MDS: add admin socket cleanup on shutdown

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #1817 from ceph/wip-osdmon-pa-wait

OSDMonitor: set next commit in mon primary-affinity reply

Reviewed-by: Sage Weil <sage@inktank.com>

OSDMonitor: set next commit in mon primary-affinity reply

Commit 8c5c55c8b47e ("mon: set next commit in mon command replies")
fixed MMonCommand replies to include the right version, but the
primary-affinity handler was authored before that.  Fix it.

Backport: firefly
Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

Merge pull request #1816 from onlyjob/docs

sample.ceph.conf: minor update

Reviewed-by: Greg Farnum <greg@inktank.com>

sample.ceph.conf: minor update

 * Moved filestore settings above [osd.*] declarations otherwise
   (if uncommented) those settings might be applied only to last
   OSD which is not very obvious.
 * Few options added.

workunits: provide some output in the dirfrag.sh test

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #1803 from onlyjob/java-gcj

Java GCJ fixes

Reviewed-by: Greg Farnum <greg@inktank.com>
Acked-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #1806 from ceph/wip-8011

ReplicatedPG: block scrub on blocked object contexts

Reviewed-by: Sage Weil <sage@inktank.com>

doc: Clarified Debian uses sysvinit.

Fixes: #7182
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added rgw print continue guidance.

Fixes: #7731
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Minor edit.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added clarifying text to CRUSH add command.

Fixes: #8322
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Omitted glance_api_version=2 to fix creating images from volumes.

Fixes: #8347

doc: Changed example to use virtio and put key usage into one line.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1802 from ceph/wip-mds-misc

Wip mds misc

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1810 from ceph/wip-fedora

doc: update instructions for RPM distros

Improve Bash completion for various tools

test: fix some templates to match new output code

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

doc: update instructions for RPM distros

Fix RPM building instructions: this has been broken since
libs3 was included inline in the ceph repo as a submodule.
"rpmbuild -tb" was concatenating the ceph.spec and
libs3.spec files, resulting in something that didn't work.

Also, the instructions suggested downloading a .tar.gz file
whereas the specfile requires a .tar.bz2 file.

Also, add a convenient yum command line for getting the compile
dependencies on Fedora 20.

Signed-off-by: John Spray <john.spray@inktank.com>

Merge remote-tracking branch 'origin/master' into wip-4354-mds-optracker

Conflicts:
src/mds/Locker.cc
src/osd/OpRequest.cc
src/osd/OpRequest.h

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: lower IO priority of storing backtrace

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge pull request #1771 from ceph/wip-5021

Wip 5021

Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1807 from ceph/wip-mds-flock

mds: reduce verbosity of handle_client_file_{readlock,setlock}

mds: reduce verbosity of handle_client_file_{readlock,setlock}

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: add a Server::submit_mdlog_entry() to provide event marking

Signed-off-by: Greg Farnum <greg@inktank.com>

ReplicatedPG: block scrub on blocked object contexts

Fixes: #8011
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1779 from ceph/wip-7553

Wip 7553

Reviewed-by: Samuel Just <sam.just@inktank.com>

Locker: mark_event in acquire_locks() when blocking or succeeding

Signed-off-by: Greg Farnum <greg@inktank.com>

Server: mark events when journaling and replying

Signed-off-by: Greg Farnum <greg@inktank.com>

MDCache: mark ops at various finish points

Signed-off-by: Greg Farnum <greg@inktank.com>

MDS: add an OpTracker and use it

Signed-off-by: Greg Farnum <greg@inktank.com>

Mutation: add an MDRequestParams struct and use that when building MDRequests

We now have a single constructor and one path to build MDRequests with.

Signed-off-by: Greg Farnum <greg@inktank.com>

MDS: add stubs for an AdminSocketHook

Signed-off-by: Greg Farnum <greg@inktank.com>

MDCache: pass the causative message to request_start_slave()

We were passing the causative MDS (as an int), but pushing down the
actual Message will help us as we set up an OpTracker.

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: remove a couple leftover declarations of MDRequest

Signed-off-by: Greg Farnum <greg@inktank.com>

doc/release-notes: v0.80.1

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1799 from ceph/wip-8305

osd: fix op ordering with pool overlay set/removed

Reviewed-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

doc: Improvements to qemu installation.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added note on Default requiretty for CentOS and others.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #1801 from ceph/wip-update-gitignore

Update gitignore entries for master

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1800 from ceph/wip-da-SCA-20140510

fixes from SCA

Reviewed-by: Sage Weil <sage@inktank.com>

prioritise use of `javac` executable (gcj provides it through alternatives).

 On Debian this fixes FTBFS when gcj-jdk and openjdk-7-jdk are installed at
 the same time because build system will use default `javac` executable
 provided by current JDK through `update-alternatives` instead of blindly
 calling GCJ when it is present.

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

pass '-classpath' option (gcj/javah ignores CLASSPATH environment variable).

 This should not affect OpenJDK which understands '-classpath' as well.

 With gcj-jdk we still get FTBFS later:

 ~~~~
    java/native/libcephfs_jni.cc:2878:55: error: invalid conversion from 'const jbyte* {aka const signed char*}' to 'jbyte* {aka signed char*}' [-fpermissive]
                 reinterpret_cast<const jbyte*>(rawAddress));
                                                           ^
    In file included from java/native/libcephfs_jni.cc:27:0:
    /usr/lib/gcc/x86_64-linux-gnu/4.8/include/jni.h:1471:8: error:   initializing argument 4 of 'void _Jv_JNIEnv::SetByteArrayRegion(jbyteArray, jsize, jsize, jbyte*)' [-fpermissive]
       void SetByteArrayRegion (jbyteArray val0, jsize val1, jsize val2, jbyte * val3)
            ^
    make[5] *** [java/native/libcephfs_jni_la-libcephfs_jni.lo] Error 1
 ~~~~

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

look for "jni.h" in gcj-jdk path, needed to find "jni.h" with gcj-jdk_4.9.0

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

mds: deny reconnect for closed session

The client that tries reconnect may have dirty caps and unsafe requests.
Allowing the reconnect attempt may compromise consistency.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: revert EMetaBlob::{fullbit,remotebit,nullbit} encoding optimization

Revert commit 40d56a97 (mds: optimize EMetaBlob::fullbit, remotebit,
nullbit encoding). This optimization creates small segments in the
result bufferlist of encoding EMetaBlob. Perf shows lots of CPU time
are used for allocating list node for bufferlist.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: properly clear new flag for stale client cap

CInode::encode_inodestat() should clear the 'new' flag of client
cap even when session is stale, because the 'new' flag prevents
Locker::issue_caps() from sending cap message to client.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: propagate inode rstat if it has never been propagated

Otherwise the 'last_dirstat_prop' of directory inode keeps in 'never'
state.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: avoid journaling unnecessary dir context

If base inode is reached, try clearing the 'maybe' list, then stop.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: cleanup usage of MDCache::predirty_journal_parent()

The sixth parameter of MDCache::predirty_journal_parent() is 'int'
with default value 0.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>