mds: EMetaBlob and its sub-parts use modern encoding now

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: EImportStart event uses modern encoding

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: EImportFinish event uses modern encoding

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: EFragment event uses modern encoding

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: remove unused EString event

While we're at it, #include LogEvent.h from each of
the log events, some of which didn't include it previously!

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: EExport event uses modern encoding

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: ECommitted now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

SnapServer: use modern encoding for server_state

Signed-off-by: Greg Farnum <greg@inktank.com>

InoTable: use modern encoding for encode_state and decode_state

Signed-off-by: Greg Farnum <greg@inktank.com>

CInode: use modern encoding for encode_export/decode_import

Signed-off-by: Greg Farnum <greg@inktank.com>

CInode: use modern encoding for encode_store

Signed-off-by: Greg Farnum <greg@inktank.com>

AnchorServer: use modern encoding for server state

Signed-off-by: Greg Farnum <greg@inktank.com>

DecayCounter: use modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: old_rstat_t now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: Capability (and sub-structs) now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: uninline Capability encoders

Signed-off-by: Sage Weil <sage@inktank.com>

mds: build dencoder with more stuff

Add libosdc and perfglue/disabled_heap_profiler to the
dencoder, because those are required for the MDS stuff
we're about to start putting in. *sigh*

Signed-off-by: Greg Farnum <greg@inktank.com>

MDSMap: mds_info_t now uses modern encoding

We have to update the older-style MDSMap encodings to generate
the previous versions for clients as well.

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: MDSMap now uses modern encoding

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: move conditional MDSMap encoding into single encode method

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mdsmap: uninline encode/decode

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: modernize SimpleLock on-wire encoding

This is a wire protocol change.

Signed-off-by: Sage Weil <sage@inktank.com>

mds: mds_table_pending_t now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: rename MDSTableServer::_pending to mds_table_pending_t

And move it from MDSTableServer into mdstypes.cc, so we can use it
in ceph-dencoder more gracefully (coming up next!).

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: MDSCacheObjectInfo now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>

mds: string_snap_t now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: session_info_t now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: move durable Session bits into session_info_t

This keeps the on-disk structure explicitly separate from the in-memory
functional stuff.

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: fnode_t now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: old_inode_t now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

remove common/types.cc

It no longer has a purpose; the functions it used to host are now
implemented in mds/mdstypes.cc and more properly belong there
anyway.

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: inode_t now uses modern encoding

And we move implementations and the dumper into mdstypes.cc (from
mdstypes.h and common/types.cc).

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: fold byte_range_t into client_writeable_range_t

As part of this, fold byte_range_t into it as a sub-struct
and eliminate its free-standing functions (it's too small
to live alone).

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: client_writeable_range_t now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: nest_info_t now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: rename struct default_file_layout to file_layout_policy_t

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: frag_info_t now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>

mds: default_file_layout now uses modern encoding

And move implementations into mdstypes.cc from CInode and common/types.
mdstypes.cc sadly lives in libcommon; as several of the types we're going
to put there are needed for Messages and similar things.

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: inode_back{trace,pointer}_t now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: move SnapRealm into its own h/cc files

Signed-off-by: Sage Weil <sage@inktank.com>

client: rename client/SnapRealm files to avoid automake build conflict

We are about to move the MDS' SnapRealm into its own files, which conflicts.
The MDS is more important, so it wins the "SnapRealm" filename and we
move the client stuff to ClientSnapRealm.[h|cc].

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: SnapInfo, snaplink_t, sr_t now use modern encoding

This commit doesn't enable the dencoder integration due
to some build and compile issues, but we'll turn it
on later.

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

 mds: Anchor now uses modern encoding

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

mds: add CEPH_FEATURE_MDSENC feature bit

This will cover the MDS cluster on upgrade, and determine which encoding
of the MDSMap they use for clients.

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>

osd: remove DecayCounter header

Neither the OSD nor the PG makes any use of this.

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #39 from dachary/master

Relax Throttle::_reset_max conditions and associated unit tests

Relax Throttle::_reset_max conditions and associated unit tests

Removes a condition in Throttle::_reset_max by which the waiting queue is only
Signal()ed if the new maximum is lower than the current maximum.
There is no evidence of a use case where such a restriction would be
useful. In addition waking up a thread when the maximum increases
gives it a chance to immediately continue the suspended process
instead of waiting for the next put().

Create a new test file covering 100% of src/Throttle.{cc,h} lines of code.
The following methods are tested:

* Throttle::Throttle with and without a maximum
* Throttle::~Throttle when each pending Cond is deleted
* Throttle::take
* Throttle::get when updating the maximum ( lower or higher ),
  when going to sleep waiting for the count to lower under
  the maximum, when going to sleep because another thread is
  already asleep waiting
* Throttle::get_or_fail when there is no maximum,
  when requesting a count that is larger than the maximum, either
  when the current value is under the maximum or above the maximum.
* Throttle::wait when used to reset the maximum and wake up
  another thread asleep waiting

All asserts checking the arguments sanity are exercised ( negative argument
for Throttle::take etc. ).
Adds the LGPLv2+ licensing terms to COPYING along with the others.
Adds a Contributors section to the AUTHORS file.

Notes:
Testing asserts outputs verbose error messages that should be silenced
but it does not seem possible.

Signed-off-by: Loic Dachary <loic@dachary.org>

os: default to 'journal aio = true'

Hooray, testing indicates this is a win!

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #36 from cmello/master

libexpat dependency

Merge pull request #38 from alram/master

Fixes in ./docs/radosgw/config.rst

Edit endpoint-create in ./doc/radosgw/config.rst

internalurl and adminurl are mandatory. Typo in publicurl.

Edit rgw keystone url in ./doc/radosgw/config.rst

Won't work with the public port, it needs to be the admin port.

Note on host in ./doc/radosgw/config.rst

Some people have configured host with a FQDN or an IP
which prevents /etc/init.d/radosgw start to launch the daemon.

doc: Updated to note bobtail supports RGW + Keystone.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

cli test: add pg deep-scrub option to test

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

Add "pg deep-scrub..." missing from ceph usage output

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

rgw: fix setting of NULL to string

Fixes: #3777
s->env->get() returns char * and not string and can return NULL.
Also, remove some old unused code.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

xattr_bench.cc: remove twice included <time.h>

Cleanup includes, remove twice included <time.h>.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-filestore-dump.cc: remove twice included <iostream>

Cleanup includes, remove twice included <iostream>.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

testmsgr.cc: remove twice included <sys/stat.h>

Cleanup includes, remove twice included <sys/stat.h>.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

perf_counters.cc: remove twice included header files

Cleanup includes, remove twice included "global/global_init.h" and
"common/ceph_context.h".

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

testxattr.cc: remove twice included <iostream>

Cleanup includes, remove twice included <iostream>.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

workload_generator.cc: remove twice included "common/debug.h"

Cleanup includes, remove twice included "common/debug.h"

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_idempotent.cc: remove twice included "os/FileStore.h"

Cleanup includes, remove twice included "os/FileStore.h".

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

tp_bench.cc: remove twice included <iostream>

Cleanup includes, remove twice included <iostream>.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

small_io_bench*.cc: remove twice included <iostream>

Cleanup includes, remove twice included <iostream>.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

MDS.cc: remove twice included common/errno.h

Cleanup includes, remove twice included common/errno.h.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon: enforce reweight be between 0..1

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Luis <joao.luis@inktank.com>

qa: smalliobenchrbd workunit

Run a bunch of parallel smalliobenchrbd processes.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/wip-rbd-bench'

Conflicts:
ceph.spec.in
debian/ceph-test.install
src/.gitignore

Merge branch 'wip-rpm-update3'

Patches to ceph.spec.in and addition of rbd-fuse package.

Merge branch 'master' of https://github.com/ceph/ceph

doc: Minor edits.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

rgw: key indexes are only link to user info

Instead of keeping multiple copies of the user info,
we just treat the key index as a pointer to the actual
user info (indexed by uid). This helps with two issues:
first, it scales better as we don't need to update the
entire set of keys whenever we make any change. Second,
it helps with the uid index atomicity.
One point to keep in mind is that both the links and the
info can be cached, so effect on performance is minimal.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: caleb miles <caleb.miles@inktank.com>

Build:  Add -n to files and description for rbd-fuse in ceph.sepc.in

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

Makefile:  Install new rdb-fuse.8 man page

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

build:  Add new rbd-fuse package

rdb-fuse is a new facility to map ceph rdb images to files.

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

Revert "Don't install rbd-fuse binary"

This reverts commit 35e5d74e5c5786bc91df5dc10b5c08c77305df4e.

-> fix build instead

rbd-fuse: quick and dirty manpage

Signed-off-by: Dan Mick <dan.mick@inktank.com>

rbd-fuse: quick and dirty manpage

Signed-off-by: Dan Mick <dan.mick@inktank.com>

ceph-filestore-dump.cc: don't use po::value<string>()->required()

Don't use po::value<string>()->required() since this breaks build on
RHEL/CentOs6. Check if the options are set as in the code of other
ceph parts.

Move some checks up in the code to validate options as soon
as possible. Remove printing 'help' twice, and check it first.

Fix type description.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Signed-off-by: David Zafman <david.zafman@inktank.com>

doc: Added more detail to SSD section. Links to performance blogs.

fixes: #3960

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #37 from alram/master

Add important note in doc/radosgw/config.rst

Add important note in doc/radosgw/config.rst

For CentOS and similar, FastCgiWrapper is turned on by default.
This causes Apache to spawn radosgw processes.

ceph-filestore-dump.cc: don't use po::value<string>()->required()

Don't use po::value<string>()->required() since this breaks build on
RHEL/CentOs6. Check if the options are set as in the code of other
ceph parts.

Move some checks up in the code to validate options as soon
as possible. Remove printing 'help' twice, and check it first.

Fix type description.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph.spec.in: fix file section for ceph-resource-agents

Create needed dirs (/usr/lib/ocf/resource.d/ceph) for the ceph-resource-agents
subpackage.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph.spec.in: extend fix for libedit-devel on special SUSE versions

Extend fix for libedit-devel on special SUSE versions, use ncurses
also on src/ocf/Makefile and src/java/Makefile

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph.spec.in: don't move libcephfs_jni files around

Don't move libcephfs_jni files around from %{_libdir} to /usr/lib/jni/
in the buildroot. They should be placed in %{_libdir} as all libs.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph.spec.in: move libcephfs_jni.so to ceph-devel

Move libcephfs_jni.so to the ceph-devel package since so-files they
shouldn't be part of the library package.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Validate format strings for CLS_ERR/CLS_LOG

cls_log needed __attribute__((format(printf..)) to allow the compiler
to crosscheck format strings and arguments.  After adding that, there
needed to be a bunch of fixups for %ll, and a few changes for missing
arguments, etc. uncovered by the checking.

Fixes: #3970
Signed-off-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

qa: update the rbd/concurrent.sh workunit

A few changes, now that a few rbd problems have been fixed.
First, the more substantive changes:
    - Generate a source file, and compare what's read back from rbd
      devices with the content of that file.
    - Write to the rbd device such that the written data spans
      an (assumed 4 MB) rbd object boundary, as well as starting
      and ending on non-page-aligned offsets.
    - Perform multiple reads on rbd devices: entirely within a range
      before any written data; beginning before but ending within
      written data; the exact written data (and validating what's
      read); beginning within written data but ending after it;
      reading after written data but within a written rbd object;
      and reading from an unwritten rbd object.
    - Have the sleep between iterations provide a non-integer value
      to avoid zero (or quantized) delays.

Also, some a little less substantive (but possibly informative):
    - Don't run with "set -x".  It produces a ton of noise that is
      not useful for this test.  This is an exerciser, looking
      really for system crashes during concurrent activity, and
      knowing which commands were (concurrently) active isn't going
      to help much in diagnosis.
    - Create two more directories, used to track the degree of
      concurrency (more or less) and the highest rbd id consumed.
      Files whose names are numbers are touched in each, and the
      highest at the end is the highest during the run.  This gets
      around issues passing environment info from sub-shells to the
      top-level shell.  As a bonus, it offers a better chance of
      avoiding problems due to concurrent update.
    - NAMESDIR is renamed NAMES_DIR, and it (and the others) is
      set up in the setup() function.
    - Increase the concurrency and iteration counts.
    - Move the default definitions before the ceph secrets stuff

Signed-off-by: Alex Elder <elder@inktank.com>

Add ceph-filestore-dump to the packaging

Feature: #3890

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

doc: v0.56.2 release notes

Signed-off-by: Sage Weil <sage@inktank.com>

osd: create tool to extract pg info and pg log from filestore

New application ceph-filestore-dump created that mounts filstore
and can dump info or log in JSON when an OSD is not running.

Feature: #3890

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

Move read_log() function to prep for next commit

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

PGMap: fix -Wsign-compare warning

Fix -Wsign-compare compiler warning:

mon/PGMap.cc: In member function 'void PGMap::apply_incremental
 (CephContext*, const PGMap::Incremental&)':
mon/PGMap.cc:247:30: warning: comparison between signed and
 unsigned integer expressions [-Wsign-compare]

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_libcephfs: fix xattr test

Ignore the ceph.*.layout xattrs.

Signed-off-by: Sage Weil <sage@inktank.com>

qa: add test for rbd map and snapshots

This tests for the behavior reported in #3964.  It passes on the current
code, but fails on 3.2 in squeeze (and 32-bit?).

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/next'

cls_rbd, cls_rgw: use PRI*64 when printing/logging 64-bit values

caused segfaults in 32-bit build

Fixes: #3961
Signed-off-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>