Merge pull request #943 from dachary/wip-formatter-newlines

common: fix perf_counters unittests for trailing newline in m_pretty

common: fix perf_counters unittests for trailing newline in m_pretty

Signed-off-by: Loic Dachary <loic@dachary.org>

common/Formatter: add newline to flushed output if m_pretty

This applies to json-pretty and xml-pretty modes.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'wip-rgw-quota'

Conflicts:
src/test/cli/radosgw-admin/help.t

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #711 from ceph/wip-objecter-errors

These two patches fix several completion Contexts which are inappropriately setting the Op rval to zero.

Reviewed-by: Greg Farnum <greg@inktank.com>

SignalHandler: fix infinite loop on BSD systems

SignalHandler::entry shouldn't poll for POLLOUT, because it never actually
writes to the pipes in question. Polling for POLLOUT causes poll(2) to
immediately return, so the function spins the CPU and never blocks. Remove
the POLLOUT flag, unnecessarily introduced in commit
8e4a78f169eda716c7d6811cb6db5c757dc67207 when switching from select() to poll().

This fixes the problem on FreeBSD and doesn't break anything (AFAICT) on
Linux. Tested on FreeBSD 9.1 amd64 and Ubuntu Server 13.04 amd64.

Fixes: #6492
Signed-off-by: Alan Somers <asomers@gmail.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

doc: Removed underscore for consistency.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #713 from dachary/master

doc: disable cephx requires auth_supported = none

radosgw-admin: add --quota-scope param to usage

and also look at it when setting quota on a bucket.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #708 from ceph/wip-truncate2

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #715 from ceph/wip-5025

Reviewed-by: Greg Farnum <greg@inktank.com>

mds: flock: fix F_GETLK

fixes #5025
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

doc: disable cephx requires auth_supported = none

    auth cluster required = none
    auth service required = none
    auth client required = none

is not enough to disable cephx if there is a

    auth_supported = cephx

line in the [global] section. It will produce the following error

    # ceph --id myself auth list
    2013-10-13 11:05:00.368638 7f812aea3700  0 librados: client.myself authentication error (1) Operation not permitted
    Error connecting to cluster: PermissionError

Changing the line to

    auth_supported = none

fixes the problem.

Signed-off-by: Loic Dachary <loic@dachary.org>

rgw: switch out param to a pointer instead of reference

following code review

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test: update cli test for radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: clarify --max-size param

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

formatter: dump_bool dumps unquoted strings

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw_quoa.{h,cc}: add copyright notice

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

doc: Created new index to incorporate side-by-side deb/rpm installs.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Created installation doc for Yum priorities. Ceph-specific packages need it.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Created new QEMU install for RPMs with provision for Ceph-specific packages.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Created new QEMU install for debian.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added new libvirt install for RPM.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Created new libvirt install for debian/ubuntu.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Removed Gatway from Ceph Object Storage. Added Yum repo information.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Moved installation portion to the installation section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Moved installation portion to the installation section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Fixed hyperlinks. Cleanup of old references to Chef.

fixes: #6493

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

osdc/Objecter: clean up completion handlers that set *prval=0

If the OSD operations returns 0, these are unnecessary.  If it does not,
then these are misleading.  In both cases here, the OSD sets the result to
zero, so this has no client-visible change.r

Signed-off-by: Sage Weil <sage@inktank.com>

osdc/Objecter: only make handlers set *prval if EIO

The normal Objecter handlers set *prval for each operation; we only need to
(re)set it if we have a special reason (like a decoding error).

Signed-off-by: Sage Weil <sage@inktank.com>

test/filestore/run_seed_to_range.sh: fix -d syntax

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #702 from ceph/wip-4405

mds: fix infinite loop of MDCache::populate_mydir().

Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

mds: avoid leaking objects when purging file.

The filer implementation does not delete stripe objects that are truncated
to zero. When purging a deleted file, we need to purge stripe objects up to
the max size the file has ever been.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

radosgw-admin: limit user bucket-level quota

We only allow it if --quota-scope=bucket is specified. This is done in
order to avoid confusion with the future user level quota command.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: can set user's bucket quota

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

doc: Minor correction.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

rgw: bucket quota threshold

Add bucket quota threshold so that when we're passed that value we
reread the bucket stats before every write and not rely on cached value.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: configurable bucket quota size

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: protect against concurrent async quota updates

Leverage the cache lru_map locking for making sure that we don't end
up with more than a single concurrent async update on the same bucket
within the same update window.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: async quota update

Asynchronously update bucket stats when a period passed, but bucket
stats are within the ttl window.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

doc: Merge cleanup.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: minor clean up.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Update from wip-doc-quickstart branch.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

mds: fix infinite loop of MDCache::populate_mydir().

make MDCache::populate_mydir() only fetch bare-bone stray dirs.
After all stray dirs are populated, call MDCache::scan_stray_dir(),
it fetches incomplete stray dirs.

Fixes: #4405
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

rgw: rearrange includes

Need to have ceph_json.h first due to some boost weirdness.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Go back to $PWD in fsstress.sh if compiling from source.

Although fsstress was being called with a static path the directory
it was writing to was in the current directory so doing a cd to the
source directory that is made in /tmp and then removing it later
caused it to be unable to write the files in a non-existent dir.

This change gets the current path first and cd's back into it after
it is done compiling fsstress.

Issue #6479.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
Reviewed-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge pull request #700 from dachary/master

doc: fix openstack rbd installation command

doc: fix openstack rbd installation command

This cannot work because client.volumes.key is not a command

   ssh {your-compute-host} client.volumes.key

replace with a tee to allow for copy/paste as well as using the

   cat client.volumes.key

in the following lines.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #699 from dachary/wip-ghobject-assert

os: stronger assert on FileStore::lfn_open

Reviewed-by: David Zafman <david.zafman@inktank.com>

os: stronger assert on FileStore::lfn_open

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge branch 'next'

Merge pull request #697 from dachary/master

This lets "make check" pass, so I like it.
Reviewed-by: Greg Farnum <greg@inktank.com>

common: unintended use of the wrong bloom_filter prototype

For encoding/decoding test purposes. The generate_test_instances should
be improved to cover more encoding/decoding cases.

Signed-off-by: Loic Dachary <loic@dachary.org>

ReplicatedPG: copy: use aggregate return code instead of individual Op return

It appears that the OSD is not filling in the individual return codes, and they
should be equivalent for all purposes we care about here (the only Op we are
doing is the copy-get, and if it fails we are getting its failure code).

Reported-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

os/FileStore: fix ENOENT error code for getattrs()

In commit dc0dfb9e01d593afdd430ca776cf4da2c2240a20 the omap xattrs code
moved up a block and r was no longer local to the block.  Translate
ENOENT -> 0 to compensate.

Fix the same error in _rmattrs().

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #610 from ceph/wip-optracker

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #695 from ceph/wip-mon-auth

fix mon double-free when dropping unhandled messages, and allow "get monmap" messages to go through without authenticating for MonCliente::get_monmap_privately()

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #693 from ceph/wip-bloom

bloom_filter improvements, cleanups

Reviewed-by: Loic Dachary <loic@dachary.org>

mon: allow MMonGetMap without authentication

This is used by the MonClient::get_monmap_privately() helper.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: do not put() unhandle message

If we return false because we aren't handling a message, we should not put
the ref.  This fixes a double-free.

Signed-off-by: Sage Weil <sage@inktank.com>

common/bloom_filter: add a few assertions and checks for bit_table_ == NULL

Signed-off-by: Sage Weil <sage@inktank.com>

common/bloom_filter: simplify compressible_bloom_filter encode/decode/dump

Re-use bloom_filter for this, as per Loic's suggestion.  Much nicer!

Signed-off-by: Sage Weil <sage@inktank.com>

common/bloom_filter: fix compress; improve argument

Fix bug in compress() when compressing to less than .5 or original.

Make the argument have sane units (target size relative to current size;
not a precentage reduction).

Signed-off-by: Sage Weil <sage@inktank.com>

common/bloom_filter: fix estimated element count for compressed filters

We need to compensate for the fact that there are fewer bits than there
used to be.

This is a crappy adjustment; it is non-linear.  It's clone enough for now,
though.

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: v0.70

Note that a bunch of stuff we thought would go in 0.70 is actually in 0.69,
so the update/release notes were adjusted accordingly.

Signed-off-by: Sage Weil <sage@inktank.com>

common/bloom_filter: methods for density, approx unique element counts

Signed-off-by: Sage Weil <sage@inktank.com>

common/bloom_filter: remember original target size

This isn't strictly needed for core functionality, but it is convenient to
know.

Signed-off-by: Sage Weil <sage@inktank.com>

common/bloom_filter: drop raw_table_size_ member

We were storing table_size_ and raw_table_size_, where one is the size in
bits and the other is the size in bytes.  This is silly.  Store only the
size in bytes.

Also, bytes are always 8 bits, so use bit shifts and drop some of that
silliness too.

Move the member declarations to the top of the class so you read them
before the methods.

Fix some annoying whitespace.

Avoid allocating a 0-length array.

Mark the encoding incompatible with v1.

Signed-off-by: Sage Weil <sage@inktank.com>

common/bloom_filter: make compressible_bloom_filter encodable

Signed-off-by: Sage Weil <sage@inktank.com>

mon/PGMap: make generated test instances obey new invariant

As of 091809b8149c7595cbcca439c5b8b75a0c42efe1 we keep an osd_map epoch
for any osd_stat update, and assert as much.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'next'

doc/release-notes: v0.67.4

Signed-off-by: Sage Weil <sage@inktank.com>

Makefile: add include/histogram.h to noinst_HEADERS

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #676 from ceph/wip-start-copy

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #667 from ceph/wip-6143

Reviewed-by: Sage Weil <sage@inktank.com>

common, os: Perform xattr handling based on detected fs type

In FileStore::_detect_fs() store discovered filesystem type in m_fs_type
Add per-filesystem filestore_max_inline_xattr_size_* variants
Add per-filesystem filestore_max_inline_xattrs_* variants
New function set_xattr_limits_via_conf()
  Set m_filestore_max_inline_xattr_size based on override or fs type
  Set m_filestore_max_inline_xattrs based on override or fs type
Handle conf change of any relevant value by calling set_xattr_limits_via_conf()
Change filestore_max_inline_xattr_size to override if non-zero
Change filestore_max_inline_xattrs to override if non-zero

Fixes: #6143
Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #692 from ceph/wip-5992-2

Wip 5992 2

Reviewed-by: Sage Weil <sage@inktank.com>

ReplicatedPG: lock snapdir obc during write

Otherwise, we won't block properly in prep_push_backfill_object.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog: on split, leave log head alone

This way last_update doesn't go backwards.

Fixes: 6447
Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: make _setattrs not return -ENOENT most of the time

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: add debugging in recover_replicas for objects added for backfill

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG,PG: use PGBackend methods for getattr

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG,PG: adapt collection_list* users to PGBackend interface

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG,ReplicatedPG: expose PGBackend to PG

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG.cc: remove leading empty space

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend,ReplicatedBackend: add interfaces for scanning the pg

This will be important since the erasure coded pg will have a different
on-disk format than the replicated backend.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: don't rescan the local collection if we can avoid it

Signed-off-by: Samuel Just <sam.just@inktank.com>

common/hobject: add is_degenerate method

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGMap: calc_min_last_epoch_clean() will now also use osd_epochs

We don't want to trim past the current osd map for any up osd.
osd_epochs provides a lower bound for that epoch for each osd.

Fixes: 5869
Signed-off-by: Samuel Just <sam.just@inktank.com>

PGMap,PGMonitor: maintain mapping of osd to recent stat epoch

Also, osd_stat will be empty for out osd.

When an osd is marked out, rather than remove it from osd_stat,
we instead 0 out the structure.

This patch also makes osd_stat_updates and osd_stat_rm private.
This should make it simpler to enforce invariants on these
mappings.

Each up osd will have a mapping since out osds are now included as
empty stats.

Signed-off-by: Samuel Just <sam.just@inktank.com>

v0.70

Merge pull request #690 from ceph/wip-fuse

a few ceph-fuse fixes

Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Yan, Zheng <zheng.z.yan@intel.com>

ReplicatedPG: copy: add op progression output

Signed-off-by: Greg Farnum <greg@inktank.com>

ReplicatedPG: copy: don't leak a ctx on failed copy ops

Signed-off-by: Greg Farnum <greg@inktank.com>

ReplicatedPG: assert that we have succeeded in do_osd_ops on copyfrom repeats

Our callback is handling errors on its own at this point.

Signed-off-by: Greg Farnum <greg@inktank.com>

ReplicatedPG: copy: switch CopyCallback to use a GenContext

Signed-off-by: Greg Farnum <greg@inktank.com>

common,os: Remove filestore_xattr_use_omap option

Now we operate just like when this was set to true

Fixes: #6143
Signed-off-by: David Zafman <david.zafman@inktank.com>

rgw: update cache atomically

Use new lru_map::find_and_update() to update internal cache state.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

lru_map: add find_and_update()

A new find_and_update() call to make atomic changes.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>