WBThrottle: rename replica nocache

We may want to influence the caching behavior for other
reasons.

Signed-off-by: Samuel Just <sam.just@inktank.com>

doc/dev/osd_internals: add wbthrottle.rst

Signed-off-by: Samuel Just <sam.just@inktank.com>

WBThrottle: add perfcounters

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::submit_push_complete don't remove the head object

The object would have had to have been removed already.  With
fd caching, this extra remove might check the wrong replay_guard
since the fd caching mechanism assumes that between any operation
on an hobject_t oid and a remove operation, all operations on that
hobject_t must refer to the same inode.

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: integrate WBThrottle

Signed-off-by: Samuel Just <sam.just@inktank.com>

os/: Add WBThrottle

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: add fd cache

Signed-off-by: Samuel Just <sam.just@inktank.com>

common/shared_cache.hpp: fix set_size()

Signed-off-by: Samuel Just <sam.just@inktank.com>

doc/dev/osd_internals: add some info about throttles

Signed-off-by: Samuel Just <sam.just@inktank.com>

common/shared_cache.hpp: add clear()

Clear clears a key/value from the cache.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #303 from ceph/wip-librbd-config-create

Reviewed-by: Sage Weil <sage.weil@inktank.com>

Merge branch 'next'

Conflicts:
debian/ceph.prerm

Merge pull request #304 from javacruft/wip-invoke-rc.d

Use invoke-rc.d in maintainer scripts

os/LevelDBStore: fix compression selection

We were always disabling compression.

Fixes: #5131
Reported-by: Sylvain Munaut <s.munaut@whatever-company.com>
Signed-off-by: Sage Weil <sage@inktank.com>

Use invoke-rc.d in maintainer scripts

Upstart configurations and sysv init scripts should be started and stopped
using invoke-rc.d in maintainer scripts.

This ensures the correct behaviour across both Debian (sysv) and Ubuntu (upstart)
and in restricted environments such as schroot where start/stop of services from
maintainer scripts is normally disabled.

Signed-off-by: James Page <james.page@ubuntu.com>

PG: kill some dead IndexedLog methods

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

OSD: kill old split code, it's been dead for a while

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>

Merge pull request #260 from ceph/wip-suppress

Reviewed-by: Gary Lowell <gary.lowell@inktank.com>

debian: stop sysvinit on ceph.prerm

Signed-off-by: Sage Weil <sage@inktank.com>

mds: better error check on sessionmap load

Would have helped with #5104.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: be a bit more verbose about osd mark down events

Put these in the cluster log; they are interesting.

Signed-off-by: Sage Weil <sage@inktank.com>

Makefile: add -Wformat-security

Clean up the formatting a bit here too.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #301 from javacruft/wip-upstart-tasks

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #300 from javacruft/wip-format-security

Fix -Werror=format-security errors

ceph df: fix si units for 'global' stats

si_t expects bytes, but it was being given kilobytes.

Signed-off-by: Mike Kelly <pioto@pioto.org>
(cherry picked from commit 0c2b738d8d07994fee4c73dd076ac9364a64bdb2)

Merge pull request #302 from pioto/ceph-df-unit-fix

Reviewed-by: Sage Weil <sage@inktank.com>

ceph df: fix si units for 'global' stats

si_t expects bytes, but it was being given kilobytes.

Signed-off-by: Mike Kelly <pioto@pioto.org>

Remove stop on from upstart tasks

Upstart tasks don't have to concept of 'stop on' as they
are not long running.

Fix -Werror=format-security errors

Fixup a couple of errors when compilng with -Werror=format-security.

libcephfs: add ceph_get_pool_name()

Convert an int pool id to a pool name.  Useful for making sense of the
output from ceph_get_*_layout()'s poolid output argument.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #253 from Elbandi/wip-getlayout

Reviewed-by: Sage Weil <sage@inktank.com>

doc: Added more glossary-compliant terms and indexing.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added another instance term to the glossary.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Minor improvements to Ceph FS landing page.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

libcephfs: get the pool name of a file path

Signed-off-by: Andras Elso <elso.andras@gmail.com>

libcephfs: get replication factor of a file handle/path

Signed-off-by: Andras Elso <elso.andras@gmail.com>

libcephfs: get file handle/path layout info

Signed-off-by: Andras Elso <elso.andras@gmail.com>

libcephfs: get stripe_unit/stripe_count/object_size/pool_id by file handle/path

Signed-off-by: Andras Elso <elso.andras@gmail.com>

libcephfs: fix typos

Signed-off-by: Andras Elso <elso.andras@gmail.com>

Client: get describe_layout by file handle/path

Signed-off-by: Andras Elso <elso.andras@gmail.com>

Merge pull request #295 from ceph/wip-5077

Reviewed-by: Joao Luis <joao.luis@inktank.com>

mon: add 'compact' command

As in, 'ceph mon tell \* compact'

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Luis <joao.luis@inktank.com>

Merge pull request #296 from dalgaaf/wip-da-CID-1021213

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

librbd/internal.cc: fix resource leak

Call release() on librados::AioCompletion to free storage before
leave the loop or call new again.

CID 1021213 (#1 of 1): Resource leak (RESOURCE_LEAK)
  leaked_storage: Variable "rados_completion" going out of scope leaks
  the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

doc: Omitted literal syntax from toc.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added fuse syntax to the fstab doc.

fixes: #3672

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge branch 'next'

sysvinit: fix enumeration of local daemons when specifying type only

- prepend $local to the $allconf list at the top
- remove $local special case for all case
- fix the type prefix checks to explicitly check for prefixes

Fugly bash, but works!

Backport: cuttlefish, bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

udev: install disk/by-partuuid rules

Wheezy's udev (175-7.2) has broken rules for the /dev/disk/by-partuuid/
symlinks that ceph-disk relies on.  Install parallel rules that work.  On
new udev, this is harmless; old older udev, this will make life better.

Fixes: #4865
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>

mon: clear pg delta after some period

If we have not pg_map updates, the delta doesn't update, and can get stuck
with the velocity right before activity stopped.  This is confusing, and
can cause incorrect health warnings about in-progress recovery.

To fix this, zero the delta if there is no activity for
'mon delta reset interval' seconds.

Fixes: #5077
Signed-off-by: Sage Weil <sage@inktank.com>

os/FileStore: fix replay guard error msgs (again)

Signed-off-by: Sage Weil <sage@inktank.com>

test_filestore_idempotent_sequence: unmount prior to deleting store

FileStoreDiff umounts the stores in its destructor.

Also, DeterministicOpSequence handles deletes its passed
object store.

Fixes: #5076
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>

Revert "test_filejournal.cc: cleanup memory in destructor"

The finish() method for Contexts calls delete this.

This reverts commit 36028916c4630ea66007760efed8fc6c441e7af5.

Fixes: #5075
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>

librbd: make image creation defaults configurable

Programs using older versions of the image creation functions can't
set newer parameters like image format and fancier striping.

Setting these options lets them use all the new functionality without
being patched and recompiled to use e.g. rbd_create3().
This is particularly useful for things like qemu-img, which does not
know how to create format 2 images yet.

Refs: #5067
backport: cuttlefish, bobtail
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

os/FileStore: print error code to log on replay guard failure

Signed-off-by: Sage Weil <sage@inktank.com>

rbd.py: fix stripe_unit() and stripe_count()

These matched older versions of the functions, but would segfault
using the current versions.

backport: cuttlefish, bobtail
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

cls_rbd: make sure stripe_unit is not larger than object size

Test a few other cases too.

backport: cuttlefish, bobtail
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

doc: Added index reference.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added glossary references and index references.

fixes: #3321

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added cluster map and CRUSH definitions.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Fixing index references.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added latency comment.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

debian: make radosgw require matching version of librados2

...indirectly via ceph-common.  We get bad behavior when they diverge, I
think because of libcommon.la being linked both statically and dynamically.

Fixes: #4997
Backport: cuttlefish, bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Gary Lowell <gary.lowell@inktank.com>

Merge pull request #291 from dalgaaf/wip-da-CID-1019548

client/Client.cc: fix/silence "logically dead code" CID-Error

Even money that this satisfies the coverity gods...

Reviewed-by: Sage Weil <sage@inktank.com>

FileJournal: adjust write_pos prior to unlocking write_lock

In committed_thru, we use write_pos to reset the header.start value in cases
where seq is past the end of our journalq.  It is therefore important that the
journalq be updated atomically with write_pos (that is, under the write_lock).

The call to align_bl() is moved into do_write in order to ensure that write_pos
is adjusted correctly prior to write_bl().

Also, we adjust pos at the end of write_bl() such that pos \in [get_top(),
header.max_size) after write_bl().

Fixes: #5020
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

client/Client.cc: fix/silence "logically dead code" CID-Error

Fix handling of 'safe' and the conditions after calling file_flush().

CID 1019548 (#1 of 1): Logically dead code (DEADCODE)
  dead_error_begin: Execution cannot reach this statement "flock.Unlock();"

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds: avoid assert after suicide()

Fixes: #5079
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #290 from dalgaaf/wip-da-SCA-cppcheck-v2

Reviewed-by: Sage Weil <sage@inktank.com>
rgw bits Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

objclass/class_debug.cc: reduce scope of 'n' in cls_log()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

tools/ceph-filestore-dump.cc: reduce scope of 'r' in export_files()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/system/systest_runnable.cc: reduce scope of 'ret' in join()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/system/st_rados_list_objects.cc: reduce scope of 'ret' in run()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/system/st_rados_create_pool.cc_ reduce scope of 'ret' in run()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/system/rados_list_parallel.cc: reduce scope of 'ret'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/librbd/test_librbd.cc: reduce scope of several variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_tools.cc: reduce scope of 'ret' in rgw_get_obj()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_common.cc: reduce scope of 'end' in two cases

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_bucket.cc: reduce scope of 'max' in rgw_remove_bucket()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_admin.cc: reduce scope of 'ret'

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_acl_s3.cc: remove local variable 'ret' from create_from_headers()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rbd_fuse/rbd-fuse.c: reduce scope of some variables in open_rbd_image()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osdc/Objecter.cc: reduce scope of skipped_map

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/chain_xattr.cc: reduce scope of local variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

src/os/LFNIndex.cc: reduce scope of suffix_len

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/HashIndex.cc: reduce scope of a local variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/FileStore.cc: reduce scope of a local variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

src/os/FlatIndex.cc: reduce scope of suffix_len

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

src/os/DBObjectMap.cc: reduce scope of some variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mount/mount.ceph.c: reduce scope of 'skip' in parse_options()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

src/mds/flock.cc: reduce scope of old_lock_to_end in two cases

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Locker.cc: reduce scope of forced_change_max

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

src/crush/mapper.c: reduce scope of some local variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

auth/Crypto.cc: reduce scope of local variable in_buf

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_rados.cc: remove not needed code

Fix for cppcheck warning:
[src/rgw/rgw_rados.cc:2390]: (warning) Assignment of function
  parameter has no effect outside the function.

Assignment of if_nomatch_str.c_str() to if_nomatch has no
effect outside the function and the functions isn't used
inside the function after the assignment.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_gc.cc: fix possible NULL pointer dereference

Fix/silence cppcheck warning:
[src/rgw/rgw_gc.cc:185] -> [src/rgw/rgw_gc.cc:181]: (error) Possible
  null pointer dereference: ctx - otherwise it is redundant to check
  it against null.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/OSD.cc: remove unused variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Fix some little/big endian issues

Ceph uses little endian, this patch fixes some endian issues
while Ceph running on big endian machine.

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Signed-off-by: Li Wang <liwang@ubuntukylin.com>

Makefle: force char to be signed

On an armv7l build, we see errors like

 warning: rgw/rgw_common.cc:626:16: comparison is always false due to limited range of data type [-Wtype-limits]

from code

      char c1 = hex_to_num(*src++);
...
      if (c1 < 0)

Force char to be signed (regardless of any weird architecture's default)
to avoid risk of this leading to misbehavior.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

Merge branch 'wip-4783'

Reviewed-by: Sam Just <sam.just@inktank.com>