msg/Message: use RefCountedObject ctor to set initial ref count

Also, drop the useless assert in the dtor (the RefCountedObject dtor does
the same).

Signed-off-by: Sage Weil <sage@inktank.com>

msg/Pipe: debug Pipe refs

Signed-off-by: Sage Weil <sage@inktank.com>

msg: debug Connection refs

Signed-off-by: Sage Weil <sage@inktank.com>

osd: debug Session refs

Signed-off-by: Sage Weil <sage@inktank.com>

common/RefCountedObject: assert nref == 0 in dtor

Signed-off-by: Sage Weil <sage@inktank.com>

common/RefCountedObject: optionally take initial ref count to ctor

Signed-off-by: Sage Weil <sage@inktank.com>

common/RefCountedObject: optionally debug

Introduce a 'refs' subsys for logging.  Print log ref count inc and dec
for any RefCountedObject that gets a cct passed to its ctor.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2047 from dachary/wip-osd-config

osd: improve tests for configuration updates

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

Merge pull request #2036 from ceph/wip-osd-map-cache-size

osd: allow osd map cache size to be adjusted at runtime

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2068 from thorstenb/wip-janitorial-clang-4

ceph_argparse_flag has no regular 3rd parameter.

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2046 from ceph/wip-8670

mon: OSDMonitor: 'osd pool' - if we can set it, we must be able to get it

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph_argparse_flag has no regular 3rd parameter.

With clang warning: 'va_start' has undefined behavior with reference
types, noticing copy-paste mistake on ceph_argparse_flag.

Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

mon: OSDMonitor: 'osd pool' - if we can set it, we must be able to get it

Add support to get the values for the following variables:
 - target_max_objects
 - target_max_bytes
 - cache_target_dirty_ratio
 - cache_target_full_ratio
 - cache_min_flush_age
 - cache_min_evict_age

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #2055 from ceph/wip-8692

mon: clear osd request queue latency info on down or up

Reviewed-by: Samuel Just <sam.just@inktank.com>

mon: clear osd request queue latency info on down or up

It doesn't make sense to warn about slow requests on OSDs that are down;
clear those stats in the monitor when we see an OSD go down (or up).

Fixes: #8692
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2054 from ceph/wip-fs-cmds-oops

Apply some accidentally-dropped commits from the wip-fs-cmds branch to be a little friendlier and pass tests.

Reviewed-by: Greg Farnum <greg@inktank.com>

mds: Update default FS name

From 'default' to 'cephfs' so that it looks
at least vaguely meaningful.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update cephtool EC pools test to respect IDs

...of data and metadata pools which are no longer
constant.

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MDSMonitor: EC check in 'fs new' like newfs

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add a check for crash_replay_interval autoset

... that after a fs new on fresh pools, crash_replay_interval
is set to the default on the data pool.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: Set crash_replay_interval automatically

Signed-off-by: John Spray <john.spray@redhat.com>

qa: Update data pool ID for vxattrs test

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #2053 from dachary/wip-signed-warnings

test: use 0U with gtest to avoid spurious warnings

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@cloudwatt.com>

Merge pull request #2052 from dachary/wip-vstart-wrapped

qa: support running under non privileged user

Reviewed-by: Florent Flament <florent.flament@cloudwatt.com>

test: use 0U with gtest to avoid spurious warnings

Signed-off-by: Loic Dachary <loic@dachary.org>

qa: support running under non privileged user

If the test is run against a cluster started with vstart.sh (which is
the case for make check), the --asok-does-not-need-root disables the use
of sudo and allows the test to run without requiring privileged user
permissions.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1942 from ceph/wip-fs-cmds

Wip fs cmds

Reviewed-by: Greg Farnum <greg@inktank.com>

osd: allow osd map cache size to be adjusted at runtime

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #2050 from ceph/wip-cephtool

qa/workunits/cephtool/test.sh: sudo ceph daemon

qa/workunits/cephtool/test.sh: sudo ceph daemon

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2049 from ceph/wip-fix-pglog-unittest

PGLog: fix clear() to avoid the IndexLog::zero() asserts

Reviewed-by: Sage Weil <sage@redhat.com>

PGLog: fix clear() to avoid the IndexLog::zero() asserts

Introduced in:
  c5b8d8105d965da852c79add607b69d5ae79a4d4
  ac11ca40b4f4525cbe9b1778b1c5d9472ecb9efa
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2003 from ferustigris/hotfix/rgw_memory_leaks

RGW memory leaks have been fixed

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2041 from ceph/wip-map-unmap

Wip map unmap

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2040 from ceph/wip-krbd-settle

krbd: rework the unmap retry loop

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2038 from ceph/wip-fsx-random-ctx

test_librbd_fsx: use private RNG context

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2048 from ceph/wip-8699

rgw: fix uninit ofs in RGWObjManifect::obj_iterator

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

rgw: fix uninit ofs in RGWObjManifect::obj_iterator

Valgrind picked this up:

  <kind>UninitCondition</kind>
  <what>Conditional jump or move depends on uninitialised value(s)</what>
  <stack>
    <frame>
      <ip>0x5145B8</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWObjManifest::obj_iterator::seek(unsigned long)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.82-354-g62027ec/src/rgw</dir>
      <file>rgw_rados.cc</file>
      <line>562</line>
    </frame>
    <frame>
      <ip>0x5672A4</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>list_multipart_parts(RGWRados*, req_state*, std::string const&amp;, std::string&amp;, int, int, std::map&lt;unsigned int, RGWUploadPartInfo, std::less&lt;unsigned int&gt;, std::allocator&lt;std::pair&lt;unsigned int const, RGWUploadPartInfo&gt; &gt; &gt;&amp;, int*, bool*, bool)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.82-354-g62027ec/src/rgw</dir>
      <file>rgw_rados.h</file>
      <line>217</line>
    </frame>
    <frame>
      <ip>0x5688EE</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWListMultipart::execute()</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.82-354-g62027ec/src/rgw</dir>
      <file>rgw_op.cc</file>
      <line>2956</line>
    </frame>
...

Fixes: #8699
Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2045 from ceph/wip-8542

qa/workunits/suites/fsx.sh: don't use zero range

Reviewed-by: Greg Farnum <greg@inktank.com>

osd: improve tests for configuration updates

Extract the default values from the actual configuration instead of
having them hardcoded. Also check that lowering the osd_map_cache_size
triggers the expected warning.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #2026 from athanatos/wip-8438-master

Wip 8438 master

Fixes: #8438
Backport: Firefly
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #2027 from athanatos/wip-7999-master

Wip 7999 master

Fixes: #7999
Backport: Firefly
Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

qa/workunits/suites/fsx.sh: don't use zero range

Zero range is not supported by cephfs.

Fixes: #8542
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2039 from dachary/wip-erasure-code-doc

erasure-code: add definitions to the glossary

Reviewed-by: Florent Flament <florent.flament@cloudwatt.com>

mon/MDSMonitor: log warning while MDS up but no FS

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MDSMonitor: fix incorrect comment

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MDSMonitor: s/enabled/get_enabled()/

Signed-off-by: John Spray <john.spray@redhat.com>

mds: Handle setting 'enabled' in upgrades

Previously was always defaulting to disabled,
should be enabled if MDS map epoch is >1 (i.e.
if an MDS was ever started).

Signed-off-by: John Spray <john.spray@redhat.com>

mds: journal-related debug messages

Signed-off-by: John Spray <john.spray@redhat.com>

test/qa: update for MDSMonitor changes

Accomodate changes:
 * data and metadata pools no longer exist by default
 * filesystem-using tests must use `fs new` to create
   the filesystem first.

Signed-off-by: John Spray <john.spray@inktank.com>

vstart: update for newfs/rmfs changes

Signed-off-by: John Spray <john.spray@inktank.com>

mon: add `ceph fs [new|rm|ls]`

Signed-off-by: John Spray <john.spray@inktank.com>

erasure-code: add definitions to the glossary

and fix a typo

Signed-off-by: Loic Dachary <loic@dachary.org>

mon: hide mdsmap in 'ceph status' if not enabled

Previously we checked if the MDS map had ever been
updated (epoch > 1), now we have an explicit flag
for whether it's enabled.

Signed-off-by: John Spray <john.spray@inktank.com>

mon: stop MDSMonitor when FS disabled

Where 'stopped' means ignoring beacons and
not performing 'tick' activities.

Signed-off-by: John Spray <john.spray@inktank.com>

mon: enable deleting pools when FS disabled

Signed-off-by: John Spray <john.spray@inktank.com>

mon: warn in newfs if crash_replay_interval=0

This is the setting we would apply to data pools
created automatically, so notify the user if they're
failing to use it for data pools they have created
by hand.

Signed-off-by: John Spray <john.spray@inktank.com>

mon: don't require 'sure' flag for new fs

Only prompt for --yes-i-really-mean-it if there
is an existing FS.

Signed-off-by: John Spray <john.spray@inktank.com>

osdmap: Don't create FS pools by default

Because many Ceph users don't use the filesystem,
don't create the 'data' and 'metadata' pools by
default -- they will be created by newfs if
they are needed.

Signed-off-by: John Spray <john.spray@inktank.com>

mds: no initial filesystem

Because not everyone uses CephFS, we would
like to avoid initially creating any data/metadata
pools for CephFS.

To avoid creating those pools, we must avoid initially
populating the MDSMap.

Signed-off-by: John Spray <john.spray@inktank.com>

mds: add 'enabled' flag to MDSMap

When 'enabled' is false, the MDSMap is effectively
null.  This allows Ceph clusters with no filesystem
or filesystem data/metadata pools.

Signed-off-by: John Spray <john.spray@inktank.com>

doc: RPM instructions correction

Signed-off-by: John Spray <john.spray@inktank.com>

krbd: rework the unmap retry loop

The retry loop in the unmap path turned out to be insufficient for
doing long fsx -K runs.  Replace it with a single retry and then call
out to 'udevadm settle' if the retry doesn't help.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

[RGW, memory leak] Memory leak in RGW has been fixed: deletion of allocated pointer to pointer to Log object has been added to "on_exit" handler.

Memory leaks detector report:

$ valgrind  --leak-check=full /usr/bin/radosgw -c /etc/ceph/ceph.conf -n
client.radosgw.gateway -
...
==16986== 8 bytes in 1 blocks are definitely lost in loss record 14 of 83
==16986==    at 0x4A075BC: operator new(unsigned long) (vg_replace_malloc.c:298)
==16986==    by 0x58980B8: ceph::log::Log::set_flush_on_exit() (in /usr/lib64/librados.so.2.0.0)
==16986==    by 0x6BE3CA: global_init(std::vector<char const*, std::allocator<char const*> >*, st
==16986==    by 0x5B6B0A: main (in /usr/bin/radosgw)
...

Signed-off-by: Andrey Kuznetsov <Andrey_Kuznetsov@epam.com>

[RGW, memory leak] Memory leak in RGW GC (losing pointer during allocating Ceph-context) has been fixed.

Memory leaks detector report:
...
==117947== 11,725 (200 direct, 11,525 indirect) bytes in 25 blocks are definitely lost in loss
record 82 of 82
==117947==    at 0x4A075BC: operator new(unsigned long) (vg_replace_malloc.c:298)
==117947==    by 0x687CC1: RGWGC::process(int, int) (in /usr/bin/radosgw)
==117947==    by 0x6884D1: RGWGC::process() (in /usr/bin/radosgw)
==117947==    by 0x68875C: RGWGC::GCWorker::entry() (in /usr/bin/radosgw)
==117947==    by 0x55F890A: Thread::_entry_func(void*) (in /usr/lib64/librados.so.2.0.0)
==117947==    by 0x34D46079D0: start_thread (in /lib64/libpthread-2.12.so)
==117947==    by 0x34D42E8B6C: clone (in /lib64/libc-2.12.so)
...

Signed-off-by: Andrey Kuznetsov <Andrey_Kuznetsov@epam.com>

[RGW, memory leaks] Memory leak in RGW initialization (Inserting new connection into connections map w/o check) has been fixed.

Memory leaks detector report:

$ valgrind  --leak-check=full /usr/bin/radosgw -c /etc/ceph/ceph.conf -n
client.radosgw.gateway -f
...
=16986== 1,262 (48 direct, 1,214 indirect) bytes in 1 blocks are definitely lost in loss record 81
of 83
==16986==    at 0x4A075BC: operator new(unsigned long) (vg_replace_malloc.c:298)
==16986==    by 0x618F0D: RGWRados::init_complete() (in /usr/bin/radosgw)
==16986==    by 0x618FE6: RGWRados::initialize() (in /usr/bin/radosgw)
==16986==    by 0x63BB23: RGWRados::initialize(CephContext*, bool) (in /usr/bin/radosgw)
==16986==    by 0x634D20: RGWStoreManager::init_storage_provider(CephContext*, bool) (in
/usr/bin/radosgw)
==16986==    by 0x5B8970: RGWStoreManager::get_storage(CephContext*, bool) (in /usr/bin/radosgw)
==16986==    by 0x5B6D5D: main (in /usr/bin/radosgw)
...

Signed-off-by: Andrey Kuznetsov <Andrey_Kuznetsov@epam.com>

map-unmap.sh: fail if 'rbd rm' fails

Fail if 'rbd rm' fails - most probably it'd fail with "image still has
watchers" and in that case it's a bug in the kernel client which we do
want to notice.  Also nuke the trap-based error handling - cleanup() is
half-baked and not really necessary here.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

map-unmap.sh: drop the get_id() logic

Take advantage of the fact that 'rbd map' will now talk to udev and
output the device that got assigned by the kernel to the newly created
mapping.  Drop the get_id() cruft, udevadm settle and chown calls.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

test_librbd_fsx: use private RNG context

It is at the core of fsx to be able to reproduce the exact op sequence
that lead to a failure.  Use reentrant glibc RNG functions to make sure
that nothing else has to chance to mess up the sequence.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

doc: dev: cache-pool.rst: fix typo s/objets/objects/

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge remote-tracking branch 'upstream/master' into wip-8438-master

Conflicts:
src/osd/ECMsgTypes.cc

osd/: in scrub, check for and remove obsolete rollback objs

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: plumb rollback_trim_to through interfaces and messages

We simply trim the roll back info up to min_last_complete_ondisk
since we cannot correctly rollback past that point anyway!

ReplicatedPG
-> PGBackend::submit_transaction
-> Backend messages
-> PGBackend::Listener::log_operation
-> append_log

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog: add rollback_info_trimmed_to_riter to IndexedLog and use in PG

Signed-off-by: Samuel Just <sam.just@inktank.com>

doc/release-notes: v0.82

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'next'

rgw: allocate enough space for bucket instance id

Fixes: #8608
Backport: dumpling, firefly
Bucket instance id is a concatenation of zone name, rados instance id,
and a running counter. We need to allocate enough space to account zone
name length.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

PGLog: fix logging whitespace in trim

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge remote-tracking branch 'upstream/master' into wip-7999-master

Conflicts:
src/os/KeyValueStore.h

PG: block outgoing peering messages until the last interval is durable

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: add machinery to temporarily buffer outgoing recovery messages

Signed-off-by: Samuel Just <sam.just@inktank.com>

os/: add async flush_commit() method to Sequencer and implementations

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2035 from ceph/wip-da-SCA-20140623

Fix for SCA and CID issues

Reviewed-by: Sage Weil <sage@redhat.com>

Revert "ceph-disk: Enable creating multiple osds per dev"

This reverts commit 200d0ae9c6e6701c06310230e180d4e739865bfe.

This breaks ceph-disk.  See #8644.  Reverting for now.

Signed-off-by: Sage Weil <sage@redhat.com>

test: use (unsigned)0 with gtest to avoid spurious warnings

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #2034 from ceph/wip-osd-ints

osd: fix int -> int32_t in misc encoded structures

Reviewed-by: Samuel Just <sam.just@inktank.com>

scratchtool.c: cleanup do_rados_getxattrs()

Cleanup error path to reduce code duplication.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd: fix pg_stat_t int -> int32_t

Still 4 bytes on i386 and x86_64.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: fix pg_shard_t int -> int32_t

Still 4 bytes on i386 and x86_64.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: fix pg_interval_t int -> int32_t

Still 4 bytes on i386 and x86_64.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2032 from sponce/wip-fixstriperoni386

Wip fixstriperoni386

Reviewed-by: Sage Weil <sage@redhat.com>

Fixed build on 32 bits platforms

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>

Added a couple of ASSERTS for avoiding coverity to complain about the possibility of a division by 0

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>

Merge pull request #2030 from ceph/wip-xattr-fix

mds: set xattr_version to 0 if replay doesn't contain xattrs

Reviewed-by: Sage Weil <sage@redhat.com>

mds: set xattr_version to 0 if replay doesn't contain xattrs

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge pull request #2019 from ceph/wip-refactor-cephtool-test

qa/workunits/cephtool/test.sh: refactor and make it modular (yey! \o/)

Reviewed-by: Loic Dachary <loic@dachary.org>

common/fd.cc: fix possible out-of-bounds write

Read max 'sizeof(target) - 1' to not write out of bound
later on the 'target[r] = 0;' call in case we read the
full PATH_MAX.

CID 1128416 (#1 of 1): Out-of-bounds write (OVERRUN)
 overrun-local: Overrunning array target of 4096 bytes
 at byte offset 4096 using index r (which evaluates to 4096).

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mount.ceph.c: fix strdup related memory leak

CID 1219611 (#1 - 5): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable saw_name going out of scope
 leaks the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

scratchtool.c: fix resource leak and error handling

Check the result of rados_ioctx_create()before using io_ctx.
Free io_ctx where needed. Fix messages at the end of testrados().

CID 1219613 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable io_ctx going out of scope leaks the
 storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>