qa/workunits/cephtool/test.sh: sudo ceph daemon

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2048 from ceph/wip-8699

rgw: fix uninit ofs in RGWObjManifect::obj_iterator

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

rgw: fix uninit ofs in RGWObjManifect::obj_iterator

Valgrind picked this up:

  <kind>UninitCondition</kind>
  <what>Conditional jump or move depends on uninitialised value(s)</what>
  <stack>
    <frame>
      <ip>0x5145B8</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWObjManifest::obj_iterator::seek(unsigned long)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.82-354-g62027ec/src/rgw</dir>
      <file>rgw_rados.cc</file>
      <line>562</line>
    </frame>
    <frame>
      <ip>0x5672A4</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>list_multipart_parts(RGWRados*, req_state*, std::string const&amp;, std::string&amp;, int, int, std::map&lt;unsigned int, RGWUploadPartInfo, std::less&lt;unsigned int&gt;, std::allocator&lt;std::pair&lt;unsigned int const, RGWUploadPartInfo&gt; &gt; &gt;&amp;, int*, bool*, bool)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.82-354-g62027ec/src/rgw</dir>
      <file>rgw_rados.h</file>
      <line>217</line>
    </frame>
    <frame>
      <ip>0x5688EE</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWListMultipart::execute()</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.82-354-g62027ec/src/rgw</dir>
      <file>rgw_op.cc</file>
      <line>2956</line>
    </frame>
...

Fixes: #8699
Backport: firefly
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2045 from ceph/wip-8542

qa/workunits/suites/fsx.sh: don't use zero range

Reviewed-by: Greg Farnum <greg@inktank.com>

qa/workunits/suites/fsx.sh: don't use zero range

Zero range is not supported by cephfs.

Fixes: #8542
Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'next'

rgw: allocate enough space for bucket instance id

Fixes: #8608
Backport: dumpling, firefly
Bucket instance id is a concatenation of zone name, rados instance id,
and a running counter. We need to allocate enough space to account zone
name length.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #2035 from ceph/wip-da-SCA-20140623

Fix for SCA and CID issues

Reviewed-by: Sage Weil <sage@redhat.com>

Revert "ceph-disk: Enable creating multiple osds per dev"

This reverts commit 200d0ae9c6e6701c06310230e180d4e739865bfe.

This breaks ceph-disk.  See #8644.  Reverting for now.

Signed-off-by: Sage Weil <sage@redhat.com>

test: use (unsigned)0 with gtest to avoid spurious warnings

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #2034 from ceph/wip-osd-ints

osd: fix int -> int32_t in misc encoded structures

Reviewed-by: Samuel Just <sam.just@inktank.com>

scratchtool.c: cleanup do_rados_getxattrs()

Cleanup error path to reduce code duplication.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd: fix pg_stat_t int -> int32_t

Still 4 bytes on i386 and x86_64.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: fix pg_shard_t int -> int32_t

Still 4 bytes on i386 and x86_64.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: fix pg_interval_t int -> int32_t

Still 4 bytes on i386 and x86_64.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2032 from sponce/wip-fixstriperoni386

Wip fixstriperoni386

Reviewed-by: Sage Weil <sage@redhat.com>

Fixed build on 32 bits platforms

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>

Added a couple of ASSERTS for avoiding coverity to complain about the possibility of a division by 0

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>

Merge pull request #2030 from ceph/wip-xattr-fix

mds: set xattr_version to 0 if replay doesn't contain xattrs

Reviewed-by: Sage Weil <sage@redhat.com>

mds: set xattr_version to 0 if replay doesn't contain xattrs

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge pull request #2019 from ceph/wip-refactor-cephtool-test

qa/workunits/cephtool/test.sh: refactor and make it modular (yey! \o/)

Reviewed-by: Loic Dachary <loic@dachary.org>

common/fd.cc: fix possible out-of-bounds write

Read max 'sizeof(target) - 1' to not write out of bound
later on the 'target[r] = 0;' call in case we read the
full PATH_MAX.

CID 1128416 (#1 of 1): Out-of-bounds write (OVERRUN)
 overrun-local: Overrunning array target of 4096 bytes
 at byte offset 4096 using index r (which evaluates to 4096).

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mount.ceph.c: fix strdup related memory leak

CID 1219611 (#1 - 5): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable saw_name going out of scope
 leaks the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

scratchtool.c: fix resource leak and error handling

Check the result of rados_ioctx_create()before using io_ctx.
Free io_ctx where needed. Fix messages at the end of testrados().

CID 1219613 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable io_ctx going out of scope leaks the
 storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

scratchtool.c: fix resource leak

CID 1219612 (#1 - 3): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable iter going out of scope leaks the
 storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

qa/workunits: cephtool: fix 'osd bench' test

Commit 7dc93a9651f602d9c46311524fc6b54c2f1ac595 fixed an incorrect
behavior with the OSD's 'osd bench' value hard-caps.  The test wasn't
appropriately modified unfortunately.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

osd: OSD: better explanation on 'max_count' calculation for 'osd bench'

'max_count' is the maximum number of bytes that we are to allow for an
'osd bench' command.  This value is a hard-cap that takes into account
a predefined throughput, the 'osd bench' duration and, for a rather large
block size, can be taken as the amount of bytes that we would ever be
able to write in that period of time.

The explanation wasn't appropriate enough, hence this patch, which
hopefully will avoid confusion in the future.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: only run heap profiler test if tcmalloc enabled

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: set +e for the tcmalloc tests

Avoids failing the tests when tcmalloc is not present

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: delete unnecessary function calls

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: disable bash debug when not running tests

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: allow running individual tests

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: cleanup state after erasure-code-profile test

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: add/remove comments

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: split into properly indented functions

The test was a big sequence of commands being run and it has been growing
organically for a while, even though it has maintained a sense of
locality with regard to the portions being tested.

This patch intends to split the commands into functions, allowing for a
better semantic context and easier expansion.  On the other hand, this
will also allow us to implement mechanisms to run specific portions of
the test instead of always having to run the whole thing just to test a
couple of lines down at the bottom (or have to creatively edit the test).

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: move test line to where it's more appropriate

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunits: cephtool: split into functions

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #1863 from ceph/wip-rgw-manifest-iter

rgw: reduce calls to rgw_obj.set_obj()

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2021 from ceph/wip-da-fix-make-check

Fixes to get "make check" run on SUSE v2

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2001 from dachary/wip-tiermsg

mon: test that pools used in tiers cannot be removed

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1938 from dachary/wip-8071

erasure-code: remove jerasure internals dependencies

Merge pull request #2025 from ceph/wip-disable-static

do_autogen.sh: --disable-static

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1967 from yuyuyu101/kvstore-fix-set-alloc-hint

Make KeyValueStore not use expected_write_size

Reviewed-by: Sage Weil <sage@redhat.com>

test/ceph-disk.sh: fix for SUSE

On SUSE 'which' returns always the full path of (shell) commands and
not e.g. './ceph-conf' as on Debian. Add check also for full
path returned by which.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osdmaptool/test-map-pgs.t: fix escaping to fix run

Run failed always running into the '|| cat $OUT' case due
to bad escaping of '\t'. This is caused by different shells
on different distros (e.g. bash on SUSE vs dash on Ubuntu).

Use 'grep -P ' and fix the regex to make it shell independet.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Revert "Give meaningful error when submodules are not checked out"

This reverts commit 87f9dbaf78ee040bbe0f56d2ada5299b150916b2.

This breaks the gitbuilders.  Maybe it needs the base dir to be explicit?

Signed-off-by: Sage Weil <sage@redhat.com>

Make <poolname> in "ceph osd tier --help" clearer.

The ceph osd tier --help info on the left always says <poolname>.
It is unclear which one to put <tierpool> on the right.

$ceph osd tier --help
osd tier add <poolname> <poolname> {--   add the tier <tierpool> to base pool
force-nonempty}                          <pool>
osd tier add-cache <poolname>            add a cache <tierpool> of size <size>
<poolname> <int[0-]>                     to existing pool <pool>
...

This patch modifies description on the right to tell which <poolname>:

osd tier add <poolname> <poolname> {--   add the tier <tierpool> (the second
 force-nonempty}                          one) to base pool <pool> (the first
                                           one)
...

Fix: http://tracker.ceph.com/issues/8256

Signed-off-by: Yilong Zhao <accelazh@gmail.com>

Merge pull request #1978 from adamcrume/master

OSD bug fix and developer error message

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2023 from adamcrume/wip-8661

doc: Fix malformed parameters in librados.h

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #1958 from jdurgin/wip-rbd-ro

qa: add script to test krbd setting ro/rw ioctl

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-mon-sanity-checks'

Reviewed-by: Sage Weil <sage@redhat.com>

mon: simplify output

Signed-off-by: Sage Weil <sage@inktank.com>

Merge commit '4225e2f6c9f69682d0e7288d7809851b61a17c70'

Conflicts:
PendingReleaseNotes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #1973 from ceph/wip-8593

mon: fix up health warnings

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #1980 from ceph/wip-osd-stats

osd: move osd_stat tracking into OSDService; fix minor bug

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2006 from ceph/wip-8603

log the command that is being run with subprocess

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #1969 from ceph/wip-vstart-conf

vstart.sh: echo CEPH_{CONF,KEYRING} exports if necessary

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2022 from ceph/wip-mon-perf

mon: shut down perfcounters last

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

do_autogen.sh: --disable-static

This will make builds go ~2x as fast when developing.

Reported-by: Luis Pabon <lpabon@redhat.com>
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1944 from athanatos/wip-8396

Move the waiting_for_pg list down into the Sessions, and coordinate the updates so that we don't get races
between the Pipe and handle_map threads.

Fixes: #8396
Fixes: #8505
Reviewed-by: Greg Farnum <greg@inktank.com>

0.82

Merge pull request #2024 from dachary/wip-8630-inject-enxio

osd: workaround race condition in tests

Reviewed-by: Sage Weil <sage@redhat.com>

osd: workaround race condition in tests

Trying to "ceph tell" a newly created OSD sometime triggers an
ENXIO. The OSD creation function used for test scripts waits for the OSD
to report that it is up (according to ceph osd tree) before
returning. That reduces (maybe close ?) the condition that triggers the
error so that the tests do not randomly fail.

http://tracker.ceph.com/issues/8630 refs: #8630

Signed-off-by: Loic Dachary <loic@dachary.org>

mon: shut down perfcounters last

In particular, remove_all_sessions updates the perfcounter stats, so we
need to stop those after.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: Fix malformed parameters in librados.h

Fixes parameters for rados_write_op_operate and rados_aio_write_op_operate

Fixes: #8661
Signed-off-by: Adam Crume <adamcrume@gmail.com>

libcephfs/test.cc: fix use after free

Don't call ceph_shutdown() after ceph_release().

CID 1219651 (#1 of 1): Use after free (USE_AFTER_FREE)
  17. deref_arg: Calling ceph_shutdown dereferences freed
  pointer cmount.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

RadosStriperImpl.cc: catch exception by reference

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_rados.h: use static_cast instead of c-style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #1996 from dachary/wip-mailmap

mailmap updates

reviewed-by individual commits

Merge pull request #2017 from ceph/wip-8654

Fix #8654 - OSD.cc: parse LSB data via lsb_release

Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

osd/OSD.cc: parse lsb release data via lsb_release

Use lsb_release tool to be portable since parsing /etc/lsb-release
is not the same between different distributions. The old code failed
e.g. for SUSE products to parse LSB information.

Fixes: #8654
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #2020 from ceph/wip-fix-rados-tool

rados.cc: fix pool alignment check

Reviewed-by: Sage Weil <sage@inktank.com>

rados.cc: fix pool alignment check

Only check pool alignment if io_ctx is initialized.

Introduced in 304b08a23a3db57010078046955a786fe3589ef8
Fixes: #8652
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge remote-tracking branch 'origin/next'

Merge pull request #2005 from ceph/wip-8624

mon: MDSMonitor: do not allow ec pools to be used for data or metadata

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: John Spray <john.spray@redhat.com>

mon: MDSMonitor: print pool name along with id during 'newfs' errors

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

qa/workunit: cephtool: test mds newfs and add_data_pool with ec pools

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: MDSMonitor: do not allow ec pools to be used for data or metadata

Fixes: #8624
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

common: LogClient: output to derr (i.e., dout(-1)) on CLOG_ERROR

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: observe conf changes and report on unsage option values

Fixes: #8605
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: sanitize options at start

Make sure some options have sane values.

Fixes: #8605
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Give meaningful error when submodules are not checked out

Signed-off-by: Adam Crume <adamcrume@gmail.com>

osd: Only normalize extent if op uses extent

Code in ReplicatedPG was reading and modifying ceph_osd_op.extent,
even though that is a member of a union and only valid for some ops.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

osd: remove OSD-specific leveldb options

OSDs will now rely on 'leveldb_*' config options.  We do keep however
leveldb's log enabled for OSDs by passing 'leveldb_log=""' as a default
argument to global_init() on ceph_osd.cc -- however, users will be able
to override this at their own discretion.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

ceph-mon: override 'leveldb_*' config options for the monitor

'leveldb_*' options are currently used both by the monitor and the osd.
However, the monitor has quite different requirements from those of the
osds.

We need to specify some default values that must squash the defaults we
have for 'leveldb_*' options, while allowing users to overriding them too.
We take this not-exactly-ideal-but-still-good-enough approach of
defining the monitor-specific defaults in the 'default arguments' to
global_init(), thus allowing the user's options to take precedence over
whatever we define.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: MonitorDBStore: remove mon-specific leveldb options.

From this point onward, users should use leveldb's options and add them
to the appropriate config sections of their configuration file.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #1998 from ceph/wip-disk-ioprio

osd: allow io priority for (background) disk threadpool to be set

Backport: firefly, dumpling
Reviewed-by: Samuel Just <sam.just@inktank.com>

mon: DataHealthService: s/mon_leveldb_size_warn/mon_data_size_warn/

Make the option name independent from the store mechanism used underneath.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #2015 from ceph/wip-8610

osd: ignore CRUSH_ITEM_NONE in compat_must_dispatch_immediately

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd: ignore CRUSH_ITEM_NONE in compat_must_dispatch_immediately

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1979 from ceph/wip-backfill-priority

osd: improve backfill prioritization

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd: allow io priority to be set for the disk_tp

The disk_tp covers scrubbing, pg deletion, and snap trimming

Signed-off-by: Sage Weil <sage@inktank.com>

common/WorkQueue: allow io priority to be set for wq

Signed-off-by: Sage Weil <sage@inktank.com>

common/Thread: allow io priority to be set for a Thread

Ideally, set this before starting the thread.  If you set it after, we
could potentially race with create() itself.

Signed-off-by: Sage Weil <sage@inktank.com>

common/io_priority: wrap ioprio_set() and gettid()

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/wip-misc-fixes'

Conflicts:
src/mon/Monitor.cc

Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #2009 from kevincox/election-int

Use sized integer.

Reviewed-by: Sage Weil <sage@inktank.com>

test/libradosstriper/striping.cc: fix potential memory leak

Fix for:
[src/test/libradosstriper/striping.cc:138]: (error) Allocation with
 getObjName, stat doesn't release it

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

MDCache.cc: init 'umaster::ls' with NULL in contructor

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

MDCache.h: init 'umaster::safe' in constructor

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>