crush/crush.c: prevent DIVIDE_BY_ZERO

Fix for:

CID 1219471 (#1 of 1): Division or modulo by zero (DIVIDE_BY_ZERO)
 divide_by_zero: In function call crush_make_uniform_bucket,
 division by expression item_weight which may be zero has undefined
 behavior.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

src/common/obj_bencher: fix some UNINIT issues

Make sure concurrentios is always >= 0 to fix these coverity
issues and to prevent bad_alloc/negative array sizes:

CID 1128404 (#1 of 1): Uninitialized scalar variable (UNINIT)
 uninit_use: Using uninitialized value index[slot].

CID 1128405 (#1 of 1): Uninitialized pointer read (UNINIT)
 uninit_use: Using uninitialized value contents[slot].

CID 1219644 (#1 of 1): Uninitialized pointer read (UNINIT)
 uninit_use: Using uninitialized value contents[slot].

CID 1219645 (#1 of 1): Uninitialized pointer read (UNINIT)
 uninit_use: Using uninitialized value contents[slot].

CID 1219646 (#1 of 1): Uninitialized scalar variable (UNINIT)
 uninit_use: Using uninitialized value index[slot].

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/MDLog.cc: fix uninitialized pointer read

Fix for Coverity issue:

CID 1238902 (#1 of 1): Uninitialized pointer read (UNINIT)
 uninit_use: Using uninitialized value this->mdlog.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

librados/librados.cc: fix UNINIT issue

Fix for issues introduced with cba4ed44f:

CID 1256099 (#1 of 1): Uninitialized pointer read (UNINIT)
 uninit_use: Using uninitialized value this->impl.

CID 1256100 (#1 of 1): Uninitialized pointer read (UNINIT)
 uninit_use: Using uninitialized value this->impl.

Don't delete a pointer in the constructor if the pointer
isn't initialized before.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/librados/snapshots.cc: fix resource leak, delete op

Fix for:

CID 1260395 (#2 of 2): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable op going out of scope leaks the storage
 it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_objectstore_tool.cc: reduce scope of variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

tools/ceph_authtool.cc: reduce scope of some variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

msg/async/AsyncMessenger.cc: reduce scope of variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

msg/async/EventSelect.cc: reduce scope of variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_monstore_tool.cc: unsigned can't be less than zero

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

common/Cycles.cc: reduce scope of variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

src/mds/MDS.cc: remove unused variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

bench/tp_bench.cc: fix warning about self init

Fix for:

[src/test/bench/tp_bench.cc:103]: (error) Member variable 'next'
 is initialized by itself.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_mon_workloadgen.cc: fix warning about self init

Fix for:

[src/test/mon/test_mon_workloadgen.cc:351]: (error) Member variable
 'whoami' is initialized by itself.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Server.cc: fix compiler warning

Fix compiler warning caused by twice checking but missing
final else block.

In file included from mds/MDS.h:20:0,
                 from mds/Server.cc:22:
mds/mdstypes.h: In member function 'void Server::handle_set_vxattr(
 MDRequestRef&, CInode*, ceph_file_layout*, std::set<SimpleLock*>,
 std::set<SimpleLock*>, std::set<SimpleLock*>)':
mds/mdstypes.h:499:42: warning: 'pi' may be used uninitialized in
 this function [-Wmaybe-uninitialized]
mds/Server.cc:3879:14: note: 'pi' was declared here

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Capability.h: init num_revoke_warnings in constructor

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

configure.ac: fix warnings about unknown commands

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

client/Client.cc: cleanup code to silence cppcheck

Remove not needed goto statement and silence cppcheck
warning about possible null pointer deref of 'in'.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_objectstore_tool.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

KeyValueStore.cc: prefer !empty() over size() for emptiness check

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

perf_counters.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_context.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

client/Client.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/librados/aio.cc: release completion before exit

CID 1242020 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable my_completion going out of scope
 leaks the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/librados/aio.cc: delete AioCompletion* before return

CID 1242018: Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable aio_completion going out of scope leaks
 the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge remote-tracking branch 'origin/wip-10270' into master

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librados/IoCtxImpl.cc
src/librados/IoCtxImpl.h

Merge branch 'wip-fast-txn'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

osd/ECBackend: make sure localt uses tbl is ec txn does

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-object-corpus: drop compat with old ObjectStore::Transaction

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix Transaction::get_data_offset bug when map layout used

add following offset:
  sizeof(__u8) +      // encode struct_v
  sizeof(__u8) +      // encode compat_v
  sizeof(__u32);      // encode len

Change-Id: I5b6662eb42aeeae64baa8699da6ce65e0b1d58c3
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: add feature CEPH_FEATURE_OSD_TRANSACTION_MAY_LAYOUT (1ULL<<47)>>)

This feature determine whether we use tbl encode for transaction of use
the new map layout.

The primary uses peer_features to determine whether transaction should
use tbl, while the replica just follow the primary.

Change-Id: I92ca6e5b59bd1acde6007ad0dffc085be17accab
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: build fields for Transaction::iterator when tbl is used

When tbl is used (for compatibility), the Transaction::begin method need
to build all fields used by iterator. That includes: coll_index,
object_index, data_bl, op_bl, etc.)

Change-Id: I48ea74fec8d052f50da254a726a9c0dffead19bc
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

ceph_perf_objectstore: fix warning

Signed-off-by: Sage Weil <sage@redhat.com>

osd: Transaction::append & Transaction::swap

Finish append and swap for new Transaction encode/decode layout.

Since append will modify the op_bl now, we changed the order of append
and swap in ReplicatedBackend::sub_op_modify and
ReplicatedBackend::submit_transaction to avoid append call on op_t, so
the op_t can be encode in message.

Change-Id: I6fb421e0defdb092fb9732eef818e90291b039f5
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: new Transaction::iterator interface

This patch add new Transaction::iterator interface according to new
encode/decode layout. The new iterator give the whole Op struct in a
single decode_op method.

All ObjectStore Impl (FileStore/MemStore/KeyValueStore) is also changed
to use the new interface.

Change-Id: I1900a6ec302890df2c4357b071e4966c26d7f037
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: add encode/deocde impl for new layout

When use_tbl is true, Transaction::encode will give the same result as
before, while when use_tbl is false, Transaction::encode will use new
field and logic to encode and all related methods such as
get_encoded_bytes, get_data_offset will do the same.

Change-Id: Ia5864e489d47f37cf496fe3fb825b21977d2d938
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: new format for Transaction encode/decode

This patch add a new fixed size struct Transaction::Op to represent
all actions.

All coll and ghobject used by the transaction are keeped in two maps:
  coll:   map<coll_t, __le32> coll_index;
  object: map<ghobject_t, __le32> object_index;

And the Op struct use the map value(__le32) to refer coll and object,
so each coll and object is only need to encode once in the transaction.

Other variable-size fields(key/value/data) is encoded in bufferlist
data_bl.

Change-Id: I52b2fcd3217a6cb35de7b309a6dd74a99478feb2
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: Add Transaction::TransactionData for fast encode/decode

TransactionData wrap the following fields:
      __le64 ops;
      __le32 largest_data_len;
      __le32 largest_data_off;
      __le32 largest_data_off_in_tbl;
      __le32 pad; //make TransactionData multiple of uint64_t

This struct can encode/decode just by a single memcpy instead of many
encode/decode operations.

Change-Id: I56df78def43bd2b80b77be0825756e133434a6e6
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: remove unused Transaction fields

We don't need sobject and pool_override anymore since we don't need to
support anything older than dumpling.

Change-Id: I22c01d4b5c6bf99765bf6bc13aecadc997d6750c
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

Merge pull request #3272 from trociny/tell-mon-version

cli: make ceph tell mon.* version work

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3196 from tchaikov/shadow-cct

client, librados, osdc: do not shadow Dispatcher::cct

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3298 from majianpeng/fix5

TestLFNIndex.cc: For root, dont do permission operations.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3283 from dachary/wip-make-debs

debian: create a repository from sources

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3301 from ktdreyer/rm-tiobench

#10152: qa: drop tiobench suite

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3300 from ceph/wip-10412

client: fix use-after-free bug in unmount()

Reviewed-by: Sage Weil <sage@redhat.com>

qa: drop tiobench suite

The tiobench software has been abandoned upstream for years. Fedora and
Debian are no longer shipping the tiobench package, so we've had to
carry the package ourselves in the Ceph project, and we're trying to
slim down our dependencies where it makes sense to do so.

Nuke the tiobench suite.

http://tracker.ceph.com/issues/10152 Refs: #10152

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #3276 from ceph/wip-repop

Add MOSDRepOp and MOSDRepOpReply

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3299 from wonzhq/compile-warn

cleanup: fix a compile warning

Reviewed-by: Sage Weil <sage@redhat.com>

client: fix use-after-free bug in unmount()

If there is error, _release_fh() already outputs error message.
No need to print extra error message

Fixes: #10412
Signed-off-by: Yan, Zheng <zyan@redhat.com>

cleanup: fix a compile warning

warning: 'pi' may be used uninitialized in this function
[-Wmaybe-uninitialized]

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

TestLFNIndex.cc: For root, dont do permission operations.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3293 from ktdreyer/help-missing-keyutils-libs-devel

configure: show package name when libkeyutils is missing

Merge pull request #3294 from ceph/wip-fix-copyfrom

osd: fix copyfrom

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3226 from ceph/wip-10319-wusui

If trusty, use older version of qemu

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd: fix object_copy_data_t compat encoding

If the omap_data map is empty, we want an empty bufferlist.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3189 from ceph/wip-10329

Wip 10329

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/PG: print last_update_applied if != last_update

Signed-off-by: Sage Weil <sage@redhat.com>

Add MOSDRepOp and MOSDRepOpReply

Add the two new message type and change the corresponding code flow as well.

Basically the idea to have MOSDRepOp is to seperate subop(read/write)
out of other subop(pull/push,etc), so that we can cleanup some unused fields in
the message type, then save some encoding/decoding overhead.

The backward compatibility is also remian, if talking with old version OSD who
doesn't support osd_client_subop/subopreply, will fall back to osd_subop/subopreply.

Sage: rename MOSDClientSubOp -> MOSDRepOp

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3206 from ceph/wip-librados-init

librados: init last_objver

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

configure: show pkg names when libkeyutils is missing

Prior to this commit, when ./configure can't find libkeyutils, it would
bail out with a terse error message.

Some of the other library checks helpfully print the DEB and RPM package
names in parentheses. Add the DEB and RPM package names to the
libkeyutils check.

Reported-by: Pankaj Garg <Pankaj.Garg@caviumnetworks.com>
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3247 from ceph/wip-10422

mon: provide encoded canonical full OSDMap from primary

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

qa: add test_tell, which currently is used only for testing 'ceph tell mon version'.

The test requires at least two monitors run.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #3291 from majianpeng/fix2

test/bufferlist: Remove chmod operation.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test/bufferlist: For root, don't do permission operation for read_file
case.

For root user, it meet those error:
    test/bufferlist.cc:1880: Failure
    Value of: bl.read_file("testfile", &error)
      Actual: 0
    Expected: -13
    test/bufferlist.cc:1884: Failure
    Value of: bl.length()
      Actual: 8
    Expected: (unsigned)4
    Which is: 4
    test/bufferlist.cc:1886: Failure
    Value of: actual
      Actual: "ABC
    ABC
    "
    Expected: "ABC\n"
    Which is: "ABC

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

qa: refactor 'ceph -w' magic in test_mon_misc by introducing helper
functions ceph_watch_start and ceph_watch_wait so they can be reused
in other tests.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

mon: make ceph tell mon.* version work

Fixes: #6767
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #3261 from tchaikov/async-msgr

AsyncMessenger: a trivial refactory and a fix

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3286 from ceph/wip-10351

mount.ceph: avoid spurious error message

Reviewed-by: Sage Weil <sage@redhat.com>

mount.ceph: avoid spurious error message

/etc/mtab in most modern distributions is a symbol link to
/proc/self/mounts.

Fixes: #10351
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3284 from ktdreyer/doc-rados-preflight-os

doc: rm reference to old Ubuntu release

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: rm reference to old Ubuntu release

Remove the reference to "Ubuntu 12.10" since this is EOL.

Clarify that we only recommend Ubuntu LTS releases.

(Since this information has a tendancy to become stale, perhaps this
whole paragraph should be removed here and we should simply point at the
main OS Recommendations page.)

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

debian: create a repository from sources

Create a debian repository suitable for inclusion in
/etc/apt/sources.list, for the current distribution and using the git
clone from the current working directory.

It is meant to quickly provide teuthology with the packages to run a
job.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3223 from objoo/master

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

AsyncMessender: use IOV_MAX instead of IOV_LEN

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

AsyncMessenger: s/sended/sent/

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

AsyncMessenger: should retry in case of EINTR

* if EINTR is returned, do_send() should return 0 or `continue`.
* reword the comment a little bit

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

msg: fix factory error message

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_msgr: enable experiemntal async msgr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3219 from yuyuyu101/wip-10383

Wip 10383: Fix unittest_msgr with AsyncMessenger bug

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3279 from ceph/wip-mon-janitorial-list

src/doc: add monitor janitorial list

Merge pull request #3278 from mslovy/wip-y04

KeyValueStore:: do_transactions: clean up code which is never used

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore:: do_transactions: clean up code which is never used
clean up ops and bytes which is never used in do_transactions()

Signed-off-by: Ning Yao <zay11022@gmail.com>

src/doc: add monitor janitorial list

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3274 from ceph/wip-10341

tools: ceph-monstore-tool: refactor cli parsing

Reviewed-by: Sage Weil <sage@redhat.com>

client: fix quota signed/unsigned warning

client/Client.cc: In member function 'bool Client::is_quota_bytes_exceeded(Inode*, uint64_t)':
client/Client.cc:10393:66: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (quota->max_bytes && (rstat->rbytes + new_bytes) > quota->max_bytes)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3275 from liewegas/wip-experimental

mark some features experimental and warn accordingly

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

common: log which experimental features are enabled.

This will appear in the logs regardless of whether or not the features
get used/enabled via other config options.

Signed-off-by: Sage Weil <sage@redhat.com>

ms: mark async messenger experimental

Signed-off-by: Sage Weil <sage@redhat.com>

os: rename keyvaluestore-dev -> keyvaluestore; mark experimental

Use a clean name for keyvaluestore (no -dev suffix), but mark as
experimental to ensure users know what they are signing up for.

Signed-off-by: Sage Weil <sage@redhat.com>

common: add 'enable experimental data corrupting features'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3265 from yuyuyu101/coverify-fix

Event: Avoid illegal multiplication for memset

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3260 from nilamdyuti/wip-doc-ceph-object-gateway

doc: Updates information for SSL configuration in Ceph Object Gateway installation.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

tools: ceph-monstore-tool: validate start/stop vals

on commands that use 'start' and 'stop'/'end' values, we should validate
them and inform the user when 'stop' > 'start'.

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

tools: ceph-monstore-tool: output number of paxos versions dumped

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

tools: ceph-monstore-tool: raw map to stdout if outfile not specified

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

tools: ceph-monstore-tool: use subcmd parser helper

reduces the noise in the code

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

tools: ceph_monstore_tool: add sub-command parser helper function

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

tools: ceph-monstore-tool: a cli that works

Fixes: #10341
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Event: Avoid illegal multiplication for memset

Because "file_events" is "FileEvent" type and adding an integral value to
this pointer automatically scales that value by the size.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>