osd/ReplicatedPG:wipe off param op from generate_subop()

Signed-off-by: jimifm <hong.zhangoole@gmail.com>

Merge pull request #10806 from tchaikov/wip-17087

test/librados: rados_connect() should succeed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/librados: rados_connect() should succeed

the "rados_osd_op_timeout" setting does not apply to the initial connection.
this regression was introduced by 6c67073

Fixes: http://tracker.ceph.com/issues/17087
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10771 from xiexingguo/xxg-wip-fix-bluerockenv

os/bluestore: misc fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10435 from ceph/wip-debian-rbdmap-service

debian: replace SysV rbdmap with systemd service

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10725 from tchaikov/wip-17023

osd: subscribe for old osdmaps when pause flag is set

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10739 from xiexingguo/xxg-wip-osdmon-2016-08-15

osd/OSDMonitor: misc. cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10753 from tchaikov/wip-16982

osd: bail out if transaction size overflows

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10751 from ceph/wip-kefu-testing

rgw: do not try to encode or decode time_t and fix compiling warnings

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #10766 from liewegas/wip-bluestore-trivial

os/bluestore/BlueFS: disable buffered io

Merge pull request #10773 from xiexingguo/xxg-wip-bluestore-2016-08-18

os/bluestore: fix decoding hash of bnode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10780 from liewegas/wip-bluestore-finishers

os/bluestore: add multiple finishers to bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10592 from michelmno/forpull

LTTng-UST disabled for openSUSE

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10788 from ceph/wip-cmake

debian and cmake cleanups

Reviewed-by: Oleh Prypin <oleh@pryp.in>

Merge pull request #10762 from dillaman/wip-16654

librbd: cache was not switching to writeback after first flush

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10741 from minshenglin/remove_non_exists_options

doc: remove the description of deleted options

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/filestore/FileJournal: bail out if transaction is too large

if a transaction is too large to fit in the FileJournal's ring buffer,
we will wait. but if its size is larger than the max_size, it's likely
due to a bug or an invalid setting. in that case, we'd better fail
earlier.

Fixes: http://tracker.ceph.com/issues/16982
Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: bail out if transaction size overflows

with a large MOSDMap message, the transaction size could be greater than
UINT_MAX. so fail early with error messages.

Fixes: http://tracker.ceph.com/issues/16982
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix assert

This assert is used to guarantee that we don't access violation
but currently has no effect.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: use hex if we are running out of space

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: use bluefs_max_prefetch for log file prefetch

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix race condition during get_fs_usage()

file_map should be accessed under the protection
of the global lock.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10591 from xiexingguo/xxg-wip-osdmon-2016-08-05

mon: misc cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10516 from wjwithagen/wip-wjw-freebsd-pidfile

src/global/pidfile.cc: Assign elements in structures individually

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10451 from songbaisen/c0

tools: fix the core dump when get the crushmap do not exist

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9694 from renhwztetecs/renhw-wip-librados-iotest

test: add the necessary judgment

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

debian: drop python-argparse dependency

and bump up the dependency of python version to 2.7.
py2.6 support was only necessary for older (now unsupported) distro
versions.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: exclude python3* packages in dh_shlibdeps

since we are not using subvar of ${shlibs:Depends} in python3-* packages,
just exclude them in dh_shlibdeps.
this silences warnings like
```
warning: dpkg-gencontrol: package python3-cephfs: unused substitution
variable ${shlibs:Depends}
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: enable dh_python3 for python3 packages

so we can use subvars like ${python3:Depends} in debian/control.
this silences the warnings like:
```
warning: dpkg-gencontrol: Depends field of package python3-cephfs:
unknown substitution variable ${python3:Depends}
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix unmatched endforeach

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix decoding hash of bnode

We encode reversed bits of hash and then write it onto disk,
therefore we shall decode and then reserse to get it back.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10782 from BlaXpirit/rados-test-py3

qa/workunits/rados/test_python.sh: Allow specifying Python executable

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/rados/test_python.sh: Allow specifying Python executable
(through an environment variable)

Signed-off-by: Oleh Prypin <oleh@pryp.in>

Merge pull request #10777 from linuxbox2/rgw-token-base64

rgw ldap:  protect rgw::from_base64 from non-base64 input

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10772 from ceph/wip-cmake

cmake: recompile erasure src for different variants

Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: add multiple finishers to bluestore

- The single finisher of a bluestore can be a bottleneck
when using an SSD as a backend device. If too much load
is given to the single finisher, client-side IO latency
increases. So we add multiple finishers to the
bluestore, which shows better performance.

- 'bluestore_shard_finishers' option is added to
be able to configure wheather finsihers is multiple or
not.

- a finisher is selected according to the shard id of a
sequencer.

- the number of finishers is decided by
osd_op_num_shards.

Signed-off-by: Ilsoo Byun <ilsoo.byun@sk.com>

os/bluestore/BlueFS: rename buffered_io option

This is not a bluestore option.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: do direct io by default (not buffered)

Somnath is doing this in his testing with good results.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: break down onode vs bnode_map size in log

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10717 from liewegas/wip-bluefs-async-compaction

os/bluestore/BlueFS: async compaction

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10698 from liewegas/wip-bluestore-bluefs-allocation

bluestore: do not use freelist to track bluefs_extents

Merge pull request #10550 from tanghaodong25/in-memory-db

kv: fix some bugs in memdb

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #10775 from xiexingguo/xxg-wip-clean-storetest

test/store_test: tidy-up SyntheticWorkloadState class

Reviewed-by: Sage Weil <sage@redhat.com>

rgw ldap:  protect rgw::from_base64 from non-base64 input

Also adds unit tests for:
1. empty output from from_base64 (turns out to be harmless)
2. random and specific non-base64 and sort strings

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

debian: replace SysV rbdmap with systemd service

Stop shipping /etc/init.d/rbdmap in the Debian packages. Ship the
rbdmap.service systemd unit file instead.

The corresponding change has already been made for RPMs, in
9224ac2ad25f7d017916f58b642c0ea25305c3e5.

For Upstart-based systems (eg Ubuntu Trusty), the Debian packages
already contain rbdmap.conf.

(This gets us a tiny bit closer to being able to remove the rbdmap SysV
script from our tree entirely.)

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

mon/PGMonitor: fix potential underflow

h.h.size() could be 0(theoretically), so "h.h.size() - 1" could underflow.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/PGMonitor: drop redundant loading of dumpcontents

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/PGMonitor: skip scrub checking if we can

The PG-scrub checking may become expensive once the cluster
is big. Since the scrub-warn related options are defaulted to be off,
we should skip this checking when it is possible, which
is good for performance.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMonitor: drop redundant check of exists()

Which is aready included by the is_up() method.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMonitor: avoid duplicated assignment of oldest_map/newest_map

As build_incremental() has done this for us.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: avoid duplicated calls when determining osdmap boundary

We have caculdated the oldest/newest map epochs already,
so we can use it directly here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: get rid of snprintf when formatting output string

We can use the standard precision() and fixed method to achieve the same
effect.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: fix potential division-by-zero error

By checking pg_num against 0, we could avoid the risk of
division-by-zero error.

Also, we limit the loop to 50 or pg_num, whichever is smaller,
when trying to generate some pg_temps randomly.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: recompile erasure src for different variants

* instead of reusing the object libraries, we should recompile jerasure
  code for different plugin flavors like neon, sse3, sse4.
* do not version plugin so, as they are not supposed to be used by
  user directly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/store_test: avoid dynamic allocation of trans

For the sake of performance.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/store_test: drop redundant transaction of callbacks

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/store_test: fix incorrect prompt given by stash callback

"clone finish" belongs to C_SyntheticOnClone instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

kv: fix iterator bug in prev

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

kv: fill 'm_key_value' after 'seek_to_first' or 'seek_to_last'

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

kv: fix iterator invalidation in memdb.

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

kv: delete store after pg destructor is called in OSD shutdown.

Using memdb for bluestore kvbackend, we will hit segfault when we use
'kill' command to shut down osd process. After destructing pg, some
reference to bluestore will be release, but bluestore has been deleted
at this time.

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #10481 from dillaman/wip-16171

librbd: request exclusive lock if current owner cannot execute op

Reviewed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #10737 from weiqiaomiao/wqm-wip-rgw-core

rgw: fix radosgw daemon core when reopen logs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10728 from xiexingguo/xxg-wip-kill-bs-warn

bluestore: initialize csum_order properly

Reviewed-by: Sage Weil <sage@redhat.com>

librbd: cache was not switching to writeback after first flush

Fixes: http://tracker.ceph.com/issues/16654
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10703 from dillaman/wip-16855

rbd-mirror: improve split-brain detection logic

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10756 from linuxbox2/fix_gc_interval

rgw_file: restore local definition of RGWLibFS gc interval
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10686 from liewegas/wip-bluefs-remove

os/bluestore/BlueFS: do not op_file_update deleted files

Tested-by: Somnath Roy <somnath.roy@sandisk.com>

rgw_file: restore local definition of RGWLibFS gc interval

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10755 from SUSE/wip-cmake-use-pie

cmake: Add -pie to CMAKE_EXE_LINKER_FLAGS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10754 from SUSE/wip-cmake-fix-rgw-soversion

cmake: Fix mismatched librgw VERSION / SOVERSION

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Add -pie to CMAKE_EXE_LINKER_FLAGS

Without this, rpmlint (on openSUSE Tumbleweed) fails with:

ceph-radosgw.x86_64: E: non-position-independent-executable
(Badness: 10000) /usr/bin/radosgw
This executable must be position independent.  Check that it
is built with -fPIE/-fpie in compiler flags and -pie in linker
flags.

Signed-off-by: Tim Serong <tserong@suse.com>

cmake: Fix mismatched librgw VERSION / SOVERSION

Without this, rpmlint (on openSUSE Tumbleweed) fails with:

librgw2.x86_64: E: shlib-policy-name-error (Badness: 10000) librgw1
Your package contains a single shared library but is not named
after its SONAME.

It seems that the VERSION/SOVERSION mismatch results in the
creation of librgw.so.1 and librgw.so.2.0.0, whereas it should
be librgw.so.2 and librgw.so.2.0.0.

Signed-off-by: Tim Serong <tserong@suse.com>

rgw: silence a compiling warning

printf() a uint64 in a portable way (i.e. C98 compliant way).

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/rbd: silence compiling warnings

silence warnings like:
```
src/test/librbd/fsx.cc:1589:21: comparison between signed and unsigned
integer expressions [-Wsign-compare]
(offset + size > monitorstart &&
^
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/filestore: silence compiling warnings

silence warnings like
```
src/os/filestore/FileStore.h:55:27: comparison between signed and
unsigned integer expressions [-Wsign-compare]
^
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-1566-ga98ddf7/src/os/filestore/BtrfsFileStoreBackend.cc:269:29:
note: in expansion of macro ‘BTRFS_SUPER_MAGIC’
if (currentfs.f_type == BTRFS_SUPER_MAGIC && basest.st_dev != st.st_dev)
{
^
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10749 from mikulely/fix-rgw-client-cb

rgw: fix flush_read_list() error msg
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10747 from oliveiradan/master

gcc 6.1.1 complains about missing include: <random>. 4.8.3 does not c…

msg/async: fix compiler warnings

fix warnings like
```
comparison between signed and unsigned integer expressions
[-Wsign-compare]
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls_rgw: fix the compiler warning

fixes the warning of
```
warning:
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-1566-ga98ddf7/src/objclass/objclass.h:35:72:
format ‘%lu’ expects argument of type ‘long unsigned int’, but argument
6 has type ‘std::basic_string::size_type {aka unsigned int}’ [-Wformat=]
cls_log(level, " %s:%d: " fmt, __FILE__, __LINE__, ##__VA_ARGS__)
^
/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-11.0.0-1566-ga98ddf7/src/cls/rgw/cls_rgw.cc:477:7:
note: in expansion of macro ‘CLS_LOG’
CLS_LOG(20, "start_key=%s len=%lu", start_key.c_str(),
start_key.size());
^
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: fix flush_read_list() error msg

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

test: gcc 6.1.1 complains about missing include: <random>. 4.8.3 does not complain about it

Signed-off-by: Daniel Oliveira <doliveira@suse.com>

mon/LogMonitor: remove "host" and "fsid" from handle_conf_change()

Because these two fields are not used and tracked.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMonitor: drop redundant check of assert

Below here version can never be identical to osdmap.epoch,
otherwise we shall have returned above.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMonitor: drop redundant null check of max_failed_since

Because the "max_failed_since < failed_since" will catch both.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: add sanity check before setting use_gmt_hitset attribute

As "use_gmt_hitset" is a tier-specified attribute which is not applicable
for normal pools.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: do not try to encode or decode time_t

time_t is `signed int` on linux, which is ambiguous when encode() or decode() on i386-linux.
see also 8f9e7b0b.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10264 from yuyuyu101/wip-async-abstract-backend

msg/async: allow other async backend implementations

Reviewed-by: Kefu Chai <kchai@redhat.com>

test_msgr: ensure wait for ms_dispatch executed

Otherwise in slow runner, ms_dispatch may not executed when assert

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Event: remove global_centers assert

In tests we allow to reset EventCenter instance in the same CephContext,
so it may let global_centers->centers to set the same position multi times.

Signed-off-by: Haomai Wang <haomai@xsky.com>

test_async_networkstack: add networkstack tests

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: fix _conn_prefix racing when stopped

When the connection is lossy and enter fault, it will dispatch reset event.
If cleanup handler is executed as well as ms_handle_reset call mark_down,
it may exists racing for "cs". cleanup handler will reset "cs" but
_conn_prefix in mark_down will access "cs".

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: readd set_priority call

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: allow EventCenter::set_owner reentry

If daemonlize we need to respawn event threads, it need to allow set_owner again

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Stack: register fork watcher

Signed-off-by: Haomai Wang <haomai@xsky.com>

common/ceph_context: add prefork/postfork wathcher support

Because daemon() will termniate all existing threads, it will make something go
wrong.

So we want to add hook at CephContext, do something before/after fork.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Stack: disable smart thread spawn now

New async msgr runtime need to spawn threads when binding, but ceph-osd will
call daemon() after binding port. So we need to respawn threads if forked.

Then thread spawn delay will increase complexity for this change and it's
really a simple strategy which help less, we disable auto spawn now.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/PosixStack: increase log level

AsyncMessenger will try to loop the bind port range, so it will produce
some addr inuse errors which is not abnormal.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Event: remove event wakeup flag

Now only dispatch external event will wakeup event thread(previously
delete_time_event will call wakeup), we only need to use
"external_num_events" to indicate whether we have extra events.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Event: debug event address when dispatching and executing

Signed-off-by: Haomai Wang <haomai@xsky.com>