osd/ReplicatedPG: for writefull, offset is zero so replace offset w/ zero.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/Replicated: Using write_update_size_and_usage for  WRITEFULL.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/Replicated: First calc crc then call write_update_size_and_usage.

Because write_update_size_and_usage will change oi.size which cause
don't calc new crc.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/ReplicatedPG: For WRITEFULL replica object, only truncate if new
size less than old size(only truncate to new size)

If new size larger or equal old size, no need do truncate.
It can diretcly overwrite.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4951 from SUSE/wip-432-missing-divergent

osd_types.cc: replace count with iterator in add_next_event function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5074 from majianpeng/mds-small-max-event

mds: fix mds crash when mds_max_log_events smaller.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4166 from sponce/wip-10759

tools: rados cli --striper

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4928 from xiaoxichen/wip-cleanup-key

os/{LevelDB,Rocks}Store: don't keep key_list in transaction

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5113 from tchaikov/wip-11840-reject-multiple-tiering

mon: disallow adding a tier on top of another tier

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mon: disallow adding a tier on top of another tier

multiple tiering is not supported at the moment

Fixes: #11840
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5112 from tchaikov/wip-12120

fix a couple build warnings

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: return EINVAL if iovcnt < 0 for p{read,write}v()

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools: fix deprecated warning

s/set_op_flags/set_op_flags2/

Fixes: #12120
Signed-off-by: Kefu Chai <kchai@redhat.com>

crush/CrushTester: fix signed/unsigned warning

Fixes: #12120
Signed-off-by: Sage Weil <sage@redhat.com>

client: fix signed/unsigned warnings in preadv code

Fixes: #12120
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5107 from yuyuyu101/wip-fix-async-14

AsyncConnection: Move sign_message to write_message

Merge pull request #5070 from ceph/wip-4161

mds: store layout on header object

AsyncConnection: Move sign_message to write_message

sign_message require correct connection feature which may wrong when preparing
message

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5105 from Oxygen31/master

doc : missing link in "quick_guide.rst"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc : missing link in "quick_guide.rst"

Fixes: #12009
Signed-off-by: Germain CHIPAUX <germain.chipaux@gmail.com>
Signed-off-by: Thomas LAUMONDAIS <thomas.laumondais@gmail.com>
Signed-off-by: Pierre CHAUMONT <pierre.chaumont31@gmail.com>
Signed-off-by: Benjamin TROQUEREAU <benjamin.troquereau@gmail.com>
Signed-off-by: Benjamin SESIA <benjamin.sesia@gmail.com>

Merge pull request #5084 from JevonQ/fix-issue-12123

tests: fix segfault issue in preadv/pwritev tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: change mds_log_max_segments type from int to unsigned.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mds: fix mds crash when mds_max_log_events smaller.

If 'mds max log events' small, it met this bug:

mds/journal.cc: 155: FAILED assert(ls != this)
 ceph version 9.0.1-1117-gb0dc971
(b0dc97198a45791c8b9661d4be8e531e63202005)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x16ef7bf]
 2: (LogSegment::try_to_expire(MDS*,
C_GatherBuilderBase<MDSInternalContextBase, MDSGather>&, int)+0x1925)
[0x15475e5]
 3: (MDLog::try_expire(LogSegment*, int)+0x66) [0x14c50a0]
 4: (MDLog::trim(int)+0xd67) [0x14c4511]
 5: (MDS::tick()+0x265) [0x1146e0b]
 6: (MDS::C_MDS_Tick::finish(int)+0x32) [0x11693b2]
 7: (Context::complete(int)+0x27) [0x11611a7]
 8: (MDSInternalContextBase::complete(int)+0x1a4) [0x14977fa]
 9: (SafeTimer::timer_thread()+0x37a) [0x16e28fa]
 10: (SafeTimerThread::entry()+0x1c) [0x16e3c56]
 11: (Thread::entry_wrapper()+0xa8) [0x16df0ee]
 12: (Thread::_entry_func(void*)+0x18) [0x16df03c]
 13: /lib64/libpthread.so.0() [0x30c9c07555]
 14: (clone()+0x6d) [0x30c9901f3d]

Check segments.size() > 1 can trim segment.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

tests: fix segfault issue in preadv/pwritev tests

Previously copied the sample data to the iov buffers not yet allocated.
Now allocate buffers before writing data into them.

Fixes: #12123
Signed-off-by: Jevon Qiao <qiaojianfeng@unitedstack.com>

Merge pull request #5094 from SUSE/wip-sharutils-dupe

ceph.spec.in: remove duplicate BuildRequires: sharutils

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5082 from ceph/wip-fedora-babeltrace

ceph.spec.in: fix babeltrace handling on Fedora

Reviewed-by: Ira Cooper ira@samba.org
Reviewed-by: Boris Ranto branto@redhat.com

Merge pull request #5098 from PierreChaumont/master

doc: homogenize Librados (Python) documentation's code samples

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: homogenize Librados (Python) documentation's code samples

I put in doc/rados/api/python.rst line numbers on every code samples in
order to make them look all the same.

Fixes: #12174
Signed off: Pierre Chaumont pierre.chaumont31@gmail.com

ceph.spec.in: remove duplicate BuildRequires: sharutils

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #4957 from SUSE/wip-python-flask

ceph.spec.in: fix python-flask dependency for SUSE

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: fix python-flask dependency for SUSE

In SLE and openSUSE, the package is called python-Flask with an upper-case F.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5086 from SUSE/wip-12173

logrotate.conf: fixes for systemd

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

logrotate.conf: fixes for systemd

Before this patch, the command 'logrotate -f /etc/logrotate.d/ceph'
was generating an error "Failed to reload ceph.target: Job type reload is not
applicable for unit ceph.target".

Before we issue systemctl reload, check that there is at least
one active ceph-* service. (The hyphen is significant.)

Since we use grep, make the grep package a dependency.

http://tracker.ceph.com/issues/12173 Fixes: #12173

Signed-off-by: Tim Serong <tserong@suse.com>
Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5091 from yehudasa/wip-multipart-parts-limit

rgw: Multipart Upload: Support to configure and enforce no of parts a…

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Multipart Upload: Support to configure and enforce no of parts allowed

Fixes: #12146
Config parameter added for no of parts limit in multipart upload and checked while
completing multipart upload

Signed-off-by: Abhishek Dixit <dixitabhi@gmail.com>

Merge pull request #4970 from SUSE/wip-12034-master

ceph.spec.in: python-argparse only in Python 2.6

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: python-argparse only in Python 2.6

argparse is a widely-used Python module for parsing command-line arguments.
Ceph makes heavy use of Python scripts, both in the build environment and on
cluster nodes and clients.

Until Python 2.6, argparse was distributed separately from Python proper.
As of 2.7 it is part of the Python standard library.

Although the python package in a given distro may or may not Provide:
python-argparse, this cannot be relied upon.

Therefore, this commit puts appropriate conditionals around Requires:
python-argparse and BuildRequires: python-argparse. It does so for Red
Hat/CentOS and SUSE only, because the last Fedora version with Python 2.6
was Fedora 13, which is EOL.

argparse is required by both the ceph and ceph-common packages, but since ceph
requires ceph-common, the argparse Requires and BuildRequires need only appear
once, under ceph-common.

http://tracker.ceph.com/issues/12034 Fixes: #12034

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5035 from yuyuyu101/wip-async-fix-13

AsyncConnection: Fix out-of-sequence problem introduced by send/write separated logic

Merge pull request #5001 from ceph/wip-11985

#11985 Fixes to MDLog (and MDS) shutdown

Merge pull request #5077 from SUSE/wip-spec-comment-clarify

ceph.spec.in: clarify two important comments

ceph.spec.in: clarify two important comments

First, the terms "common" and "specific" are vague. Second,
"common" can easily be confused with the ceph-common subpackage.

Fix this by rephrasing to "distro-unconditional dependencies" and
"distro-conditional dependencies", respectively.

Third, move the "distro-unconditional dependencies" header so it
is above the part where these dependencies are actually defined.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5033 from wuxingyi/fixuserquota

rgw: fix empty json response when getting user quota

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4942 from SUSE/wip-11991

rpm: add missing Java conditionals

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

mds: fix MDLog shutdown process

We must join threads before completing ::shutdown,
because otherwise these threads might try to use
torn-down resources like the objecter.

The replay/recovery threads may be blocking on
journaler calls like wait_for_readable, so we
must signal them using Journaler::shutdown.  In
order for that to be safe, we must also protect
the assignment of ::journaler from the threads
using the mds_lock.

Fixes: #11985
Signed-off-by: John Spray <john.spray@redhat.com>

mds: drop MDSIOContext on mds->stopping

`stopping` is true once the MDS has entered suicide().  During
this phase, any MDSIOContexts that are called (e.g. from objecter,
journaler) become no-ops.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: refine shutdown, add ::stopping

Add a ::stopping flag, set at start of suicide(),
that other contexts must inspect after taking
mds_lock.

This guards against the possibility of multiple
threads entering suicide, and more generally
against the possibility of other procedures
starting while we're in the middle of shutting down.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc/Journaler: add a shutdown() method

Because consumers may be blocked on the on_readable
condition, it is necessary for Journaler to have
an explicit shutdown method that fires the completion

Signed-off-by: John Spray <john.spray@redhat.com>

common/Thread: print msg on join errors

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add a ceph_file_layout wrapper for dencoder

So that we can decode it from tests.  This should go away
at some point when encoding of these raw-encoded structures
is modernized.

Signed-off-by: John Spray <john.spray@redhat.com>

test/ceph-dencoder: add boilerplate

... mainly just for the modelines.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: store layout on header object

This is surprisingly simple because we were already redundantly
calling store_backtrace whenever the layout changed!  That was
a side effect of the way add_old_pool is handled, the backtrace
version is bumped to latest even if the "old" pool is the
current one.

The upshot is that if we accept the existing behaviour of
also unnecessarily updating the 'parent' xattr, keeping
the new 'layout' xattr update requires no new dirty flags.  This
is a twitchy enough behaviour that new tests are needed to guard
against regressions though.

Fixes: #4161
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5058 from SUSE/wip-fix-release-notes-blooper

release-notes.txt: fix version number in section heading

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5073 from ceph/wip-12105

mds: update CInode::oldest_snap during migration

Reviewed-by: John Spray <john.spray@redhat.com>

mds: update CInode::oldest_snap during migration

Fixes: #12105
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #5036 from ceph/wip-dirfrag-msgs

mds: tweak dirfrag asok messages

ceph.spec.in: rm reference to EOL Fedoras (< 20)

Fedora 19 is EOL, so there's no need to refer to Fedoras less than 20.
"%{fedora}" effectively implies "Fedora 20 and above".

Reported-by: Ira Cooper <ira@samba.org>
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: package rbd-replay-prep on all Fedoras

This reverts the change in commit
85517d611b7bf4cb6cbffcd2c65303be0d038264. Since we BuildRequire:
libbabeltrace-devel, autoconf will see that babeltrace is available
during the build, and make will build/install the rbd-replay-prep
utility.

This change also simplifies Fedora selection logic, because Fedora 19 is
EOL, so "%{fedora}" implies "Fedora 20 and above".

Reported-by: Ira Cooper <ira@samba.org>
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4981 from ceph/wip-fedora-build-fixes

Fix various issues with fedora (f21+) builds

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4898 from SUSE/wip_ceph_spec_sharutils

ceph.spec.in:BuildRequires sharutils

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

ceph.spec.in:BuildRequires sharutils

The uudecode binary is used to build Java-related components, and
uudecode is provided by the sharutils package on all supported
RPM platforms. When building with "--without=cephfs_java",
sharutils is not needed.

Thanks to Nathan Cutler <ncutler@suse.cz> for going into the
details with me.

On OBS without this patch we get the error message:

[  170s] -----------------------------------------------------------------
[  170s] ----- building ceph.spec (user abuild)
[  170s] -----------------------------------------------------------------
[  170s] -----------------------------------------------------------------
[  170s] + exec rpmbuild -ba --define '_srcdefattr (-,root,root)' --nosignature --define 'disturl obs://build.opensuse.org/home:osynge:ceph:wip:wip_obs_fedora/Fedora_20/c0bbbc1e62228ca956ac3d367edc4fba-master' /home/abuild/rpmbuild/SOURCES/ceph.spec
[  170s] error: Failed build dependencies:
[  170s]    sharutils is needed by ceph-1:2+git.1435043747.c1bd02c-1.1.x86_64

With this patch we can build fedora 22 and fedora 20 rpms fine.

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #5042 from ceph/wip-final-kw

mds/Migrator: don't use c++11 kw 'final' as var name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5066 from ceph/wip-gitignore-add-ceph_perf_msgr

src/.gitignore: add ceph_perf_msgr_{server,client}

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4840 from tchaikov/wip-11833-crush-dump-tree

mon: add an "osd crush dump tree" command

Reviewed-by: Min Chen <minchen@ubuntukylin.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

src/.gitignore: add ceph_perf_msgr_{server,client}

Signed-off-by: Kefu Chai <kchai@redhat.com>

Fixed inclusion of ceph-helpers file in tests after it was moved to qa/workunits

Merge pull request #4965 from xinxinsh/wip-12025

mon/OSDMonitor : error out if pgp_num > pg_num

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4923 from dachary/wip-make-check-verbosity

tests: display the output of failed make check runs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5032 from guangyy/wip-cleanup

osd: remove duplicate checks

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5064 from vuhuong/wip-xio

xio: sync works

Reviewed-by: Kefu Chai <kchai@redhat.com>

move pgp_num, pg_num check to prepare_new_pool method

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

xio: fix to work with commit 626360aa

Fix xio to work with commit 626360aa
"msg, ceph_osd: Support feature bits for all message type's local connection"

Signed-off-by: Vu Pham <vu@mellanox.com>

xio: fix to work with the merge of pull request #4707

fix xio to work with the merge of pull request
" Removed unnecessary inclusion of iostream in several files #4707"

Signed-off-by: Vu Pham <vu@mellanox.com>

xio: sync to work with accellio v1.4

latest master's HEAD - tag v1-4

Signed-off-by: Vu Pham <vu@mellanox.com>

Merge pull request #4747 from ceph/wip-damaged-fixes

MDS metadata damage handling

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

release-notes.txt: fix version number in section heading

From context it is obvious that this section is about upgrading from Giant, not
Firefly, so change the version number to match Giant.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5054 from SUSE/wip-release-notes-syntax

doc/release-notes.rst: add missing word to complete sentence

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/release-notes.rst: add missing word to complete sentence

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tests : Fixed radosstriper tests to introduce the needed calls to setup and teardown

In particular port 7112 was replaced by 7113 for the monitor as 7112 is taken by test-erasure-eio now
And calls to setup and teardown have been added around the testing shell code

tests : Fixed broken Makefiles after integration of lttng into rados.

The symbols of lttng were exposed twice leading to error when launching executables using libradosstriper.

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>

tests: fixed environment for osd-class-dir

osd-class-dir was not set when activating osds in the test environment leading to failures with 'operation not supported' message when trying to lock objects

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>

rados: Added --striper option to the command line

Fixes: #10759
the new --striper option allows to use the striping API provided by libradosstriper.

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>

Merge pull request #4792 from kylinstorage/wip-writeback-throttling-for-cache-tiering

Wip writeback throttling for cache tiering

This patch is to do write back throttling for cache tiering, which is similar to what the Linux kernel does for  page cache write back.  A paramter 'cache_target_dirty_high_ratio'  (default 0.6) is introduced as the high speed flushing threshold, while leave the 'cache_target_dirty_ratio' (default 0.4) to represent the low speed  threshold. The flush speed is controlled by limiting the parallelism of flushing. The maximum parallelism under low speed is half of the parallelism under high speed. If there is at least one PG such that the dirty ratio beyond the high threshold, full speed mode is entered; If there is no PG such that dirty ratio beyond the low threshold, idle mode is entered; In other cases, slow speed mode is entered.

Signed-off-by: Mingxin Liu <mingxinliu@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>
Suggested-by: Nick Fisk <nick@fisk.me.uk>
Tested-by: Kefu Chai <kchai@redhat.com>

rgw: fix empty json response when getting user quota

Fixes: #12117
Signed-off-by: wuxingyi <wuxingyi@letv.com>

Merge pull request #4290 from XinzeChi/wip-filestore-lat

osd: dout more log info to track io request

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5049 from ceph/wip-librbd-reduce-tests

librbd: reduce unit test execution time

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3973 from XinzeChi/wip-scrub-pending

osd: check pending or active scrub before sched_scrub

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4847 from ceph/wip-11661

osd: Improve feature bit handling and remove ancient checking including CHUNKY_SCRUB

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

tests: reduce duplicate librbd unit test case coverage

The test run for RBD_FEATURES=13 is fully covered by
RBD_FEATURES=45 and the pre deep-flatten cases are covered
by RBD_FEATURES=0 and 1.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

lockdep: do not automatically collect all backtraces

It is expensive to collect backtraces every time a lock is
checked in order to provide cycle backtraces.  The backtraces
can be forced on for specific locks or globally via the new
config option "lockdep_force_backtrace".

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5041 from oritwas/wip-12095

rgw: force content_type for swift bucket stats request

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

tests: move librbd valgrind test to teuthology

This test can take up to 20 minutes to execute, which is an order of
magnitude longer than other unit tests.  This is slowing down the
build bots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: force content_type for swift bucket stats request

Fixes: 12095
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

mds/Migrator: don't use c++11 kw 'final' as var name

If for no other reason than that it looks weird
in an editor.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: tweak dirfrag asok messages

Signed-off-by: John Spray <john.spray@redhat.com>

AsyncConnection: Fix msgr send bytes perf counter statistic

prepare_send_message may be called twice, we need to move to write_message

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

AsyncConnection: set out seq for message when sending ready

Otherwise it will result in higher priority but later message to be issued
firstly, then peer will receive out-of-order message seq

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test_msgr: Random send different priority message to test sequence

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

osd: remove duplicate checks

Within pg_has_reset_since, it checkes if the PG is being deleted or not,
no need a duplicate check

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

ceph_osd: Add required feature bits related to this branch to osd_required mask

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: CEPH_FEATURE_INDEP_PG_MAP feature now required

Feature present since at least the Dumpling release.
A later commit will add it to the osd_required mask

Signed-off-by: David Zafman <dzafman@redhat.com>