ceph.spec.in: add missing -%{release}

We have it everywhere else and I can't think of any reason why
it should be omitted here.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #4976 from t-miyamae/wip-mshec-fixlogic2

erasure code: fix shec's recovery issues found in teuthology test

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4813 from dachary/wip-11667-erasure-code-add-osd

tests: verify erasure code read / write after remapping

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: verify erasure code read / write after remapping

Implement the following scenario:

 * create an erasure coded pool and write an object

 * mark two osd used to store the object out
 * wait until the PG is clean
 * read the object and compare it with the original

 * mark the two osd back in
 * wait until the PG is clean
 * read the object and compare it with the original

The scenario is run with the jerasure plugin only because the code path
involved does not vary depending on the erasure code plugin.

http://tracker.ceph.com/issues/11667 Fixes: #11667

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh get_osds with no trailing whitespace

Trim the trailing whitespace of the get_osds output to help with
comparisons.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: improve shell output readability

Modify the test traces to include the file name in addition to the
function and line name. It makes it easier to locate the faulty line
without going back to the test name.

Format the trace lines to be emacs friendly (filename:lineno) so that
C-x ` or C-c C-c jumps to the right file and the right line when running
the test with M-x compile.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

erasure code: shec's gtest for minimum_to_decode() arguments

TestErasureCodeShec_arguments.cc added here verifies shec's fixed logic
to pass Teuthology test.

Signed-off-by: Shotaro Kawaguchi <kawaguchi.s@jp.fujitsu.com>

erasure code: fix shec's recovery issues found in teuthology test

Fixed shec's recovery issues caused by unexpected arguments in
minimum_to_decode() / decode() functions.
We misunderstood as follows.

 (1) want_to_read and available_chunks do not share osds.
 (2) osds that are not included in want_to_read are always included
     in available_chunks.

Signed-off-by: Takanori Nakao <nakao.takanori@jp.fujitsu.com>

erasure code: move shec.cc into ErasureCodeShec.cc

Signed-off-by: Takanori Nakao <nakao.takanori@jp.fujitsu.com>

tests: ceph-helpers.sh reduce kill_daemon verbosity

When a test fails, the script returns immediately and kill_daemon
function is called to cleanup. It is quite verbose and requires
scrolling hundreds of lines back to find the actual error
message. Turn off the shell trace to reduce the verbosity and improve
error output readability.

The kill_daemon cannot just turn off set -x because it may be called by
a test, not just at the end of the run. Instead the kill_daemon function
checks if tracing is activated and temporarily disables it.

Also get rid of the find standard error that commonly happens when
kill_daemon is called to verify there are no leftovers and the test
directory does not exist.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5109 from ceph/wip-rgw-orphans-2

Wip rgw orphans 2

Merge pull request #5151 from majianpeng/make-morereaable-dump-ops

common/TrackedOp: Make get_duration get correctly value.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5166 from yuyuyu101/wip-async-fix-17

AsyncConnection: Only prepare message when it support fast dispatch

Merge pull request #5157 from rajukv/wip-xio-rajukv

XIO: Add missing fastpath events to OSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

AsyncConnection: Only prepare message when it support fast dispatch

Not all messages support reencode now, we just avoid them prepare firstly when
call "send_message"

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

common/TrackedOp: Make get_duration get correctly value.

For get_duration(), it should return the duration of Op. For ops which in
flight, the duration is the time current time subtrace the
initiated_time. For the completed Ops, it used the last event(done)
subtrace the initiated_time.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4837 from xinxinsh/wip-kv-zero

in _zero() function, we should remove strips to minimize keyvalue DB update

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

XIO: Add missing fastpath events to OSD

Without this incoming fastpath messages will be dropped at OSD due to missing OSD::Session

Signed-off-by: Raju Kurunkad <raju.kurunkad@sandisk.com>

generic check return code of get operation

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #5138 from majianpeng/cleanup

common/TrackedOp: code cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5148 from tchaikov/wip-fix-facility-syslog

mon/LogMonitor: use the configured facility if log to syslog

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge remote-tracking branch 'origin/next'

Merge pull request #5142 from cernceph/wip-nukememmax

mds: nuke the unused mds_mem_max option

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5154 from ceph/wip-rbd-queue-depth

rbd: recognize queue_depth option

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5146 from ceph/wip-12215-next

librbd: don't attempt to invalidate an object map in R/O mode

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5145 from ceph/wip-12214-next

librbd: fix TestObjectMap.InvalidateFlagInMemoryOnly test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5131 from ceph/wip-12069-next

librbd: propagate errors from image close

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5108 from ceph/wip-12079

Wip 12079

rgw: conversion tool to fix broken multipart objects

Fixes: #12079
Broken multipart objects: multipart objects that created on 0.94.2
and that start with underscore have a bad locator on their tail objects.
This extends the tool that was needed for older issue we've had with
hammer multipart objects (that start with underscore). The same usage
applies:

$ radosgw-admin bucket check --check-head-obj-locator \
                             --bucket=<bucket> [--fix]

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5153 from oritwas/fix-cmake-3

Add libradosstriper to cmake

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5149 from tchaikov/wip-fix-erasure-hides-parse

erasure-code: do not hide overloaded ErasureCode::parse()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rbd: recognize queue_depth option

.. and also stress the difference between libceph and rbd mapping
options.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #5115 from ceph/wip-12037

Add unit/tox structure for ceph python script

Reviewed-by: Travis Rhoden <trhoden@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Add libradosstriper to cmake

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

erasure-code: do not hide overloaded ErasureCode::parse()

* this change fixes the warning from clang:
  ErasureCodeJerasure::parse' hides overloaded virtual function
  [-Woverloaded-virtual]
* some erasure codecs' ErasureCode::parse() rewrites the profile
  using the default values when parsing it if the corresponding
  items are not specified. and we don't call ErasureCode::parse()
  via its children's references. so no need to make it a virtual
  function.
* and ErasureCode::parse() is used as a helper function by its
  children, so make it `protected`
* and parse() in ErasureCode's children is but a helper function
  called by ctor, descendants' parse() and init(). so make them
  protected or private accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/TrackedOp: check tracking_enabled for event "initiated/done".

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mon/LogMonitor: use the configured facility if log to syslog

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/TrackedOp: clean up code make look good.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4394 from majianpeng/tools-copyget-flags

add fadvise flags for rados/rbd

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4937 from majianpeng/replicated-misc-fix

ReplicatedPG: small bugs and clean up

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5141 from XinzeChi/wip-crush-rule

mon/PGMonitor: bug fix pg monitor get crush rule

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: don't attempt to invalidate an object map in R/O mode

The ImageWatcher is not initialized when in R/O mode, which
resulted in a NULL pointer dereference.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: read op should return number of bytes read

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: fixed TestObjectMap.InvalidateFlagInMemoryOnly

librados and librados_test_stub return different result codes
for a read full object operation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5135 from yuyuyu101/wip-async-fix-16

AsyncConnection: Fix incorrect sign message behavior

Merge pull request #5144 from tchaikov/wip-fix-tools-rados

tools: fix segfault tools/rados introduced by 0f7aeee

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

obj_bencher: remove trailing space

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rados: fix the segfault introduced in 0f7aeee

prefix and run_name should not be initialized by a temporary string

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5134 from stiopaa1/ThreadAddConsts

common/Thread: added const to 2 functions

Reviewed-by: Loic Dachary <ldachary@redhat.com>

AsyncConnection: Fix incorrect sign message behavior

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5128 from ceph/wip-fix-doc

doc: fix the build

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mon/PGMonitor: bug fix pg monitor get crush rule

when some rules have been deleted before, the index in array of crush->rules
is not always equals to crush_ruleset of pool.

Fixes: #12210
Reported-by: Ning Yao <zay11022@gmail.com>
Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #5071 from cxwshawn/master

obj_bencher: check run_name and prefix for empty string instead of NULL

Reviewed-by: Kefu Chai <kchai@redhat.com>

obj_bencher: check run_name and prefix for empty string instead of NULL

change aio_bench and clean_up parameter const char * to const std::string & format.
In rest_bench.cc, aio_bench used run_name.c_str(), so this format will always be empty
string not NULL, so the condition statement
const std::string run_name_meta = (run_name == NULL ? BENCH_LASTRUN_METADATA : std::string(run_name));
is wrong!

test fix:
    before:
        ./rest-bench --seconds 1 -t 2 -b 100 write --api-host=radosgw.com --bucket=test_rm --access-key=FTL7TSJAGXX5KKDQHMJM  --secret=123456879
        use s3cmd ls s3://test_rm , we can a lot of objects in this bucket, objects are not cleaned up.
    after changes, do the same procedure, objects are cleaned up.

Signed-off-by: shawn chen <cxwshawn@gmail.com>

common/Thread: added const to 2 functions

Changed get_thread_id and am_self to const member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

rbd: explicitly close images to check for unexpected errors

While errors shouldn't occur during image close, it is possible that
they could occur.  In this case, the user should be informed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind: RBD close can now raise an exception on close

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: closing images now returns a result code

Added new librbd::Image::close method to allow checking the close result
when using the C++ librbd library.  rbd_close is no longer hard-coded to
return 0.

Fixes: #12069
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mds: nuke the unused mds_mem_max option

mds mem max has been a noop since argonaut. Nuke it because it
is misleading.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

doc: indent warning messages in add-or-rm-osds.rst

otherwise they are rendered as normal text

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix the link in dev/quick_guide

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: for writefull, offset is zero so replace offset w/ zero.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/Replicated: Using write_update_size_and_usage for  WRITEFULL.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/Replicated: First calc crc then call write_update_size_and_usage.

Because write_update_size_and_usage will change oi.size which cause
don't calc new crc.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #5125 from majianpeng/print-blocked-op

mon/PGMonitor: Make blocked Op message more readable.

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/PGMonitor: Make blocked Op message more readable.

W/O this patch:
root@dev:/var/log/ceph# ceph health detail
HEALTH_WARN 1 pgs stuck unclean; 2 requests are blocked > 32 sec; 1 osds
have slow requests; recovery 5/115 objects degraded (4.348%); recovery
1/38 unfound (2.632%); too few PGs per OSD (15 < min 30)
pg 2.1 is stuck unclean for 899.708271, current state active, last
acting [2,3,0,1]
1 ops are blocked > 1048.58 sec
1 ops are blocked > 262.144 sec
1 ops are blocked > 1048.58 sec on osd.2
1 ops are blocked > 262.144 sec on osd.2
1 osds have slow requests
recovery 5/115 objects degraded (4.348%)
recovery 1/38 unfound (2.632%)
too few PGs per OSD (15 < min 30)

W/ this patch:
ceph health detail
HEALTH_WARN 1 pgs stuck unclean; 2 requests are blocked > 32 sec; 1 osds
have slow requests; recovery 5/115 objects degraded (4.348%); recovery
1/38 unfound (2.632%); too few PGs per OSD (15 < min 30)
pg 2.1 is stuck unclean for 427.103877, current state active, last
acting [2,3,0,1]
1 ops are blocked > 524.288 sec on osd.2
1 ops are blocked > 131.072 sec on osd.2
1 osds have slow requests
recovery 5/115 objects degraded (4.348%)
recovery 1/38 unfound (2.632%)
too few PGs per OSD (15 < min 30

Later messages looks better.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/ReplicatedPG: For WRITEFULL replica object, only truncate if new
size less than old size(only truncate to new size)

If new size larger or equal old size, no need do truncate.
It can diretcly overwrite.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4951 from SUSE/wip-432-missing-divergent

osd_types.cc: replace count with iterator in add_next_event function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5074 from majianpeng/mds-small-max-event

mds: fix mds crash when mds_max_log_events smaller.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4166 from sponce/wip-10759

tools: rados cli --striper

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4928 from xiaoxichen/wip-cleanup-key

os/{LevelDB,Rocks}Store: don't keep key_list in transaction

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5113 from tchaikov/wip-11840-reject-multiple-tiering

mon: disallow adding a tier on top of another tier

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

test: ignore symlinked ceph.py file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

test: ignore symlinked ceph_disk.py file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph.in: linter cleanup, remove unused imports

Signed-off-by: Alfredo Deza <adeza@redhat.com>

test: create python test files for ceph cli

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mon: disallow adding a tier on top of another tier

multiple tiering is not supported at the moment

Fixes: #11840
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5112 from tchaikov/wip-12120

fix a couple build warnings

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: return EINVAL if iovcnt < 0 for p{read,write}v()

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools: fix deprecated warning

s/set_op_flags/set_op_flags2/

Fixes: #12120
Signed-off-by: Kefu Chai <kchai@redhat.com>

crush/CrushTester: fix signed/unsigned warning

Fixes: #12120
Signed-off-by: Sage Weil <sage@redhat.com>

client: fix signed/unsigned warnings in preadv code

Fixes: #12120
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5107 from yuyuyu101/wip-fix-async-14

AsyncConnection: Move sign_message to write_message

Merge pull request #5070 from ceph/wip-4161

mds: store layout on header object

AsyncConnection: Move sign_message to write_message

sign_message require correct connection feature which may wrong when preparing
message

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5105 from Oxygen31/master

doc : missing link in "quick_guide.rst"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc : missing link in "quick_guide.rst"

Fixes: #12009
Signed-off-by: Germain CHIPAUX <germain.chipaux@gmail.com>
Signed-off-by: Thomas LAUMONDAIS <thomas.laumondais@gmail.com>
Signed-off-by: Pierre CHAUMONT <pierre.chaumont31@gmail.com>
Signed-off-by: Benjamin TROQUEREAU <benjamin.troquereau@gmail.com>
Signed-off-by: Benjamin SESIA <benjamin.sesia@gmail.com>

Merge pull request #5084 from JevonQ/fix-issue-12123

tests: fix segfault issue in preadv/pwritev tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: change mds_log_max_segments type from int to unsigned.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mds: fix mds crash when mds_max_log_events smaller.

If 'mds max log events' small, it met this bug:

mds/journal.cc: 155: FAILED assert(ls != this)
 ceph version 9.0.1-1117-gb0dc971
(b0dc97198a45791c8b9661d4be8e531e63202005)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x16ef7bf]
 2: (LogSegment::try_to_expire(MDS*,
C_GatherBuilderBase<MDSInternalContextBase, MDSGather>&, int)+0x1925)
[0x15475e5]
 3: (MDLog::try_expire(LogSegment*, int)+0x66) [0x14c50a0]
 4: (MDLog::trim(int)+0xd67) [0x14c4511]
 5: (MDS::tick()+0x265) [0x1146e0b]
 6: (MDS::C_MDS_Tick::finish(int)+0x32) [0x11693b2]
 7: (Context::complete(int)+0x27) [0x11611a7]
 8: (MDSInternalContextBase::complete(int)+0x1a4) [0x14977fa]
 9: (SafeTimer::timer_thread()+0x37a) [0x16e28fa]
 10: (SafeTimerThread::entry()+0x1c) [0x16e3c56]
 11: (Thread::entry_wrapper()+0xa8) [0x16df0ee]
 12: (Thread::_entry_func(void*)+0x18) [0x16df03c]
 13: /lib64/libpthread.so.0() [0x30c9c07555]
 14: (clone()+0x6d) [0x30c9901f3d]

Check segments.size() > 1 can trim segment.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

tests: fix segfault issue in preadv/pwritev tests

Previously copied the sample data to the iov buffers not yet allocated.
Now allocate buffers before writing data into them.

Fixes: #12123
Signed-off-by: Jevon Qiao <qiaojianfeng@unitedstack.com>

rgw: api adjustment following a rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: orphans, fix check on number of shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: orphans, change default number of shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: change error output related to orphans

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: orphan, fix truncated detection

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: simplify orphan command

No need for --init-search

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: stat orphan objects before reporting leakage

We don't want to report new objects as leaked, because they might just
be written, and waiting to be linked to a new object. The number of
seconds prior to the job initialization can be cofigured through
--orphan-stale-secs (by default it's 24h).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: orphans finish command

A command to remove orphans data

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cannot re-init an orphan scan job

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>