UnittestBuffer: Add bufferlist zero test case

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4837 from xinxinsh/wip-kv-zero

in _zero() function, we should remove strips to minimize keyvalue DB update

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

generic check return code of get operation

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #5138 from majianpeng/cleanup

common/TrackedOp: code cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5148 from tchaikov/wip-fix-facility-syslog

mon/LogMonitor: use the configured facility if log to syslog

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge remote-tracking branch 'origin/next'

Merge pull request #5142 from cernceph/wip-nukememmax

mds: nuke the unused mds_mem_max option

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5154 from ceph/wip-rbd-queue-depth

rbd: recognize queue_depth option

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5146 from ceph/wip-12215-next

librbd: don't attempt to invalidate an object map in R/O mode

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5145 from ceph/wip-12214-next

librbd: fix TestObjectMap.InvalidateFlagInMemoryOnly test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5131 from ceph/wip-12069-next

librbd: propagate errors from image close

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5108 from ceph/wip-12079

Wip 12079

rgw: conversion tool to fix broken multipart objects

Fixes: #12079
Broken multipart objects: multipart objects that created on 0.94.2
and that start with underscore have a bad locator on their tail objects.
This extends the tool that was needed for older issue we've had with
hammer multipart objects (that start with underscore). The same usage
applies:

$ radosgw-admin bucket check --check-head-obj-locator \
                             --bucket=<bucket> [--fix]

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5153 from oritwas/fix-cmake-3

Add libradosstriper to cmake

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5149 from tchaikov/wip-fix-erasure-hides-parse

erasure-code: do not hide overloaded ErasureCode::parse()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rbd: recognize queue_depth option

.. and also stress the difference between libceph and rbd mapping
options.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #5115 from ceph/wip-12037

Add unit/tox structure for ceph python script

Reviewed-by: Travis Rhoden <trhoden@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Add libradosstriper to cmake

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

erasure-code: do not hide overloaded ErasureCode::parse()

* this change fixes the warning from clang:
  ErasureCodeJerasure::parse' hides overloaded virtual function
  [-Woverloaded-virtual]
* some erasure codecs' ErasureCode::parse() rewrites the profile
  using the default values when parsing it if the corresponding
  items are not specified. and we don't call ErasureCode::parse()
  via its children's references. so no need to make it a virtual
  function.
* and ErasureCode::parse() is used as a helper function by its
  children, so make it `protected`
* and parse() in ErasureCode's children is but a helper function
  called by ctor, descendants' parse() and init(). so make them
  protected or private accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/TrackedOp: check tracking_enabled for event "initiated/done".

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mon/LogMonitor: use the configured facility if log to syslog

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/TrackedOp: clean up code make look good.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4394 from majianpeng/tools-copyget-flags

add fadvise flags for rados/rbd

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4937 from majianpeng/replicated-misc-fix

ReplicatedPG: small bugs and clean up

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5141 from XinzeChi/wip-crush-rule

mon/PGMonitor: bug fix pg monitor get crush rule

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: don't attempt to invalidate an object map in R/O mode

The ImageWatcher is not initialized when in R/O mode, which
resulted in a NULL pointer dereference.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: read op should return number of bytes read

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: fixed TestObjectMap.InvalidateFlagInMemoryOnly

librados and librados_test_stub return different result codes
for a read full object operation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5135 from yuyuyu101/wip-async-fix-16

AsyncConnection: Fix incorrect sign message behavior

Merge pull request #5144 from tchaikov/wip-fix-tools-rados

tools: fix segfault tools/rados introduced by 0f7aeee

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

obj_bencher: remove trailing space

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rados: fix the segfault introduced in 0f7aeee

prefix and run_name should not be initialized by a temporary string

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5134 from stiopaa1/ThreadAddConsts

common/Thread: added const to 2 functions

Reviewed-by: Loic Dachary <ldachary@redhat.com>

AsyncConnection: Fix incorrect sign message behavior

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5128 from ceph/wip-fix-doc

doc: fix the build

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mon/PGMonitor: bug fix pg monitor get crush rule

when some rules have been deleted before, the index in array of crush->rules
is not always equals to crush_ruleset of pool.

Fixes: #12210
Reported-by: Ning Yao <zay11022@gmail.com>
Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #5071 from cxwshawn/master

obj_bencher: check run_name and prefix for empty string instead of NULL

Reviewed-by: Kefu Chai <kchai@redhat.com>

obj_bencher: check run_name and prefix for empty string instead of NULL

change aio_bench and clean_up parameter const char * to const std::string & format.
In rest_bench.cc, aio_bench used run_name.c_str(), so this format will always be empty
string not NULL, so the condition statement
const std::string run_name_meta = (run_name == NULL ? BENCH_LASTRUN_METADATA : std::string(run_name));
is wrong!

test fix:
    before:
        ./rest-bench --seconds 1 -t 2 -b 100 write --api-host=radosgw.com --bucket=test_rm --access-key=FTL7TSJAGXX5KKDQHMJM  --secret=123456879
        use s3cmd ls s3://test_rm , we can a lot of objects in this bucket, objects are not cleaned up.
    after changes, do the same procedure, objects are cleaned up.

Signed-off-by: shawn chen <cxwshawn@gmail.com>

common/Thread: added const to 2 functions

Changed get_thread_id and am_self to const member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

rbd: explicitly close images to check for unexpected errors

While errors shouldn't occur during image close, it is possible that
they could occur.  In this case, the user should be informed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind: RBD close can now raise an exception on close

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: closing images now returns a result code

Added new librbd::Image::close method to allow checking the close result
when using the C++ librbd library.  rbd_close is no longer hard-coded to
return 0.

Fixes: #12069
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mds: nuke the unused mds_mem_max option

mds mem max has been a noop since argonaut. Nuke it because it
is misleading.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

doc: indent warning messages in add-or-rm-osds.rst

otherwise they are rendered as normal text

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix the link in dev/quick_guide

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: for writefull, offset is zero so replace offset w/ zero.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/Replicated: Using write_update_size_and_usage for  WRITEFULL.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/Replicated: First calc crc then call write_update_size_and_usage.

Because write_update_size_and_usage will change oi.size which cause
don't calc new crc.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #5125 from majianpeng/print-blocked-op

mon/PGMonitor: Make blocked Op message more readable.

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/PGMonitor: Make blocked Op message more readable.

W/O this patch:
root@dev:/var/log/ceph# ceph health detail
HEALTH_WARN 1 pgs stuck unclean; 2 requests are blocked > 32 sec; 1 osds
have slow requests; recovery 5/115 objects degraded (4.348%); recovery
1/38 unfound (2.632%); too few PGs per OSD (15 < min 30)
pg 2.1 is stuck unclean for 899.708271, current state active, last
acting [2,3,0,1]
1 ops are blocked > 1048.58 sec
1 ops are blocked > 262.144 sec
1 ops are blocked > 1048.58 sec on osd.2
1 ops are blocked > 262.144 sec on osd.2
1 osds have slow requests
recovery 5/115 objects degraded (4.348%)
recovery 1/38 unfound (2.632%)
too few PGs per OSD (15 < min 30)

W/ this patch:
ceph health detail
HEALTH_WARN 1 pgs stuck unclean; 2 requests are blocked > 32 sec; 1 osds
have slow requests; recovery 5/115 objects degraded (4.348%); recovery
1/38 unfound (2.632%); too few PGs per OSD (15 < min 30)
pg 2.1 is stuck unclean for 427.103877, current state active, last
acting [2,3,0,1]
1 ops are blocked > 524.288 sec on osd.2
1 ops are blocked > 131.072 sec on osd.2
1 osds have slow requests
recovery 5/115 objects degraded (4.348%)
recovery 1/38 unfound (2.632%)
too few PGs per OSD (15 < min 30

Later messages looks better.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/ReplicatedPG: For WRITEFULL replica object, only truncate if new
size less than old size(only truncate to new size)

If new size larger or equal old size, no need do truncate.
It can diretcly overwrite.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4951 from SUSE/wip-432-missing-divergent

osd_types.cc: replace count with iterator in add_next_event function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5074 from majianpeng/mds-small-max-event

mds: fix mds crash when mds_max_log_events smaller.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4166 from sponce/wip-10759

tools: rados cli --striper

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4928 from xiaoxichen/wip-cleanup-key

os/{LevelDB,Rocks}Store: don't keep key_list in transaction

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5113 from tchaikov/wip-11840-reject-multiple-tiering

mon: disallow adding a tier on top of another tier

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

test: ignore symlinked ceph.py file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

test: ignore symlinked ceph_disk.py file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph.in: linter cleanup, remove unused imports

Signed-off-by: Alfredo Deza <adeza@redhat.com>

test: create python test files for ceph cli

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mon: disallow adding a tier on top of another tier

multiple tiering is not supported at the moment

Fixes: #11840
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5112 from tchaikov/wip-12120

fix a couple build warnings

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: return EINVAL if iovcnt < 0 for p{read,write}v()

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools: fix deprecated warning

s/set_op_flags/set_op_flags2/

Fixes: #12120
Signed-off-by: Kefu Chai <kchai@redhat.com>

crush/CrushTester: fix signed/unsigned warning

Fixes: #12120
Signed-off-by: Sage Weil <sage@redhat.com>

client: fix signed/unsigned warnings in preadv code

Fixes: #12120
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5107 from yuyuyu101/wip-fix-async-14

AsyncConnection: Move sign_message to write_message

Merge pull request #5070 from ceph/wip-4161

mds: store layout on header object

AsyncConnection: Move sign_message to write_message

sign_message require correct connection feature which may wrong when preparing
message

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5105 from Oxygen31/master

doc : missing link in "quick_guide.rst"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc : missing link in "quick_guide.rst"

Fixes: #12009
Signed-off-by: Germain CHIPAUX <germain.chipaux@gmail.com>
Signed-off-by: Thomas LAUMONDAIS <thomas.laumondais@gmail.com>
Signed-off-by: Pierre CHAUMONT <pierre.chaumont31@gmail.com>
Signed-off-by: Benjamin TROQUEREAU <benjamin.troquereau@gmail.com>
Signed-off-by: Benjamin SESIA <benjamin.sesia@gmail.com>

Merge pull request #5084 from JevonQ/fix-issue-12123

tests: fix segfault issue in preadv/pwritev tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: change mds_log_max_segments type from int to unsigned.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mds: fix mds crash when mds_max_log_events smaller.

If 'mds max log events' small, it met this bug:

mds/journal.cc: 155: FAILED assert(ls != this)
 ceph version 9.0.1-1117-gb0dc971
(b0dc97198a45791c8b9661d4be8e531e63202005)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x16ef7bf]
 2: (LogSegment::try_to_expire(MDS*,
C_GatherBuilderBase<MDSInternalContextBase, MDSGather>&, int)+0x1925)
[0x15475e5]
 3: (MDLog::try_expire(LogSegment*, int)+0x66) [0x14c50a0]
 4: (MDLog::trim(int)+0xd67) [0x14c4511]
 5: (MDS::tick()+0x265) [0x1146e0b]
 6: (MDS::C_MDS_Tick::finish(int)+0x32) [0x11693b2]
 7: (Context::complete(int)+0x27) [0x11611a7]
 8: (MDSInternalContextBase::complete(int)+0x1a4) [0x14977fa]
 9: (SafeTimer::timer_thread()+0x37a) [0x16e28fa]
 10: (SafeTimerThread::entry()+0x1c) [0x16e3c56]
 11: (Thread::entry_wrapper()+0xa8) [0x16df0ee]
 12: (Thread::_entry_func(void*)+0x18) [0x16df03c]
 13: /lib64/libpthread.so.0() [0x30c9c07555]
 14: (clone()+0x6d) [0x30c9901f3d]

Check segments.size() > 1 can trim segment.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

tests: fix segfault issue in preadv/pwritev tests

Previously copied the sample data to the iov buffers not yet allocated.
Now allocate buffers before writing data into them.

Fixes: #12123
Signed-off-by: Jevon Qiao <qiaojianfeng@unitedstack.com>

Merge pull request #5094 from SUSE/wip-sharutils-dupe

ceph.spec.in: remove duplicate BuildRequires: sharutils

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5082 from ceph/wip-fedora-babeltrace

ceph.spec.in: fix babeltrace handling on Fedora

Reviewed-by: Ira Cooper ira@samba.org
Reviewed-by: Boris Ranto branto@redhat.com

Merge pull request #5098 from PierreChaumont/master

doc: homogenize Librados (Python) documentation's code samples

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: homogenize Librados (Python) documentation's code samples

I put in doc/rados/api/python.rst line numbers on every code samples in
order to make them look all the same.

Fixes: #12174
Signed off: Pierre Chaumont pierre.chaumont31@gmail.com

rgw: only scan for objects not in namespace

Fixes: #11984
The tool should only work on the head objects, and these are not inside
any namespace.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph.spec.in: remove duplicate BuildRequires: sharutils

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #4957 from SUSE/wip-python-flask

ceph.spec.in: fix python-flask dependency for SUSE

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: fix python-flask dependency for SUSE

In SLE and openSUSE, the package is called python-Flask with an upper-case F.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5086 from SUSE/wip-12173

logrotate.conf: fixes for systemd

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

logrotate.conf: fixes for systemd

Before this patch, the command 'logrotate -f /etc/logrotate.d/ceph'
was generating an error "Failed to reload ceph.target: Job type reload is not
applicable for unit ceph.target".

Before we issue systemctl reload, check that there is at least
one active ceph-* service. (The hyphen is significant.)

Since we use grep, make the grep package a dependency.

http://tracker.ceph.com/issues/12173 Fixes: #12173

Signed-off-by: Tim Serong <tserong@suse.com>
Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5091 from yehudasa/wip-multipart-parts-limit

rgw: Multipart Upload: Support to configure and enforce no of parts a…

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Multipart Upload: Support to configure and enforce no of parts allowed

Fixes: #12146
Config parameter added for no of parts limit in multipart upload and checked while
completing multipart upload

Signed-off-by: Abhishek Dixit <dixitabhi@gmail.com>

Merge pull request #5089 from rzarzynski/wip-enforce_content-type_for_swift-next

rgw [next]: enforce Content-Type in Swift responses.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4970 from SUSE/wip-12034-master

ceph.spec.in: python-argparse only in Python 2.6

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5090 from ceph/wip-12176-next

librbd: assertion failure race condition if watch disconnected

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5080 from ceph/wip-12165-next

librbd: prevent object map updates from being interrupted

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: assertion failure race condition if watch disconnected

It's possible for librbd's watch of the header object to be reset by
connection issues just prior to the image being removed.  This will
causes an assertion failure which assumes at least one watcher on the
image.

Fixes: #12176
Backport: hammer, firefly
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: enforce Content-Type in Swift responses.

Swift sends Content-Type HTTP header even if the response
doesn't contain body. We have this behaviour implemented
until applying some changes in end_header() function.
Unfortunately, lack of Content-Type causes early exits in
many Tempest's tests for Swift API verification.

Fixes: #12157
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

ceph.spec.in: python-argparse only in Python 2.6

argparse is a widely-used Python module for parsing command-line arguments.
Ceph makes heavy use of Python scripts, both in the build environment and on
cluster nodes and clients.

Until Python 2.6, argparse was distributed separately from Python proper.
As of 2.7 it is part of the Python standard library.

Although the python package in a given distro may or may not Provide:
python-argparse, this cannot be relied upon.

Therefore, this commit puts appropriate conditionals around Requires:
python-argparse and BuildRequires: python-argparse. It does so for Red
Hat/CentOS and SUSE only, because the last Fedora version with Python 2.6
was Fedora 13, which is EOL.

argparse is required by both the ceph and ceph-common packages, but since ceph
requires ceph-common, the argparse Requires and BuildRequires need only appear
once, under ceph-common.

http://tracker.ceph.com/issues/12034 Fixes: #12034

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5035 from yuyuyu101/wip-async-fix-13

AsyncConnection: Fix out-of-sequence problem introduced by send/write separated logic

Merge pull request #5081 from ceph/wip-11851

rgw: error out if frontend did not send all data

Merge pull request #5001 from ceph/wip-11985

#11985 Fixes to MDLog (and MDS) shutdown

rgw: error out if frontend did not send all data

Fixes: #11851
The civetweb mg_write() doesn't return error when it can't flush all data
to the user, it just sends the total number of bytes written. Modified the
client io to return total number of bytes and return an error if didn't
send anything.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: prevent object map updates from being interrupted

Object map updates were being canceled in-flight when the exclusive lock
is released.  This resulted in an ERESTART error code bubbling up to
AioRequest.

Fixes: 12165
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5077 from SUSE/wip-spec-comment-clarify

ceph.spec.in: clarify two important comments