rgw: Make vstart.sh print out swift user info

Currently vstart.sh only print out S3 user info. This patch updates it
to print swift user info too. Also changes the swift account/user/password
to test/tester/testing which align with openstack-swift better:
https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L270

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge pull request #5197 from ceph/wip-cot-meta

Wip cot meta

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5198 from tchaikov/wip-doc-erasure-fix

doc/erasure-code: fix couple typos

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/erasure-code: fix couple typos

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools, test: Some ceph-objectstore-tool error handling fixes

Improve various error messages generated with invalid syntax
Add test cases for most of these error messages

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add debug argument to the ceph-objectstore-tool test

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Check for valid --op earlier so we can get a better error message

Signed-off-by: David Zafman <dzafman@redhat.com>

tools, test: Add ceph-objectstore-tool to operate on the meta collection

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Fix newlines in output of --op list

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Fix dump-super which doesn't require pgid

A valid pgid had to be specified in order for superblock to be displayed,
but it was otherwise ignored.
Fix message that didn't include dump-super

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Check and specify commands that require the pgid specification

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #4404 from rzarzynski/wip-11431

rgw: fix lack of account name attribute in XML-formatted bucket listing of Swift account

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5078 from rzarzynski/wip-12158

rgw: send Content-Length in response for GET on Swift account.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge branch 'wip-temp'

Conflicts:
src/osd/ReplicatedPG.cc
src/test/osd/RadosModel.h

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

osd, tools: Always filter temp objects since not being exported

Can't use object_locator_to_pg() directly with a temp object
because the pool is negative.

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Don't export temporary objects until we have persistent-temp objects

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: fix temp clearing in OSD

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5175 from david-z/wip-mds-dump-crash

Fix mds dump_ops_in_flight crashing ocassionally

Fix mds dump_ops_in_flight crashing ocassionally

Merge pull request #4977 from theanalyst/rados/wip-9737

rados/tool: handle --snapid correctly

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5185 from yuyuyu101/wip-store-test-zero

StoreTest: Add zero test for SyntheticTest

Merge pull request #3273 from majianpeng/fix4

cleanup code for RecoveyMessages

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #4143 from majianpeng/ec-notsupp-omap

fix bugs about cache tier.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4329 from majianpeng/filestore-copy

Filestore copy:using splice instead read/write for clone

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #4458 from wonzhq/dup-promote

osd/ReplicatedPG: need to queue op when avoiding dup promotion in maybe_handle_cache

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5075 from SUSE/wip-logrotate-requires

packaging: add find and which dependencies

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

packaging: add find and which dependencies

The postrotate script in src/logrotate.conf uses the which and find utilities
to do its work. Although logrotate itself is only a Recommends, I think which
and find are so ubiquitous that it makes sense to have them as hard
dependencies.

Also, I checked and the package names which and findutils are the same on all
the RPM distros we are currently targeting in the spec file.

In Debian, find is also in a package called findutils while the package
containing 'which' is called debianutils.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5187 from SUSE/wip-12261

ceph.spec.in: install 95-ceph-osd.rules, mount.ceph, and mount.fuse.c…

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5181 from SUSE/wip-snappy-devel

ceph.spec.in: snappy-devel for all supported distros

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5180 from SUSE/wip-add-missing-specfile-bit

ceph.spec.in: add missing -%{release}

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5076 from SUSE/wip-logrotate-hard-dependency

packaging: make logrotate a hard dependency across all distros

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
 Conflicts:
debian/control

ceph.spec.in: install 95-ceph-osd.rules, mount.ceph, and mount.fuse.ceph properly on SUSE

http://tracker.ceph.com/issues/12261 Fixes: #12261

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5079 from SUSE/wip-12166

ceph.spec.in: use %_udevrulesdir to eliminate conditionals

Merge pull request #5126 from SUSE/wip-12201

ceph.spec.in: /var/run/ceph fixes

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

ceph.spec.in: use _udevrulesdir to eliminate conditionals

The conditionals governing where 50-rbd.rules is installed were not doing the
right thing on SUSE distros.

Start using the %_udevrulesdir RPM macro, while taking care that it is defined
and set to the right value. Use it to eliminate some conditionals around other
udev rules files as well.

http://tracker.ceph.com/issues/12166 Fixes: #12166

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5172 from ceph/wip-rm-git-build-dep

Remove git build-time dependency

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5167 from zaitcev/wip-rgw-doc-layout

rgw: Document the layout of pools and objects

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5184 from majianpeng/compiler-fix

Compiler fix warning message

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw: fix signed/unsigned compare warning.

rgw/rgw_op.cc: In member function 'virtual void
RGWCompleteMultipart::execute()':
rgw/rgw_op.cc:3133:27: warning: comparison between signed and unsigned
integer expressions [-Wsign-compare]
   if (parts->parts.size() > s->cct->_conf->rgw_multipart_part_upload_limit) {

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

test: fix signed/unsigned compare warning.

 In file included from test/objectstore/chain_xattr.cc:29:0:
../src/gmock/gtest/include/gtest/gtest.h: In instantiation of
‘testing::AssertionResult testing::internal::CmpHelperEQ(const char*,
const char*, const T1&, const T2&) [with T1 = long unsigned int; T2 =
int]’:
../src/gmock/gtest/include/gtest/gtest.h:1484:23:   required from
‘static testing::AssertionResult
testing::internal::EqHelper<lhs_is_null_literal>::Compare(const char*,
const char*, const T1&, const T2&) [with T1 = long unsigned int; T2 =
int; bool lhs_is_null_literal = false]’
test/objectstore/chain_xattr.cc:205:314:   required from here
../src/gmock/gtest/include/gtest/gtest.h:1448:16: warning: comparison
between signed and unsigned integer expressions [-Wsign-compare]
   if (expected == actual) {

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #5176 from zhouyuan/rest_bench_https

tools: Fix rest_bench to support https

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5174 from yuyuyu101/fix-bufferlist-zero

Buffer: Fix bufferlist::zero bug

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph.spec.in: snappy-devel for all supported distros

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: make /var/run/ceph conditional

/var/run/ceph should only be installed on Fedora, RH/CentOS, and
all supported SUSE distros that do not have systemd (e.g. SLE11-SP3).

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: add missing -%{release}

We have it everywhere else and I can't think of any reason why
it should be omitted here.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5179 from yghannam/yghannam-workunit-fixes-pjd

Workunits : suites/pjd.sh : Do make clean so make can build on curren…

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5178 from yghannam/yghannam-workunit-fixes-chmod

Workunits : fs/misc/chmod.sh : Include ACL characters in permission c…

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Workunits : fs/misc/chmod.sh : Include ACL characters in permission check.

Signed-off-by: Yazen Ghannam <yazen.ghannam@linaro.org>

Workunits : suites/pjd.sh : Do make clean so make can build on current arch.

Signed-off-by: Yazen Ghannam <yazen.ghannam@linaro.org>

Fix rest_bench to support https

Fixes: #3968
There's typo in the code which has compared 'http' twice

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

buffer: Fix bufferlist::zero bug with special case

Fixes: #12252
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

UnittestBuffer: Add bufferlist zero test case

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Fix mds dump_ops_in_flight crashing ocassionally

rgw: Document the layout of pools and objects

This has to be kept in sync as the layout changes. Like any other
document, really.

Signed-off-by: Pete Zaitcev <zaitcev@redhat.com>

Merge pull request #4976 from t-miyamae/wip-mshec-fixlogic2

erasure code: fix shec's recovery issues found in teuthology test

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4813 from dachary/wip-11667-erasure-code-add-osd

tests: verify erasure code read / write after remapping

Reviewed-by: Kefu Chai <kchai@redhat.com>

Remove git build-time dependency

The git command is currently only needed in the src/make_version script.
However, we can simply avoid it by assuming that if the git command is
not installed then the current dir is not a git repo.

Signed-off-by: Boris Ranto <branto@redhat.com>

StoreTest: Add zero test for SyntheticTest

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

tests: verify erasure code read / write after remapping

Implement the following scenario:

 * create an erasure coded pool and write an object

 * mark two osd used to store the object out
 * wait until the PG is clean
 * read the object and compare it with the original

 * mark the two osd back in
 * wait until the PG is clean
 * read the object and compare it with the original

The scenario is run with the jerasure plugin only because the code path
involved does not vary depending on the erasure code plugin.

http://tracker.ceph.com/issues/11667 Fixes: #11667

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh get_osds with no trailing whitespace

Trim the trailing whitespace of the get_osds output to help with
comparisons.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: improve shell output readability

Modify the test traces to include the file name in addition to the
function and line name. It makes it easier to locate the faulty line
without going back to the test name.

Format the trace lines to be emacs friendly (filename:lineno) so that
C-x ` or C-c C-c jumps to the right file and the right line when running
the test with M-x compile.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

erasure code: shec's gtest for minimum_to_decode() arguments

TestErasureCodeShec_arguments.cc added here verifies shec's fixed logic
to pass Teuthology test.

Signed-off-by: Shotaro Kawaguchi <kawaguchi.s@jp.fujitsu.com>

erasure code: fix shec's recovery issues found in teuthology test

Fixed shec's recovery issues caused by unexpected arguments in
minimum_to_decode() / decode() functions.
We misunderstood as follows.

 (1) want_to_read and available_chunks do not share osds.
 (2) osds that are not included in want_to_read are always included
     in available_chunks.

Signed-off-by: Takanori Nakao <nakao.takanori@jp.fujitsu.com>

erasure code: move shec.cc into ErasureCodeShec.cc

Signed-off-by: Takanori Nakao <nakao.takanori@jp.fujitsu.com>

tests: ceph-helpers.sh reduce kill_daemon verbosity

When a test fails, the script returns immediately and kill_daemon
function is called to cleanup. It is quite verbose and requires
scrolling hundreds of lines back to find the actual error
message. Turn off the shell trace to reduce the verbosity and improve
error output readability.

The kill_daemon cannot just turn off set -x because it may be called by
a test, not just at the end of the run. Instead the kill_daemon function
checks if tracing is activated and temporarily disables it.

Also get rid of the find standard error that commonly happens when
kill_daemon is called to verify there are no leftovers and the test
directory does not exist.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5109 from ceph/wip-rgw-orphans-2

Wip rgw orphans 2

Merge pull request #5151 from majianpeng/make-morereaable-dump-ops

common/TrackedOp: Make get_duration get correctly value.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5166 from yuyuyu101/wip-async-fix-17

AsyncConnection: Only prepare message when it support fast dispatch

Merge pull request #5157 from rajukv/wip-xio-rajukv

XIO: Add missing fastpath events to OSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

AsyncConnection: Only prepare message when it support fast dispatch

Not all messages support reencode now, we just avoid them prepare firstly when
call "send_message"

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

common/TrackedOp: Make get_duration get correctly value.

For get_duration(), it should return the duration of Op. For ops which in
flight, the duration is the time current time subtrace the
initiated_time. For the completed Ops, it used the last event(done)
subtrace the initiated_time.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4837 from xinxinsh/wip-kv-zero

in _zero() function, we should remove strips to minimize keyvalue DB update

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

XIO: Add missing fastpath events to OSD

Without this incoming fastpath messages will be dropped at OSD due to missing OSD::Session

Signed-off-by: Raju Kurunkad <raju.kurunkad@sandisk.com>

generic check return code of get operation

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #5138 from majianpeng/cleanup

common/TrackedOp: code cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5148 from tchaikov/wip-fix-facility-syslog

mon/LogMonitor: use the configured facility if log to syslog

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge remote-tracking branch 'origin/next'

Merge pull request #5142 from cernceph/wip-nukememmax

mds: nuke the unused mds_mem_max option

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5154 from ceph/wip-rbd-queue-depth

rbd: recognize queue_depth option

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5146 from ceph/wip-12215-next

librbd: don't attempt to invalidate an object map in R/O mode

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5145 from ceph/wip-12214-next

librbd: fix TestObjectMap.InvalidateFlagInMemoryOnly test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5131 from ceph/wip-12069-next

librbd: propagate errors from image close

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5108 from ceph/wip-12079

Wip 12079

rgw: conversion tool to fix broken multipart objects

Fixes: #12079
Broken multipart objects: multipart objects that created on 0.94.2
and that start with underscore have a bad locator on their tail objects.
This extends the tool that was needed for older issue we've had with
hammer multipart objects (that start with underscore). The same usage
applies:

$ radosgw-admin bucket check --check-head-obj-locator \
                             --bucket=<bucket> [--fix]

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5153 from oritwas/fix-cmake-3

Add libradosstriper to cmake

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5149 from tchaikov/wip-fix-erasure-hides-parse

erasure-code: do not hide overloaded ErasureCode::parse()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rbd: recognize queue_depth option

.. and also stress the difference between libceph and rbd mapping
options.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #5115 from ceph/wip-12037

Add unit/tox structure for ceph python script

Reviewed-by: Travis Rhoden <trhoden@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Add libradosstriper to cmake

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

erasure-code: do not hide overloaded ErasureCode::parse()

* this change fixes the warning from clang:
  ErasureCodeJerasure::parse' hides overloaded virtual function
  [-Woverloaded-virtual]
* some erasure codecs' ErasureCode::parse() rewrites the profile
  using the default values when parsing it if the corresponding
  items are not specified. and we don't call ErasureCode::parse()
  via its children's references. so no need to make it a virtual
  function.
* and ErasureCode::parse() is used as a helper function by its
  children, so make it `protected`
* and parse() in ErasureCode's children is but a helper function
  called by ctor, descendants' parse() and init(). so make them
  protected or private accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/TrackedOp: check tracking_enabled for event "initiated/done".

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mon/LogMonitor: use the configured facility if log to syslog

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/TrackedOp: clean up code make look good.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4394 from majianpeng/tools-copyget-flags

add fadvise flags for rados/rbd

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4937 from majianpeng/replicated-misc-fix

ReplicatedPG: small bugs and clean up

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5141 from XinzeChi/wip-crush-rule

mon/PGMonitor: bug fix pg monitor get crush rule

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: don't attempt to invalidate an object map in R/O mode

The ImageWatcher is not initialized when in R/O mode, which
resulted in a NULL pointer dereference.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: read op should return number of bytes read

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: fixed TestObjectMap.InvalidateFlagInMemoryOnly

librados and librados_test_stub return different result codes
for a read full object operation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5135 from yuyuyu101/wip-async-fix-16

AsyncConnection: Fix incorrect sign message behavior