Merge pull request #4241 from rzarzynski/wip-11285

rgw: rectify broken statistics on Swift account listing.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4432 from big-henry/wip-rgw-listparts

rgw: fix ListParts response

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4476 from dachary/wip-releases

doc: add giant v0.87.2 to the release timeline

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

config: Document that 'mds_max_file_size' is only read on FS creation

Afterwards you have to change this through the monitors

Merge pull request #4037 from osynge/wip-fix-disk-zap-sgdisk-invocation

Fix "disk zap" sgdisk invocation

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Fix "disk zap" sgdisk invocation

Fixes #11143

If the metadata on the disk is truly invalid, sgdisk would fail to zero
it in one go, because --mbrtogpt apparently tried to operate on the
metadata it read before executing --zap-all.

Splitting this up into two separate invocations to first zap everything
and then clear it properly fixes this issue.

Based on patch by Lars Marowsky-Bree <lmb@suse.com> in ceph-deploy.
Created by Vincent Untz <vuntz@suse.com>

Signed-off-by: Owen Synge <osynge@suse.com>
Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

Merge pull request #4513 from SUSE/wip-suse-run-make-check

run-make-check.sh: fix a copule of SUSE oddities

Reviewed-by: Loic Dachary <ldachary@redhat.com>

run-make-check.sh: stricter check for yum/apt-get/zypper

Believe it or not, it's possible (although not necessarily likely) to
have both yum and zypper installed on openSUSE.  In this case the
previous implementation using `which` would return a string containing
both commands, which doesn't work.

Signed-off-by: Tim Serong <tserong@suse.com>

run-make-check.sh: use /sbin/modprobe

Rationale: In some cases (e.g.: by default on openSUSE), `sudo` won't
add /sbin to $PATH.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #4509 from wonzhq/miss-arg

librbd: missing an argument when calling invoke_async_request

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: missing an argument when calling invoke_async_request

This prevents the master branch from compiling.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #4186 from ceph/wip-7787

librbd: add fast diff feature

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4381 from ceph/wip-system-retcode

ceph-fuse: check return value on system() invocation

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #4457 from ceph/wip-client-osd-caps

client: check OSD caps before read/write

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-fuse: check return value on system() invocation

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4495 from rzarzynski/wip-11501

rgw: fix broken account listing of Swift API.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4487 from ceph/wip-11464

Revert "osd: For object op, first check object whether unfound."

Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: fix broken account listing of Swift API.

Fixes: #11501
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #4445 from tchaikov/wip-doxygen-comments-in-throttle

doc: doxygen comments in throttle

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4488 from ceph/wip-async-fix-6

Event: Delete driver after cleanup

Event: Delete driver after cleanup

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

rgw: fix ListParts response

The response XML element name should be 'ListPartsResult'.

Fixes: #11494
Signed-off-by: Henry Chang <henry@bigtera.com>

qa/workunits/post-file: pick a dir that's readable by world

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "osd: For object op, first check object whether unfound."

This reverts commit b16b080332616f12852c15cd27e022a83ed86319.

It is normal for objects to be unfound for brief periods of time
after peering--we do not want to EIO in that case.

Fixes: #11464
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4483 from xinxinsh/wip-keyvalue-perfcounter

add perf counter for keyvalue db

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4480 from wonzhq/indent

osd/ReplicatedPG: fix an indent in find_object_context

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4474 from ceph/wip-11482

mds: remove caps from revoking list when caps are voluntarily released

Reviewed-by: John Spray <john.spray@redhat.com>

osd/ReplicatedPG: fix an indent in find_object_context

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

doc: add giant v0.87.2 to the release timeline

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mds: remove caps from revoking list when caps are voluntarily released

Fixes: #11482
Signed-off-by: Yan, Zheng <zyan@redhat.com>

add perf counter for rocksdb to evaluate latency of get and transaction commit

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add perf counter for leveldb to evaluate latency of get&commit

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #4454 from ceph/wip-11447

rgw: use correct oid when creating gc chains

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4471 from ceph/wip-11477

rbd: correct issues with cli integration tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4470 from ceph/wip-11475

librbd: correct issues discovered by Python unit tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/release-notes: v0.87.2

Signed-off-by: Sage Weil <sage@redhat.com>

client: check OSD caps before read/write

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tests: rbd cli integration test should explicitly select image format

RBD format 2 is now the default image format, so tests involving the old
format should explicitly request the old format.

Fixes: #11477
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not attempt to retrieve metadata for old format images

An unnecessary error message is being logged due to a failure to retrieve
metadata for old-format images -- which don't support metadata.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: flatten should return -EROFS if image is read-only

Fixes: #11475
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allow snapshots to be created when snapshot is active

The librbd API previously permitted the creation of snapshots while
the image context was associated to another snapshot.  A recent code
cleanup broke that ability, so this re-introduces it.  The code change
also allows minor cleanup with rebuild_object_map.

Fixes: #11475
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4455 from liewegas/wip-civetweb-gitignore

civetweb: add .gitignore

Merge pull request #4434 from ceph/wip-7385

librbd: updated cache max objects calculation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4395 from ceph/wip-11381

Fix clear_pipe after reaping progress

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4440 from ceph/wip-11442

Wip 11442

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4401 from yuyuyu101/wip-async-fix-5

Wip async fix 5

rgw_admin: add --remove-bad flag to bucket check

Add this flag so that the bad object will be removed (should be called
only after user has verified that objects content is correct).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4438 from oritwas/wip-10295

rgw: civetweb should use unique request id

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

AsyncMessenger: Don't need to join thread if not started

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Fix clear_pipe after reaping progress

In pipe.cc:1353 we stop this connection and we will let reader and write threads stop. If now reader and writer quit ASAP and we call queue_reap to trigger the reap progress. Now we haven't call "connection_state->clear_pipe(this)" in pipe.cc:1379, so we may assert failure  here.

Fixes: #11381
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

civetweb: add .gitignore

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4165 from liewegas/wip-post-file

qa/workunits/post-file.sh: s/base-files/aclocal/

rgw: use correct oid when creating gc chains

Fixes: #11447
Backport: hammer

When creating gc chain, use the appropriate oid, otherwise objects will
leak.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4439 from ceph/wip-11453-rgw-init-as-root

#11453 init-radosgw: run RGW as root

Reviewed-by: Sage Weil <sage@redhat.com>

init-radosgw: run RGW as root

The ceph-radosgw service fails to start if the httpd package is not
installed. This is because the init.d file attempts to start the RGW
process with the "apache" UID. If a user is running civetweb, there is
no reason for the httpd or apache2 package to be present on the system.

Switch the init scripts to use "root" as is done on Ubuntu.

http://tracker.ceph.com/issues/11453 Refs: #11453

Reported-by: Vickey Singh <vickey.singh22693@gmail.com>
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rgw: civetweb should use unique request id

max_req_id was moved to RGWRados and changed to atomic64_t.

The same request id resulted in gc giving the same idtag to all objects
resulting in a leakage of rados objects. It only kept the last deleted object in
it's queue, the previous objects were never freed.

Fixes: 10295
Backport: Hammer, Firefly

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #4451 from ceph/wip-kfsx-flush-on-discard

test_librbd_fsx: flush before discard in krbd mode

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test_librbd_fsx: flush before discard in krbd mode

We need to sync dirty pages before issuing BLKDISCARD, explained in
a comment.

Fixes: #11448
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #4448 from tchaikov/wip-check-libxml2-and-libcurl

configure.ac: check for libcurl and libxml2 if build libs3

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4446 from osynge/wip-expat

libexpat is now used for rgw

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4444 from big-henry/wip-throttle

common: fix Throttle - unable to reenable throttling after disabling it

Reviewed-by: Kefu Chai <kchai@redhat.com>

Throttle: reset max only if max changed

Save some CPU cycles.

Signed-off-by: Henry Chang <henry@bigtera.com>

Throttle: fix wait/get() with new max

We were unable to set a new non-zero max if the original max was 0.
Fix it. Also, add test cases for it.

Signed-off-by: Henry Chang <henry@bigtera.com>

configure.ac: check for libcurl and libxml2 if build libs3

Fixes: #11458
Signed-off-by: Kefu Chai <kchai@redhat.com>

Throttle: add more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

Throttle: improve doxygen comments in Throttle

* reindent a line

Signed-off-by: Kefu Chai <kchai@redhat.com>

libexpat is now used for rgw

Check for expat.h
check libexpat has method XML_Parse.

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #4435 from ceph/wip-11451

rgw: fix s3 list buckets

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw-admin: a tool to fix object locator issue

Objects that start with underscore need to have an object locator,
this is due to an old behavior that we need to retain. Some objects
might have been created without the locator. This tool creates a new
rados object with the appropriate locator.

Syntax:

$ ./radosgw-admin bucket check --check-head-obj-locator \
                               --bucket=<bucket> [--fix]

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set a special object locator if object starts with underscore

Fixes: #11442
Backport: hammer

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix s3 list buckets

Fixes: #11451
Backport: hammer
Got broken in commit:7dd54fa3621c04c8ea5723fb1bc06b91d81a0c6c.
Resurrect the option to list unlimited number of buckets using the S3
api.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: updated cache max objects calculation

The previous calculation was based upon the image's object size.
Since the cache stores smaller bufferheads, the object size is not
a good indicator of cache usage and was resulting in objects being
evicted from the cache too often.  Instead, base the max number of
objects on the memory load required to store the extra metadata
for the objects.

Fixes: #7385
Backport: firefly, hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4426 from liewegas/wip-11330

logrotate: prefer service over invoke-rc.d

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4425 from ceph/wip-10243

civetweb: update max num of threads

Merge pull request #4421 from dachary/wip-11441-osd-bench

tests: osd-bench.sh must be verbose on failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4430 from zhouyuan/fix_make_debs

misc: fix make debs script

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4427 from big-henry/wip-mds-fix-integer-truncation

mds: fix integer truncation

misc: fix make debs script

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge pull request #4428 from wkennington/master.pgrefdebugging

configure.ac: fix the disablement of pgrefdebugging

Reviewed-by: Kefu Chai <kchai@redhat.com>

Fix the disablement of pgrefdebugging

Currently, if you add the flag --disable-pgrefdebugging during configure
it enables pgrefdebugging. The only way for it to actually be disabled
is to remove the flag entirely. This patch makes the flag respect the
--enable and --disable options correctly. Therefore,
--enable-pgrefdebugging is the only way to enable this now and
--enable-pgrefdebugging=no and --disable-pgrefdebugging work as
expected.

Signed-off-by: William A. Kennington III <william@wkennington.com>

mds: fix integer truncation

Client ID is a 64-bit integer. Unlike Ceph daemons, it may go beyond
2147483647 and cause problems.

Signed-off-by: Henry Chang <henry@bigtera.com>

Merge pull request #4376 from majianpeng/needs-recovery

osd: fix bugs for PG::needs_recovery()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4269 from majianpeng/add-recoverable

osd: For ec pool recovery, only for can recoverable object.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4222 from majianpeng/scrub-fix1

osd/PG: check scrub state when handle CEPH_OSD_OP_SCRUB_MAP.

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: add test-case for repair unrecovery-ec pg.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: Remove the duplicated func MissingLoc::get_all_missing.

It is the same as MissingLoc::get_needs_recovery.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: For object op, first check object whether unfound.

If object unfound, asap return -EIO.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: Fix ec pg repair endless when met unrecover object.

In repair_object, if bad_peer is replica, it don't add soid in
MissingLoc for ec pool. If there are more bad replica for ec pool
which cause object can't recover, the later recoverying will endless.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

PG: For needs_recovery, asap return if need recover.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

PG: remove the duplicated code.

assert(!actingbackfill.empty()) is equal to
assert(actingbackfill.end() != actingbackfill.begin()).

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

PG: fix logic error: don't set need_recovery if peer_missing don't have acting osd.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

logrotate.conf: prefer service over invoke-rc.d

On trusty 14.04, service works but invoke-rc.d does not (but is present).

Fixes: #11330
Reported-by: Wim Heirman <wim@heirman.net>
Signed-off-by: Sage Weil <sage@redhat.com>

tests: osd-bench.sh must be verbose on failure

When a command fail as it should but it does not contain the expected
output, display the entire output to confirm it's a false negative.

http://tracker.ceph.com/issues/11441 Fixes: #11441

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: convert osd-bench.sh to ceph-helpers.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-helpers: implement test_expect_failure

To display the output in case the command did not fail with the expected
output.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4399 from trociny/fix-osd_df_avgutil

mon: osd df: fix average_util calculation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4386 from ceph/wip-10657

mds: warn when clients are not advancing their oldest_client_tid

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4331 from ceph/wip-mds-misc

Wip mds misc

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4424 from ceph/wip-doc-nuke-setmap

doc: don't mention ceph osd setmap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

civetweb: update max num of threads

Fixes: #10243
cherry-picked upstream fixed into submodule

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>