Merge pull request #3564 from guangyy/wip-10198

osd: break the handling of PG removal event into multiple iterations

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3004 from guangyy/wip-10183

Add a new tick timer which does not need to hold osd_lock, put sched_scrub to that tick timer

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3659 from SUSE/wip-fix-calamari-health

Always provide summary for non-healthy cluster.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3702 from mslovy/wip-fineGrain-submitManagerLock

FileStore:: queue_transactions, fine-grain submitManager lock

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3919 from guangyy/wip-9558

osd : populate the needs_recovery_map source fast when only one peer has missing

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #4493 from ceph/wip-use-get-linux-version

use get_linux_version() instead ad-hoc uname().release parsing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4461 from ceph/wip-11372

librbd: add valgrind memory checks for unit tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4241 from rzarzynski/wip-11285

rgw: rectify broken statistics on Swift account listing.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4432 from big-henry/wip-rgw-listparts

rgw: fix ListParts response

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

pybind: fix valgrind warning on rbd_get_parent_info call

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc: invalid read of freed memory

The bytes not in cache stat was potentially reading the bh length
from a deleted bufferhead.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

krbd: fix incorrect types in the krbd API

The C API functions were referencing the C++ CephContext
instead of the C rados_config_t.  Additionally, the ceph
namespace was missing on the Formatter class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

fsx: cleanup crypto library at exit

Also made small tweaks so that it can be compiled under
a C++ compiler.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

autotools: detect presence of valgrind

Conditionally enable running librbd unit tests under valgrind.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: add support for running API tests under valgrind

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: run librbd valgrind test during 'make check'

Fixes: #11372
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

valgrind: update valgrind suppressions for lttng-ust

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4476 from dachary/wip-releases

doc: add giant v0.87.2 to the release timeline

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

config: Document that 'mds_max_file_size' is only read on FS creation

Afterwards you have to change this through the monitors

Merge pull request #4037 from osynge/wip-fix-disk-zap-sgdisk-invocation

Fix "disk zap" sgdisk invocation

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Fix "disk zap" sgdisk invocation

Fixes #11143

If the metadata on the disk is truly invalid, sgdisk would fail to zero
it in one go, because --mbrtogpt apparently tried to operate on the
metadata it read before executing --zap-all.

Splitting this up into two separate invocations to first zap everything
and then clear it properly fixes this issue.

Based on patch by Lars Marowsky-Bree <lmb@suse.com> in ceph-deploy.
Created by Vincent Untz <vuntz@suse.com>

Signed-off-by: Owen Synge <osynge@suse.com>
Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

Merge pull request #4513 from SUSE/wip-suse-run-make-check

run-make-check.sh: fix a copule of SUSE oddities

Reviewed-by: Loic Dachary <ldachary@redhat.com>

run-make-check.sh: stricter check for yum/apt-get/zypper

Believe it or not, it's possible (although not necessarily likely) to
have both yum and zypper installed on openSUSE.  In this case the
previous implementation using `which` would return a string containing
both commands, which doesn't work.

Signed-off-by: Tim Serong <tserong@suse.com>

run-make-check.sh: use /sbin/modprobe

Rationale: In some cases (e.g.: by default on openSUSE), `sudo` won't
add /sbin to $PATH.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #4509 from wonzhq/miss-arg

librbd: missing an argument when calling invoke_async_request

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: missing an argument when calling invoke_async_request

This prevents the master branch from compiling.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #4186 from ceph/wip-7787

librbd: add fast diff feature

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4381 from ceph/wip-system-retcode

ceph-fuse: check return value on system() invocation

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #4457 from ceph/wip-client-osd-caps

client: check OSD caps before read/write

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-fuse: check return value on system() invocation

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4495 from rzarzynski/wip-11501

rgw: fix broken account listing of Swift API.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4487 from ceph/wip-11464

Revert "osd: For object op, first check object whether unfound."

Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: fix broken account listing of Swift API.

Fixes: #11501
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

use get_linux_version() instead ad-hoc uname().release parsing

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #4445 from tchaikov/wip-doxygen-comments-in-throttle

doc: doxygen comments in throttle

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4488 from ceph/wip-async-fix-6

Event: Delete driver after cleanup

Event: Delete driver after cleanup

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

rgw: fix ListParts response

The response XML element name should be 'ListPartsResult'.

Fixes: #11494
Signed-off-by: Henry Chang <henry@bigtera.com>

qa/workunits/post-file: pick a dir that's readable by world

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "osd: For object op, first check object whether unfound."

This reverts commit b16b080332616f12852c15cd27e022a83ed86319.

It is normal for objects to be unfound for brief periods of time
after peering--we do not want to EIO in that case.

Fixes: #11464
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4483 from xinxinsh/wip-keyvalue-perfcounter

add perf counter for keyvalue db

Reviewed-by: Sage Weil <sage@redhat.com>

librbd: TaskFinisher should finish all queued tasks

The destructor wasn't waiting for all Finisher tasks
to complete before stopping the thread.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: librados completions are not properly released

Changes associated with rebuilding the object map can result
in the leaking of librados AioCompletions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: fix valgrind errors with librbd unit test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: librbd should release global data before exit

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: cleanup singleton memory allocation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4480 from wonzhq/indent

osd/ReplicatedPG: fix an indent in find_object_context

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4474 from ceph/wip-11482

mds: remove caps from revoking list when caps are voluntarily released

Reviewed-by: John Spray <john.spray@redhat.com>

osd/ReplicatedPG: fix an indent in find_object_context

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

doc: add giant v0.87.2 to the release timeline

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mds: remove caps from revoking list when caps are voluntarily released

Fixes: #11482
Signed-off-by: Yan, Zheng <zyan@redhat.com>

add perf counter for rocksdb to evaluate latency of get and transaction commit

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add perf counter for leveldb to evaluate latency of get&commit

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #4454 from ceph/wip-11447

rgw: use correct oid when creating gc chains

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4471 from ceph/wip-11477

rbd: correct issues with cli integration tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4470 from ceph/wip-11475

librbd: correct issues discovered by Python unit tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/release-notes: v0.87.2

Signed-off-by: Sage Weil <sage@redhat.com>

client: check OSD caps before read/write

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tests: rbd cli integration test should explicitly select image format

RBD format 2 is now the default image format, so tests involving the old
format should explicitly request the old format.

Fixes: #11477
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not attempt to retrieve metadata for old format images

An unnecessary error message is being logged due to a failure to retrieve
metadata for old-format images -- which don't support metadata.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: flatten should return -EROFS if image is read-only

Fixes: #11475
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allow snapshots to be created when snapshot is active

The librbd API previously permitted the creation of snapshots while
the image context was associated to another snapshot.  A recent code
cleanup broke that ability, so this re-introduces it.  The code change
also allows minor cleanup with rebuild_object_map.

Fixes: #11475
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4455 from liewegas/wip-civetweb-gitignore

civetweb: add .gitignore

Merge pull request #4434 from ceph/wip-7385

librbd: updated cache max objects calculation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4395 from ceph/wip-11381

Fix clear_pipe after reaping progress

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4440 from ceph/wip-11442

Wip 11442

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4401 from yuyuyu101/wip-async-fix-5

Wip async fix 5

rgw_admin: add --remove-bad flag to bucket check

Add this flag so that the bad object will be removed (should be called
only after user has verified that objects content is correct).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4438 from oritwas/wip-10295

rgw: civetweb should use unique request id

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

AsyncMessenger: Don't need to join thread if not started

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Fix clear_pipe after reaping progress

In pipe.cc:1353 we stop this connection and we will let reader and write threads stop. If now reader and writer quit ASAP and we call queue_reap to trigger the reap progress. Now we haven't call "connection_state->clear_pipe(this)" in pipe.cc:1379, so we may assert failure  here.

Fixes: #11381
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

civetweb: add .gitignore

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4165 from liewegas/wip-post-file

qa/workunits/post-file.sh: s/base-files/aclocal/

rgw: use correct oid when creating gc chains

Fixes: #11447
Backport: hammer

When creating gc chain, use the appropriate oid, otherwise objects will
leak.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4439 from ceph/wip-11453-rgw-init-as-root

#11453 init-radosgw: run RGW as root

Reviewed-by: Sage Weil <sage@redhat.com>

init-radosgw: run RGW as root

The ceph-radosgw service fails to start if the httpd package is not
installed. This is because the init.d file attempts to start the RGW
process with the "apache" UID. If a user is running civetweb, there is
no reason for the httpd or apache2 package to be present on the system.

Switch the init scripts to use "root" as is done on Ubuntu.

http://tracker.ceph.com/issues/11453 Refs: #11453

Reported-by: Vickey Singh <vickey.singh22693@gmail.com>
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rgw: civetweb should use unique request id

max_req_id was moved to RGWRados and changed to atomic64_t.

The same request id resulted in gc giving the same idtag to all objects
resulting in a leakage of rados objects. It only kept the last deleted object in
it's queue, the previous objects were never freed.

Fixes: 10295
Backport: Hammer, Firefly

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #4451 from ceph/wip-kfsx-flush-on-discard

test_librbd_fsx: flush before discard in krbd mode

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test_librbd_fsx: flush before discard in krbd mode

We need to sync dirty pages before issuing BLKDISCARD, explained in
a comment.

Fixes: #11448
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #4448 from tchaikov/wip-check-libxml2-and-libcurl

configure.ac: check for libcurl and libxml2 if build libs3

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4446 from osynge/wip-expat

libexpat is now used for rgw

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4444 from big-henry/wip-throttle

common: fix Throttle - unable to reenable throttling after disabling it

Reviewed-by: Kefu Chai <kchai@redhat.com>

Throttle: reset max only if max changed

Save some CPU cycles.

Signed-off-by: Henry Chang <henry@bigtera.com>

Throttle: fix wait/get() with new max

We were unable to set a new non-zero max if the original max was 0.
Fix it. Also, add test cases for it.

Signed-off-by: Henry Chang <henry@bigtera.com>

configure.ac: check for libcurl and libxml2 if build libs3

Fixes: #11458
Signed-off-by: Kefu Chai <kchai@redhat.com>

Throttle: add more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

Throttle: improve doxygen comments in Throttle

* reindent a line

Signed-off-by: Kefu Chai <kchai@redhat.com>

libexpat is now used for rgw

Check for expat.h
check libexpat has method XML_Parse.

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #4435 from ceph/wip-11451

rgw: fix s3 list buckets

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw-admin: a tool to fix object locator issue

Objects that start with underscore need to have an object locator,
this is due to an old behavior that we need to retain. Some objects
might have been created without the locator. This tool creates a new
rados object with the appropriate locator.

Syntax:

$ ./radosgw-admin bucket check --check-head-obj-locator \
                               --bucket=<bucket> [--fix]

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set a special object locator if object starts with underscore

Fixes: #11442
Backport: hammer

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix s3 list buckets

Fixes: #11451
Backport: hammer
Got broken in commit:7dd54fa3621c04c8ea5723fb1bc06b91d81a0c6c.
Resurrect the option to list unlimited number of buckets using the S3
api.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: updated cache max objects calculation

The previous calculation was based upon the image's object size.
Since the cache stores smaller bufferheads, the object size is not
a good indicator of cache usage and was resulting in objects being
evicted from the cache too often.  Instead, base the max number of
objects on the memory load required to store the extra metadata
for the objects.

Fixes: #7385
Backport: firefly, hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4426 from liewegas/wip-11330

logrotate: prefer service over invoke-rc.d

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4425 from ceph/wip-10243

civetweb: update max num of threads

Merge pull request #4421 from dachary/wip-11441-osd-bench

tests: osd-bench.sh must be verbose on failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4430 from zhouyuan/fix_make_debs

misc: fix make debs script

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4427 from big-henry/wip-mds-fix-integer-truncation

mds: fix integer truncation

misc: fix make debs script

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge pull request #4428 from wkennington/master.pgrefdebugging

configure.ac: fix the disablement of pgrefdebugging

Reviewed-by: Kefu Chai <kchai@redhat.com>