osd/ReplicatedPG: fix an indent in find_object_context

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #4454 from ceph/wip-11447

rgw: use correct oid when creating gc chains

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4471 from ceph/wip-11477

rbd: correct issues with cli integration tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4470 from ceph/wip-11475

librbd: correct issues discovered by Python unit tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/release-notes: v0.87.2

Signed-off-by: Sage Weil <sage@redhat.com>

tests: rbd cli integration test should explicitly select image format

RBD format 2 is now the default image format, so tests involving the old
format should explicitly request the old format.

Fixes: #11477
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not attempt to retrieve metadata for old format images

An unnecessary error message is being logged due to a failure to retrieve
metadata for old-format images -- which don't support metadata.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: flatten should return -EROFS if image is read-only

Fixes: #11475
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allow snapshots to be created when snapshot is active

The librbd API previously permitted the creation of snapshots while
the image context was associated to another snapshot.  A recent code
cleanup broke that ability, so this re-introduces it.  The code change
also allows minor cleanup with rebuild_object_map.

Fixes: #11475
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4455 from liewegas/wip-civetweb-gitignore

civetweb: add .gitignore

Merge pull request #4434 from ceph/wip-7385

librbd: updated cache max objects calculation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4395 from ceph/wip-11381

Fix clear_pipe after reaping progress

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4440 from ceph/wip-11442

Wip 11442

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4401 from yuyuyu101/wip-async-fix-5

Wip async fix 5

rgw_admin: add --remove-bad flag to bucket check

Add this flag so that the bad object will be removed (should be called
only after user has verified that objects content is correct).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4438 from oritwas/wip-10295

rgw: civetweb should use unique request id

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

AsyncMessenger: Don't need to join thread if not started

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Fix clear_pipe after reaping progress

In pipe.cc:1353 we stop this connection and we will let reader and write threads stop. If now reader and writer quit ASAP and we call queue_reap to trigger the reap progress. Now we haven't call "connection_state->clear_pipe(this)" in pipe.cc:1379, so we may assert failure  here.

Fixes: #11381
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

civetweb: add .gitignore

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4165 from liewegas/wip-post-file

qa/workunits/post-file.sh: s/base-files/aclocal/

rgw: use correct oid when creating gc chains

Fixes: #11447
Backport: hammer

When creating gc chain, use the appropriate oid, otherwise objects will
leak.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4439 from ceph/wip-11453-rgw-init-as-root

#11453 init-radosgw: run RGW as root

Reviewed-by: Sage Weil <sage@redhat.com>

init-radosgw: run RGW as root

The ceph-radosgw service fails to start if the httpd package is not
installed. This is because the init.d file attempts to start the RGW
process with the "apache" UID. If a user is running civetweb, there is
no reason for the httpd or apache2 package to be present on the system.

Switch the init scripts to use "root" as is done on Ubuntu.

http://tracker.ceph.com/issues/11453 Refs: #11453

Reported-by: Vickey Singh <vickey.singh22693@gmail.com>
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rgw: civetweb should use unique request id

max_req_id was moved to RGWRados and changed to atomic64_t.

The same request id resulted in gc giving the same idtag to all objects
resulting in a leakage of rados objects. It only kept the last deleted object in
it's queue, the previous objects were never freed.

Fixes: 10295
Backport: Hammer, Firefly

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #4451 from ceph/wip-kfsx-flush-on-discard

test_librbd_fsx: flush before discard in krbd mode

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test_librbd_fsx: flush before discard in krbd mode

We need to sync dirty pages before issuing BLKDISCARD, explained in
a comment.

Fixes: #11448
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #4448 from tchaikov/wip-check-libxml2-and-libcurl

configure.ac: check for libcurl and libxml2 if build libs3

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4446 from osynge/wip-expat

libexpat is now used for rgw

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4444 from big-henry/wip-throttle

common: fix Throttle - unable to reenable throttling after disabling it

Reviewed-by: Kefu Chai <kchai@redhat.com>

Throttle: reset max only if max changed

Save some CPU cycles.

Signed-off-by: Henry Chang <henry@bigtera.com>

Throttle: fix wait/get() with new max

We were unable to set a new non-zero max if the original max was 0.
Fix it. Also, add test cases for it.

Signed-off-by: Henry Chang <henry@bigtera.com>

configure.ac: check for libcurl and libxml2 if build libs3

Fixes: #11458
Signed-off-by: Kefu Chai <kchai@redhat.com>

libexpat is now used for rgw

Check for expat.h
check libexpat has method XML_Parse.

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #4435 from ceph/wip-11451

rgw: fix s3 list buckets

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw-admin: a tool to fix object locator issue

Objects that start with underscore need to have an object locator,
this is due to an old behavior that we need to retain. Some objects
might have been created without the locator. This tool creates a new
rados object with the appropriate locator.

Syntax:

$ ./radosgw-admin bucket check --check-head-obj-locator \
                               --bucket=<bucket> [--fix]

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set a special object locator if object starts with underscore

Fixes: #11442
Backport: hammer

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix s3 list buckets

Fixes: #11451
Backport: hammer
Got broken in commit:7dd54fa3621c04c8ea5723fb1bc06b91d81a0c6c.
Resurrect the option to list unlimited number of buckets using the S3
api.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: updated cache max objects calculation

The previous calculation was based upon the image's object size.
Since the cache stores smaller bufferheads, the object size is not
a good indicator of cache usage and was resulting in objects being
evicted from the cache too often.  Instead, base the max number of
objects on the memory load required to store the extra metadata
for the objects.

Fixes: #7385
Backport: firefly, hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4426 from liewegas/wip-11330

logrotate: prefer service over invoke-rc.d

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4425 from ceph/wip-10243

civetweb: update max num of threads

Merge pull request #4421 from dachary/wip-11441-osd-bench

tests: osd-bench.sh must be verbose on failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4430 from zhouyuan/fix_make_debs

misc: fix make debs script

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4427 from big-henry/wip-mds-fix-integer-truncation

mds: fix integer truncation

misc: fix make debs script

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge pull request #4428 from wkennington/master.pgrefdebugging

configure.ac: fix the disablement of pgrefdebugging

Reviewed-by: Kefu Chai <kchai@redhat.com>

Fix the disablement of pgrefdebugging

Currently, if you add the flag --disable-pgrefdebugging during configure
it enables pgrefdebugging. The only way for it to actually be disabled
is to remove the flag entirely. This patch makes the flag respect the
--enable and --disable options correctly. Therefore,
--enable-pgrefdebugging is the only way to enable this now and
--enable-pgrefdebugging=no and --disable-pgrefdebugging work as
expected.

Signed-off-by: William A. Kennington III <william@wkennington.com>

mds: fix integer truncation

Client ID is a 64-bit integer. Unlike Ceph daemons, it may go beyond
2147483647 and cause problems.

Signed-off-by: Henry Chang <henry@bigtera.com>

Merge pull request #4376 from majianpeng/needs-recovery

osd: fix bugs for PG::needs_recovery()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4269 from majianpeng/add-recoverable

osd: For ec pool recovery, only for can recoverable object.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4222 from majianpeng/scrub-fix1

osd/PG: check scrub state when handle CEPH_OSD_OP_SCRUB_MAP.

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: add test-case for repair unrecovery-ec pg.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: Remove the duplicated func MissingLoc::get_all_missing.

It is the same as MissingLoc::get_needs_recovery.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: For object op, first check object whether unfound.

If object unfound, asap return -EIO.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: Fix ec pg repair endless when met unrecover object.

In repair_object, if bad_peer is replica, it don't add soid in
MissingLoc for ec pool. If there are more bad replica for ec pool
which cause object can't recover, the later recoverying will endless.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

PG: For needs_recovery, asap return if need recover.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

PG: remove the duplicated code.

assert(!actingbackfill.empty()) is equal to
assert(actingbackfill.end() != actingbackfill.begin()).

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

PG: fix logic error: don't set need_recovery if peer_missing don't have acting osd.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

logrotate.conf: prefer service over invoke-rc.d

On trusty 14.04, service works but invoke-rc.d does not (but is present).

Fixes: #11330
Reported-by: Wim Heirman <wim@heirman.net>
Signed-off-by: Sage Weil <sage@redhat.com>

tests: osd-bench.sh must be verbose on failure

When a command fail as it should but it does not contain the expected
output, display the entire output to confirm it's a false negative.

http://tracker.ceph.com/issues/11441 Fixes: #11441

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: convert osd-bench.sh to ceph-helpers.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-helpers: implement test_expect_failure

To display the output in case the command did not fail with the expected
output.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4399 from trociny/fix-osd_df_avgutil

mon: osd df: fix average_util calculation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4386 from ceph/wip-10657

mds: warn when clients are not advancing their oldest_client_tid

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4331 from ceph/wip-mds-misc

Wip mds misc

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4424 from ceph/wip-doc-nuke-setmap

doc: don't mention ceph osd setmap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

civetweb: update max num of threads

Fixes: #10243
cherry-picked upstream fixed into submodule

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4420 from tchaikov/wip-man-fix-desc-in-name

man: fix the description in NAME section

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4409 from ceph/wip-11437

rgw: don't use end_marker for namespaced object listing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4408 from ceph/wip-11436

rgw: adjust return code if can't find upload

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4407 from ceph/wip-11435

rgw: fail if parts not specified on complete-multipart-upload

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: don't mention ceph osd setmap

ceph osd setmap command was disabled and subsequently removed 5 years
ago.  See 9aadd41b2095 ("mon: disable 'osd setmap'") and 6d292397aa26
("mon: remove dead 'setmap' code").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #4423 from ceph/wip-11370

librbd: better handling for duplicate flatten requests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: better handling for duplicate flatten requests

A proxied flatten request could be replayed, resulting in a
-EINVAL error code being generated on the second attempt. Filter
out that error if it is known the parent did exist before the
op started.

Fixes: #11370
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4372 from xiaoxichen/fix_kv_compact_on_mount

Fix kv compact on mount

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4418 from oritwas/wip-11438

rgw: force content-type header for swift account responses without body

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

client: add failure injection for not advancing oldest_tid

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: include size of completed_requests in session dump

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: warn when clients are not advancing their oldest_client_tid

Fixes: #10657
Signed-off-by: Yan, Zheng <zyan@redhat.com>

rgw: force content-type header for swift account responses without body

This regression was introduced in commit 4cfeca03ade21861ab70ca759f94eb244c16cb39

Fixes: #11438
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

man: fix the description in NAME section

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4370 from guangyy/wip-ec-test-fix

qa : misc fixes to stabilize test-erasure-code suite

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4411 from hjwsm1989/wip-monstore-pool

mon: ceph-monstore-tool must do out_store.close() in "store-copy" command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4417 from osynge/wip-LimitNOFILE

Increase max files open limit for OSD daemon.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tools: ceph-monstore-tool must do out_store.close()

this change fixes the "store-copy" command.
Like the bug reported in http://tracker.ceph.com/issues/10093.

Signed-off-by: huangjun <hjwsm1989@gmail.com>

Increase max files open limit for OSD daemon.

Under heavy load the number of file descriptors opened
by the OSD can go beyond the 64K file limit. This patch
increases the default to 128K.

Signed-off-by: Owen Synge <osynge@suse.com>

client: inlcude inline data in FLUSHSNAP cap message

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: flush dirty snap data before allowing new writes

this guarantees that new writes do not overwrite existing dirty
snap data.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly update capability's client_follow

If snap inode exists, capability's client_follow has already been
updated by MDCache::cow_inode(). So we only update client_follow
when the inode was not COWed.

Besides, we need to check return of SnapRealm::get_snap_following().
It can return CEPH_NOSNAP, which is clearly invalid for client_follows.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix Locker::_do_null_snapflush

the snapid check in Locker::_do_null_snapflush() prevents us from
flushing old snap inodes. The check does not make sense.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: do null snap flush when processing embeded cap releases

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: make sure snap inode's last match existing snapshots

MDCache::pick_inode_snap() and snap caps flushing code require
this.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: remove stray dentry from delayed list when it's queued for purging

this avoids assert(!dn->state_test(CDentry::STATE_PURGING)) in
eval_stray() being triggered when processing the delayed list.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: move snap inode to new dirfrag's dirty_rstat_inodes

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: keep auth pin on CDir when marking it dirty

Signed-off-by: Yan, Zheng <zyan@redhat.com>

osd/PG: check scrub state when handle CEPH_OSD_OP_SCRUB_MAP.

There is a chance which cause scrub stop before replica-node send
scrub-map.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4257 from rzarzynski/wip-10683

rgw: rectify support for GET on Swift's account with "limit" parameter == 0

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't use end_marker for namespaced object listing

Fixes: #11437
Don't deal with end_marker if there's a namespace set for the object
listing. The cur_end_marker.empty() check is incorrect if no end_marker
is provided, but a namespace is set. Just avoid this issue entirely, as
it's not needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust return code if can't find upload

Fixes: #11436
Return NoSuchUpload (instead of NoSuchKey).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fail if parts not specified on complete-multipart-upload

Fixes: #11435
A complete multipart upload should not succeed if parts were not
specified. Also, adjust return codes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4406 from ceph/wip-11433

librbd: always initialize perf counters

Reviewed-by: Jason Dillaman <dillaman@redhat.com>