Merge pull request #17245 from jdurgin/wip-dup-perf

osd/PGLog: write only changed dup entries

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17225 from liewegas/wip-bluestore-fsck-leaks

os/bluestore: print leaked extents to debug output

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17242 from neha-ojha/wip-21023

mon: warn about using osd new instead of osd create

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17248 from tchaikov/wip-20987

mon/OSDMonitor: check last_scan_epoch instead when sending creates

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17246 from tchaikov/wip-20913

osd/PGBackend: release a msg using msg->put() not delete

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17029 from xiexingguo/wip-crush-rule-rename

mon: "ceph osd crush rule rename" support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17103 from C2python/clear_attr

common: Destroy attr of RWLock after initialized

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17126 from xiexingguo/wip-nicenum

common/types: make numbers a bit nicer when displaying space usage

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17182 from tchaikov/wip-obj-bencher

common/obj_bencher: set {min,max}_iops if runtime < 1 sec

Reviewed-by: PCzhangPC <pengcheng.zhang@easystack.cn>

Merge pull request #17221 from xiexingguo/wip-pg-sparse-read

osd/PrimaryLogPG: fix sparse read won't trigger repair correctly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17222 from xiexingguo/wip-propagate-eagain

osd/PrimaryLogPG: do_osd_ops - propagate EAGAIN/EINPROGRESS on failok

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17258 from liewegas/wip-21127

qa/standalone/scrub/osd-scrub-repair: fix grep pattern

Reviewed-by: David Zafman <dzafman@redhat.com>

qa/standalone/scrub/osd-scrub-repair: fix grep pattern

PGMap shows

    ss << pg_sum.stats.sum.num_objects_unfound
       << "/" << pg_sum.stats.sum.num_objects << " objects unfound (" << b << "%)";

but we were grepping for "1/1 unfound" instead of "1/1 objects
unfound".

Introduced by fe81b7e3a5034ce855303f93f3e413f3f2dc74a8.

Fixes: http://tracker.ceph.com/issues/21127
Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: check creating_pgs.last_scan_epoch instead when sending creates

we cannot be sure that the creating_pgs_by_osd_epoch is in sync with
creating_pgs.pgs even if mapping.get_epoch() is less or equal to
creating_pgs_epoch. because, 1) the access to mapping.epoch is not
protected by lock, 2) even worse, the mapping might be not finished yet
when we are trying to send pg-creates to subscribers.

so instead of comparing creating_pgs_epoch with mapping's epoch, we
should compare it with creating_pgs.last_scan_epoch. the former is
updated once the creating_pgs_by_osd_epoch is updated with the latest
mapping's epoch and creating_pgs.pgs; the latter is updated with current
osdmap's epoch when creating_pgs is being updated with the inc osd map.
if we are using a creating_pgs_epoch in sync, creating_pgs_epoch should
be creating_pgs.last_scan_epoch + 1.

Fixes: http://tracker.ceph.com/issues/20785
Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "osd: adjust osd_min_pg_log_entries"

This reverts commit 1c4df03394312fe67f36448613d8b54cb1a0e2c9.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd/PGLog: write only changed dup entries

Writing all of the dup entries whenever one is changed causes a large
regression in performance.

Instead, keep track of ranges that need to be cleared and written
after log merging (dirty_{to,from}_dups) and the earliest dup entry we
haven't written yet during normal operation (write_from_dups). This
parallels the way we track unpersisted log entries.

Check that the correct set of dups is persisted by adding a roundtrip
through memstore to each merge_log unit test.

Fixes: http://tracker.ceph.com/issues/21026
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16960 from jcsp/wip-20692

mgr/dashboard: fix FS status on old MDS daemons

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PGBackend: release a msg using msg->put() not delete

fix the regression introduced by 1c18b5cb

Fixes: http://tracker.ceph.com/issues/20913
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: print leaked extents to debug output

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17142 from Songweibin/wip-unused-variable

osd/PG: drop unused variable "oldest_update" in PG.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17171 from xiexingguo/wip-dedup-ec-want-acting

osd/PG: two cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17243 from liewegas/wip-min-size-stuck

qa/overrides/2-size-2-min-size: whitelist REQUEST_STUCK

qa/overrides/2-size-2-min-size: whitelist REQUEST_STUCK

With min_size == size things can get stuck a very long time.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: warn about using osd new instead of osd create

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #17224 from liewegas/wip-ceph-deploy-mgr

qa/tasks/ceph_deploy: gatherkeys before mgr deploy

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #17083 from xiexingguo/wip-swap-bucket

crush: force rebuilding shadow hierarchy after swapping buckets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17033 from mychoxin/yuanxin_put_assert_to_base

os/bluestore: move assert of read/write to base class

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17058 from xiexingguo/wip-crush-compiler

crush: fix CrushCompiler won't compile maps with empty shadow tree

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17034 from xiexingguo/wip-class-ls-osd

mon/OSDMonitor: add plain output for "crush class ls-osd" command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17179 from xiexingguo/wip-crush-smoke-test

mon/OSDMonitor: fix improper input/testing range of crush somke testing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17183 from tchaikov/wip-20913

osd/PGBackend: delete reply if fails to complete delete request

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17227 from liewegas/wip-jewel-x

qa/suites/upgrade/jewel-x/parallel: tolerate laggy mgr

qa/suites/upgrade/jewel-x/parallel: tolerate laggy mgr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17226 from liewegas/wip-jewel-x

qa/suites/upgrade/jewel-x/stress-split: tolerate sloppy past_intervals

qa/suites/upgrade/jewel-x/stress-split: tolerate sloppy past_intervals

This is harmless in general, esp during upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_deploy: gatherkeys before mgr deploy

Otherwise we may be missing the bootstrap-mgr key.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17190 from amitkumar50/cov-test-1

test: Checking fd for negative before closing

Reviewed-by: Sage Weil <sage@redhat.com>

test: Checking fd for negative before closing

Fixes the coverity issue:

** 1395317 Argument cannot be negative
161. negative_return_fn: Function open("bufferlist", 0) returns a
negative number.
162. var_assign: Assigning: signed variable fd = open.

Signed-off-by: Amit Kumar amitkuma@redhat.com

osd/PrimaryLogPG: do_osd_ops - propagate EAGAIN/EINPROGRESS on failok

These are not really errors and ignoring these error codes can
cause potential problems.

Propagating these errors to high-level callers such as execute_ctx()
where they can be taken good care of should instead be the preferred
option.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PrimaryLogPG: fix sparse read won't trigger repair correctly

Normal reads instead of hole-reads should trigger auto-repair.
Also we shall quit on successfully triggering a repair, or else
we are at risk of doing the op multiple times.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #17205 into master

* refs/remotes/upstream/pull/17205/head:
client: fix locking in Client::getcwd

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17204 into master

* refs/remotes/upstream/pull/17204/head:
fuse: use c++ allocations for group list

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17199 into master

* refs/remotes/upstream/pull/17199/head:
client: fix compat version on MStatfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #17121 from zmedico/PGPool-update-optimize-with-interval_set-swap

PGPool::update: optimize with interval_set.swap

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17014 from xiexingguo/wip-various-weight-set-fix

crush: various weight-set fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17099 from xiexingguo/wip-stuck-op-count

mon/PGMap: fix "0 stuck requests are blocked > 4096 sec" warn

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17203 from liewegas/wip-jewel-x

qa/suites/upgarde/jewel-x/parallel: tolerate mgr warning

Merge pull request #17207 from ceph/wip-bz1484002

ceph-volume: use unique logical volumes

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #12429 from yehudasa/wip-meta-stream

rgw: stream metadata full sync init

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16926 from fangyuxiangGL/infinitely-send-data-log-list

rgw: break sending data-log list infinitely

Merge pull request #16612 from cbodley/wip-20668

rgw: fixes for multisite replication of encrypted objects

ceph-volume tests centos7 use the new ansible syntax for lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests use the new ansible syntax for lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create tests for the get_lv helper method

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create tests for the new arg validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util create a validator module for argparse

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare enforce usage of vg/lv when preparing lvm devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/suites/upgrade/jewel-x/parallel: tolerate OBJECT_MISPLACED

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgarde/jewel-x/parallel: tolerate mgr warning

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #17175 into master

* refs/remotes/upstream/pull/17175/head:
client: add blank space between function name and inode number for log

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17150 into master

* refs/remotes/upstream/pull/17150/head:
doc/cephfs: s/rados_max_object_size/osd_max_object_size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

client: fix locking in Client::getcwd

Currently, it doesn't take the client_lock at all, which is problematic
as make_request may very well end up unlocking it. Rename the current
function to _getcwd, and add a new getcwd wrapper that takes the mutex
before calling _getcwd.

This fixes: http://tracker.ceph.com/issues/21082

Signed-off-by: Jeff Layton <jlayton@redhat.com>

ceph-volume lvm.common update help values for vg/lv usage

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.create update docstring for vg/lv usage

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare update docstring for vg/lv usage

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17162 from cbodley/wip-qa-rgw-pool-application

qa/rgw: use 'ceph osd pool application enable' on created pools

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #17163 into master

* refs/remotes/upstream/pull/17163/head:
mon: get writeable osdmap for added data pool

Reviewed-by: Sage Weil <sage@redhat.com>

fuse: use c++ allocations for group list

Valgrind is unhappy about our turning on supplimentary group handling
with fuse by default. The problem is that we end up calling delete to
free the supplimentary gids list, but fuse uses malloc to allocate it.

Note that I was initially concerned that I needed to use malloc and
free there to handle the case of userland calling ceph_userperm_new,
but we leave freeing the pointer up to the caller in that case.

Convert fuse to use new/delete to allocate and free the group lists
instead.

Tracker: http://tracker.ceph.com/issues/21065
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #17075 from ivancich/wip-adjust-dup-ops-options

osd: adjust osd_min_pg_log_entries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

client: fix compat version on MStatfs

Fixes: http://tracker.ceph.com/issues/21078
Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #16959 into master

* refs/remotes/upstream/pull/16959/head:
client: skip lookupname if writing to unlinked file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17053 into master

* refs/remotes/upstream/pull/17053/head:
qa: add chown test clearing S_ISGID and S_ISUID
ceph-fuse: load supplementary groups by default
client: clear suid/sgid bits on non-zero write
client: add missing space in debug msg
cephfs: clear suid/sgid if regular file is exe
client: refactor clear set uid/gid if -1

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17122 into master

* refs/remotes/upstream/pull/17122/head:
mds: check cap string only if !allow_all

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17111 into master

* refs/remotes/upstream/pull/17111/head:
qa: add health whitelist for kcephfs suite

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17054 into master

* refs/remotes/upstream/pull/17054/head:
mds: Initializing uninitiazed members SnapServer

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17184 from PCzhangPC/rmconv

rbd:rm unnecessary conversion from string to char* in image-meta function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17186 from trociny/wip-librados_test_stub-zero-snap

test/librados_test_stub: pass snap context to zero op

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd:rm unnecessary conversion from string to char* in image-meta function

the APIs in librbd receive string type as parameter, so there is no need to convert string to char* in image-meta function

Signed-off-by: PCzhangPC <pengcheng.zhang@easystack.cn>

osd/PG: calc_replicated_acting - better tips

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: calc_ec_acting - kill "usable"

which has no consumers.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PGBackend: delete reply if fails to complete delete request

if any of the objects fails to be deleted due to pg reset after latest
osdmap, the pg recovery delete reply won't be sent to the primary OSD.
in that case, we should delete the reply.

Fixes: http://tracker.ceph.com/issues/20913
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librados_test_stub: pass snap context to zero op

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

common/obj_bencher: set {min,max}_iops if runtime < 1 sec

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: fix improper input/testing range of crush somke testing

CrushTester::test() will reset testing range to [0, 1023] whenever
min_x or max_x is negative and the constructor of CrushTester will
always default min_x and max_x to -1.

Thus to set the test range correctly, you have to specify both min_x and max_x.
Local test shows this patch shall decrease the time consumed by the crush
smoke testing to approximate 1/20 of those without this.

For exmaple:
crush somke test duration: 0.668354 seconds ->
crush somke test duration: 0.012592 seconds

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: move assert of read/write to base class

Signed-off-by: mychoxin <mychoxin@gmail.com>

Merge pull request #17154 from ktdreyer/doc-cat

doc: eliminate useless cat statement

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: get writeable osdmap for added data pool

Continuation of: 435717791ec499f71c9d1485b1e4e63239a343e2

Fixes: http://tracker.ceph.com/issues/21064
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: add blank space between function name and inode number for log

Fixes: http://tracker.ceph.com/issues/21069
Signed-off-by: Zhu Shangzhong <zhu.shangzhong@zte.com.cn>

Merge pull request #16127 from ZVampirEM77/wip-obj-acl-bug

rgw: fix obj copied from remote gateway acl full_control issue

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17041 from dengxiafubi/EAS-1

rgw: exit early if rgw_bucket_set_attrs() fails

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #17065 from tchaikov/wip-20785

mon/OSDMonitor: do not send_pg_creates with stale info

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17168 from ceph/wip-volume-man-fix

doc: include ceph-disk and ceph-disk-volume man pages in index

Reviewed-by: Dan Mick <dmick@redhat.com>

doc add ceph-volume and ceph-volume-systemd man pages to CMakeLists file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: include ceph-disk and ceph-disk-volume man pages in index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16836 from heyingstar/bug-ms_dump_on_send-doubledata

messages/MOSDOp: a fixes of encode_payload

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #17079 from joscollin/wip-use-after-free

msg: drop the unnecessary polling_stop()

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16344 from rzarzynski/wip-rgwqa-tempest

rgw, qa: integrate Tempest to verify RadosGW's compliance with Swift API

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: enable 'rgw' application on created pools

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: add optional 'application' to pool creation helpers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16818 from theanalyst/rgw-lc-add-filter-2

rgw_lc: add support for optional filter argument and make ID optional

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>