Merge pull request #17254 from joscollin/wip-cleanup-rgw-override

rgw: add missing override in list_keys_init()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17311 from hjwsm1989/wip-monop-tracker

mon: use ceph_clock_now if message is self-generated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17330 from xiexingguo/wip-class-rename-idempotent

mon/OSDMonitor: make 'osd crush class rename' idempotent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16863 from mikulely/fix-curl-referer

rgw: fix opslog can't record referrer when using curl as client

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #16860 from mikulely/fix-opslog

rgw: fix opslog cannot record remote_addr

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #16933 from amitkumar50/cov-rgw-3

rgw: Initialize is_master, max_aio, size

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #17185 from oritwas/wip-rgw-20777

rgw-admin: require --yes-i-really-mean-it when using --inconsistent_index

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16774 from yanyixing/fix-sidenav

mgr/dashboard: fix rbd's pool sub menu

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17283 from liupan1111/wip-final-nbd

rbd-nbd: cleanup for NBDServer shut down

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #17269 from liewegas/wip-health

mon: fix two stray legacy get_health() callers

Reviewed-by: Kefu Chai <kchai@redhat.com

Merge pull request #17328 from tchaikov/wip-kill-gcc-warning

test/unittest_pg_log: silence gcc warning

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17146 from shangzhong/wip-add-line-break

common: add line break for "ceph daemon TYPE.ID version"

Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17091 from liupan1111/wip-txc-add

os/bluestore: support calculate cost when using spdk.

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #17251 from Songweibin/wip-unused-variable

os/bluestore: drop unused param "what" in apply()

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: make 'osd crush class rename' idempotent

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/unittest_pg_log: silence gcc warning

this silences following warning:

In file included from ceph/src/test/osd/TestPGLog.cc:24:0:
ceph/src/googletest/googletest/include/gtest/gtest.h:430:16:
warning: ‘virtual void testing::Test::SetUp()’ was hidden
[-Woverloaded-virtual]
   virtual void SetUp();
                ^~~~~
ceph/src/test/osd/TestPGLog.cc:2703:8: warning:   by ‘void
PGLogTrimTest::SetUp(unsigned int, unsigned int, unsigned int)’
[-Woverloaded-virtual]
   void SetUp(unsigned min_entries, unsigned max_entries, unsigned
dup_track) {
        ^~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17326 from neha-ojha/wip-21023-contd

doc: fix order of options in osd new

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17180 from tchaikov/wip-literals-for-options

common/options: use user-defined literals for default values

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16066 from tanghaodong25/bluefs-sync-write

os/bluestore/BlueFS: don't need wait for aio when using _sync_write

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15479 from yonghengdexin735/wip-zzz-const

messages/MDentryLink: add const to member function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17256 from scienceluo/wip-luo-add-log-branch

auth: add err reason for log info in load function

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fix order of options in osd new

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #17307 from liewegas/wip-powercycle-slow-trim

qa/suites/powercycle/osd/whitelist_health: whitelist slow trimming

qa/suites/powercycle/osd/whitelist_health: whitelist more

"2017-08-26 16:09:27.704418 mon.a mon.0 172.21.15.169:6789/0 876 : cluster [WRN] MDS health message (mds.0): Behind on trimming (66/30)" in cluster log

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16857 from yanyixing/chartfix

pybind: fix chart size become bigger when refresh

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17267 from linuxbox2/wip-lock-qlane

rgw_file:  fix LRU lane lock in evict_block()

mon: use ceph_clock_now if message is self-generated

Signed-off-by: huangjun <huangjun@xsky.com>

Merge pull request #17293 from tchaikov/wip-jewel-x-singleton

qa/suites/rados/upgrade/jewel-x-singleton: tolerate sloppy past_intervals

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17019 from dachary/wip-doc-ceph-disk

doc: document ceph-disk prepare class hierarchy

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/upgrade/jewel-x-singleton: tolerate sloppy past_intervals

See-also: d5d5d7d1
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17284 from liupan1111/wip-nvme-cleanup

bluestore/NVMEDevice: code cleanup

Merge pull request #17253 from hjwsm1989/fix-qa-bug

qa: fix the potential delay of pg state change

Reviewed-by: Sage Weil <sage@redhat.com>

bluestore/NVMEDevice: remove useless IOSegment, and set
dpdk_thread_adaptor as static.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
Signed-off-by: Ziye Yang <optimistyzy@intel.com>

bluestore/NVMEDevice: remove useless aio_stop in class NVMEDevice

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

os/bluestore/NVMEDevice: remove not used BufferedExtents.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

rbd-nbd: don't need call "stop()" seperately, and ~NBDServer will do it.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

Merge pull request #17055 from dillaman/wip-21017

mgr/dashboard: fix duplicate images listed on iSCSI status page

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17216 from Yan-waller/wip-walle-0824imagectx

test/librbd: fix mock method macro of set_journal_policy

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17275 from ceph/revert-17268-wip-21089

Revert "os/bluestore: compensate for bad freelistmanager size/blocks metadata"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16494 from asomers/bin_bash

misc: Fix bash path in shebangs

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Revert "os/bluestore: compensate for bad freelistmanager size/blocks metadata"

Merge pull request #17268 from liewegas/wip-21089

os/bluestore: compensate for bad freelistmanager size/blocks metadata

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17271 from xiexingguo/wip-ls-by-class

mon/MonCommands: fix copy-and-paste error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17272 from scienceluo/wip-luo-silence-buiding-warning-branch

common: silence jenkins's buiding warning in obj_bencher.cc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17200 from optimistyzy/823_queue_depth

os/bluestore/NVMEDevice: fix the nvme queue depth issue

common: silence jenkins's buiding warning in obj_bencher.cc

Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>

mon/MonCommands: fix copy-and-paste error

Class is definitely required by default for the "crush rule ls-by-class"
command.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17270 from liewegas/wip-luminous-note

doc/release-notes: clarify purpose of require-osd-release

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

NVMEDevice: fix the nvme queue depth issue

We need to track the queue depth usage for each qpair to
prevent submit too many I/Os. If there are too many requests,
we need to poll the qpair, to make sure there is enough slots
for next I/O submit.

Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

doc/release-notes: clarify purpose of require-osd-release

Signed-off-by: Sage Weil <sage@redhat.com>

mon: fix two stray legacy get_health() callers

These need to call the new method when appropriate.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: compensate for bad freelistmanager size/blocks metadata

This repairs bluestores created before http://tracker.ceph.com/issues/21089
was fixed in f6f1ae3724d593d3709d982c973ec18a25a47b6e.

In both cases, the freelistmanager's size is off by one block (4k).  In
one case, it is just a matter of fixing the size and twiddling the trailing
bit.  In the second case, the size delta causes freelistmanager to need
a new row, which means the blocks count also changes, and we have lots
of bits to zero (all but one in the new row).

Both are silently corrected by fsck in this patch.

Fixes: http://tracker.ceph.com/issues/21089
Signed-off-by: Sage Weil <sage@redhat.com>

rgw_file:  fix LRU lane lock in evict_block()

Found by "Supriti Singh" <Supriti.Singh@suse.com>.

Fixes http://tracker.ceph.com/issues/21141

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #17245 from jdurgin/wip-dup-perf

osd/PGLog: write only changed dup entries

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17225 from liewegas/wip-bluestore-fsck-leaks

os/bluestore: print leaked extents to debug output

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17242 from neha-ojha/wip-21023

mon: warn about using osd new instead of osd create

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17248 from tchaikov/wip-20987

mon/OSDMonitor: check last_scan_epoch instead when sending creates

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17246 from tchaikov/wip-20913

osd/PGBackend: release a msg using msg->put() not delete

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17029 from xiexingguo/wip-crush-rule-rename

mon: "ceph osd crush rule rename" support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17103 from C2python/clear_attr

common: Destroy attr of RWLock after initialized

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17126 from xiexingguo/wip-nicenum

common/types: make numbers a bit nicer when displaying space usage

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17182 from tchaikov/wip-obj-bencher

common/obj_bencher: set {min,max}_iops if runtime < 1 sec

Reviewed-by: PCzhangPC <pengcheng.zhang@easystack.cn>

Merge pull request #17221 from xiexingguo/wip-pg-sparse-read

osd/PrimaryLogPG: fix sparse read won't trigger repair correctly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17222 from xiexingguo/wip-propagate-eagain

osd/PrimaryLogPG: do_osd_ops - propagate EAGAIN/EINPROGRESS on failok

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17258 from liewegas/wip-21127

qa/standalone/scrub/osd-scrub-repair: fix grep pattern

Reviewed-by: David Zafman <dzafman@redhat.com>

qa/standalone/scrub/osd-scrub-repair: fix grep pattern

PGMap shows

    ss << pg_sum.stats.sum.num_objects_unfound
       << "/" << pg_sum.stats.sum.num_objects << " objects unfound (" << b << "%)";

but we were grepping for "1/1 unfound" instead of "1/1 objects
unfound".

Introduced by fe81b7e3a5034ce855303f93f3e413f3f2dc74a8.

Fixes: http://tracker.ceph.com/issues/21127
Signed-off-by: Sage Weil <sage@redhat.com>

doc: document ceph-disk prepare class hierarchy

Signed-off-by: Loic Dachary <loic@dachary.org>

auth: add err log info for load function

Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>

rgw: add missing override in list_keys_init()

Added missing override in list_keys_init().

Signed-off-by: Jos Collin <jcollin@redhat.com>

qa: fix the potential delay of pg state change

If start osd process first and then mark it in, the
pg state may remain all active+clean when doing
wait_for_clean() check, which may fail the next
osd_scrub_pgs() process.
So faster pg state change by marking osd in first.

Signed-off-by: huangjun <huangjun@xsky.com>

os/bluestore: drop useless variable of "what" in apply func

Signed-off-by: songweibin <song.weibin@zte.com.cn>

mon/OSDMonitor: check creating_pgs.last_scan_epoch instead when sending creates

we cannot be sure that the creating_pgs_by_osd_epoch is in sync with
creating_pgs.pgs even if mapping.get_epoch() is less or equal to
creating_pgs_epoch. because, 1) the access to mapping.epoch is not
protected by lock, 2) even worse, the mapping might be not finished yet
when we are trying to send pg-creates to subscribers.

so instead of comparing creating_pgs_epoch with mapping's epoch, we
should compare it with creating_pgs.last_scan_epoch. the former is
updated once the creating_pgs_by_osd_epoch is updated with the latest
mapping's epoch and creating_pgs.pgs; the latter is updated with current
osdmap's epoch when creating_pgs is being updated with the inc osd map.
if we are using a creating_pgs_epoch in sync, creating_pgs_epoch should
be creating_pgs.last_scan_epoch + 1.

Fixes: http://tracker.ceph.com/issues/20785
Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "osd: adjust osd_min_pg_log_entries"

This reverts commit 1c4df03394312fe67f36448613d8b54cb1a0e2c9.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd/PGLog: write only changed dup entries

Writing all of the dup entries whenever one is changed causes a large
regression in performance.

Instead, keep track of ranges that need to be cleared and written
after log merging (dirty_{to,from}_dups) and the earliest dup entry we
haven't written yet during normal operation (write_from_dups). This
parallels the way we track unpersisted log entries.

Check that the correct set of dups is persisted by adding a roundtrip
through memstore to each merge_log unit test.

Fixes: http://tracker.ceph.com/issues/21026
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16960 from jcsp/wip-20692

mgr/dashboard: fix FS status on old MDS daemons

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PGBackend: release a msg using msg->put() not delete

fix the regression introduced by 1c18b5cb

Fixes: http://tracker.ceph.com/issues/20913
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: print leaked extents to debug output

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17142 from Songweibin/wip-unused-variable

osd/PG: drop unused variable "oldest_update" in PG.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17171 from xiexingguo/wip-dedup-ec-want-acting

osd/PG: two cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17243 from liewegas/wip-min-size-stuck

qa/overrides/2-size-2-min-size: whitelist REQUEST_STUCK

qa/overrides/2-size-2-min-size: whitelist REQUEST_STUCK

With min_size == size things can get stuck a very long time.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: warn about using osd new instead of osd create

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #17224 from liewegas/wip-ceph-deploy-mgr

qa/tasks/ceph_deploy: gatherkeys before mgr deploy

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #17083 from xiexingguo/wip-swap-bucket

crush: force rebuilding shadow hierarchy after swapping buckets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17033 from mychoxin/yuanxin_put_assert_to_base

os/bluestore: move assert of read/write to base class

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17058 from xiexingguo/wip-crush-compiler

crush: fix CrushCompiler won't compile maps with empty shadow tree

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17034 from xiexingguo/wip-class-ls-osd

mon/OSDMonitor: add plain output for "crush class ls-osd" command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17179 from xiexingguo/wip-crush-smoke-test

mon/OSDMonitor: fix improper input/testing range of crush somke testing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17183 from tchaikov/wip-20913

osd/PGBackend: delete reply if fails to complete delete request

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17227 from liewegas/wip-jewel-x

qa/suites/upgrade/jewel-x/parallel: tolerate laggy mgr

qa/suites/upgrade/jewel-x/parallel: tolerate laggy mgr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17226 from liewegas/wip-jewel-x

qa/suites/upgrade/jewel-x/stress-split: tolerate sloppy past_intervals

qa/suites/upgrade/jewel-x/stress-split: tolerate sloppy past_intervals

This is harmless in general, esp during upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_deploy: gatherkeys before mgr deploy

Otherwise we may be missing the bootstrap-mgr key.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17190 from amitkumar50/cov-test-1

test: Checking fd for negative before closing

Reviewed-by: Sage Weil <sage@redhat.com>

test: Checking fd for negative before closing

Fixes the coverity issue:

** 1395317 Argument cannot be negative
161. negative_return_fn: Function open("bufferlist", 0) returns a
negative number.
162. var_assign: Assigning: signed variable fd = open.

Signed-off-by: Amit Kumar amitkuma@redhat.com

common/options: use user-defined literals for default intervals

for better readablility

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: use user-defined literals for default sizes

for better readablity.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: do_osd_ops - propagate EAGAIN/EINPROGRESS on failok

These are not really errors and ignoring these error codes can
cause potential problems.

Propagating these errors to high-level callers such as execute_ctx()
where they can be taken good care of should instead be the preferred
option.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PrimaryLogPG: fix sparse read won't trigger repair correctly

Normal reads instead of hole-reads should trigger auto-repair.
Also we shall quit on successfully triggering a repair, or else
we are at risk of doing the op multiple times.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>