Merge pull request #15915 from liupan1111/wip-add-perfs

src/OSD: add more useful perf counters for performance tuning.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15917 from xiexingguo/wip-upmap-fixes

mon/OSDMonitor: a couple of upmap and other fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15935 from zphj1987/master

ceph-conf: fix typo in usage: 'mon add' should be 'mon addr'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15885 from liewegas/wip-radosbench-timeout

qa/tasks/radosbench: increase timeout

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #15882 from rzarzynski/wip-20372

rgw: fixes for AWSBrowserUploadAbstractor auth

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15931 from cbodley/wip-rgw-slo-compression

rgw: use uncompressed size for range_to_ofs() in slo/dlo

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15928 from liewegas/wip-rgw-warning

rgw/rgw_op: fix whitespace and indentation warning

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15920 from wido/docs-mgr-dashboard

docs: mgr dashboard

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>

doc: Mgr Dashboard docs update

More instructions on how to deploy the dashboard using ceph-mgr

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #15919 from dillaman/wip-test-cls-rbd

test: fix compile warning in ceph_test_cls_rbd

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15934 from myoungwon/omw-mailmap

mailmap: add Myoungwon Oh's mailmap and affiliation

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15914 from wjwithagen/remotes/origin/wip-wjw-freebsd-osd-scrub-repair-3

test/osd/osd-scrub-repair.sh: Fix diff options on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15856 from cbodley/wip-rgw-vstart-s3tests

vstart: remove rgw_enable_static_website

Merge pull request #15863 from gmayyyha/paxosservice-use-func

mon/PaxosService: use __func__ instead of hard code function name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15759 from ovh/bp-optimized-bs-key-escape

os/bluestore: optimized (encode|decode)_escaped

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15765 from jimmyway/wip-ec-plan-cleanup

osd/ECTransaction: cleanup the redundant check which works in overwrite IO context

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-conf: fix typo in usage: 'mon add' should be 'mon addr'

src/test/cli/ceph-conf/help.t
src/tools/ceph_conf.cc

fix add to addr

Signed-off-by: Peng Zhang zphj1987@gmail.com

mailmap: add Myoungwon Oh's mailmap and affiliation

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

rgw: use uncompressed size for range_to_ofs() in slo/dlo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rgw_op: fix some more whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

rgw/rgw_op: fix indentation warning

/home/sage/src/ceph2/src/rgw/rgw_op.cc: In member function ‘virtual void RGWDeleteObjTags::execute()’:
/home/sage/src/ceph2/src/rgw/rgw_op.cc:777:3: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
   if (s->object.empty())
   ^~
/home/sage/src/ceph2/src/rgw/rgw_op.cc:780:5: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘if’
     rgw_obj obj;
     ^~~~~~~

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/radosbench: increase timeout

The current timeout isn't enough in some cases (powercycle thrashing leaves
osds down for a long time because rebooting is so slow).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14997 from ivancich/wip-bring-in-dmclock-p2

osd/PG: Add two new mClock implementations of the PG sharded operator queue

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15569 from aclamk/compress_crash_bug_refactor

rgw: Compress crash bug refactor

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15859 from smithfarm/wip-swift-task-move-master

tests: move swift.py task from teuthology to ceph, phase one (master)

Merge pull request #15555 from liewegas/wip-19023

osd/PG: set clean when last_epoch_clean is updated

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15862 from yuyuyu101/wip-omap-enoent

kv/RocksDBStore: abort if rocksdb EIO, don't return incorrect result

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15866 from tchaikov/wip-set-secrets

auth/RotatingKeyRing: use std::move() to set secrets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15905 from tchaikov/wip-kill-clang-warnings

test, ceph-osdomap-tool: kill clang warnings

Reviewed-by: Sage Weil <sage@redhat.com>

src/OSD: add more useful perf counters for performance tuning.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

Merge pull request #13753 from theanalyst/wip/f/rgw/tagging

rgw: implement get/put object tags for S3

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15889 from joscollin/wip-1412614-uninitialized-scalar-field

tools/rbd_mirror: initialize non-static class member m_do_resync in ImageReplayer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: fix compile warning in ceph_test_cls_rbd

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: add use_gmt_hitset to "osd pool get" cmd filter

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: drop debug_fake_ec_pool from "osd pool set" cmd filter

It is dead!

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: fix wrong sanity check order for allow_ec_overwrites

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: make blacklist expire configurable; fix potential races

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: fix wrong comment, wrong indention, wrong output etc.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: strict comparison condition

To match the specifications of straw_calc_version.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/osd_type: map.find() -> map.count() for existense checking

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: kill dead code

This option is now filtered out automatically, so these codes are unaccessable.

./bin/ceph osd unset sortbitwise
Invalid command:  sortbitwise not in full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent
osd unset full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent :  unset <key>
Error EINVAL: invalid command

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: make pg-up-items result a bit nicer

~#: ./bin/ceph osd pg-upmap-items 0.7 0 2 1 3 4 5
set 0.7 pg_upmap_items mapping to [0->2,1->3,4->5]

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: reject updating upmap if number of osd overflows/underflows

This can avoid, e.g.:

~#:./bin/ceph osd pg-upmap 0.7 1 2 3 4
set 0.7 pg_upmap mapping to [1,2,3,4]

~#:./bin/ceph pg dump
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE        STATE_STAMP                VERSION REPORTED UP        UP_PRIMARY ACTING    ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP                LAST_DEEP_SCRUB DEEP_SCRUB_STAMP
0.7           0                  0        0         0       0     0   0        0      peering 2017-06-24 10:11:03.938102     0'0    53:18 [1,2,3,4]          1 [1,2,3,4]              1        0'0 2017-06-24 09:38:50.162197             0'0 2017-06-24 09:38:50.162197

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: make apply_upmap handle multiple pg-upmap-items correctly

Was:
PG_STAT  UP      UP_PRIMARY ACTING  ACTING_PRIMARY
0.7      [1,0,4]          1 [1,0,4]              1

~#:./bin/ceph osd pg-upmap-items 0.7 0 2 1 3 4 5
set 0.7 pg_upmap_items mapping to [0,2,1,3,4,5]

PG_STAT  UP      UP_PRIMARY ACTING  ACTING_PRIMARY
0.7      [1,2,4]          1 [1,2,4]              1

Now:
PG_STAT  UP      UP_PRIMARY ACTING  ACTING_PRIMARY
0.7      [1,0,4]          1 [1,0,4]              1

~#:./bin/ceph osd pg-upmap-items 0.7 0 2 1 3 4 5
set 0.7 pg_upmap_items mapping to [0,2,1,3,4,5]

PG_STAT  UP      UP_PRIMARY ACTING  ACTING_PRIMARY
0.7      [3,2,5]          3 [3,2,5]              3

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/osd/osd-scrub-repair.sh: Fix diff options

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

radosgw: refactor compression code

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

radosgw, compression: added unit test

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

rgw: connection reset/crashed when download large zero object with compression enable

large zero object has a large compression rate,
even 4M compressed data can decompress several GB data.
Handle so much data in single process lead strange issue.

Fixes: http://tracker.ceph.com/issues/20098
Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

Merge pull request #15445 from wjwithagen/wip-wjw-freebsd-osd-scrub-repair-2

test/osd/osd-scrub-repair.sh: disable ec_overwrite tests on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15902 from joscollin/wip-1411820-Division-or-modulo-by-zero

test: Division by zero in Legacy::encode_n()

Reviewed By: Brad Hubbard <bhubbard@redhat.com>

ceph-osdomap-tool: early return and remove never-read store statement

* silence clang analyzer's warning of: "Value stored to 'r' is never
  read"
* replace the "goto" statement with early return to improve the
  readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15903 from Adirl/data_path_fixes

msg/async/rdma: Data path fixes

Reviewed-by: Haomai Wang <haomai@xsky.com>

test/test_async_driver: silence clang static analyzer warning

* silence the warning of: Value stored to 'r' is never read
* update the gtest assertions to be semantically more correct.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: swift.py: tweak imports

The ".." form only works within the teuthology repo. With swift.py now in the
Ceph repo, we have to be explicit.

Error message was: "ValueError: Attempted relative import beyond toplevel
package

Signed-off-by: Nathan Cutler <ncutler@suse.com>

test: Division by zero in Legacy::encode_n()

Fixes the Coverity Scan Report:
CID 1411820 (#1 of 1): Division or modulo by zero (DIVIDE_BY_ZERO)
9. divide_by_zero: In expression bl.length() * i / sum, division by expression sum which may be zero has undefined behavior.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15826 from xiexingguo/wip-bluestore-20170622

os/bluestore: cleanup min_alloc_size; some formatting nits

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15865 from tchaikov/wip-mgr-pg-command

mgr: enable ceph_send_command() to send pg command

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15875 from xiexingguo/wip-osdmon-20170623

mon/OSDMonitor: "osd crush class rename" support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15890 from tchaikov/wip-19741

osdc/Objecter: release message if it's not handled

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15891 from minggr/wip-sdata_op_ordering_lock

osd: unlock sdata_op_ordering_lock with sdata_lock hold to avoid miss…

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15899 from theanalyst/doc/release-typo

doc: Luminous release notes typo fixes

Merge pull request #15901 from joscollin/wip-CID1412577-Division-or-modulo-by-float-zero

mon: Division by zero in PGMapDigest::dump_pool_stats_full()

Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15900 from smithfarm/wip-docdir-fix

build/ops: rpm: sane packaging of %{_docdir}/ceph directory

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: allocated reged mem can be greater than requires len

Change-Id: I3ec42afca39c2ad93fc66ed69b3609b68f315e34
Signed-off-by: Adir lev <adirl@mellanox.com>

msg/async/rdma: if failed to send push csi to the back

Change-Id: If6c72917b6e3a836279b86a4d5e9e54383d7b4bd
Signed-off-by: Adir lev <adirl@mellanox.com>

mon: Division by zero in PGMapDigest::dump_pool_stats_full()

Fixes The Coverity Scan Report:
CID 1412577 (#1 of 1): Division or modulo by float zero (DIVIDE_BY_ZERO)
35. divide_by_zero: In expression (float)mk / k, division by expression k which may be zero has undefined behavior.

Signed-off-by: Jos Collin <jcollin@redhat.com>

build/ops: rpm: sane packaging of %{_docdir}/ceph directory

91648044163 introduced a build error:

ceph-common-12.1.0+git.1498286248.2fcedc7b3d-1.1.x86_64.rpm: directories not
owned by a package:
 - /usr/share/doc/packages/ceph

The %docdir directive is a way of flagging anything in that directory as being
documentation. It does not actually package the directory. And we don't need
it because we're not dumping a large number of files into this directory.
For more information, see the "Directory-related Directives" section of
http://ftp.rpm.org/max-rpm/s1-rpm-inside-files-list-directives.html

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: Luminous release notes typo fixes

Suggested-by: Roger Brown <rogerbrown@gmail.com>
Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #15876 from gmayyyha/config-unused-opt

common/config_opts: drop unused opt

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

kv/RocksDBStore: abort if rocksdb EIO, don't return incorrect result

If the underlying disk is missing, the current logic won't check the actual
reason why Get/Set failed, it will result to client get a empty key/value
pair which is not expected. The correct logic should be abort
right now. Otherwise, it will leads to undefined behavior.

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15108 from guihecheng/rgw_file-fix

rgw_file: fix misuse of make_key_name before make_fhk

Merge pull request #15893 from liewegas/wip-valgrind-everywhere

qa: do not restrict valgrind runs to centos

qa: do not restrict valgrind runs to centos

This reverts 693bd238510e69569cc3461f84b04c8667bc11da, which was
added in response to http://tracker.ceph.com/issues/18126. But
we updated the Ubuntu packages in sepia so it should be good to go.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15572 from rzarzynski/wip-rgw-17779

rgw: external auth engines of S3 honor rgw_keystone_implicit_tenants.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #14624 from ceph/wip-s3a-hadoop

qa/tasks: S3A hadoop task to test s3a with Ceph

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: unlock sdata_op_ordering_lock with sdata_lock hold to avoid missing wakeup signal

We are running mysql on top of rbd. sysbench qps occasionally drops to zero
with the INSERT benchmark.

Debug code captured >2s latency between PG::queue_op() and OSD::dequeue_op().
We finally found out that the latency came from below code in OSD::ShardedOpWQ::_process(),

sdata->sdata_cond.WaitInterval(sdata->sdata_lock,
      utime_t(osd->cct->_conf->threadpool_empty_queue_max_wait, 0));

"threadpool_empty_queue_max_wait" is 2s by default.

Normally, it should not sleep for 2s since the comming IO requests will wakeup it.
But there is a small timing window that it missed the wakeup signal actually.
For example,

     msgr-worker-0 thread                    tp_osd_tp thread
     OSD::ShardedOpWQ::_enqueue              OSD::ShardedOpWQ::_process
     ---------------------------             ---------------------------
T1:                                          sdata_op_ordering_lock.Lock()
T2:  sdata_op_ordering_lock.Lock()
                                             "queue empty"
                                             sdata_op_ordering_lock.Unlock()
     "insert op"
     sdata_op_ordering_lock.Unlock()

T3:  sdata_lock.Lock()
T4:                                          sdata_lock.Lock()
     "send wakeup signal"
     sdata_lock.Unock()
                                             // here the wakeup signal has no effect actually
                                             // becuase it has not slept yet.

                                             // then, it sleeps.
                                             WaitInterval(2s)

This patch unlocks sdata_op_ordering_lock with sdata_lock hold in OSD::ShardedOpWQ::_process(),
then the timeline becomes,

     msgr-worker-0 thread                    tp_osd_tp thread
     OSD::ShardedOpWQ::_enqueue              OSD::ShardedOpWQ::_process
     ---------------------------             ---------------------------
T1:                                          sdata_op_ordering_lock.Lock()
T2:  sdata_op_ordering_lock.Lock()
                                             "queue empty"
                                             sdata_lock.Lock()
T3:                                          sdata_op_ordering_lock.Unlock()
     "insert op"
     sdata_op_ordering_lock.Unlock()
     sdata_lock.Lock()

T4:                                          WaitInterval(2s) -> it actually unlocks sdata_lock
     "send wakeup signal"
     sdata_lock.Unock()
                                             //got signal, wakeup immediately

With this one line change, we can avoid occasional high latency.

Signed-off-by: Ming Lin <ming.lin@alibaba-inc.com>

Merge pull request #15665 from oritwas/wip-rgw-reshard-old-bucket

rgw: auto reshard old buckets

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

osdc/Objecter: release message if it is not handled

Fixes: http://tracker.ceph.com/issues/19741
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15821 from jdurgin/wip-20302

qa/suites/powercycle/osd/tasks/radosbench: consume less space

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15154 from jcsp/wip-multimds-stable

Remove "experimental" warnings from multimds

tools/rbd_mirror: initialize Non-static class member m_do_resync in ImageReplayer

Fixes the Coverity Scan Report:
CID 1412614 (#2-1 of 2): Uninitialized scalar field (UNINIT_CTOR)
7. uninit_member: Non-static class member m_do_resync is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: multimds is no longer experimental

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update cephtool test for multimds on by default

Signed-off-by: John Spray <john.spray@redhat.com>

qa: no longer need to explicitly enable multimds

Signed-off-by: John Spray <john.spray@redhat.com>

mds: enable multimds by default in new filesystems

Signed-off-by: John Spray <john.spray@redhat.com>

mon: remove experimental warning on multimds

Signed-off-by: John Spray <john.spray@redhat.com>

mds: don't call StrayManager::eval_stray() for undefined inode

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: drop locks before waiting for export targets

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: handle MDirUpdate race

mds may try discover several times for MDirUpdate, rename may kick
in and cause MDCache::path_traverse() to return error.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't forge replica dirfrag

MDCache::forge_replica_dir() set wrong dir_auth if the forged replica
dirfrag is subtree root.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: avoid submitting log entry while adjusting subtree map

MDCache::eval_subtree_root() may tigger scatter-gather process, which
submits log entry. Submitting log entry while adjusting subtree map is
bad, because subtree map in intermediate state may get used/logged.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't mark nestlock dirty on improper inode

If inode is replica and it has no auth subtree dirfrag, We should
not mark its nestlock dirty.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: create subtree root immediately after directory tree becomes frozen

When a directory tree become frozen, its WAIT_FROZEN contexts are
executed asynchronously. Before Migrator::export_frozen() set export
bounds, MDCache::try_subtree_merge_at() can merge newly imported
subtree into the frozen directory tree. This causes problem if there
are auth pins in newly imported subtree.

The fix is creating subtree root immediately after directory tree
becomes frozen. The new subtree root has dir_auth 'me, me', so it's
not meregeable.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix stray dentry replication in cache rejoin ack

To replicate s stray dentry, we need to replicate all its ancestors.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #15795 from myoungwon/wip-print-ignore_redirect

osd/osd_types: add flag name (IGNORE_REDIRECT)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15656 from aclamk/download_err_with_comp_followup

rgw: use 64-bit offsets for compression

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15848 from xiexingguo/wip-fix-rmcc

src/vstart.sh: kill dead upmap option

Merge pull request #15851 from liewegas/wip-luminous-notes

doc/release-notes: update luminous notes

doc/release-notes: 'osd crush class rename' is coming

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: ceph tell <foo> help

Signed-off-by: Sage Weil <sage@redhat.com>