librbd: fix missing write block validation in IO work queue

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: properly close and destroy journal when dynamically disabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rbd: test dynamic features with cache disabled

The journaling logic follows different paths depending on the
cache configuration.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks/qemu: rbd cache is enabled by default

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: unit tests for librbd IO work queue failure path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: cleanup interface between IO work queue and IO requests

This will help mock the testing of the IO work queue

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: improve the ability to mock PointerWQ classes

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: exclusive lock failures should bubble up to IO

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: directly inform IO work queue when locks are required

Due to lock dependency issues between librbd locks and the the thread
pool lock, it's impossible to directly determine if the lock is
required within the _void_dequeue method. Therefore, this state needs
to be internally tracked by the work queue.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: clean up variable naming in IO work queue

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: convert ImageRequestWQ to template

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15955 from liewegas/wip-20431

qa/tasks/dump_stuck: fix for active+clean+remapped

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/dump_stuck: fix for active+clean+remapped

In d24a8886658c2d8882275d69c6409717a62701be we made remapped a clean
state but didn't fix this test.

Fixes: http://tracker.ceph.com/issues/20431
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15874 from elliswuuuuuuuu/remove_unuse_config

common/config_opt: remove unused config

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15864 from joscollin/wip-CID1412839-uninitialized-pointer-field

common: initialize array in struct BackTrace

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15881 from joscollin/wip-1412776-explicit-null-dereferenced

common: Passing null pointer option_name to operator << in md_config_t::parse_option()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15896 from Yan-waller/wip-waller-mgrcleanup0624

mgr: drop repeated log info. and unnecessary write permission

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15910 from wido/cest-api-fork

ceph-rest-api: Various REST API fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15915 from liupan1111/wip-add-perfs

src/OSD: add more useful perf counters for performance tuning.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15917 from xiexingguo/wip-upmap-fixes

mon/OSDMonitor: a couple of upmap and other fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15935 from zphj1987/master

ceph-conf: fix typo in usage: 'mon add' should be 'mon addr'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15885 from liewegas/wip-radosbench-timeout

qa/tasks/radosbench: increase timeout

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #15882 from rzarzynski/wip-20372

rgw: fixes for AWSBrowserUploadAbstractor auth

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15931 from cbodley/wip-rgw-slo-compression

rgw: use uncompressed size for range_to_ofs() in slo/dlo

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15928 from liewegas/wip-rgw-warning

rgw/rgw_op: fix whitespace and indentation warning

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15920 from wido/docs-mgr-dashboard

docs: mgr dashboard

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>

doc: Mgr Dashboard docs update

More instructions on how to deploy the dashboard using ceph-mgr

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #15919 from dillaman/wip-test-cls-rbd

test: fix compile warning in ceph_test_cls_rbd

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15934 from myoungwon/omw-mailmap

mailmap: add Myoungwon Oh's mailmap and affiliation

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15914 from wjwithagen/remotes/origin/wip-wjw-freebsd-osd-scrub-repair-3

test/osd/osd-scrub-repair.sh: Fix diff options on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15856 from cbodley/wip-rgw-vstart-s3tests

vstart: remove rgw_enable_static_website

Merge pull request #15863 from gmayyyha/paxosservice-use-func

mon/PaxosService: use __func__ instead of hard code function name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15759 from ovh/bp-optimized-bs-key-escape

os/bluestore: optimized (encode|decode)_escaped

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15765 from jimmyway/wip-ec-plan-cleanup

osd/ECTransaction: cleanup the redundant check which works in overwrite IO context

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-conf: fix typo in usage: 'mon add' should be 'mon addr'

src/test/cli/ceph-conf/help.t
src/tools/ceph_conf.cc

fix add to addr

Signed-off-by: Peng Zhang zphj1987@gmail.com

mailmap: add Myoungwon Oh's mailmap and affiliation

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

rgw: use uncompressed size for range_to_ofs() in slo/dlo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rgw_op: fix some more whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

rgw/rgw_op: fix indentation warning

/home/sage/src/ceph2/src/rgw/rgw_op.cc: In member function ‘virtual void RGWDeleteObjTags::execute()’:
/home/sage/src/ceph2/src/rgw/rgw_op.cc:777:3: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
   if (s->object.empty())
   ^~
/home/sage/src/ceph2/src/rgw/rgw_op.cc:780:5: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘if’
     rgw_obj obj;
     ^~~~~~~

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/radosbench: increase timeout

The current timeout isn't enough in some cases (powercycle thrashing leaves
osds down for a long time because rebooting is so slow).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14997 from ivancich/wip-bring-in-dmclock-p2

osd/PG: Add two new mClock implementations of the PG sharded operator queue

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15569 from aclamk/compress_crash_bug_refactor

rgw: Compress crash bug refactor

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15859 from smithfarm/wip-swift-task-move-master

tests: move swift.py task from teuthology to ceph, phase one (master)

Merge pull request #15555 from liewegas/wip-19023

osd/PG: set clean when last_epoch_clean is updated

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15862 from yuyuyu101/wip-omap-enoent

kv/RocksDBStore: abort if rocksdb EIO, don't return incorrect result

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15866 from tchaikov/wip-set-secrets

auth/RotatingKeyRing: use std::move() to set secrets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15905 from tchaikov/wip-kill-clang-warnings

test, ceph-osdomap-tool: kill clang warnings

Reviewed-by: Sage Weil <sage@redhat.com>

src/OSD: add more useful perf counters for performance tuning.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

Merge pull request #13753 from theanalyst/wip/f/rgw/tagging

rgw: implement get/put object tags for S3

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15889 from joscollin/wip-1412614-uninitialized-scalar-field

tools/rbd_mirror: initialize non-static class member m_do_resync in ImageReplayer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: fix compile warning in ceph_test_cls_rbd

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: add use_gmt_hitset to "osd pool get" cmd filter

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: drop debug_fake_ec_pool from "osd pool set" cmd filter

It is dead!

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: fix wrong sanity check order for allow_ec_overwrites

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: make blacklist expire configurable; fix potential races

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: fix wrong comment, wrong indention, wrong output etc.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: strict comparison condition

To match the specifications of straw_calc_version.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/osd_type: map.find() -> map.count() for existense checking

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: kill dead code

This option is now filtered out automatically, so these codes are unaccessable.

./bin/ceph osd unset sortbitwise
Invalid command:  sortbitwise not in full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent
osd unset full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent :  unset <key>
Error EINVAL: invalid command

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: make pg-up-items result a bit nicer

~#: ./bin/ceph osd pg-upmap-items 0.7 0 2 1 3 4 5
set 0.7 pg_upmap_items mapping to [0->2,1->3,4->5]

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: reject updating upmap if number of osd overflows/underflows

This can avoid, e.g.:

~#:./bin/ceph osd pg-upmap 0.7 1 2 3 4
set 0.7 pg_upmap mapping to [1,2,3,4]

~#:./bin/ceph pg dump
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE        STATE_STAMP                VERSION REPORTED UP        UP_PRIMARY ACTING    ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP                LAST_DEEP_SCRUB DEEP_SCRUB_STAMP
0.7           0                  0        0         0       0     0   0        0      peering 2017-06-24 10:11:03.938102     0'0    53:18 [1,2,3,4]          1 [1,2,3,4]              1        0'0 2017-06-24 09:38:50.162197             0'0 2017-06-24 09:38:50.162197

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: make apply_upmap handle multiple pg-upmap-items correctly

Was:
PG_STAT  UP      UP_PRIMARY ACTING  ACTING_PRIMARY
0.7      [1,0,4]          1 [1,0,4]              1

~#:./bin/ceph osd pg-upmap-items 0.7 0 2 1 3 4 5
set 0.7 pg_upmap_items mapping to [0,2,1,3,4,5]

PG_STAT  UP      UP_PRIMARY ACTING  ACTING_PRIMARY
0.7      [1,2,4]          1 [1,2,4]              1

Now:
PG_STAT  UP      UP_PRIMARY ACTING  ACTING_PRIMARY
0.7      [1,0,4]          1 [1,0,4]              1

~#:./bin/ceph osd pg-upmap-items 0.7 0 2 1 3 4 5
set 0.7 pg_upmap_items mapping to [0,2,1,3,4,5]

PG_STAT  UP      UP_PRIMARY ACTING  ACTING_PRIMARY
0.7      [3,2,5]          3 [3,2,5]              3

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/osd/osd-scrub-repair.sh: Fix diff options

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rest api: Set application variable when not executed as __main__

This allows for using this Python file as a WSGI module as it expects
a variable called application.

Signed-off-by: Wido den Hollander <wido@42on.com>

rest api: Set default ceph.conf and id options

This allows for easy execution of the REST API:

  $ /usr/bin/ceph-rest-api

Signed-off-by: Wido den Hollander <wido@42on.com>

rest api: Listen on :: (IPv6) instead of 0.0.0.0

On Linux systems a socket listening on :: also listens on
all IPv4 addresses.

This makes sure the API is usable over IPv6 as well.

Signed-off-by: Wido den Hollander <wido@42on.com>

common: Passing null pointer option_name to operator << in md_config_t::parse_option()

Fixes the Coverity Scan Report:
CID 1412776 (#1 of 1): Explicit null dereferenced (FORWARD_NULL)19. var_deref_model: Passing null pointer option_name to operator <<, which dereferences it.

Fixed the review comments too in this commit.

Signed-off-by: Jos Collin <jcollin@redhat.com>

radosgw: refactor compression code

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

radosgw, compression: added unit test

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

rgw: connection reset/crashed when download large zero object with compression enable

large zero object has a large compression rate,
even 4M compressed data can decompress several GB data.
Handle so much data in single process lead strange issue.

Fixes: http://tracker.ceph.com/issues/20098
Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

Merge pull request #15445 from wjwithagen/wip-wjw-freebsd-osd-scrub-repair-2

test/osd/osd-scrub-repair.sh: disable ec_overwrite tests on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: drop unnecessary write permission

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

mgr: use readable errno strings

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #15902 from joscollin/wip-1411820-Division-or-modulo-by-zero

test: Division by zero in Legacy::encode_n()

Reviewed By: Brad Hubbard <bhubbard@redhat.com>

ceph-osdomap-tool: early return and remove never-read store statement

* silence clang analyzer's warning of: "Value stored to 'r' is never
  read"
* replace the "goto" statement with early return to improve the
  readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15903 from Adirl/data_path_fixes

msg/async/rdma: Data path fixes

Reviewed-by: Haomai Wang <haomai@xsky.com>

test/test_async_driver: silence clang static analyzer warning

* silence the warning of: Value stored to 'r' is never read
* update the gtest assertions to be semantically more correct.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: swift.py: tweak imports

The ".." form only works within the teuthology repo. With swift.py now in the
Ceph repo, we have to be explicit.

Error message was: "ValueError: Attempted relative import beyond toplevel
package

Signed-off-by: Nathan Cutler <ncutler@suse.com>

test: Division by zero in Legacy::encode_n()

Fixes the Coverity Scan Report:
CID 1411820 (#1 of 1): Division or modulo by zero (DIVIDE_BY_ZERO)
9. divide_by_zero: In expression bl.length() * i / sum, division by expression sum which may be zero has undefined behavior.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15826 from xiexingguo/wip-bluestore-20170622

os/bluestore: cleanup min_alloc_size; some formatting nits

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15865 from tchaikov/wip-mgr-pg-command

mgr: enable ceph_send_command() to send pg command

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15875 from xiexingguo/wip-osdmon-20170623

mon/OSDMonitor: "osd crush class rename" support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15890 from tchaikov/wip-19741

osdc/Objecter: release message if it's not handled

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15891 from minggr/wip-sdata_op_ordering_lock

osd: unlock sdata_op_ordering_lock with sdata_lock hold to avoid miss…

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15899 from theanalyst/doc/release-typo

doc: Luminous release notes typo fixes

Merge pull request #15901 from joscollin/wip-CID1412577-Division-or-modulo-by-float-zero

mon: Division by zero in PGMapDigest::dump_pool_stats_full()

Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15900 from smithfarm/wip-docdir-fix

build/ops: rpm: sane packaging of %{_docdir}/ceph directory

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: allocated reged mem can be greater than requires len

Change-Id: I3ec42afca39c2ad93fc66ed69b3609b68f315e34
Signed-off-by: Adir lev <adirl@mellanox.com>

msg/async/rdma: if failed to send push csi to the back

Change-Id: If6c72917b6e3a836279b86a4d5e9e54383d7b4bd
Signed-off-by: Adir lev <adirl@mellanox.com>

mon: Division by zero in PGMapDigest::dump_pool_stats_full()

Fixes The Coverity Scan Report:
CID 1412577 (#1 of 1): Division or modulo by float zero (DIVIDE_BY_ZERO)
35. divide_by_zero: In expression (float)mk / k, division by expression k which may be zero has undefined behavior.

Signed-off-by: Jos Collin <jcollin@redhat.com>

build/ops: rpm: sane packaging of %{_docdir}/ceph directory

91648044163 introduced a build error:

ceph-common-12.1.0+git.1498286248.2fcedc7b3d-1.1.x86_64.rpm: directories not
owned by a package:
 - /usr/share/doc/packages/ceph

The %docdir directive is a way of flagging anything in that directory as being
documentation. It does not actually package the directory. And we don't need
it because we're not dumping a large number of files into this directory.
For more information, see the "Directory-related Directives" section of
http://ftp.rpm.org/max-rpm/s1-rpm-inside-files-list-directives.html

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: Luminous release notes typo fixes

Suggested-by: Roger Brown <rogerbrown@gmail.com>
Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

mgr: remove repeated function log

In the front of this file, '__func__' has been added to dout_prefix as below:

"  #define dout_prefix *_dout << "mgr.server " << __func__ << " "    "

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #15876 from gmayyyha/config-unused-opt

common/config_opts: drop unused opt

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

kv/RocksDBStore: abort if rocksdb EIO, don't return incorrect result

If the underlying disk is missing, the current logic won't check the actual
reason why Get/Set failed, it will result to client get a empty key/value
pair which is not expected. The correct logic should be abort
right now. Otherwise, it will leads to undefined behavior.

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15108 from guihecheng/rgw_file-fix

rgw_file: fix misuse of make_key_name before make_fhk

Merge pull request #15893 from liewegas/wip-valgrind-everywhere

qa: do not restrict valgrind runs to centos

qa: do not restrict valgrind runs to centos

This reverts 693bd238510e69569cc3461f84b04c8667bc11da, which was
added in response to http://tracker.ceph.com/issues/18126. But
we updated the Ubuntu packages in sepia so it should be good to go.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15572 from rzarzynski/wip-rgw-17779

rgw: external auth engines of S3 honor rgw_keystone_implicit_tenants.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #14624 from ceph/wip-s3a-hadoop

qa/tasks: S3A hadoop task to test s3a with Ceph

Reviewed-by: Casey Bodley <cbodley@redhat.com>