osd: Execute crush_location_hook when configured in ceph.conf

CrushLocation only invoked update_from_conf() which meant that
update_from_hook() was never executed.

By called init_on_startup() both functions are executed so that
the hook is called when configured.

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #15920 from wido/docs-mgr-dashboard

docs: mgr dashboard

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>

doc: Mgr Dashboard docs update

More instructions on how to deploy the dashboard using ceph-mgr

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #15919 from dillaman/wip-test-cls-rbd

test: fix compile warning in ceph_test_cls_rbd

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15934 from myoungwon/omw-mailmap

mailmap: add Myoungwon Oh's mailmap and affiliation

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15914 from wjwithagen/remotes/origin/wip-wjw-freebsd-osd-scrub-repair-3

test/osd/osd-scrub-repair.sh: Fix diff options on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15856 from cbodley/wip-rgw-vstart-s3tests

vstart: remove rgw_enable_static_website

Merge pull request #15863 from gmayyyha/paxosservice-use-func

mon/PaxosService: use __func__ instead of hard code function name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15759 from ovh/bp-optimized-bs-key-escape

os/bluestore: optimized (encode|decode)_escaped

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15765 from jimmyway/wip-ec-plan-cleanup

osd/ECTransaction: cleanup the redundant check which works in overwrite IO context

Reviewed-by: Sage Weil <sage@redhat.com>

mailmap: add Myoungwon Oh's mailmap and affiliation

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #14997 from ivancich/wip-bring-in-dmclock-p2

osd/PG: Add two new mClock implementations of the PG sharded operator queue

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15569 from aclamk/compress_crash_bug_refactor

rgw: Compress crash bug refactor

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15859 from smithfarm/wip-swift-task-move-master

tests: move swift.py task from teuthology to ceph, phase one (master)

Merge pull request #15555 from liewegas/wip-19023

osd/PG: set clean when last_epoch_clean is updated

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15862 from yuyuyu101/wip-omap-enoent

kv/RocksDBStore: abort if rocksdb EIO, don't return incorrect result

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15866 from tchaikov/wip-set-secrets

auth/RotatingKeyRing: use std::move() to set secrets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15905 from tchaikov/wip-kill-clang-warnings

test, ceph-osdomap-tool: kill clang warnings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13753 from theanalyst/wip/f/rgw/tagging

rgw: implement get/put object tags for S3

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15889 from joscollin/wip-1412614-uninitialized-scalar-field

tools/rbd_mirror: initialize non-static class member m_do_resync in ImageReplayer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: fix compile warning in ceph_test_cls_rbd

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/osd/osd-scrub-repair.sh: Fix diff options

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

radosgw: refactor compression code

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

radosgw, compression: added unit test

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

rgw: connection reset/crashed when download large zero object with compression enable

large zero object has a large compression rate,
even 4M compressed data can decompress several GB data.
Handle so much data in single process lead strange issue.

Fixes: http://tracker.ceph.com/issues/20098
Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

Merge pull request #15445 from wjwithagen/wip-wjw-freebsd-osd-scrub-repair-2

test/osd/osd-scrub-repair.sh: disable ec_overwrite tests on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15902 from joscollin/wip-1411820-Division-or-modulo-by-zero

test: Division by zero in Legacy::encode_n()

Reviewed By: Brad Hubbard <bhubbard@redhat.com>

ceph-osdomap-tool: early return and remove never-read store statement

* silence clang analyzer's warning of: "Value stored to 'r' is never
  read"
* replace the "goto" statement with early return to improve the
  readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15903 from Adirl/data_path_fixes

msg/async/rdma: Data path fixes

Reviewed-by: Haomai Wang <haomai@xsky.com>

test/test_async_driver: silence clang static analyzer warning

* silence the warning of: Value stored to 'r' is never read
* update the gtest assertions to be semantically more correct.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: swift.py: tweak imports

The ".." form only works within the teuthology repo. With swift.py now in the
Ceph repo, we have to be explicit.

Error message was: "ValueError: Attempted relative import beyond toplevel
package

Signed-off-by: Nathan Cutler <ncutler@suse.com>

test: Division by zero in Legacy::encode_n()

Fixes the Coverity Scan Report:
CID 1411820 (#1 of 1): Division or modulo by zero (DIVIDE_BY_ZERO)
9. divide_by_zero: In expression bl.length() * i / sum, division by expression sum which may be zero has undefined behavior.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15826 from xiexingguo/wip-bluestore-20170622

os/bluestore: cleanup min_alloc_size; some formatting nits

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15865 from tchaikov/wip-mgr-pg-command

mgr: enable ceph_send_command() to send pg command

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15875 from xiexingguo/wip-osdmon-20170623

mon/OSDMonitor: "osd crush class rename" support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15890 from tchaikov/wip-19741

osdc/Objecter: release message if it's not handled

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15891 from minggr/wip-sdata_op_ordering_lock

osd: unlock sdata_op_ordering_lock with sdata_lock hold to avoid miss…

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15899 from theanalyst/doc/release-typo

doc: Luminous release notes typo fixes

Merge pull request #15901 from joscollin/wip-CID1412577-Division-or-modulo-by-float-zero

mon: Division by zero in PGMapDigest::dump_pool_stats_full()

Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15900 from smithfarm/wip-docdir-fix

build/ops: rpm: sane packaging of %{_docdir}/ceph directory

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: allocated reged mem can be greater than requires len

Change-Id: I3ec42afca39c2ad93fc66ed69b3609b68f315e34
Signed-off-by: Adir lev <adirl@mellanox.com>

msg/async/rdma: if failed to send push csi to the back

Change-Id: If6c72917b6e3a836279b86a4d5e9e54383d7b4bd
Signed-off-by: Adir lev <adirl@mellanox.com>

mon: Division by zero in PGMapDigest::dump_pool_stats_full()

Fixes The Coverity Scan Report:
CID 1412577 (#1 of 1): Division or modulo by float zero (DIVIDE_BY_ZERO)
35. divide_by_zero: In expression (float)mk / k, division by expression k which may be zero has undefined behavior.

Signed-off-by: Jos Collin <jcollin@redhat.com>

build/ops: rpm: sane packaging of %{_docdir}/ceph directory

91648044163 introduced a build error:

ceph-common-12.1.0+git.1498286248.2fcedc7b3d-1.1.x86_64.rpm: directories not
owned by a package:
 - /usr/share/doc/packages/ceph

The %docdir directive is a way of flagging anything in that directory as being
documentation. It does not actually package the directory. And we don't need
it because we're not dumping a large number of files into this directory.
For more information, see the "Directory-related Directives" section of
http://ftp.rpm.org/max-rpm/s1-rpm-inside-files-list-directives.html

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: Luminous release notes typo fixes

Suggested-by: Roger Brown <rogerbrown@gmail.com>
Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #15876 from gmayyyha/config-unused-opt

common/config_opts: drop unused opt

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

kv/RocksDBStore: abort if rocksdb EIO, don't return incorrect result

If the underlying disk is missing, the current logic won't check the actual
reason why Get/Set failed, it will result to client get a empty key/value
pair which is not expected. The correct logic should be abort
right now. Otherwise, it will leads to undefined behavior.

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15108 from guihecheng/rgw_file-fix

rgw_file: fix misuse of make_key_name before make_fhk

Merge pull request #15893 from liewegas/wip-valgrind-everywhere

qa: do not restrict valgrind runs to centos

qa: do not restrict valgrind runs to centos

This reverts 693bd238510e69569cc3461f84b04c8667bc11da, which was
added in response to http://tracker.ceph.com/issues/18126. But
we updated the Ubuntu packages in sepia so it should be good to go.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15572 from rzarzynski/wip-rgw-17779

rgw: external auth engines of S3 honor rgw_keystone_implicit_tenants.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #14624 from ceph/wip-s3a-hadoop

qa/tasks: S3A hadoop task to test s3a with Ceph

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: unlock sdata_op_ordering_lock with sdata_lock hold to avoid missing wakeup signal

We are running mysql on top of rbd. sysbench qps occasionally drops to zero
with the INSERT benchmark.

Debug code captured >2s latency between PG::queue_op() and OSD::dequeue_op().
We finally found out that the latency came from below code in OSD::ShardedOpWQ::_process(),

sdata->sdata_cond.WaitInterval(sdata->sdata_lock,
      utime_t(osd->cct->_conf->threadpool_empty_queue_max_wait, 0));

"threadpool_empty_queue_max_wait" is 2s by default.

Normally, it should not sleep for 2s since the comming IO requests will wakeup it.
But there is a small timing window that it missed the wakeup signal actually.
For example,

     msgr-worker-0 thread                    tp_osd_tp thread
     OSD::ShardedOpWQ::_enqueue              OSD::ShardedOpWQ::_process
     ---------------------------             ---------------------------
T1:                                          sdata_op_ordering_lock.Lock()
T2:  sdata_op_ordering_lock.Lock()
                                             "queue empty"
                                             sdata_op_ordering_lock.Unlock()
     "insert op"
     sdata_op_ordering_lock.Unlock()

T3:  sdata_lock.Lock()
T4:                                          sdata_lock.Lock()
     "send wakeup signal"
     sdata_lock.Unock()
                                             // here the wakeup signal has no effect actually
                                             // becuase it has not slept yet.

                                             // then, it sleeps.
                                             WaitInterval(2s)

This patch unlocks sdata_op_ordering_lock with sdata_lock hold in OSD::ShardedOpWQ::_process(),
then the timeline becomes,

     msgr-worker-0 thread                    tp_osd_tp thread
     OSD::ShardedOpWQ::_enqueue              OSD::ShardedOpWQ::_process
     ---------------------------             ---------------------------
T1:                                          sdata_op_ordering_lock.Lock()
T2:  sdata_op_ordering_lock.Lock()
                                             "queue empty"
                                             sdata_lock.Lock()
T3:                                          sdata_op_ordering_lock.Unlock()
     "insert op"
     sdata_op_ordering_lock.Unlock()
     sdata_lock.Lock()

T4:                                          WaitInterval(2s) -> it actually unlocks sdata_lock
     "send wakeup signal"
     sdata_lock.Unock()
                                             //got signal, wakeup immediately

With this one line change, we can avoid occasional high latency.

Signed-off-by: Ming Lin <ming.lin@alibaba-inc.com>

Merge pull request #15665 from oritwas/wip-rgw-reshard-old-bucket

rgw: auto reshard old buckets

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

osdc/Objecter: release message if it is not handled

Fixes: http://tracker.ceph.com/issues/19741
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15821 from jdurgin/wip-20302

qa/suites/powercycle/osd/tasks/radosbench: consume less space

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15154 from jcsp/wip-multimds-stable

Remove "experimental" warnings from multimds

tools/rbd_mirror: initialize Non-static class member m_do_resync in ImageReplayer

Fixes the Coverity Scan Report:
CID 1412614 (#2-1 of 2): Uninitialized scalar field (UNINIT_CTOR)
7. uninit_member: Non-static class member m_do_resync is not initialized in this constructor nor in any functions that it calls.

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: multimds is no longer experimental

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update cephtool test for multimds on by default

Signed-off-by: John Spray <john.spray@redhat.com>

qa: no longer need to explicitly enable multimds

Signed-off-by: John Spray <john.spray@redhat.com>

mds: enable multimds by default in new filesystems

Signed-off-by: John Spray <john.spray@redhat.com>

mon: remove experimental warning on multimds

Signed-off-by: John Spray <john.spray@redhat.com>

mds: don't call StrayManager::eval_stray() for undefined inode

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: drop locks before waiting for export targets

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: handle MDirUpdate race

mds may try discover several times for MDirUpdate, rename may kick
in and cause MDCache::path_traverse() to return error.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't forge replica dirfrag

MDCache::forge_replica_dir() set wrong dir_auth if the forged replica
dirfrag is subtree root.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: avoid submitting log entry while adjusting subtree map

MDCache::eval_subtree_root() may tigger scatter-gather process, which
submits log entry. Submitting log entry while adjusting subtree map is
bad, because subtree map in intermediate state may get used/logged.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't mark nestlock dirty on improper inode

If inode is replica and it has no auth subtree dirfrag, We should
not mark its nestlock dirty.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: create subtree root immediately after directory tree becomes frozen

When a directory tree become frozen, its WAIT_FROZEN contexts are
executed asynchronously. Before Migrator::export_frozen() set export
bounds, MDCache::try_subtree_merge_at() can merge newly imported
subtree into the frozen directory tree. This causes problem if there
are auth pins in newly imported subtree.

The fix is creating subtree root immediately after directory tree
becomes frozen. The new subtree root has dir_auth 'me, me', so it's
not meregeable.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix stray dentry replication in cache rejoin ack

To replicate s stray dentry, we need to replicate all its ancestors.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #15795 from myoungwon/wip-print-ignore_redirect

osd/osd_types: add flag name (IGNORE_REDIRECT)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15656 from aclamk/download_err_with_comp_followup

rgw: use 64-bit offsets for compression

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15848 from xiexingguo/wip-fix-rmcc

src/vstart.sh: kill dead upmap option

Merge pull request #15851 from liewegas/wip-luminous-notes

doc/release-notes: update luminous notes

doc/release-notes: 'osd crush class rename' is coming

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: ceph tell <foo> help

Signed-off-by: Sage Weil <sage@redhat.com>

doc/start/os-recommendations: update

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: note debian stretch addition

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: sleep settings

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: link to EC docs

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: update RGW metadata

Signed-off-by: Sage Weil <sage@redhat.com>

dev/release-notes: various updates from other PR

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: notes on new CLI commands

Signed-off-by: Sage Weil <sage@redhat.com>

mon: 'mon feature list' -> 'mon feature ls'

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: update luminous notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15764 from tchaikov/wip-20342

qa/suites/upgrade/hammer-jewel-x: add luminous.yaml

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15877 from wjwithagen/wip-wjw-vstart-ceph-mgr-restfull

vstart.sh: Work around mgr restfull not available

mon/OSDMonitor: slightly nice error output if set-device-class failed

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: set result code properly if we fail to process "swap-bucket"

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: "osd crush class rename" support

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

./src/vstart.sh: Work around mgr restfull not available

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

common/config_opts: drop unused opt

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

test/osd/osd-scrub-repair.sh: Adjust for FreeBSD

Fixes 2 problems:
 -  Do not test Bluestore on FreeBSD, since that does not work (yet)
    And all erasure code overwrite tests are executed on BlueStore OSDs
    Erasure code overwrites are unsafe on Filestore, see:
    http://docs.ceph.com/docs/master/rados/operations/erasure-code/#erasure-coding-with-overwrites

 -  the JQ expression errors out with:
    (version 1.5-1-g940132e-dirty)
    ====
    jq: error (at :232): Cannot iterate over null (null)
    Traceback (most recent call last):
    File "", line 1, in
    File "/usr/lib64/python2.7/json/init.py", line 338, in loads
    return _default_decoder.decode(s)
    File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File "/usr/lib64/python2.7/json/decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
    ValueError: No JSON object could be decoded
    ====
    Adding a ? to the jq expression allows to proceed on null blocks.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

auth/RotatingKeyRing: use std::move() to set secrets

the param will be thrown away anyway. see
CephxClientHandler::handle_response().

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge branch 'master' of /home/smithfarm/src/ceph/upstream/teuthology into wip-swift-task-move-master

tests: move swift.py task to qa/tasks

In preparation for moving this task from ceph/teuthology.git into ceph/ceph.git

The move is necessary because jewel-specific changes are needed, yet teuthology
does not maintain a separate branch for jewel. Also, swift.py is a
Ceph-specific task so it makes more sense to have it in Ceph.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

osdc/Objecter: pass vector by const reference

so we can pass temporary object to it as parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: enable ceph_send_command() to send pg command

Signed-off-by: Kefu Chai <kchai@redhat.com>

crypto: allow PK11 module to load even if it's already initialized

there is chance that other pieces of application loads PK11 module
already and does not finalize it before calling common_init_finish().

also, upon fork, PK11 module resets its entire status including `nsc_init`,
by which PK11 module tell if it is initialized or not. so the behavior
of NSS_InitContext() could be different before and after fork. that's
another reason to ignore CKR_CRYPTOKI_ALREADY_INITIALIZED error (see
NSS_GetError()).

Fixes: http://tracker.ceph.com/issues/19741
Signed-off-by: Kefu Chai <kchai@redhat.com>