rest api: Set application variable when not executed as __main__

This allows for using this Python file as a WSGI module as it expects
a variable called application.

Signed-off-by: Wido den Hollander <wido@42on.com>

rest api: Set default ceph.conf and id options

This allows for easy execution of the REST API:

  $ /usr/bin/ceph-rest-api

Signed-off-by: Wido den Hollander <wido@42on.com>

rest api: Listen on :: (IPv6) instead of 0.0.0.0

On Linux systems a socket listening on :: also listens on
all IPv4 addresses.

This makes sure the API is usable over IPv6 as well.

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #15445 from wjwithagen/wip-wjw-freebsd-osd-scrub-repair-2

test/osd/osd-scrub-repair.sh: disable ec_overwrite tests on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15902 from joscollin/wip-1411820-Division-or-modulo-by-zero

test: Division by zero in Legacy::encode_n()

Reviewed By: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15903 from Adirl/data_path_fixes

msg/async/rdma: Data path fixes

Reviewed-by: Haomai Wang <haomai@xsky.com>

test: Division by zero in Legacy::encode_n()

Fixes the Coverity Scan Report:
CID 1411820 (#1 of 1): Division or modulo by zero (DIVIDE_BY_ZERO)
9. divide_by_zero: In expression bl.length() * i / sum, division by expression sum which may be zero has undefined behavior.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15826 from xiexingguo/wip-bluestore-20170622

os/bluestore: cleanup min_alloc_size; some formatting nits

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15865 from tchaikov/wip-mgr-pg-command

mgr: enable ceph_send_command() to send pg command

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15875 from xiexingguo/wip-osdmon-20170623

mon/OSDMonitor: "osd crush class rename" support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15890 from tchaikov/wip-19741

osdc/Objecter: release message if it's not handled

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15891 from minggr/wip-sdata_op_ordering_lock

osd: unlock sdata_op_ordering_lock with sdata_lock hold to avoid miss…

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15899 from theanalyst/doc/release-typo

doc: Luminous release notes typo fixes

Merge pull request #15901 from joscollin/wip-CID1412577-Division-or-modulo-by-float-zero

mon: Division by zero in PGMapDigest::dump_pool_stats_full()

Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15900 from smithfarm/wip-docdir-fix

build/ops: rpm: sane packaging of %{_docdir}/ceph directory

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: allocated reged mem can be greater than requires len

Change-Id: I3ec42afca39c2ad93fc66ed69b3609b68f315e34
Signed-off-by: Adir lev <adirl@mellanox.com>

msg/async/rdma: if failed to send push csi to the back

Change-Id: If6c72917b6e3a836279b86a4d5e9e54383d7b4bd
Signed-off-by: Adir lev <adirl@mellanox.com>

mon: Division by zero in PGMapDigest::dump_pool_stats_full()

Fixes The Coverity Scan Report:
CID 1412577 (#1 of 1): Division or modulo by float zero (DIVIDE_BY_ZERO)
35. divide_by_zero: In expression (float)mk / k, division by expression k which may be zero has undefined behavior.

Signed-off-by: Jos Collin <jcollin@redhat.com>

build/ops: rpm: sane packaging of %{_docdir}/ceph directory

91648044163 introduced a build error:

ceph-common-12.1.0+git.1498286248.2fcedc7b3d-1.1.x86_64.rpm: directories not
owned by a package:
 - /usr/share/doc/packages/ceph

The %docdir directive is a way of flagging anything in that directory as being
documentation. It does not actually package the directory. And we don't need
it because we're not dumping a large number of files into this directory.
For more information, see the "Directory-related Directives" section of
http://ftp.rpm.org/max-rpm/s1-rpm-inside-files-list-directives.html

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: Luminous release notes typo fixes

Suggested-by: Roger Brown <rogerbrown@gmail.com>
Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #15876 from gmayyyha/config-unused-opt

common/config_opts: drop unused opt

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15108 from guihecheng/rgw_file-fix

rgw_file: fix misuse of make_key_name before make_fhk

Merge pull request #15893 from liewegas/wip-valgrind-everywhere

qa: do not restrict valgrind runs to centos

qa: do not restrict valgrind runs to centos

This reverts 693bd238510e69569cc3461f84b04c8667bc11da, which was
added in response to http://tracker.ceph.com/issues/18126. But
we updated the Ubuntu packages in sepia so it should be good to go.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15572 from rzarzynski/wip-rgw-17779

rgw: external auth engines of S3 honor rgw_keystone_implicit_tenants.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #14624 from ceph/wip-s3a-hadoop

qa/tasks: S3A hadoop task to test s3a with Ceph

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: unlock sdata_op_ordering_lock with sdata_lock hold to avoid missing wakeup signal

We are running mysql on top of rbd. sysbench qps occasionally drops to zero
with the INSERT benchmark.

Debug code captured >2s latency between PG::queue_op() and OSD::dequeue_op().
We finally found out that the latency came from below code in OSD::ShardedOpWQ::_process(),

sdata->sdata_cond.WaitInterval(sdata->sdata_lock,
      utime_t(osd->cct->_conf->threadpool_empty_queue_max_wait, 0));

"threadpool_empty_queue_max_wait" is 2s by default.

Normally, it should not sleep for 2s since the comming IO requests will wakeup it.
But there is a small timing window that it missed the wakeup signal actually.
For example,

     msgr-worker-0 thread                    tp_osd_tp thread
     OSD::ShardedOpWQ::_enqueue              OSD::ShardedOpWQ::_process
     ---------------------------             ---------------------------
T1:                                          sdata_op_ordering_lock.Lock()
T2:  sdata_op_ordering_lock.Lock()
                                             "queue empty"
                                             sdata_op_ordering_lock.Unlock()
     "insert op"
     sdata_op_ordering_lock.Unlock()

T3:  sdata_lock.Lock()
T4:                                          sdata_lock.Lock()
     "send wakeup signal"
     sdata_lock.Unock()
                                             // here the wakeup signal has no effect actually
                                             // becuase it has not slept yet.

                                             // then, it sleeps.
                                             WaitInterval(2s)

This patch unlocks sdata_op_ordering_lock with sdata_lock hold in OSD::ShardedOpWQ::_process(),
then the timeline becomes,

     msgr-worker-0 thread                    tp_osd_tp thread
     OSD::ShardedOpWQ::_enqueue              OSD::ShardedOpWQ::_process
     ---------------------------             ---------------------------
T1:                                          sdata_op_ordering_lock.Lock()
T2:  sdata_op_ordering_lock.Lock()
                                             "queue empty"
                                             sdata_lock.Lock()
T3:                                          sdata_op_ordering_lock.Unlock()
     "insert op"
     sdata_op_ordering_lock.Unlock()
     sdata_lock.Lock()

T4:                                          WaitInterval(2s) -> it actually unlocks sdata_lock
     "send wakeup signal"
     sdata_lock.Unock()
                                             //got signal, wakeup immediately

With this one line change, we can avoid occasional high latency.

Signed-off-by: Ming Lin <ming.lin@alibaba-inc.com>

Merge pull request #15665 from oritwas/wip-rgw-reshard-old-bucket

rgw: auto reshard old buckets

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

osdc/Objecter: release message if it is not handled

Fixes: http://tracker.ceph.com/issues/19741
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15821 from jdurgin/wip-20302

qa/suites/powercycle/osd/tasks/radosbench: consume less space

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15154 from jcsp/wip-multimds-stable

Remove "experimental" warnings from multimds

doc: multimds is no longer experimental

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update cephtool test for multimds on by default

Signed-off-by: John Spray <john.spray@redhat.com>

qa: no longer need to explicitly enable multimds

Signed-off-by: John Spray <john.spray@redhat.com>

mds: enable multimds by default in new filesystems

Signed-off-by: John Spray <john.spray@redhat.com>

mon: remove experimental warning on multimds

Signed-off-by: John Spray <john.spray@redhat.com>

mds: don't call StrayManager::eval_stray() for undefined inode

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: drop locks before waiting for export targets

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: handle MDirUpdate race

mds may try discover several times for MDirUpdate, rename may kick
in and cause MDCache::path_traverse() to return error.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't forge replica dirfrag

MDCache::forge_replica_dir() set wrong dir_auth if the forged replica
dirfrag is subtree root.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: avoid submitting log entry while adjusting subtree map

MDCache::eval_subtree_root() may tigger scatter-gather process, which
submits log entry. Submitting log entry while adjusting subtree map is
bad, because subtree map in intermediate state may get used/logged.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't mark nestlock dirty on improper inode

If inode is replica and it has no auth subtree dirfrag, We should
not mark its nestlock dirty.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: create subtree root immediately after directory tree becomes frozen

When a directory tree become frozen, its WAIT_FROZEN contexts are
executed asynchronously. Before Migrator::export_frozen() set export
bounds, MDCache::try_subtree_merge_at() can merge newly imported
subtree into the frozen directory tree. This causes problem if there
are auth pins in newly imported subtree.

The fix is creating subtree root immediately after directory tree
becomes frozen. The new subtree root has dir_auth 'me, me', so it's
not meregeable.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix stray dentry replication in cache rejoin ack

To replicate s stray dentry, we need to replicate all its ancestors.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #15795 from myoungwon/wip-print-ignore_redirect

osd/osd_types: add flag name (IGNORE_REDIRECT)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15656 from aclamk/download_err_with_comp_followup

rgw: use 64-bit offsets for compression

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15848 from xiexingguo/wip-fix-rmcc

src/vstart.sh: kill dead upmap option

Merge pull request #15851 from liewegas/wip-luminous-notes

doc/release-notes: update luminous notes

doc/release-notes: 'osd crush class rename' is coming

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: ceph tell <foo> help

Signed-off-by: Sage Weil <sage@redhat.com>

doc/start/os-recommendations: update

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: note debian stretch addition

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: sleep settings

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: link to EC docs

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: update RGW metadata

Signed-off-by: Sage Weil <sage@redhat.com>

dev/release-notes: various updates from other PR

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: notes on new CLI commands

Signed-off-by: Sage Weil <sage@redhat.com>

mon: 'mon feature list' -> 'mon feature ls'

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: update luminous notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15764 from tchaikov/wip-20342

qa/suites/upgrade/hammer-jewel-x: add luminous.yaml

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15877 from wjwithagen/wip-wjw-vstart-ceph-mgr-restfull

vstart.sh: Work around mgr restfull not available

mon/OSDMonitor: slightly nice error output if set-device-class failed

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: set result code properly if we fail to process "swap-bucket"

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: "osd crush class rename" support

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

./src/vstart.sh: Work around mgr restfull not available

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

common/config_opts: drop unused opt

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

test/osd/osd-scrub-repair.sh: Adjust for FreeBSD

Fixes 2 problems:
 -  Do not test Bluestore on FreeBSD, since that does not work (yet)
    And all erasure code overwrite tests are executed on BlueStore OSDs
    Erasure code overwrites are unsafe on Filestore, see:
    http://docs.ceph.com/docs/master/rados/operations/erasure-code/#erasure-coding-with-overwrites

 -  the JQ expression errors out with:
    (version 1.5-1-g940132e-dirty)
    ====
    jq: error (at :232): Cannot iterate over null (null)
    Traceback (most recent call last):
    File "", line 1, in
    File "/usr/lib64/python2.7/json/init.py", line 338, in loads
    return _default_decoder.decode(s)
    File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File "/usr/lib64/python2.7/json/decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
    ValueError: No JSON object could be decoded
    ====
    Adding a ? to the jq expression allows to proceed on null blocks.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

osdc/Objecter: pass vector by const reference

so we can pass temporary object to it as parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: enable ceph_send_command() to send pg command

Signed-off-by: Kefu Chai <kchai@redhat.com>

crypto: allow PK11 module to load even if it's already initialized

there is chance that other pieces of application loads PK11 module
already and does not finalize it before calling common_init_finish().

also, upon fork, PK11 module resets its entire status including `nsc_init`,
by which PK11 module tell if it is initialized or not. so the behavior
of NSS_InitContext() could be different before and after fork. that's
another reason to ignore CKR_CRYPTOKI_ALREADY_INITIALIZED error (see
NSS_GetError()).

Fixes: http://tracker.ceph.com/issues/19741
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15832 from Adirl/pd_assert

msg/async/rdma: remove assert from ibv_dealloc_pd in ProtectionDomain

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15283 from donglinpeng/copyPartUpload

rgw: multipart copy-part remove '/' for s3 java sdk request header.
(make check failures not related)

Merge pull request #15857 from theanalyst/doc/v12.1.0/release-notes-3

doc: v12.1.0 release notes notable changes addition again

doc: v12.1.0 release notes notable changes addition again

commmits from 9f259cb923e7474229bb2e5e54b59f3b7f068bc5..luminous && sort

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #15819 from linuxbox2/wip-clru-lockfix

rgw_file: avoid a recursive lane lock in LRU drain
(failing make check is unrelated)

rgw_file: avoid a recursive lane lock in LRU drain

While highly unlikely in normal fs operations, draining an
entire filesystem could induce a recursion on object delete/unref,
when provoked by cohort_lru::drain().

Adjusted to use intrusive slist rather than std:vector<T*>
after review.

Found running librgw_file_nfsns unit test.

Fixes: http://tracker.ceph.com/issues/20374
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

radosgw: fixed sizes of integers used to operate on offsets and lengths in compression,
         followup of http://tracker.ceph.com/issues/20231

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Merge pull request #15814 from liewegas/wip-everyone-hates-btrfs

qa/objectstore/filestore-btrfs: test btrfs on trusty only

src/vstart.sh: kill dead upmap option

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15820 from liewegas/wip-20303

mon/PGMap: slightly better debugging around pgmap updates

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15846 from xiexingguo/wip-remap-upmap

mon/OSDMonitor: _apply_remap -> _apply_upmap; less code redundancy

Reviewed-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: less code redundancy

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15805 from jcsp/wip-mds-flush-upgrade

client: check for luminous MDS before sending FLUSH_MDLOG

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15800 from ukernel/wip-20340

mds: save  projected path into inode_t::stray_prior_path

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15776 from ukernel/wip-20338

osdc/Journaler: fix memory leak in Journaler::_issue_read()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15430 from jcsp/wip-20165

mds: fix hangs involving re-entrant calls to journaler

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15548 from ukernel/wip-20196

mds: improvements for stray reintegration

Reviewed-by: John Spray <john.spray@redhat.com>

mon/MonCommand: fix description of "osd pg-upmap"

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: _apply_remap -> _apply_upmap

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15796 from lixiaoy1/fio_ceph_objectstore

fio_ceph_objectstore: Print db_statistics when rocksdb_perf is enabled

Merge pull request #15753 from pritha-srivastava/wip-rgw-s3tests-conf

rgw: Changes for s3test config file, to add user under a tenant.
Reviewed-by: Casey Bodely <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

msg/async/rdma: remove assert from ibv_dealloc_pd in ProtectionDomain
destructor

issue: 1044307

Change-Id: I2fe2099c5f4403f14877e0fbe5d4868dd34d4f9b
Signed-off-by: DanielBar-On <danielbo@mellanox.com>

os/bluestore: avoid initializing finishers in constructor

This is a follow up change of https://github.com/ceph/ceph/pull/15666,
which I forget to update.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add "0x" prefix for hex outputs

Which is the normal case.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: non-atomic min_alloc_size_order

This is not necessary because we now write min_alloc_size into disk
during mkfs.

There are two acaccompanying changes too:
(1) don't call _set_alloc_sizes() during mkfs() any more, which is
not necessary.
(2) don't update min_alloc_size during _set_alloc_sizes(), we instead
update min_alloc_size_order each time we successfully read min_alloc_size
from disk.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: check all users bucket for resharding, this will reshard bucket created in older versions

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore: drop _open_alloc() from mkfs

This is useless as the fsck() process will do necessary validations later.
And dropping this can make mkfs slightly faster.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: prefix open_statfs with "_"

As it is an internal method only.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15317 from liupan1111/wip-fix-list

rbd-nbd: display pool/image/snap information in list output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15803 from mikulely/cleanup-rgw_rados_doc

rgw: update function doc in rgw_rados.h  and rgw_rados.cc

Reviewed-by: Jos Collin <jcollin@redhat.com>