osd: restructure and simplify internal fullness checks

First, eliminate the useless nearfull failsafe--all it did was
generate a log message, which we can do based on the OSDMap
states.

Add some new helpers.

Unify the cluster nearfull/full vs failsafe states so that
failsafe is a "really" full state that is more severe than
full, so we have NONE, NEARFULL, FULL, FAILSAFE.

Pull the full/nearfull ratios out of the OSDMap (remember that
we require luminous mons, so these will be initialized).

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: disable old 'pg set_[near]full_ratio ...' in luminous

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: change [near]full_ratio tests

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: implement new 'osd set-[near]full-ratio ...' commands

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: initialize osdmap ratios from pgmap on upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: set osdmap ratios on mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: handle MOSDFull messages from OSDs

Signed-off-by: Sage Weil <sage@redhat.com>

osd: rename failsafe [near]full getters appropriately

...and make most of these methods private to clarify the public
interface

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add [near]full_ratio to OSDMap[::Incremental]

This used to live in PGMap; we're moving it here for luminous
(which makes more sense anyway!).

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add per-osd FULL and NEARFULL state bits

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13425 from dzafman/wip-15912

osd: various changes for preventing internal ENOSPC condition

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13783 from cbodley/wip-18980

rgw: if user.email is empty, dont try to delete

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13379 from jecluis/wip-mon-luminous-features

mon: enable luminous monmap feature on full quorum

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13582 from guzhongyan/cleanup_execute_ctx

osd: correct the func name in execute_ctx() log messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13699 from yuyuyu101/wipasync-name

msg/async: set thread name for msgr worker

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13726 from dachary/wip-crushtool-json

crush: add --dump to crushtool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13730 from idryomov/wip-osd-rm-incremental

osd/OSDMap: don't set weight to IN when OSD is destroyed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13715 from wangzhengyong/master

os/bluestore: remove unused local variable "pos"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13728 from ifed01/wip-bluestore-fix-smallwrite

os/bluestore: fix a bug in small write handling on sharded extents

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13736 from rzarzynski/wip-bs-bitmap-exact-type-in-leaf

os/bluestore: leverage the type knowledge in BitMapAreaLeaf.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13743 from rzarzynski/wip-bs-bitmap-zone-direct-vector

os/bluestore: remove unneeded indirection in BitMapZone.

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13228 from trociny/wip-qa-rbd_mirror-opts

qa/workunits/rbd: tweak rbd-mirror config to spead up testing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13772 from Liuchang0812/wip-add-override-in-mgr-headers

mgr: add override in headers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13765 from guihecheng/rgwfs-time-attrs

rgw_file: posix style atime,ctime,mtime

Merge pull request #13712 from linuxbox2/wip-dtor-fhcache

rgw_file: RGWFileHandle dtor must also cond-unlink from FHCache

Merge pull request #13645 from yehudasa/wip-rgw-support-ragweed

rgw: add apis to support ragweed
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #13413 from zhaochao/fix-mailmap-affiation

mailmap: fix Zhao Chao affiliation.

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: delete_system_obj() fails on empty object name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: if user.email is empty, dont try to delete

Fixes: http://tracker.ceph.com/issues/18980
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13767 from tchaikov/wip-vstart-no-multiple-mgr-x

vstart.sh: do not add host for mgr.* section if not $overwrite_conf

Reviewed-by: Sage Weil <sage@redhat.com>

rgw_file: RGWFileHandle dtor must also cond-unlink from FHCache

Formerly masked in part by the reclaim() action, direct-delete now
substitutes for reclaim() iff its LRU lane is over its high-water
mark, and in particular, like reclaim() the destructor is certain
to see handles still interned on the FHcache when nfs-ganesha is
recycling objects from its own LRU.

Fixes: http://tracker.ceph.com/issues/19112
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

mgr: add override in headers

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #13584 from theanalyst/doc/release-luminous

doc: releases update the luminous, hammer, jewel release dates

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

os/bluestore: "noid" is not always necessary in clone op

Signed-off-by: wangzhengyong@cmss.chinamobile.com

vstart.sh: do not add host for mgr.* section if not $overwrite_conf

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13678 from liewegas/wip-readable

test/encoding: fix readable.sh bugs; fix ceph-object-corpus

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file: posix style atime,ctime,mtime

As an ganesha FSAL backend, rgw_file should properly maintain
the atime,ctime,mtime properly against operations such as:
(read,write) for file
(create,unlink,mkdir,rmdir,rename) for dir
(setattr) for file and dir

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

Merge pull request #13762 from Liuchang0812/wip-remove-unnecessary-code-in-mon

mon: remove unnecessary function declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: remove unnecessary function declaration

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12566 from Liuchang0812/wip-rados

tools/rados: remove useless function declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12421 from cernceph/wip-dvanders-split

os/filestore/HashIndex: be loud about splits

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12627 from liupan1111/wip-fix-remove-when-full

rbd: When Ceph cluster becomes full, should allow user to remove rbd …

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13289 from songbaisen/s13

rados:  we need to get the latest osdmap when pool does not exists

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

os/bluestore: remove unneeded indirection in BitMapZone.

BitMapZone aggregates BmapEntries through a pointer to independently-
allocated std::vector which in turn has its own pointer to the real
data. This indirection is unnecessary as std::vector is pretty cheap
in the terms of memory overhead. For instance, on x86-64 & GCC 5.4.0
sizeof(std::vector) == 24, which translates to 3 raw pointers.

Stripping the indirection could have positive impact on processor's
caches and allow to minimize the number of memory accesses.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13599 from kylinstorage/cleanup_bitmapallocator

os/bluestore/BitMapAllocator: rm unused variable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13744 from smithfarm/wip-19129

build/ops: add psmisc dependency to ceph-base (deb and rpm)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13750 from Saumay-Agrawal/master

mon: Update OSDMon.cc comments

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13751 from Saumay-Agrawal/patch-1

doc: Update sample.ceph.conf

Reviewed-by: Sage Weil <sage@redhat.com>

Update sample.ceph.conf

I have fixed the cleanup issue #19090: Wrong hard-coded URLs
Please add these changes to your documentation.

Update OSDMon.cc

I have fixed the issue cleanup #19090: Wrong hard-coded URLs
I have replaced the wrong URLs with correct URLs redirecting to the Tunables headline under Crush Map documentation.
Please add it to the documentation.

Merge pull request #13748 from tchaikov/wip-disable-dwz

rpm: disable dwz to speed up valgrind

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mon: test 'mon feature' cli

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa/workunits/ceph-helpers: add wait_for_quorum()

Takes optional timeout and desired quorum size

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: enable persistent monmap features on full quorum

We will now only enable persistent features automatically
when ALL the monitors in the monmap are in the quorum.
 #noMonitorLeftBehind

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: better 'mon features' cli

Allows listing supported and currently set monmap features, as well as
setting and unsetting them.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #13571 from trociny/wip-18784

rbd-mirror A/A: leader should track up/down rbd-mirror instances

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13746 from tahia-khan/wip-doc-quick-ceph-deploy

doc: deletes duplicated word and clarifies an example

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13689 from Liuchang0812/wip-19107-cleanup-confutil

common: use ref to avoid unnecessary memory copy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13739 from Liuchang0812/fix-unittest-hostname

test: unittest_hostname compile error on freebsd

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #13693 from Liuchang0812/wip-add-override-in-mon-headers

mon: add override in headers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13701 from dillaman/wip-19010

librbd: simplify image open/close semantics

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

common: use ref to avoid unnecessary memory copy

Fixes: http://tracker.ceph.com/issues/19107
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

doc: deletes duplicated word and clarifies an example

Signed-off-by: Tahia Khan <tahia.khan@gmail.com>

Merge pull request #13679 from dillaman/wip-librbd-image-cache-clip

librbd: image-extent cache needs to clip out-of-bounds read buffers

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mon: add override in headers

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

build/ops: add psmisc dependency to ceph-base (deb and rpm)

Fixes: http://tracker.ceph.com/issues/19129
Signed-off-by: Nathan Cutler <ncutler@suse.com>

os/bluestore: switch to alloc_dbg_assert in BmapEntityListIter::next().

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

os/bluestore: inline for the sake of BitMapAreaLeaf::child_check_n_lock.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

mailmap: fix Zhao Chao affiliation.

Signed-off-by: Zhao Chao <zhaochao1984@gmail.com>

Merge pull request #13427 from ly798/update-info

mailmap: Liu Yang affiliation

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #13738 from yuyuyu101/wip-fix-ib

msg/async/rdma: fix potential racing connection usage

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #13655 from Adirl/ulimit

msg/async/rdma: check ulimit

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: check ulimit

On RDMA, memlock should be unlimited. If it is not then
warn the user

issue: 973654

Change-Id: If1e07722aaca9df88714828ca3f7398a2504e262
Signed-off-by: Sarit Zubakov <saritz@mellanox.com>
Signed-off-by: Adir Lev <adirl@mellanox.com>

Merge pull request #13510 from yuyuyu101/wip-rdma-buffer-size

common/config_opts: improve rdma buffer size to 128k

test: unittest_hostname compile error on freebsd

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #13713 from drunkard/master

doc: misc minor fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

rpm: disable dwz to speed up valgrind

Fixes: http://tracker.ceph.com/issues/19099
Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: fix incorrect method name

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: keep qp_conns lock to prevent connection alive

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: remove unused variables

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13525 from yuyuyu101/wip-infiniband-fork

msg/async/rdma: make Infiniband can be forkable

msg/async/rdma: restart Infiniband resources to handle fork properly

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13666 from Adirl/chunk_size

msg/async/rdma: rename chunk_size to buffer_size

Reviewed-by: Haomai Wang <haomai@xsky.com>

doc: add missed commas

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

doc: add mark for quoted content

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

doc: command typo fix, s/radosgw_admin/radosgw-admin

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

doc: fix aiming of command ping

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

doc: hyperlink typo fix in dev/index

The doc compiler complains about this.

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

os/bluestore: leverage the type knowledge in BitMapAreaLeaf.

We're sure the only element type BitMapAreaLeaf aggregates is
BitMapZone. There is no business to go through vptrs and thus
prohibit compiler to inline in a method that is crucial for
overall performance of the BitMap allocator.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13680 from yuyuyu101/wip-tx-zerocopy

msg/async/rdma: refactor tx handle flow to get rid of locks

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #13706 from zmc/wip-openstack-volumes

qa/suites/ceph-deploy: Drop OpenStack volume count

Merge pull request #13657 from badone/wip-rgw-setuid-no-coredump

rgw: set dumpable flag after setuid post ff0e521

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #12998 from Liuchang0812/wip-11081

common: fix that $host always expands to localhost instead of actual hostname

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13651 from rzarzynski/wip-bs-bitmap-nocct-in-entry

os/bluestore: remove CephContext* from BmapEntry.

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13662 from majianpeng/aio-memset

os/aio: remove the redundant memset(struct iocb)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13668 from songbaisen/m3

osd: the osd should not share map with others when it is in stopping state

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13702 from liewegas/wip-trackedop-desc

common/TrackedOp: make TrackedOp::reset_desc() safe

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/ceph-deploy: Drop OpenStack volume count

Looks like we only need two per node, since there is only one OSD per
node, and ceph-deploy wants two disks per OSD to account for the
journal.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #13729 from dillaman/wip-librbd-compile-warnings

librbd: eliminate compiler warnings

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

osd/OSDMap: don't set weight to IN when OSD is destroyed

Since commit 4e28f9e63644 ("osd/OSDMap: clear osd_info, osd_xinfo on
osd deletion"), weight is set to IN when OSD is deleted.  This changes
the result of applying an incremental for clients, not just OSDs.
Because CRUSH computations are obviously affected, pre-4e28f9e63644
servers disagree with post-4e28f9e63644 clients on object placement,
resulting in misdirected requests.

Fixes: http://tracker.ceph.com/issues/19119
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>