qa/suites/upgrade/jewel-x: do not thrash cluster full during upgrade

The mon command varies.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suite/rados/upgrade: do not thrash full cluster during upgrade

The mon command varies!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13615 from liewegas/wip-osd-full

mon,osd: new mechanism for managing full and nearfull OSDs for luminous

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #13759 from liewegas/wip-19133

osdc/Objecter: resend RWORDERED ops on full

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #13734 from liewegas/wip-jewel-x

qa/suite/upgrade/jewel-x: various fixes

Reviewed-by: Yuri Weinstein <yweinstei@redhat.com>

Merge pull request #13846 from rzarzynski/wip-qa-rgw-start-apache-first

qa/tasks/rgw.py: start Apache before RadosGW.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/ceph_manager: use new luminous set-full-ratio etc

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13410 from yehudasa/wip-tracing-fix

tracing: don't include oid when tracing at dequeue_op()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13839 from theanalyst/release/10.2.6/changelog

doc: add changelog for v10.2.6 Jewel release

doc: add changelog for v10.2.6 Jewel release

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #13704 from batrick/mds-counter-unify

mds: remove some redundant object counters

Reviewed-by: Yan, Zheng <zyan@redhat.com>

osdc/Objecter: resend RWORDERED ops on full

Our condition for respecting the FULL flag is complex, and involves
the WRITE | RWORDERED flags vs the FULL_FORCE | FULL_TRY flags.  Previously,
we could block a read bc of RWORDRED but not resend it later.

Fix by capturing the complex condition in a respects_full() bool and using
it both for the blocking-on-send and resending-on-possibly-notfull-later
checks.

Fixes: http://tracker.ceph.com/issues/19133
Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/thrashosds: chance_thrash_cluster_full

Induce a momentarily full cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13832 from linuxbox2/wip-rgw-fs_inst

rgw_file:  fix fs_inst progression

Merge pull request #10240 from songbaisen/b2

mon: remove the redudant jugement in paxosservice is_writeable function

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file:  fix fs_inst progression

Reported by Gui Hecheng<guimark@126.com>.  This change is a
variation on proposed fix by Dan Gryniewicz<dang@redhat.com>
to take root_fh.state.dev as fs_inst for new handles.

Fixes: http://tracker.ceph.com/issues/19214
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

qa/tasks/rgw.py: start Apache before RadosGW.

At the end of start_rgw() we wait till establishing HTTP connections
with RadosGW become possible. However, if RadosGW uses the FastCGI,
the condition can't be fulfilled without spawning HTTP server first.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13323 from yehudasa/wip-18079-2

librados: use cursor for nobjects listing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13262 from batrick/multimds-thrasher

Add multimds:thrash sub-suite and fix bugs in thrasher for multimds

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13560 from wjwithagen/wip-wjw-ceph-disk-tests

ceph-disk/tests: Certain partition types do not work on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13810 from yuyuyu101/wip-rdma-inflight

msg/async/rdma: destroy QueuePair if needed

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #13742 from liupan1111/wip-cleanup-journal

os/filestore: use existing variable for same func.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12177 from kylinstorage/wip-remove-unneeded-loop

os/filestore/FileStore.cc: remove unneeded loop

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13741 from rzarzynski/wip-bs-fastcrc32-in-rocks

os/bluestore: enable SSE-assisted CRC32 calculations in RocksDB

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13768 from tchaikov/wip-clang-fixes

librados, osd: clang fixes

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13794 from liewegas/wip-clog-newlines

common: remove \n on clog messages

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13796 from liewegas/wip-debian-base-dbg

debian/control: add ceph-base-dbg

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/upgrade/jewel-x/parallel: upgrade mons before osds

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x/parallel: expand workload matrix

These should run independently against a racing upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13799 from optimistyzy/36_1

bluestore, NVMEDEVICE: Specify the max io completion in conf

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #13806 from vshankar/rbd-internal-api-move

test/librbd: move tests using non-public api to internal

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

test/cli/osdmaptool: fix osdmap output

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: generate health warnings for luminous

Note that this tells us how many OSDs are full or nearfull; it
does not include detailed warnings telling you exactly what the
utilization is because we don't have the full osd_stat_t
available.  We leave it to ceph-mgr to generate those health
messages.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: stop generating health warnings with luminous

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: set cluster flags based on osd flags (luminous)

For luminous, set cluster flags based on osd flags.  Until
require_luminous is set, stick with the old pgmap-based behavior.
Move the new check to encode_pending so that the cluster flag is
set in the same epoch that the osd state(s) change.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: request a fullness state change during tick if needed

Signed-off-by: Sage Weil <sage@redhat.com>

osd: require fullness state changes (as needed) before boot

This ensures that we don't have a down osd that is marked full
go up, then realize it's not actually full, and then clear its
full flag.  That would result in a cluster full blip that isn't
needed. This can easily happen if the full_ratio in the osdmap is
increased while the OSD is down.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: restructure and simplify internal fullness checks

First, eliminate the useless nearfull failsafe--all it did was
generate a log message, which we can do based on the OSDMap
states.

Add some new helpers.

Unify the cluster nearfull/full vs failsafe states so that
failsafe is a "really" full state that is more severe than
full, so we have NONE, NEARFULL, FULL, FAILSAFE.

Pull the full/nearfull ratios out of the OSDMap (remember that
we require luminous mons, so these will be initialized).

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: disable old 'pg set_[near]full_ratio ...' in luminous

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: change [near]full_ratio tests

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: implement new 'osd set-[near]full-ratio ...' commands

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: initialize osdmap ratios from pgmap on upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: set osdmap ratios on mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: handle MOSDFull messages from OSDs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13755 from liewegas/wip-19131

osd/osd_internal_types: wake snaptrimmer on put_read lock, too

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13411 from yehudasa/wip-vstart-rgw-fix

vstart: don't configure rgw_dns_name

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/upgrade/jewel-x/stress-split-erasure-code: box thrashosds

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x/stress-split: finish client.0 upgrade too

Signed-off-by: Sage Weil <sage@redhat.com>

osd: rename failsafe [near]full getters appropriately

...and make most of these methods private to clarify the public
interface

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add [near]full_ratio to OSDMap[::Incremental]

This used to live in PGMap; we're moving it here for luminous
(which makes more sense anyway!).

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add per-osd FULL and NEARFULL state bits

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13425 from dzafman/wip-15912

osd: various changes for preventing internal ENOSPC condition

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13783 from cbodley/wip-18980

rgw: if user.email is empty, dont try to delete

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13379 from jecluis/wip-mon-luminous-features

mon: enable luminous monmap feature on full quorum

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13582 from guzhongyan/cleanup_execute_ctx

osd: correct the func name in execute_ctx() log messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13699 from yuyuyu101/wipasync-name

msg/async: set thread name for msgr worker

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13726 from dachary/wip-crushtool-json

crush: add --dump to crushtool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13730 from idryomov/wip-osd-rm-incremental

osd/OSDMap: don't set weight to IN when OSD is destroyed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13715 from wangzhengyong/master

os/bluestore: remove unused local variable "pos"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13728 from ifed01/wip-bluestore-fix-smallwrite

os/bluestore: fix a bug in small write handling on sharded extents

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13736 from rzarzynski/wip-bs-bitmap-exact-type-in-leaf

os/bluestore: leverage the type knowledge in BitMapAreaLeaf.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13743 from rzarzynski/wip-bs-bitmap-zone-direct-vector

os/bluestore: remove unneeded indirection in BitMapZone.

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/rdma: destroy QueuePair if needed

Signed-off-by: Haomai Wang <haomai@xsky.com>

test / librbd: move non-public api usage tests to internal

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test / librbd: create_image_full_pp() / clone_image_pp() helpers

Signed-off-by: Venky Shankar <vshankar@redhat.com>

bluestore, NVMEDEVICE: Specify the max io completion in conf

I do not think that 4 is meaninful. If a value needs to be determined,
it should be configured in the configuration file.

Signed-off-by: optimistyzy <optimistyzy@gmail.com>

os/filestore: use existing variable to assign newsize.

Signed-off-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #13228 from trociny/wip-qa-rbd_mirror-opts

qa/workunits/rbd: tweak rbd-mirror config to spead up testing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common: remove \n on clog messages

Confirmed these aren't needed, and are inconsistently included.
Remove them where they do appear.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13772 from Liuchang0812/wip-add-override-in-mgr-headers

mgr: add override in headers

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: enable SSE-assisted CRC32 calculations in RocksDB.

By default RocksDB extensively employs CRC32. It has two paths
for the checksum calculation:
 * rocksdb::crc32c::Slow_CRC32,
 * rocksdb::crc32c::Fast_CRC32.

The fast path depends on a run-time discovery of CPU capabilities
AND a compile-time define __SSE4_2__. Although my systems really
offer SSE4.2 support, the macro was undefined resulting in poor
RocksDB performance visible especially during WAL transactions.

The patch (awkwardly) adds the -msse4.2 to CXXFLAGS for RocksDB.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

debian/control: add ceph-base-dbg

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x: remove kraken references

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13765 from guihecheng/rgwfs-time-attrs

rgw_file: posix style atime,ctime,mtime

Merge pull request #13712 from linuxbox2/wip-dtor-fhcache

rgw_file: RGWFileHandle dtor must also cond-unlink from FHCache

Merge pull request #13645 from yehudasa/wip-rgw-support-ragweed

rgw: add apis to support ragweed
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #13413 from zhaochao/fix-mailmap-affiation

mailmap: fix Zhao Chao affiliation.

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: delete_system_obj() fails on empty object name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: if user.email is empty, dont try to delete

Fixes: http://tracker.ceph.com/issues/18980
Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/suites/rados/thrash: add no-thrash item to matrix

Run all of our cases with no thrashing.  This would have uncovered
http://tracker.ceph.com/issues/19131 much sooner!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13767 from tchaikov/wip-vstart-no-multiple-mgr-x

vstart.sh: do not add host for mgr.* section if not $overwrite_conf

Reviewed-by: Sage Weil <sage@redhat.com>

rgw_file: RGWFileHandle dtor must also cond-unlink from FHCache

Formerly masked in part by the reclaim() action, direct-delete now
substitutes for reclaim() iff its LRU lane is over its high-water
mark, and in particular, like reclaim() the destructor is certain
to see handles still interned on the FHcache when nfs-ganesha is
recycling objects from its own LRU.

Fixes: http://tracker.ceph.com/issues/19112
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

mgr: add override in headers

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

osd/PGLog: always use ldpp_dout() to print with dpp

lderr(dpp->get_cct()) could crash if dpp is null.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13584 from theanalyst/doc/release-luminous

doc: releases update the luminous, hammer, jewel release dates

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

os/bluestore: "noid" is not always necessary in clone op

Signed-off-by: wangzhengyong@cmss.chinamobile.com

librados: rados_inconsistent_pg_list() check "buf" param for null before copying to it

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: do not add host for mgr.* section if not $overwrite_conf

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13678 from liewegas/wip-readable

test/encoding: fix readable.sh bugs; fix ceph-object-corpus

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file: posix style atime,ctime,mtime

As an ganesha FSAL backend, rgw_file should properly maintain
the atime,ctime,mtime properly against operations such as:
(read,write) for file
(create,unlink,mkdir,rmdir,rename) for dir
(setattr) for file and dir

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

Merge pull request #13762 from Liuchang0812/wip-remove-unnecessary-code-in-mon

mon: remove unnecessary function declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: remove unnecessary function declaration

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12566 from Liuchang0812/wip-rados

tools/rados: remove useless function declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/filestore/FileStore.cc: remove unneeded loop

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

Merge pull request #12421 from cernceph/wip-dvanders-split

os/filestore/HashIndex: be loud about splits

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12627 from liupan1111/wip-fix-remove-when-full

rbd: When Ceph cluster becomes full, should allow user to remove rbd …

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13289 from songbaisen/s13

rados:  we need to get the latest osdmap when pool does not exists

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

os/bluestore: remove unneeded indirection in BitMapZone.

BitMapZone aggregates BmapEntries through a pointer to independently-
allocated std::vector which in turn has its own pointer to the real
data. This indirection is unnecessary as std::vector is pretty cheap
in the terms of memory overhead. For instance, on x86-64 & GCC 5.4.0
sizeof(std::vector) == 24, which translates to 3 raw pointers.

Stripping the indirection could have positive impact on processor's
caches and allow to minimize the number of memory accesses.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13599 from kylinstorage/cleanup_bitmapallocator

os/bluestore/BitMapAllocator: rm unused variable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13744 from smithfarm/wip-19129

build/ops: add psmisc dependency to ceph-base (deb and rpm)

Reviewed-by: Kefu Chai <kchai@redhat.com>