mon: rate limit on health check update logging

Fixes: http://tracker.ceph.com/issues/20888
Signed-off-by: John Spray <john.spray@redhat.com>

mon: add helper for emitting logs by health status

We were starting to repeat our little "if(status==HEALTH_ERR"
blocks in too many places.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #17397 from ZVampirEM77/wip-doc

doc: fix doc/radosgw/admin.rst typos

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16521 from myoungwon/wip-objecter-finishers

osd: add multiple objecter finishers

Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16586 from tianshan/fix_logrotate

logrotate: add systemd reload in logrotate in case of centos minimal without killall

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16643 from Aran85/wip-mon-deadlock-on-shutdown

kv/leveldb: fix deadlock when close db

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16733 from liupan1111/wip-bluefs-fio

os: add compile option to build libbluefs.so

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17244 from liewegas/wip-21121

osd: subscribe to new osdmap while waiting_for_healthy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16925 from gmayyyha/mgr-osd-metadata

mgr: implement completion of osd MetadataUpdate

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17036 from majianpeng/add-pmem-submodule

submodule: make libmpem as a submodule.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17354 from liewegas/wip-bluefs-compaction

os/bluestore/BlueFS: compact log even when sync_metadata sees no work

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17357 from Liuchang0812/wip-bluefs-not-assert-eexist

os/bluestore: do not core dump when BlueRocksEnv gets EEXIST error

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17360 from tchaikov/wip-crush-cleanup

crush: various CrushWrapper cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17379 from liewegas/wip-div-p

qa/suites/rados/singleton/diverget_priors*: broaden whitelist

Merge pull request #17401 from liewegas/wip-disable-pidfile

test/CMakeLists: disable test-pidfile.sh

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17390 from tchaikov/wip-test-coredump

test: Don't dump core when using EXPECT_DEATH

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

doc: fix doc/radosgw/admin.rst typos

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

test/CMakeLists: disable test-pidfile.sh

Signed-off-by: Sage Weil <sage@redhat.com>

test: Don't dump core when using EXPECT_DEATH

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17362 from C2python/update_example

doc: Revise the Example of Bucket Policy

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Adam Emerson <aemerson@redhat.com>

Merge pull request #17387 from tchaikov/wip-package-kvstoretool-man

rpm,deb: package ceph-kvstore-tool man page

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17145 from optimistyzy/822_bl

os/bluestore/NVMEDevice: change write_bl to bl

Reviewed-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: change write_bl to bl

With this patch, both read and write can use this field in
task.

Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

rpm,deb: package ceph-kvstore-tool man page

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17382 from liewegas/wip-integration

src/script/build-integration-branch

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/script/build-integration-branch

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17333 from liewegas/wip-luminous-mon-bits

mon: final luminous compatset feature and osdmap flag

Merge pull request #17376 from dillaman/ritz303-wip-doc-20437

doc/rbd: iSCSI Gateway Documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17217 from tchaikov/wip-19605

osd/PG: discard msgs from down peers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17339 from amitkumar50/cov-osd-2

osd: Deleting dead code PrimaryLogPG.cc

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

mon: set purged_snapdirs OSDMap flag once snapsets have all converted

This makes it easier to test whether the upgrade + conversion has
completed.  In particular, mimic+ will be able to simply test for this
flag without waiting for complete PG stats to know whether it is safe to
upgrade beyond luminous.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: CEPH_MON_FEATURE_INCOMPAT_LUMINOUS

- set this once we have a luminous quorum
- require luminous if we have this set locally
- include this on mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17247 from liewegas/wip-post-luminous-cleanup

osd: clean up pre-luminous compat cruft (part 1 of many)

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/singleton/diverget_priors*: broaden whitelist

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17071 from ivancich/wip-dup-ops-perf-regression

osd: speed up get_key_name

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17206 from rzarzynski/wip-rgw-21085

rgw: fix accessing expired memory in PrefixableSignatureHelper.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17250 from shashalu/fix-get-bucket-location

rgw: return bucket's location no matter which zonegroup it located in.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17294 from tchaikov/wip-clang-warnings

rgw,unittest_bit_alloc: silence clang analyzer warning

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17343 from yaozongyou/rgw-admin-check-parameters-for-friendly-prompt

rgw-admin: check input parameters for friendly prompt.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: tweaks for the RBD iSCSI configuration documentation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15994 from chuang-he/fix_decrypt

rgw: Data encryption is not follow the AWS agreement

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rbd: Replaced numbered steps with a # for dynamic numbering

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the content based on engineering feedback

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the content based on engineering feedback

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the content based on engineering feedback

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the requirements based on engineering feedback

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI gateway requirements content

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI gateway overview content

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI gateway 'gwtop' monitoring content

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI targets, Ansible and CLI docs

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI initiators, RHEL, Windows, and ESX docs

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the "toctree" adding the iSCSI gateway overview doc

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the unmodified iSCSI initiator images for ESX and Windows 2016

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

osd: add multiple objecter finishers
Bluestore and Filestore already adapt multiple finishers to improve IO performance.
But Objecter finisher (for tier approach) is still single thread. This can be a
bottleneck if we generate many IOs as Bluestore and Filestore because most of
completion processes (proxy, writeback) are handled by a objecter finisher.

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

doc: Revise the Example of Bucket Policy
Signed-off-by: Wen Zhang zhangwen1@unionpay.com

crush: add more constness to CrushWrapper

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush: get largest bucket id faster

std::map<> is sorted, so we can always get the largest key by reading
it's last element.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17358 from tchaikov/wip-doc-disable-ofp

doc/dev/perf: add doc on disabling -fomit-frame-pointer

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16357 from Blackclaws/master

ceph-disk activate unlocks bluestore data partition

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16312 from tchaikov/wip-store-event-if-added

common,mds,mgr,mon,osd: store event only if it's added

Reviewed-by: Sage Weil <sage@redhat.com>

doc/dev/perf: add doc on disabling -fomit-frame-pointer

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: do not core dump when BlueRocksEnv gets EEXIST error

in `rocksdb::repairdb`, rocksdb needs create a directory `lost` to archive all files.
we will get a EEXIST error when there is a `lost` directory already.

Fixes: http://tracker.ceph.com/issues/20871
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #15847 from jan--f/doc-fix-multimds-cmds

doc: add allow_multimds and fs_name parameter

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #16210 from aclamk/fix-radosgw-nfs-sync-hint

rgw,nfs: Add hint to use -o sync when mouting

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #16542 from linuxbox2/wip-rados-urls

librados.h:  add LIBRADOS_SUPPORTS_APP_METADATA

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore/BlueFS: compact log even when sync_metadata sees no work

It's possible that when sync_metadata() is called there won't be any new
log data to flush because it was already flushed for other reasons (e.g.,
because fsync was called).  However, the log may still be large and in
need of compaction.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17346 from idryomov/wip-krbd-xfstests-update

qa/suites/krbd: rbd_xfstests job overhaul

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16771 from yanyixing/fix-base

mgr/dashboard: fix pool size base conversion

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16988 from mikulely/wip-mstart

mstart.sh: support read CLUSTERS_LIST from env var

Reviewed-by: Sage Weil <sage@redhat.com.>

Merge pull request #17057 from mikulely/enhance-vstart

vstart.sh: quote cmd params when display executing cmd

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17092 from Liuchang0812/wip-man-kvstore-tool

doc: add ceph-kvstore-tool's man

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17119 from nrdmn/wip-dashboard-httproot

pybind/mgr/dashboard: add url_prefix

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17135 from amitkumar50/cov-tools-1

tools: Initialization of *server, command variables

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17143 from smithfarm/wip-50-too-short

doc: SubmittingPatches: clarify PR title section

Merge pull request #17177 from liupan1111/wip-fix-rm

osd: allow FULL_TRY after failsafe

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17220 from amitkumar50/cov-test-2

test: Check to avoid divide by zero

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17313 from mslovy/wip-fix-storetest

os: kstore fix unittest for FiemapHole

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17329 from xiexingguo/wip-21162

mon/OSDMonitor: make 'osd crush rule rename' idempotent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17296 from amitkumar50/cov-test-4

test: assert check for negative returns

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17297 from amitkumar50/cov-os-2

os: Removing dead code from LFNIndex.cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17334 from chenliuzhong/modify-fault_rang-parameter-in-reshard

os/bluestore: ExtentMap::reshard - fix wrong shard length

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

include/ceph_features: enable SERVER_MIMIC feature bit

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17337 from scienceluo/wip-luo-fix-cinder-conf-branch

doc: move glance_api_version option to the right place

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17341 from amitkumar50/cov-common-1

common: logically dead code inside shunique_lock.h

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16915 from theanalyst/doc-12.2.0-major-changes

doc: 12.2.0 major release announcements

doc: release notes: changelog for 12.2.0

Amalgamated all the dev & RC changelogs + the changelog we saw post
12.1.4 into one large 12.2.0 changelog.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

osd: speed up get_key_name

Reduce string manipulation and creation of an extra string by
streamlining code for get_key_name in both eversion_t and
pg_log_dup_t.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #17308 from jcsp/wip-21157

mgr: fix crash in MonCommandCompletion

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #17312 from cbodley/wip-21155

qa: create_cache_pool no longer runs 'pool application enable'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17125 from xiexingguo/wip-add-bucket-with-loc

mon/OSDMonitor: add location option for "crush add-bucket" command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17239 from zmedico/PGPool-update-eliminate-expensive-union_of

osd/PG: PGPool::update: avoid expensive union_of

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17276 from yuyuyu101/wip-mon-loop

msg/async: don't stuck into resetsession/retrysession loop

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17288 from yuyuyu101/wip-21143

msg/async: don't kill connection if replacing

Reviewed-by: Sage Weil <sage@redhat.com>

common: logically dead code inside shunique_lock.h

Fixes the coverity issue:

** 1352097 Structurally dead code

CID 1352097 (#1 of 1): Structurally dead code (UNREACHABLE)
unreachable: This code cannot be reached: return ceph::shunique_lock<...

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

osd: Deleting dead code PrimaryLogPG.cc

Fixes the coverity issue:

** 1415775 Logically dead code
Here r cannot be less than 0.
CID 1415775 (#1 of 1): Logically dead code (DEADCODE)
dead_error_line: Execution cannot reach this statement: return r;

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #17254 from joscollin/wip-cleanup-rgw-override

rgw: add missing override in list_keys_init()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/krbd: new rbd_xfstests job definition

ext4 seems to be a better choice for our purposes -- less test churn,
rather small and reliable exclude list.

All excluded tests but generic/050 fail with no krbd in the mix, most
have popped up on linux-ext4 list at least once.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/rbd.xfstests: take exclude list from yaml

Different filesystems (and further, different configurations of the
same filesystem) need different exclude lists.  Hard coding the list in
a wrapper script is inflexible.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/run_xfstests.sh: quit building xfstests on test nodes

xfstests is a pain to build on trusty, xenial and centos7 with a single
script.  It is also very sensitive to dependencies, which again need to
be managed on all those distros -- different sets of supported commands
and switches, some versions have known bugs, etc.

Download a pre-built, statically linked tarball and use it instead.
The tarball was generated using xfstests-bld by Ted Ts'o, with a number
of tweaks by myself (mostly concerning the build environment).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/run_xfstests.sh: drop *_MKFS_OPTIONS variables

AFAICT ./check doesn't query EXT4_MKFS_OPTIONS or BTRFS_MKFS_OPTIONS,
We don't need anything special for xfs, so remove all of them to avoid
confusion.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>