os/bluestore: separate finisher for deferred_try_submit

Reusing finishers[0], which is used for completions back into the OSD,
is deadlock-prone: the OSD code might block trying to submit new IO or
while waiting for some other bluestore work to complete.

Fixes: http://tracker.ceph.com/issues/21207
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17401 from liewegas/wip-disable-pidfile

test/CMakeLists: disable test-pidfile.sh

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17390 from tchaikov/wip-test-coredump

test: Don't dump core when using EXPECT_DEATH

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

test/CMakeLists: disable test-pidfile.sh

Signed-off-by: Sage Weil <sage@redhat.com>

test: Don't dump core when using EXPECT_DEATH

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17362 from C2python/update_example

doc: Revise the Example of Bucket Policy

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Adam Emerson <aemerson@redhat.com>

Merge pull request #17387 from tchaikov/wip-package-kvstoretool-man

rpm,deb: package ceph-kvstore-tool man page

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17145 from optimistyzy/822_bl

os/bluestore/NVMEDevice: change write_bl to bl

Reviewed-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: change write_bl to bl

With this patch, both read and write can use this field in
task.

Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

rpm,deb: package ceph-kvstore-tool man page

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17382 from liewegas/wip-integration

src/script/build-integration-branch

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/script/build-integration-branch

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17333 from liewegas/wip-luminous-mon-bits

mon: final luminous compatset feature and osdmap flag

Merge pull request #17376 from dillaman/ritz303-wip-doc-20437

doc/rbd: iSCSI Gateway Documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17217 from tchaikov/wip-19605

osd/PG: discard msgs from down peers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17339 from amitkumar50/cov-osd-2

osd: Deleting dead code PrimaryLogPG.cc

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

mon: set purged_snapdirs OSDMap flag once snapsets have all converted

This makes it easier to test whether the upgrade + conversion has
completed.  In particular, mimic+ will be able to simply test for this
flag without waiting for complete PG stats to know whether it is safe to
upgrade beyond luminous.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: CEPH_MON_FEATURE_INCOMPAT_LUMINOUS

- set this once we have a luminous quorum
- require luminous if we have this set locally
- include this on mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17247 from liewegas/wip-post-luminous-cleanup

osd: clean up pre-luminous compat cruft (part 1 of many)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17071 from ivancich/wip-dup-ops-perf-regression

osd: speed up get_key_name

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17206 from rzarzynski/wip-rgw-21085

rgw: fix accessing expired memory in PrefixableSignatureHelper.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17250 from shashalu/fix-get-bucket-location

rgw: return bucket's location no matter which zonegroup it located in.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17294 from tchaikov/wip-clang-warnings

rgw,unittest_bit_alloc: silence clang analyzer warning

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17343 from yaozongyou/rgw-admin-check-parameters-for-friendly-prompt

rgw-admin: check input parameters for friendly prompt.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: tweaks for the RBD iSCSI configuration documentation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15994 from chuang-he/fix_decrypt

rgw: Data encryption is not follow the AWS agreement

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rbd: Replaced numbered steps with a # for dynamic numbering

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the content based on engineering feedback

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the content based on engineering feedback

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the content based on engineering feedback

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the requirements based on engineering feedback

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI gateway requirements content

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI gateway overview content

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI gateway 'gwtop' monitoring content

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI targets, Ansible and CLI docs

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI initiators, RHEL, Windows, and ESX docs

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the "toctree" adding the iSCSI gateway overview doc

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the unmodified iSCSI initiator images for ESX and Windows 2016

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc: Revise the Example of Bucket Policy
Signed-off-by: Wen Zhang zhangwen1@unionpay.com

Merge pull request #17358 from tchaikov/wip-doc-disable-ofp

doc/dev/perf: add doc on disabling -fomit-frame-pointer

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16357 from Blackclaws/master

ceph-disk activate unlocks bluestore data partition

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16312 from tchaikov/wip-store-event-if-added

common,mds,mgr,mon,osd: store event only if it's added

Reviewed-by: Sage Weil <sage@redhat.com>

doc/dev/perf: add doc on disabling -fomit-frame-pointer

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15847 from jan--f/doc-fix-multimds-cmds

doc: add allow_multimds and fs_name parameter

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #16210 from aclamk/fix-radosgw-nfs-sync-hint

rgw,nfs: Add hint to use -o sync when mouting

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #16542 from linuxbox2/wip-rados-urls

librados.h:  add LIBRADOS_SUPPORTS_APP_METADATA

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17346 from idryomov/wip-krbd-xfstests-update

qa/suites/krbd: rbd_xfstests job overhaul

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16771 from yanyixing/fix-base

mgr/dashboard: fix pool size base conversion

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16988 from mikulely/wip-mstart

mstart.sh: support read CLUSTERS_LIST from env var

Reviewed-by: Sage Weil <sage@redhat.com.>

Merge pull request #17057 from mikulely/enhance-vstart

vstart.sh: quote cmd params when display executing cmd

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17092 from Liuchang0812/wip-man-kvstore-tool

doc: add ceph-kvstore-tool's man

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17119 from nrdmn/wip-dashboard-httproot

pybind/mgr/dashboard: add url_prefix

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17135 from amitkumar50/cov-tools-1

tools: Initialization of *server, command variables

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17143 from smithfarm/wip-50-too-short

doc: SubmittingPatches: clarify PR title section

Merge pull request #17177 from liupan1111/wip-fix-rm

osd: allow FULL_TRY after failsafe

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17220 from amitkumar50/cov-test-2

test: Check to avoid divide by zero

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17313 from mslovy/wip-fix-storetest

os: kstore fix unittest for FiemapHole

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17329 from xiexingguo/wip-21162

mon/OSDMonitor: make 'osd crush rule rename' idempotent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17296 from amitkumar50/cov-test-4

test: assert check for negative returns

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17297 from amitkumar50/cov-os-2

os: Removing dead code from LFNIndex.cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17334 from chenliuzhong/modify-fault_rang-parameter-in-reshard

os/bluestore: ExtentMap::reshard - fix wrong shard length

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

include/ceph_features: enable SERVER_MIMIC feature bit

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17337 from scienceluo/wip-luo-fix-cinder-conf-branch

doc: move glance_api_version option to the right place

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17341 from amitkumar50/cov-common-1

common: logically dead code inside shunique_lock.h

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16915 from theanalyst/doc-12.2.0-major-changes

doc: 12.2.0 major release announcements

doc: release notes: changelog for 12.2.0

Amalgamated all the dev & RC changelogs + the changelog we saw post
12.1.4 into one large 12.2.0 changelog.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

osd: speed up get_key_name

Reduce string manipulation and creation of an extra string by
streamlining code for get_key_name in both eversion_t and
pg_log_dup_t.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #17308 from jcsp/wip-21157

mgr: fix crash in MonCommandCompletion

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #17312 from cbodley/wip-21155

qa: create_cache_pool no longer runs 'pool application enable'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17125 from xiexingguo/wip-add-bucket-with-loc

mon/OSDMonitor: add location option for "crush add-bucket" command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17239 from zmedico/PGPool-update-eliminate-expensive-union_of

osd/PG: PGPool::update: avoid expensive union_of

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17276 from yuyuyu101/wip-mon-loop

msg/async: don't stuck into resetsession/retrysession loop

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17288 from yuyuyu101/wip-21143

msg/async: don't kill connection if replacing

Reviewed-by: Sage Weil <sage@redhat.com>

common: logically dead code inside shunique_lock.h

Fixes the coverity issue:

** 1352097 Structurally dead code

CID 1352097 (#1 of 1): Structurally dead code (UNREACHABLE)
unreachable: This code cannot be reached: return ceph::shunique_lock<...

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

osd: Deleting dead code PrimaryLogPG.cc

Fixes the coverity issue:

** 1415775 Logically dead code
Here r cannot be less than 0.
CID 1415775 (#1 of 1): Logically dead code (DEADCODE)
dead_error_line: Execution cannot reach this statement: return r;

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #17254 from joscollin/wip-cleanup-rgw-override

rgw: add missing override in list_keys_init()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/krbd: new rbd_xfstests job definition

ext4 seems to be a better choice for our purposes -- less test churn,
rather small and reliable exclude list.

All excluded tests but generic/050 fail with no krbd in the mix, most
have popped up on linux-ext4 list at least once.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/rbd.xfstests: take exclude list from yaml

Different filesystems (and further, different configurations of the
same filesystem) need different exclude lists.  Hard coding the list in
a wrapper script is inflexible.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/run_xfstests.sh: quit building xfstests on test nodes

xfstests is a pain to build on trusty, xenial and centos7 with a single
script.  It is also very sensitive to dependencies, which again need to
be managed on all those distros -- different sets of supported commands
and switches, some versions have known bugs, etc.

Download a pre-built, statically linked tarball and use it instead.
The tarball was generated using xfstests-bld by Ted Ts'o, with a number
of tweaks by myself (mostly concerning the build environment).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/run_xfstests.sh: drop *_MKFS_OPTIONS variables

AFAICT ./check doesn't query EXT4_MKFS_OPTIONS or BTRFS_MKFS_OPTIONS,
We don't need anything special for xfs, so remove all of them to avoid
confusion.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/run_xfstests.sh: kill FS_TYPE variable

./check expects FSTYP, so use that throughout.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: create_cache_pool no longer runs 'pool application enable'

Fixes: http://tracker.ceph.com/issues/21155
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17311 from hjwsm1989/wip-monop-tracker

mon: use ceph_clock_now if message is self-generated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17330 from xiexingguo/wip-class-rename-idempotent

mon/OSDMonitor: make 'osd crush class rename' idempotent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16863 from mikulely/fix-curl-referer

rgw: fix opslog can't record referrer when using curl as client

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mon/OSDMonitor: allow require-osd-release mimic

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16860 from mikulely/fix-opslog

rgw: fix opslog cannot record remote_addr

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #16933 from amitkumar50/cov-rgw-3

rgw: Initialize is_master, max_aio, size

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: check input parameters for friendly prompt.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #17185 from oritwas/wip-rgw-20777

rgw-admin: require --yes-i-really-mean-it when using --inconsistent_index

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: 12.2.0 major release announcements

Reordered the RC releases sections back to their respective components,
added a ceph-mon section, added links to documentation wherever
possible, and a few forgotten RGW announcements. Also cleared up the
PendingReleaseNotes upto this point

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

os/bluestore:reshard-fix wrong shard length

change fault_range parameter from 'needs_reshard_end' to 'needs_reshard_end-needs_reshard_begin'
be given needs_reshard_end may be out of bounds

Signed-off-by: chenliuzhong <liuzhong.chen@easystack.cn>

doc: move glance_api_version option to the right place

Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>

pybind: fix filesystem page's pool size display

Signed-off-by: Yixing Yan <yanyx@umcloud.com>

pybind: using default column text-align same as others

Signed-off-by: Yixing Yan <yanyx@umcloud.com>

pybind: fix health page's pool size display

Signed-off-by: Yixing Yan <yanyx@umcloud.com>

Merge pull request #16774 from yanyixing/fix-sidenav

mgr/dashboard: fix rbd's pool sub menu

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart.sh: quote cmd params when display executing cmd

vstart.sh will echo executed cmd to the stdout:

/root/jiaying/ceph/build/bin/radosgw -c /root/jiaying/ceph/build/ceph.conf --log-file=/root/jiaying/ceph/build/out/rgw.j.log --debug-ms=1 -n client.rgw --rgw_frontends=civetweb port=8000

As params contain whitespaces, pasting the above cmd into the shell didn't start rgw
on the port 8000. Now the vstart.sh output is like the following:

'/root/jiaying/ceph/build/bin/radosgw' '-c' '/root/jiaying/ceph/build/ceph.conf' '--log-file=/root/jiaying/ceph/build/out/rgw.j.log' '--debug-ms=1' '-n' 'client.rgw' '--rgw_frontends=civetweb port=8000'

We can paste above cmd to the shell to restart rgw, it's more convenient for debugging.

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Merge pull request #17283 from liupan1111/wip-final-nbd

rbd-nbd: cleanup for NBDServer shut down

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

doc: add ceph-kvstore-tool's man

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>