Merge pull request #26089 from taoCH/wip-librgw-create-bucket-bug

rgw: unexpected crash when creating bucket in librgw

Merge PR #27564 into master

* refs/pull/27564/head:
ceph-objectstore-tool: Rename dump-import to dump-export
osd: Fix fast path to use version bumped value

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27546 into master

* refs/pull/27546/head:
mgr/progress: fix typo
mgr/progress: behave if pgs disappear (due to a racing pg merge)

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #27036 from votdev/issue_24453

mgr/dashboard: Manager should complain about wrong dashboard certificate

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #27433 from rhcs-dashboard/wip-36243-master

mgr/dashboard: Add frontend code documentation

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #27530 from yuvalif/fix_pubsub_duplicates

rgw/pubsub: handle subscription conf errors better

Merge pull request #27552 from tchaikov/wip-corpus.rst

doc/corpus: update to adapt the change from autotools to cmake

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Changcheng Liu <changcheng.liu@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/corpus: update to adapt the change from autotools to cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27493 from yuvalif/pubsub_zone_conf_doc

rgw/pubsub: clarify pubsub zone configuration

Merge PR #27585 into master

* refs/pull/27585/head:
messages: remove MNop

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27578 from cbodley/wip-rgw-pubsub-warning

rgw: silence a -Wunused-function warning in pubsub

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #27178 from pritha-srivastava/wip-rgw-iam

rgw: Adding 'iam' namespace for Role and User Policy related REST APIs.

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #27553 from theanalyst/doc/releases/12.2.12

doc: releases/luminous: release notes for 12.2.12

Reviewed-By: Nathan Cutler <ncutler@suse.com>

doc: releases/12.2.12: fix changelog to add issues without trackers

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27491 from sebastian-philipp/rook-service-ls-rgw

mgr/rook: Added missing `rgw` daemons in `service ls`

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #27462 from sebastian-philipp/rook-fix-rgw

mgr/rook: Fix RGW creation

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #27339 from david-z/wip-fix-rbd-map-hang

krbd: fix rbd map hang due to udev return subsystem unordered

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

messages: remove MNop

MNop was introduced along with xio messenger. since we've dropped
xiomessenger in #27021, there is no need to keep MNop around anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27517 into master

* refs/pull/27517/head:
mon: make mon_osd_down_out_subtree_limit update at runtime

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #27458 into master

* refs/pull/27458/head:
bluestore: fix for FreeBSD iocb structure

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27485 into master

* refs/pull/27485/head:
mgr/ActivePyModules: handle_command - fix broken lock

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27506 into master

* refs/pull/27506/head:
src/tests/crushtool: Add test for an empty default
crush/CrushCompiler: Fix __replacement_assert

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27271 from idryomov/wip-krbd-wac-on-dm

qa: add device mapper and lvm test cases for stable pages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27533 from dillaman/wip-39257

rbd-mirror: periodically flush IO and commit positions

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #27229 from dillaman/wip-librbd-writearound

librbd: introduce new default write-around cache policy

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #27446 from trociny/wip-rbd-io-simple-scheduler-improvements

librbd: dispatch delayed requests only if read intersects

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27513 from ddiss/librados_silence_unused_vars

librbd: silence -Wunused-variable warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: silence a -Wunused-function warning in pubsub

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ceph-objectstore-tool: Rename dump-import to dump-export

If user specifies dump-import it will still work, but isn't
in the usage that way.

Fixes: http://tracker.ceph.com/issues/39284
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix fast path to use version bumped value

Fix the code so that the version is in one place
for both the slow and fast path.

Caused by: d2ca3d2feb442f97ca89023c7d01178d96f517a6

Fixes: http://tracker.ceph.com/issues/39281
Signed-off-by: David Zafman <dzafman@redhat.com>

mgr/dashboard: Add frontend code documentation

Adds compodoc as auto-documentation tool for Angular frontend code.

Fixes: http://tracker.ceph.com/issues/36243
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #27470 from cbodley/wip-39160

rgw: don't recalculate etags for slo/dlo

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #27528 from cbodley/wip-qa-rgw-no-debuginfo

qa/rgw: disable debuginfo packages

librbd: write-around cache should honor FUA op flag

If force unit access is specified, the cache should be bypassed just
like other non-optimized IO through the cache.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: support new bench 'full-seq' io pattern option

This ensures IOs are issued in sequential order regardless of the current
IO thread setting.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: releases/luminous: release notes for 12.2.12

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/progress: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/progress: behave if pgs disappear (due to a racing pg merge)

If one of the PGs we care about disappears, consider it complete.

Fixes: http://tracker.ceph.com/issues/38157
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27473 into master

* refs/pull/27473/head:
mgr/DaemonServer: prevent pgp_num reductions from outpacing pg_num merges

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27477 into master

* refs/pull/27477/head:
os/bluestore: check return value of func _open_db_and_around.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #27499 into master

* refs/pull/27499/head:
qa/standalone/osd/osd-markdown: fix dup command disabling

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27516 into master

* refs/pull/27516/head:
mgr/rook: Fix Python 2 regression

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Gabriel Brascher <gabriel@apache.org>

Merge pull request #26816 from iotcg/doc

doc: correct example to use vstart to run up cluster

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27535 from yuriw/wip-yuriw-wip-yuriw-crontab-master

qa/tests: added rados on master, reduced fs, rbd, multimds

Merge pull request #27428 from cyx1231st/wip-seastar-msgr-write

crimson/net: extract do_write_dispatch_sweep()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tests: added rados on master, reduced fs, rbd, multimds

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #27515 from tchaikov/wip-seastar-fix-monc

test/crimson: fix build of unittest_seastar_monc

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

src/tests/crushtool: Add test for an empty default

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #27524 from tchaikov/wip-39164

install-deps.sh: install '*rpm-macros'

Reviewed-by: Boris Ranto <branto@redhat.com>

rbd-mirror: periodically flush prior to updating status

This ensure the most up-to-date commit position is included in the
status report.

Fixes: http://tracker.ceph.com/issues/39257
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: simplify image replayer IO / commit position flush path

There is no current user of the callback context provided to the
flush method and there are no longer any reason to have the flush
support methods be virtual.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27443 from Polynomial-C/tinfo

cmake: Fix build against ncurses with separate libtinfo

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27527 into master

* refs/pull/27527/head:
qa/tests: run luminous-x and mimic-x 2 times a week but with high priority

Reviewed-by: Sage Weil <sage@redhat.com>

rgw/pubsub: handle subscription conf errors better

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

librbd: enable the simple IO scheduler by default

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch to write-around cache policy by default

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new alternative write-around cache layer

This is a replacement for the ObjectCacher-based writethrough/
writeback cache. It permits a configurable maximum byte size
of in-flight writes (bytes) to the OSDs where the writes are
immediately completed back to the caller. Any IO errors are
propagated on the next flush request.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: pass journal tid through flush object dispatch spec

This permits moving the journal flush and commit logic down to the
journal object dispatch layer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove special case for journal replay via writeback

With the future addition of a writearound cache (and to a much
lesser extent the IO scheduler), the code will become easier
to maintain via a single IO path that periodically injects IO
flushes to ensure the data is consistently committed to disk.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: moved object cacher writeback handler to cache sub-directory

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/buffer: hexdump should skip lines that are the same as previous

Previously, it would only skip similar lines if the data was all zeroed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: run luminous-x and mimic-x 2 times a week but with high priority
hope is it will help with locking 5 nodes for tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #27345 into master

* refs/pull/27345/head:
cephfs-shell: Change type of d_name to bytes array
cephfs-shell: Revert "cephfs.pyx: add py3 compatibility"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

install-deps.sh: install '*rpm-macros'

so `yum-builddep` can have access to the latest macros for preparing the
build dependencies

Fixes: http://tracker.ceph.com/issues/39164
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27504 from tchaikov/wip-qa-use-python36-for-py3-tests

qa/tasks/ceph_deploy: install python3.6 instead of python3.4 for py3 tests

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

mon: make mon_osd_down_out_subtree_limit update at runtime

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: Fix Python 2 regression

Fixes: http://tracker.ceph.com/issues/39250
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #27323 into master

* refs/pull/27323/head:
msg/async: avoid creating unnecessary AsyncConnectionRef

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27444 from ifed01/wip-ifed-fix-21312

os/bluestore: call fault_range prior to looking for blob to reuse

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27445 from ifed01/wip-ifed-fix-39143

os/ceph-bluestore-tool: bluefs-bdev-expand asserts if no WAL

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

crimson: add move ctor for CephContext and mon::Client

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: add dummy_auth_handler to test_monc

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

Merge pull request #27438 from Devp00l/wip-issue-37701

mgr/dashboard: Removes distracting search behavior

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

librbd: silence -Wunused-variable warnings

Signed-off-by: David Disseldorp <ddiss@suse.de>

rgw/pubsub: more info on notification deletion compatibility

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #27348 from Devp00l/wip-issue-39086

mgr/dashboard: Enable read only users to read again

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

cmake: Fix build against ncurses with separate libtinfo

Signed-off-by: Lars Wendler <polynomial-c@gentoo.org>

cephfs-shell: Change type of d_name to bytes array

By reverting commit 5106582 'd_name' is always a bytes array. This produces
type error wherever 'd_name' is used with 'str' type. In such cases decode it.

Fixes: https://tracker.ceph.com/issues/39060
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Revert "cephfs.pyx: add py3 compatibility"

This reverts commit 5106582fc7edae7f39161cf89e566c020fcfa0ce.

Signed-off-by: Varsha Rao <varao@redhat.com>

krbd: fix rbd map hang due to udev return subsystem unordered

The order of subsystem returned by udev_device_get_subsystem
might not be same order as adding subsystem by
udev_monitor_filter_add_match_subsystem_devtype. So if block
event is returned first and rbd event is returned next, then
further poll will get nothing back until timed-out.

Fixes: http://tracker.ceph.com/issues/39089
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

crush/CrushCompiler: Fix __replacement_assert

When compiled with _GLIBCXX_ASSERTIONS we see an assert due to UB of
passing the address of an empty vector. Use vector's data member
function instead of address of array syntax.

Fixes: http://tracker.ceph.com/issues/39174
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

qa/tasks/ceph_deploy: install python3.6 instead of python3.4 for py3 tests

EPEL7 has switched over to python3.6 as the main python3. and we started
packaging python bindings for python3.6 since
https://github.com/ceph/ceph-build/pull/1283

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/ActivePyModules: handle_command - fix broken lock

by unlocking properly when module is not available.

Fixes: http://tracker.ceph.com/issues/39235
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #27495 into master

* refs/pull/27495/head:
doc/releases/nautilus: fix config update step

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/standalone/osd/osd-markdown: fix dup command disabling

The ceph cli tool checks for the presence of the variable, not its value.

Fixes: http://tracker.ceph.com/issues/38359
Signed-off-by: Sage Weil <sage@redhat.com>

rgw/pubsub: fix comments from PR #27493

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

doc/releases/nautilus: fix config update step

'ceph ... > /etc/ceph/ceph.conf' fails because bash truncates ceph.conf
before the ceph cli invocation is able to use it to connect to the cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27290 from sebastian-philipp/rook-system-namespace

mgr/rook: Support other system namespaces

Reviewed-by: Blaine Gardner <blaine.gardner@suse.com>
Reviewed-by: Rohan Gupta <rohanrgupta1996@gmail.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #27408 from nathan-weinberg/reformat

mgr/dashboard: Updated existing E2E tests to match new format

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

rgw/pubsub: clarify pubsub zone configuration

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge PR #27490 into master

* refs/pull/27490/head:
PendingReleaseNotes: note on python3.6 changes

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/rook: Added missing `rgw` daemons in `service ls`

Fixes: http://tracker.ceph.com/issues/39171
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

PendingReleaseNotes: note on python3.6 changes

Fixes: http://tracker.ceph.com/issues/39164
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27406 from rhcs-dashboard/fix-39125-master

mgr/dashboard: Add whitelist to guard

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27289 from sebastian-philipp/rook-fix-osd-create

mgr/rook: Fix creation of bluestore OSDs

Reviewed-by: Blaine Gardner <blaine.gardner@suse.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

os/bluestore: check return value of func _open_db_and_around.

In develop, met the following error messages:

2019-04-08 10:24:54.259 7faa25114d80  1 bdev create path /var/lib/ceph/mnt/osd-device-6-data/block.db type kernel
2019-04-08 10:24:54.259 7faa25114d80  1 bdev(0x55ef0840ce00 /var/lib/ceph/mnt/osd-device-6-data/block.db) open path /var/lib/ceph/mnt/osd-device-6-data/block.db
2019-04-08 10:24:54.259 7faa25114d80 -1 bdev(0x55ef0840ce00 /var/lib/ceph/mnt/osd-device-6-data/block.db) _lock flock failed on /var/lib/ceph/mnt/osd-device-6-data/block.db
2019-04-08 10:24:54.259 7faa25114d80 -1 bdev(0x55ef0840ce00 /var/lib/ceph/mnt/osd-device-6-data/block.db) open failed to lock /var/lib/ceph/mnt/osd-device-6-data/block.db: (11) Resource temporarily unavailable
2019-04-08 10:24:54.259 7faa25114d80 -1 bluestore(/var/lib/ceph/mnt/osd-device-6-data) _minimal_open_bluefs add block device(/var/lib/ceph/mnt/osd-device-6-data/block.db) returned: (11) Resource temporarily unavailable
2019-04-08 10:24:54.259 7faa25114d80  1 fbmap_alloc 0x55ef0754ef00 shutdown
2019-04-08 10:24:54.259 7faa25114d80  1 freelist shutdown
2019-04-08 10:24:54.259 7faa25114d80  1 bluestore(/var/lib/ceph/mnt/osd-device-6-data) _upgrade_super from 2, latest 2
2019-04-08 10:24:54.259 7faa25114d80  1 bluestore(/var/lib/ceph/mnt/osd-device-6-data) _upgrade_super done
2019-04-08 10:24:54.264 7faa25114d80 -1 *** Caught signal (Segmentation fault) **
 in thread 7faa25114d80 thread_name:ceph-osd

 ceph version 14.2.0 (3a54b2b6d167d4a2a19e003a705696d4fe619afc) nautilus (stable)
 1: (()+0xf5d0) [0x7faa21d655d0]
 2: (BlueStore::_open_collections(int*)+0x5b) [0x55eefd1f511b]
 3: (BlueStore::_mount(bool, bool)+0x6c6) [0x55eefd22d1c6]
 4: (OSD::init()+0x3aa) [0x55eefcda4e4a]
 5: (main()+0x14fa) [0x55eefcc4137a]
 6: (__libc_start_main()+0xf5) [0x7faa20b483d5]
 7: (()+0x5646c5) [0x55eefcd356c5]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mgr/DaemonServer: prevent pgp_num reductions from outpacing pg_num merges

If we are merging lots of pgs down to a much smaller number of pgs, and
the pgs are able to move quickly (faster than the merges happen), we can
end up with too many pgs on a small number of osds, triggering the max
pgs per osd limits.

Avoid this by preventing the pgp_num reductions from getting too far
out in front of the merges themselves.  Basically, cap the delta between
pgp_num and pg_num to the max_misplaced ratio.  We are already limiting
the movement caused by pgp_num by max_misplaced; this effectively just
makes sure that the actual merging (and pg_num reductions) are keeping
up.

Fixes: http://tracker.ceph.com/issues/38786
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: don't recalculate etags for slo/dlo

for slo, we've already calculated this as 'lo_etag' in get_params()
for dlo, the local 'etag' already contains the hash of an empty string

the calls to complete_etag() were calling hash.Final() a second time on
the same hash without a hash.Restart() in between. this apparently
worked fine with NSS, but with OpenSSL the second call to Final()
returns a different value

Fixes: https://tracker.ceph.com/issues/39160
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #27464 into master

* refs/pull/27464/head:
msg/async/frames_v2.h: fix warning

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27357 from dillaman/wip-39021

librbd: fixed several race conditions related to copyup

Reviewed-by: Mykola Golub <mgolub@suse.com>