os/bluestore: fix flush_commit locking

We were updating the txc state to KV_DONE and queuing the oncommits
waiters without holding any locks.  This was mostly fine, *except* that
Collection|OpSequencer::flush_commit(Context *) was looking at the state
(under qlock) and also adding items to oncommits.

The flush_commit() method is only used in 2 places: osd bench, and the
PG reset_interval_flush outgoing message blocking machinery (which is
a bit ick). The first we could get rid of, but the second is hard to
remove (despite its ick factor).

The simple fix is to take qlock while updating the state value and
working with oncommits.

Fixes: http://tracker.ceph.com/issues/21480
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21932 from yuriw/wip-yuriw-add-dollar-rgw

tests/qa: Adding $ distro mix - rgw

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22011 from dillaman/wip-rbd-release-notes

doc/releases/mimic: tweak RBD major features

Merge PR #22017 into master

* refs/pull/22017/head:
ptl-tool: skip empty comment body

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: skip empty comment body

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21924 into master

* refs/pull/21924/head:
align lspools output in file-layouts and manual-freebsd-deployment
Fixed configuration setting standby_replay
Fixed the order of configuration

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22016 from LenzGr/dashboard-mimic-reasenotes

doc/releases/mimic: Updated dashboard description

doc/releases/mimic: Updated dashboard description

Fixed typo and slightly reworked the description of the mgr
dashboard in the "Major changes" section. Updated the
description in the "Notable changes" section as well by adding
a note about creating an SSL certificate.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

doc/releases/mimic: tweak RBD major features

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #21842 into master

* refs/pull/21842/head:
MDSMonitor: clean up use of pending fsmap in uncommitted ops
MDSMonitor: refactor last_beacons to use mono_clock
mds: refactor MDSMap init
mds: refactor FSMap init
mds: refactor Filesystem init
mds: move compat set methods to MDSMap

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #21986 from cbodley/wip-24124

Revert "common: add an async SharedMutex"

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #21987 from liewegas/wip-mimic-upgrade

doc/releases/mimic: upgrade steps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/releases/mimic: upgrade steps

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21809 from tchaikov/wip-22420

mon,osd: dump "compression_algorithms" in "mon metadata"

Reveiwed-by: John Spray <john.spray@redhat.com>

Merge pull request #21933 from tspmelo/fix-unittest-warnings

mgr/dashboard: Fix unit tests cli warnings

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21967 from votdev/bug_24097

mgr/dashboard: Navbar dropdown button does not respond for mobile browsers

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21971 from tspmelo/remove-husky

Mgr/dashboard: Remove husky package

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21895 from yuriw/wip-yuriw-add-dollar1

tests/qa: trying $ distro mix

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21949 from yuriw/wip-yuriw-crontab

qa/tests: added --filter-out="ubuntu_14.04"

Merge PR #21966 into master

* refs/pull/21966/head:
mds: handle imported session race
mds: check for session import race

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #20975 from theanalyst/doc/releases/mimic-dev-2

doc: mimic rc1 release notes

mds: handle imported session race

Thanks to Yan Zheng for identifying and reviewing the fix.

Fixes: http://tracker.ceph.com/issues/24087
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: check for session import race

Credit to Yan Zheng for identifying the race condition [1].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1572555#c7

Test-for: http://tracker.ceph.com/issues/24072

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Revert "common: add an async SharedMutex"

This reverts commit 67fe5d25a7bd57841d5d1a2636c4e46ae97ac3e7.

Fixes: http://tracker.ceph.com/issues/24124
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Mgr/dashboard: Remove husky package

It seems pretty-quick is not restricted to the root of the dashboard frontend
and this could bring undesired changes to some files.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21785 from votdev/info_no_rgw

mgr/dashboard: Display notification if RGW is not configured

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #21975 from ukernel/wip-revert-23837

Revert "ceph-fuse: Delete inode's bufferhead was in Tx state would le…

Revert "ceph-fuse: Delete inode's bufferhead was in Tx state would lead a assert fail"

This reverts commit 07e3bceea78dc8ecd76abb1cafca5c9d1fde521e.
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mgr/dashboard: Display useful info if RGW is not configured

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add ModuleStatusGuardService route guard

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add info panel component

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #20118 from b-ranto/wip-selinux-ldconfig

selinux: Allow ceph to execute ldconfig

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21943 from tchaikov/wip-24077

qa/tasks/mgr: fix test_pool.py

Reviewed-by: Ricardo Dias <rdias@suse.com>

selinux: Allow ceph to block suspend

Ceph is now trying to block suspend in certain cases, we need to update
the policy accordingly.

Signed-off-by: Boris Ranto <branto@redhat.com>

selinux: Allow ceph to execute ldconfig

The ceph-volume testing showed that the ceph daemons can run ldconfig in
a corner case when they are forbidden access to some files. This patch
allows ceph to execute ldconfig in Enforcing mode.

Fixes: https://tracker.ceph.com/issues/22302
Signed-off-by: Boris Ranto <branto@redhat.com>

mgr/dashboard: Navbar dropdown button does not respond for mobile-like browsers

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #20731 from yuyuyu101/wip-opti-ac

msg/async: avoid put message within write_lock

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21946 from dillaman/wip-24092

rbd: disable cache for actions that open multiple images

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/tests: added --filter-out="ubuntu_14.04"

This is needed for all "client-upgrade-*" suites running on mimic

Fixes https://tracker.ceph.com/issues/2402

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21804 from dillaman/wip-24000

osdc: self-managed snapshot helper should catch decode exception

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #21929 from majianpeng/fio-ceph-objectstore

test/fio: add option single_pool_mode in ceph-bluestore.fio

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #21905 from jcsp/wip-24022

osd: write "bench" output to stdout

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21950 from liewegas/wip-23899

osd: clean up smart probe

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21940 from dmick/wip-remove-cephd

cmake: Remove embedded 'cephd' code

Reviewed-by: Bassam Tabbara <Bassam.Tabbara@Quantum.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: fix test_pool.py

* per command desc of "osd pool set", "val" parameter should be a "string"
  not a number. so update the test_pool.py accordingly.
* also as a follow-up of 65925511bc, we need to update the test
  accordingly to reflect the change in the error's str representation.

Fixes: http://tracker.ceph.com/issues/24077
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #21615 into master

* refs/pull/21615/head:
ceph-fuse: Delete inode's bufferhead was in Tx state would lead a assert fail

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21769 into master

* refs/pull/21769/head:
mds: properly reconnect client caps after loading inodes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21883 into master

* refs/pull/21883/head:
mds: tighten conditions of calling rejoin_gather_finish()
mds: avoid calling rejoin_gather_finish() two times successively

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21923 into master

* refs/pull/21923/head:
mds: set could_consume to false when no purge queue item actually executed

Reviewed-by: Zheng Yan <zyan@redhat.com>

tests/qa: adding rados/.. dirs

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

osd: clean up smart probe code a bit

Signed-off-by: Sage Weil <sage@redhat.com>

Remove embedded 'cephd' code

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #21942 from tchaikov/wip-qa-ceph-disk-no-mon-config

qa/workunits/ceph-disk: --no-mon-config

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21945 from tchaikov/wip-dashboard-errno

mgr/dashboard: set errno via the parent class

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21947 from yuriw/wip-yuriw-crontab

qa/tests: added client-upgrade-luminous suit

qa/tests: added client-upgrade-luminous suit

Fixes https://tracker.ceph.com/issues/24021
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rbd: disable cache for actions that open multiple images

Fixes: http://tracker.ceph.com/issues/24092
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: set errno via the parent class

update the test accordingly. see also rados.Error.__str__()

Signed-off-by: Ricardo Dias <rdias@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21524 from dillaman/wip-23517

test/librados_test_stub: deterministically load cls shared libraries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21415 from ErwanAliasr1/evelu-cephdisk

ceph-disk: get_partition_dev() should fail until get_dev_path(partnam…

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/ceph-disk: --no-mon-config

port a095ec8b to the qa/workunits

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add release notes for v13.1.0 mimic

Adding mimic.rst and dropping related changes from PendingReleaseNotes. Also
added a few ref. labels from the major changes section

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

align lspools output in file-layouts and manual-freebsd-deployment

align as per https://github.com/ceph/ceph/commit/50dc08cffd99fd3540d708f6774646806e839553

Signed-off-by: Jos Collin <jcollin@redhat.com>

osd: fix osd smart data collection segment fault issue.
output.read_fd() could return zero length. it needs to
be handled gracefully.

always call invoke join() if spawn() is successfully returned.
otherwise there will be assert failure in ~SubProcess().

Fixes: http://tracker.ceph.com/issues/23899
Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

tests/qa: Adding $ distro mix - rgw

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21292 from zy751713126/snapshot_count

rbd: add snapshot count in rbd info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd/info: add snapshot count in rbd info

Signed-off-by: Zheng Yin <zhengyin@huayun.com>

test/fio: add option single_pool_mode in ceph-bluestore.fio

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ceph_release: nautilus

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/mimic'

ceph_release: we are an rc now

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Fix unit tests cli warnings

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21930 from dillaman/wip-rbd-backports-mimic

rbd: backport of mimic bug fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21893 from cbodley/wip-rgw-req-state-time

rgw: use coarse_real_clock for req_state::time

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #21931 from yuriw/wip-yuriw-fix-mimic-subset

tests/qa - fix mimic subset for nightlies

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests/qa - fix mimic subset for nightlies

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21680 from cbodley/wip-rm-replica-log

rgw: remove all traces of cls replica_log

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Revert "stop.sh: use --no-mon-config when trying to unmap rbd devices"

This reverts commit 6c722cd5f3f829ccf14280e7e36387bafd819c7a.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3c6b01574766ca128818461d6630866cc0c0405b)

rbd: don't load config overrides from monitor initially

The overrides will be loaded when we connect to the cluster via librados
and the current approach prevents running 'rbd help' without a running
cluster.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 46eafdabde57736d2a8b63f55cab94461885a957)

rbd-mirror: ensure remote demotion is replayed locally

The bootstrap process cannot immediately quit if it notices the remote
image is not primary. Instead, it needs to continue if the local image is
still chained to the remote.

Fixes: http://tracker.ceph.com/issues/24009
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d35bc94319c6fa267e973e533f4591568289ee96)

rbd-mirror: fix potential race between image sync and shut down

Fixes: http://tracker.ceph.com/issues/24008
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4f2dbcefc5e3547926b8d34135a808266bd059b8)

rbd-mirror: ensure instance replayer is stoped before destructing

Fixes: http://tracker.ceph.com/issues/24008
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9ba94f303072374b5c7c9265c3a493fddce2f8ca)

Merge pull request #21831 from tchaikov/wip-23966

osdc: invoke notify finish context on linger commit failure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21898 from adamemerson/wip-fedora28

src: Eliminate new warnings in Fedora 28

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21861 from adamemerson/wip-clangtasm

osd,mon,pybind: Make able to compile with Clang

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21912 from dillaman/wip-24064

rbd: disk-usage can now optionally compute exact on-disk usage

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21910 from dillaman/wip-rbd-help

rbd: don't load config overrides from monitor initially

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21823 from dillaman/wip-24009

rbd-mirror: ensure remote demotion is replayed locally

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21817 from dillaman/wip-24008

rbd-mirror: fixed potential crashes during shut down

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21853 from rishabh-d-dave/volclient-keyargs-for-all-exceptions

rados.pyx: make all exceptions accept keyword arguments

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21886 from jcsp/wip-no-install-hello

cmake: do not install `hello` demo module

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21782 from wido/mgr-telegraf

mgr/telegraf: Telegraf module for Ceph Mgr

Reveiwed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: set could_consume to false when no purge queue item actually executed

Fixes: http://tracker.ceph.com/issues/24073
Signed-off-by: Xuehan Xu <xuxuehan@360.cn>

Merge pull request #21918 from smithfarm/wip-24003-mimic

script: build-integration-branch: avoid Unicode error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Fixed configuration setting standby_replay

Signed-off-by: Jos Collin <jcollin@redhat.com>

Fixed the order of configuration

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #21891 from cbodley/wip-cmake-crypto-plugins

cmake: move crypto_plugins target

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Revert "stop.sh: use --no-mon-config when trying to unmap rbd devices"

This reverts commit 6c722cd5f3f829ccf14280e7e36387bafd819c7a.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21770 from zy751713126/remove-redundant-code

rbd: add the judgment of resizing the image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21914 from cbodley/wip-common-async

common: add a generic async Completion for use with boost::asio

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>