os/bluestore: fix race between SharedBlobSet::lookup and SharedBlob::put

A                             B
SharedBlobSet::lookup()
  takes lock
  nref is not 0
                              SharedBlob::put()
                                --nref
returns SharedBlobRef,
  ++nref
                                takes cache lock
                                SharedBlobSet::remove
                                  takes lock
                                  removes
                                deletes SharedBlob

-> A ends up with a ref to deleted SharedBlob

Fix by verifying that nref is still zero in SharedBlobSet::remove(),
while we are holding the SharedBlobSet::lock.  The lock ensures that we
have increased the ref for the lookup before entering remove, so we can
verify that nref is still zero before removing it.  If not, we have
raced, and put() bails out and does nothing.

Fixes: http://tracker.ceph.com/issues/36526
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24689 into master

* refs/pull/24689/head:
qa/tasks/ceph_manager: fix get_stuck_pgs from pg dump change
Merge PR #24625 into nautilus
qa/suites/rados/mgr/tasks/module_selftest: whitelist 'foo bar security'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20004 from mogeb/steady-clock-tools-rados

librados: use steady clock for rados_mon_op_timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24658 from tchaikov/wip-18202-rebased

blkdev: Rework API and add FreeBSD support

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

qa/tasks/ceph_manager: fix get_stuck_pgs from pg dump change

Fixes 95b7d2340c04dc7cf90085c89606b8c85a8f2803

Fixes: http://tracker.ceph.com/issues/36485
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24184 into master

* refs/pull/24184/head:
mgr/DaemonServer: remove any upmaps on merging PGs
mgr/DaemonServer: prevent merge if either pg is remapped|upmap
mgr/DaemonServer: move pending merge check for more consistent code
qa/suites/rados/thrash*/thrashers/careful.yaml: thrash with mgr controller
mgr/DaemonServer: add option to bypass careful throttling for thrasher
PendingReleaseNotes: note about mgr/balancer/max_misplaced change
mgr/DaemonServer: remove stale/misleading check
mgr/DaemonServer: throttle pgp_num changes based on misplaced %
mgr/DaemonServer: block pg_num decrease(merge) until pgp_num is reduced
mgr/DaemonServer: adjust_pgs(): cosmetic change to debug output
mon/PGMap: add get_recovery_stats()
mgr/balancer: mgr/balancer/max_misplaced -> pg_max_misplaced
pybind/mgr/mgr_module: add get_option()
mgr/DaemonServer: allow pg_num increases that abort pending merges
mon/OSDMonitor: resent pre-nautilus client ops on aborted merge
mon/OSDMonitor: make pgp_num track pg_num more consistently

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #24654 into master

* refs/pull/24654/head:
osd: remove unused parameter 'dev' in OSD::mkfs function

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: remove any upmaps on merging PGs

Remove any pg_upmap[_items] on pgs that are merging to ensure that they
land on the same OSDs.

This is a bit sloppy: we *could* set the source upmap to match the target
upmap (vs potentially moving both PGs to a third location, and/or then
having the balancer move the resulting PG somewhere else again), but for
now assume upmaps are not a common case and Keep It Simple.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: prevent merge if either pg is remapped|upmap

Remapping means they could be on different OSDs.

Fixes: http://tracker.ceph.com/issues/36166
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: move pending merge check for more consistent code

No functional change, but this makes the code simpler to read.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash*/thrashers/careful.yaml: thrash with mgr controller

Thrash such that we still exercise the careful throttling in the mgr.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: add option to bypass careful throttling for thrasher

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20581 from chrone81/patch-1

doc: Fix EC k=3 m=2 profile overhead calculation example.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22349 from gregsfortytwo/wip-24368-osd-restarts

systemd: only restart 3 times in 30 minutes, as fast as possible

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24646 from dillaman/wip-36500

rbd-mirror: always attempt to restart canceled status update task

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24670 from falcon78921/wip-doc-multisite-link

doc: updated wording from federated to multisite

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24595 from cbodley/wip-24505

rgw: user stats account for resharded buckets

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #24673 from tchaikov/wip-librbd-test-os-cleanup

librbd,test: remove unused context_cb() function, silence GCC warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24645 from cbodley/wip-36496

cls/user: cls_user_remove_bucket writes modified header

Merge PR #24611 into master

* refs/pull/24611/head:
mgr/diskprediction: Solve predictor model file not backward compatibility

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24623 into master

* refs/pull/24623/head:
mgr: Mutex::Locker -> std::lock_guard
kv: Mutex::Locker -> std::lock_guard
librados: Mutex::Locker -> std::lock_guard
mon: Mutex::Locker -> std::lock_guard
osd: Mutex::Locker -> std::lock_guard

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #24605 from tchaikov/wip-use-ccache-the-old-way

cmake: fix compilation with distcc and other compiler wrappers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

blkdev: FreeBSD support

Signed-off-by: Alan Somers <asomers@gmail.com>

blkdev: overhaul API for better portability

* Turn the API into a class so it will work with GoogleMock
* Take file descriptors for all methods, instead of a mix of file
  descriptors, path names, and canonical device names.  It's more
  consistent and it will work better with FreeBSD.
* Split get_device_by_fd into block_device_partition and
  block_device_wholedisk
* Remove dead code

Signed-off-by: Alan Somers <asomers@gmail.com>

blkdev: move test_get_blkdev_size to test_get_blkdev_props

Also, enhance blkdev test program to show more properties than just size

Signed-off-by: Alan Somers <asomers@gmail.com>

blkdev: Add dedicated accessors for all property functions

Hide get_block_device_string_property and get_block_device_int_property
to lower risk of misuse.

Signed-off-by: Alan Somers <asomers@gmail.com>

blkdev: add an enum type for block device properties

Signed-off-by: Alan Somers <asomers@gmail.com>

Merge pull request #21155 from runsisi/wip-auth-cleanup

auth/cephx: minor code cleanup

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24672 from SUSE/wip-orchestrator-result-property

mgr/orchestrator: use result property in Completion classes

Merge pull request #24314 from rhcs-dashboard/dashboards

mgr/dashboard: Grafana dashboard updates and additions

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #24628 from Devp00l/wip-issue-36445

mgr/dashboard: Add missing test requirement "werkzeug"

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #24669 from falcon78921/wip-docs-36527

doc: add spacing to subcommand references

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19386 from ganeshmaharaj/qat-ceph-crypto

crypto: QAT based Encryption for RGW

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: remove unused context_cb() function

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/objectstore: silence "-Wsign-compare" warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: use result property in Completion classes

The _Completion class defined a get_result() method, but the orchestrator
CLI and Rook module both just expected a result property, so let's run with
the property version.

Signed-off-by: Tim Serong <tserong@suse.com>

auth/cephx: remove unused member method

Signed-off-by: runsisi <runsisi@zte.com.cn>

auth: remove unused forward declaration

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #24618 from CharlotteRose/wip-number-eviction-events

doc: numbered eviction situations

Reviewed-by: Jos Collin <jcollin@redhat.com>

crypto: Merge compilation of crypto accelerators under a single cmake.

- Control ISA-L and QAT under a crypto makefile
- Allow conditional compile of QAT.

Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>

crypto: QAT Wrapper for Ceph Encryption

- Crypto plugin created and currently handles only AES_256_CBC
- Contents
-- Poller thread for QAT HW
-- Contiguous memory allocator using QAT's usdm driver
-- Do in-place crypto operation reducing mem usage.
-- Enable multi-QAT instance to allow parallel operations.

Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>

doc: updated wording from federated to multisite

Updated the terminology from federated architecture
to multisite configuration. Updated the reference link for multisite
configuration.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: add spacing to subcommand references

Added appropriate spacing for subcommand
code references.

Fixes: https://tracker.ceph.com/issues/36527
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #24531 from alfredodeza/wip-rm36386

ceph-volume remove version reporting from help menu

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24453 from cbodley/wip-rgw-putobjprocessor

rgw: refactor PutObjProcessor stack

Merge pull request #23304 from smithfarm/wip-23776

 tools: ceph-authtool: report correct number of caps when creating keyring

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc: numbered eviction situations

Signed-off-by: Scoots Hamilton <scoots@redhat.com>

Merge pull request #19571 from Fusl/patch-1

doc: doc/radosgw/s3.rst: Adding AWS S3 `Storage Class` as `Not Supported`

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24660 from tchaikov/wip-crimson-cleanup

crimson: crimson/mon: remove timeout support from mon::Client::authenticate()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24626 from jtlayton/wip-mgr-urlparse

mgr/rook: fix urljoin import

Merge pull request #24604 from yangdongsheng/token_fix

common/Throttle: TokenBucketThrottle: use reference to m_blockers.front()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24653 from mynaramana/claydoc

doc: erasure-code-clay fixes typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22162 from liuzhanhao/master

doc: Update cpp.rst to accommodate the new APIs in libs3

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: remove unused parameter 'dev' in OSD::mkfs function

Signed-off-by: Yao Guotao <yaoguot@gmail.com>

doc: erasure-code-clay fixes typos

Signed-off-by: Myna <mynaramana@gmail.com>

tests/ceph-authtool: add reproducer for "with 0 caps" issue

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tools: ceph-authtool: report correct number of caps when creating

Fixes: https://tracker.ceph.com/issues/23776
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd-mirror: always attempt to restart canceled status update task

Fixes: http://tracker.ceph.com/issues/36500
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/user: cls_user_remove_bucket writes modified header

if cls_user_remove_bucket() modifies the header with dec_header_stats(),
it should write the updated header back

Fixes: http://tracker.ceph.com/issues/36496
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #24503 into master

* refs/pull/24503/head:
qa: increase timeout for cleanup

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #24455 into master

* refs/pull/24455/head:
qa: use timeout for fs asok operations

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #24615 into master

* refs/pull/24615/head:
cmake: mark gcc 7 as minimum C++ compiler (due to std++17)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24617 from p-na/fix-python3-issue

mgr/dashboard: Fix Python3 issue

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crimson/mon: remove timeout support from mon::Client::authenticate()

as the timeout parameter is not used by OSD.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: reuse common_mountcephfs_objs in crimson-common

so we won't built them twice.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #24102 into master

* refs/pull/24102/head:
global: fix err_to_stderr can't work when it was configured to false
osd: replace cout/derr with dout(0) to avoid writing trivial logs to syslog file

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24236 into master

* refs/pull/24236/head:
osd: Log error and abort if fails to pick address

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #24359 into master

* refs/pull/24359/head:
qa/tests: update ansible version to 2.6 for master branch testing.
qa/tests: use lvm as default for ceph-ansible testing, this should also work with raw devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge PR #24603 into master

* refs/pull/24603/head:
crush: get "ceph osd crush class create/rm" back

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24613 into master

* refs/pull/24613/head:
OSD: fix a typo err when increase recvoery bytes statistics

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24586 from mynaramana/claydoc

erasure-code: fixes alignment issue when clay code is used with jerasure, cauchy_orig

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24625 into nautilus

* refs/pull/24625/head:
qa/suites/rados/mgr/tasks/module_selftest: whitelist 'foo bar security'

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mgr/rook: fix urljoin import

This is the correct syntax to ensure that we get urljoin, no matter
the python version.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #24608 from cyx1231st/wip-seastar-msgr-accepting

crimson/net: implement accepting/connecting states

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: implement accepting/connecting states

- introduce the new accepting/connecting connection states.
- return ConnectionRef immediately when connect().
- manage the ownership of the accepting connections.
- manage the ownership of the registered connections.
- encapsulate a Socket class because it is not created when constructing
  a SocketConnection, and allow it to be replaced in the future.
- refactor related interfaces.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge PR #24624 into master

* refs/pull/24624/head:
client: Mutex::Locker -> std::lock_guard
mds: Mutex::Locker -> std::lock_guard

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

erasure-code: fixing bugs in clay-code plugin

* Fixes alignment issue while using clay code with jerasure, cauchy_orig
* Fixes ceph_erasure_code_non_regression to adapt to new decode

Signed-off-by: Myna <mynaramana@gmail.com>

qa/suites/rados/mgr/tasks/module_selftest: whitelist 'foo bar security'

Avoid failures like

"2018-10-16 20:36:00.437153 mgr.y (mgr.25609) 6 : cluster [SEC] foo bar security" in cluster log

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Fix Python3 issue

Which results in a 500 error when trying to access the `Performance
Counter` tab on the OSD list.

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #24629 from cyx1231st/wip-seastar-fix

crimson/net: fix compile errors in test_alien_echo.cc

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: fix compile errors

test_alien_echo.cc is using common_init_finish, but it is not available
if defined WITH_SEASTAR.

Dispatcher::ms_verify_authorizer() is removed by PR#24095.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #24607 from p-na/fix-osd-list-status-labels

mgr/dashboard: Fix spaces around status labels on OSD list

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24614 from s0nea/wip-dashboard-configs-textarea-vertical-resize

mgr/dashboard: configs textarea disallow horizontal resize

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24631 from Devp00l/wip-issue-36466

mgr/dashboard: Add left padding to helper icon

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add left padding to helper icon

Fixes: https://tracker.ceph.com/issues/36466
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add missing test requirement "werkzeug"

Fixes: https://tracker.ceph.com/issues/36445
Signed-off-by: Stephan Müller <smueller@suse.com>

doc: erasure-code cleanup of clay plugin documentation

Signed-off-by: Myna <mynaramana@gmail.com>

mgr/diskprediction: Solve predictor model file not backward compatibility

Use pickle library instead of joblib library. It can avoid library not compatible with different operation system.

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>

TokenBucketThrottle: use reference to m_blockers.front() and then update it

As we need to update the blocker below, then we need to use reference to front object,
rather than a new object of blocker.

Fixes: http://tracker.ceph.com/issues/36475
Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge PR #24578 into master

* refs/pull/24578/head:
pybind/ceph_argparse.py: do not create file for validating CephFilepath

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #24565 into master

* refs/pull/24565/head:
mgr: python 3 compat fixes

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #24558 into master

* refs/pull/24558/head:
pybind/mgr: Fix Python 3 imports in diskprediction & insights

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge PR #24528 into master

* refs/pull/24528/head:
common/condition_variable_debug: fix wait hooks
common/mutex_debug: remove no-op before/after hooks
common/mutex_debug: do lockdep post-lock step in caller
os/bluestore: {Mutex,Cond} -> ceph::{mutex,condition_variable}
os/bluestore: std::recursive_mutex -> ceph::recursive_mutex
os/bluestore: re-add is_locked assert
os/bluestore: std::{mutex,condition_variable} -> ceph::{mutex,condition_variable}
os/bluestore: use deduction for lock_guard<>, unique_lock<>

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge PR #24151 into master

* refs/pull/24151/head:
mgr/devicehealth: use is_valid_daemon_name helper
mgr/devicehealth: generalize to mon and osd daemons
mon: implement 'smart [devid]' tell command
mgr: parse mon metadata properly
mon: report device id used by mon
common/blkdev: add get_device_by_path
common/blkdev: migrate block_device_run_smartctl from OSD.cc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #24566 into master

* refs/pull/24566/head:
osd,mon: keep last_epoch_started along with last_epoch_clean premerge

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr: Mutex::Locker -> std::lock_guard

Signed-off-by: Sage Weil <sage@redhat.com>

kv: Mutex::Locker -> std::lock_guard

Signed-off-by: Sage Weil <sage@redhat.com>

librados: Mutex::Locker -> std::lock_guard

Signed-off-by: Sage Weil <sage@redhat.com>

client: Mutex::Locker -> std::lock_guard

Signed-off-by: Sage Weil <sage@redhat.com>

mds: Mutex::Locker -> std::lock_guard

Signed-off-by: Sage Weil <sage@redhat.com>