common: fix races when visiting a TrackedOp.

Before the commit following race can happen:
  ```
  A           : OpTracker::visit_ops_in_flight(..., callable leaking TrackedOpRef outside)
  A           : Mutex::Locker::Locker(sdata->ops_in_flight_lock_sharded)
  A with lock : (nref > 0) == true

  B           : TrackedOp::put(), nref := 0 // updating the counter is done without the lock
  B           : OpTracker::unregister_inflight_op()
  B           : Mutex::Locker::Locker(sdata->ops_in_flight_lock_sharded)

  A with lock : visit() -> TrackedOp::get(), nref := 1
  A with lock : Mutex::Locker::~Locker()

  B with lock : boost::intrusive::list::iterator_to(op)
  B with lock : boost::intrusive::list::erase(iter)
  B with lock : Mutex::Locker::~Locker()

  A           : TrackedOp::put(), nref := 0
  A           : OpTracker::unregister_inflight_op()
  A           : Mutex::Locker::Locker(sdata->ops_in_flight_lock_sharded)
  A with lock : boost::intrusive::list::iterator_to(op) // oops as op doesn't belong to the list anymore
  ```

Fixes: https://tracker.ceph.com/issues/24664
Related-To: https://tracker.ceph.com/issues/24037
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #22830 from ricardoasmarques/fix-24756

mgr/dashboard: Fix RBD object size dropdown options

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22644 from Devp00l/wip-forms-helper-service

 mgr/dashboard: CdFormGroup

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix RBD object size dropdown options

Fixes: https://tracker.ceph.com/issues/24756
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #22461 from tspmelo/wip-osd-flags-ui

mgr/dashboard: Add UI for Cluster-wide OSD Flags configuration

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #22757 from votdev/improve_str_to_bool

mgr/dashboard: Improve str_to_bool

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #22837 from dzafman/wip-fix-timeout

test: test_get_timeout_delays() fix

Reviewed-by: Erwan Velu <erwan@redhat.com>

test: test_get_timeout_delays() fix

Caused by: 7b0d1c8b8acff2a7010bfb0400df09786033ac63

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #22165 into master

* refs/pull/22165/head:
qa: add one-off clusters to qa/cephfs/clusters
qa: allocate more space for VM disk
qa/cephfs/clusters/*: bigger cinder volumses

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Move validators into forms directory

Now that 'forms' directory exists, the validators are moved to
it. Angular also does this with it's validators ('@angular/forms').

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in deletion modal

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RGW user capability

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RGW subuser modal

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RGW user s3 key modal

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RGW user form

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RGW bucket form

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RBD snapshot form

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use forms in RBD form

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: CdFormBuilder

Now the form builder can be used to build CdFormGroups.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: CdFormGroup

CdFormGroup extends 'FormGroup' with a few new methods that will help
form development.

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #22828 from ceph/wip-conf-error-message

ceph-volume: provide a nice errror message when missing ceph.conf

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22548 from ovh/bp-cls-otp-build

cls: build cls_otp only WITH_RADOSGW

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22764 from ceph/wip-rm24504

ceph-volume: refuse to zap mapper devices

Reviewed-by: Alfredo Deza

ceph-volume: always ignore a missing ceph conf in main.py

Now that we have a nice error message when a ceph.conf is missing
and we try to use values from it, maintaining a list of commands that
don't need ceph.conf isn't as helpful. We had actually missed 'simple
trigger' when we first implemented this causing all our luminous tests
for simple to fail when we backported.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22808 from jcsp/wip-mgr-doc-initial-modules

doc/dashboard: don't advise mgr_initial_modules

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

doc/dashboard: don't advise mgr_initial_modules

This setting tends to confuse people, as it's only respected
on the very first startup of the cluster.  Instead, mention
it (with appropriate caveats) on the general mgr admin
page.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #22802 from bradfitz/its

doc: fix some it's -> its typos

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: fix some it's -> its typos

Signed-off-by: Brad Fitzpatrick <brad@danga.com>

Merge pull request #22761 from fullerdj/wip-djf-24686

osd/filestore: Change default filestore_merge_threshold to -10

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: show a nice error message when ceph.conf is not loaded

If the ceph configuration file is not loaded correctly and then values
from it are used then an undescript error message is shown, e.g.

AttributeError: 'property' object has no attribute 'get'

With this change that same error condition shows the message:

RuntimeError: No valid ceph configuration file was loaded.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #21772 from oritwas/wip-rgw-test-bi-list

rgw: test bi list

Merge pull request #22769 from idryomov/wip-rbdmap-remote-fs-pre

systemd/rbdmap.service: order us before remote-fs-pre.target

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #22760 from s0nea/wip-dashboard-config-option-flags

mgr/dashboard: add supported flag information to config options documentation

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add UI for Cluster-wide OSD Flags configuration

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22772 from joscollin/wip-doc-dashboard-ref

doc: s/doc/ref for dashboard urls

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22776 from tchaikov/wip-thread-setaffinity

crimson/thread: pin thread pool to given CPU

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/thread: pin thread pool to given CPU

to take the full advantage of seastar reactor, we need to confine the
alien threads to given CPU core(s). in current setting, it's assumed
that alien threads do not perform CPU-bound tasks, so a single core
would suffice. we can always change the interface to pass a cpu_set_t
or a std::bitset if one CPU core is not enough.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #22111 into master

* refs/pull/22111/head:
cephfs-journal-tool: check fsid when import purge queue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22718 into master

* refs/pull/22718/head:
qa/workunits/suites/ffsb.sh: fix strncpy bug in ffsb

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add one-off clusters to qa/cephfs/clusters

To consistently handle openstack configuration.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: allocate more space for VM disk

Otherwise daemons may run out of space due to logging.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs/clusters/*: bigger cinder volumses

Fixes: http://tracker.ceph.com/issues/24238
Signed-off-by: Sage Weil <sage@redhat.com>

osd/filestore: Change default filestore_merge_threshold to -1

Performance evaluations of medium to large size Ceph clusters have
demonstrated negligible performance impact from unnecessarily deep
directory hierarchies but significant performance impact from filestore
split and merge activity. Disable merges by default.

Fixes: http://tracker.ceph.com/issues/24686
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #22736 from tchaikov/wip-seastar-lockless-shared-lru

common/shared_cache: add lockless SharedLRU

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: s/doc/ref for dashboard urls

Replaced :doc: with :ref: as per https://github.com/ceph/ceph/pull/22750#discussion_r198895040.

Signed-off-by: Jos Collin <jcollin@redhat.com>

systemd/rbdmap.service: order us before remote-fs-pre.target

If "/usr/bin/rbdmap unmap-all" notices a file system mounted on top of
an rbd device, it will call umount, interfering with systemd shutdown
logic.  Make sure we aren't invoked until all _netdev mounts are dealt
with by systemd.

Fixes: http://tracker.ceph.com/issues/24713
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #22771 from tchaikov/wip-test-librados-ns

test/librados: fix LibRadosList.ListObjectsNS

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

lvm zap: refuse to zap mapper devices

Fixes: http://tracker.ceph.com/issues/24504
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

adds the util.disk.is_mapper_device method

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22086 from ifed01/wip-ifed-new-bluefs-pcounter

os/bluestore: introduce new BlueFS perf counter to track the amount of

Reviewed-by: Sage Weil <sage@redhat.com>

test/librados: fix LibRadosList.ListObjectsNS

rados_ioctx_get_namespace() returns the length of namespace string on
success, not 0.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21634 from smithfarm/wip-bci-complain-dupe

scripts: backport-create-issue: complain about duplicates and support mimic

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: add flag info to config options documentation page

Add information about the supported flags (introduced by
https://github.com/ceph/ceph/pull/22595) to the config options
documentation page.

Fixes: https://tracker.ceph.com/issues/24605
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #22756 from wjwithagen/wjw-fix-missing-vector

mgr: `#include <vector>` for clang

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22752 from tchaikov/wip-freebsd-w/o-rdma

cmake,do_freebsd.sh: disable rdma features

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #22743 from rzarzynski/wip-common-drop-static_assert.h

common: drop static_assert.h as it looks unused.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22720 from rzarzynski/wip-build-static-ceph-common-in-osd

cmake: link ceph-osd with common statically

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22727 from dzafman/wip-21664

qa/standalone/scrub: When possible show side-by-side diff in addition to regular diff

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22763 from dzafman/wip-remove-sudo

qa: Don't use sudo when moving logs

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22441 from ErwanAliasr1/evelu-makecheck

Improving make check reliability

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

qa: Don't use sudo when moving logs

Caused by: f0964beac5b8b54bc44a11bfb50bdea1f2ae8825

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #22565 from tchaikov/wip-seastar-thread-pool

crimson/thread: add thread pool

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: add crimson static library

libcrimson consolidates the object library of crimson_{net,thread} and
seastar_{buffer,net,thread}_objs. by offering a static library glueing
seastar,  we can link against these libraries and its dependencies in a
simpler way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/thread: add a thread pool impl

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22007 from dalgaaf/wip-da-SCA-20180514

rgw, common: Fixes SCA issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: extract the RDMA libraries interface properties out

Signed-off-by: Kefu Chai <kchai@redhat.com>

do_freebsd.sh: disable rdma features

FreeBSD does have libibiverb and librdmacm, but we disable them anyway
for minimal dependency.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: consolidate WITH_{VERBS,RDMACM}

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22759 from ricardoasmarques/wip-fix-rbd-features-style

mgr/dashboard: Fix RBD features style

Reviewed-by: Tiago Melo <tmelo@suse.com>

systemd/rbdmap.service: remove a dependency on local-fs.target

We don't require anything outside of rootfs.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: Improve str_to_bool

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #22750 from joscollin/wip-dashboard-intro

doc: updated the overview and glossary for dashboard

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Fix RBD features style

Signed-off-by: Ricardo Marques <rimarques@suse.com>

doc: Updated dashboard doc references

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: updated the overview and glossary for dashboard

* Updated the overview of dashboard to avoid many paragraphs,
some outdated statements, etc.
* Updated glossary, glossary references and urls
* Dropped unnecessary term *module* in possible places
* Unified the new dashboard name as *Ceph Manager Dashboard*

Signed-off-by: Jos Collin <jcollin@redhat.com>

mgr: fix Clangs/FreeBSD missing vector

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #22702 from dzafman/wip-standalone-log

qa: For teuthology copy logs to teuthology expected location

Reviewed-by: Neha Ojha <nojha@redhat.com>

cmake: remove stale comments

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22726 from yuriw/wip-yuriw-crontab

qa/tests - added mimic-p2p suite

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #22560 into master

* refs/pull/22560/head:
qa/fs/upgrade/snaps: whitelist inconsistent rstat
mds: track snapshot count in rstat
mds: add MAY_SNAPSHOT in MDSAuthCaps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22719 into master

* refs/pull/22719/head:
qa: fix symlinks indirectly pointing at qa to .qa
qa: add .qa helper link

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22631 from tianshan/fix_notify_thread

rgw: fix meta and data notify thread miss stop cr manager

Reviewed-by: Casey Bodley <cbodley@redhat.com>

common: drop static_assert.h as it looks unused.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #22632 from tianshan/fix_index_zones_trace

rgw: fix index complete miss zones_trace set

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22660 from cfanz/wip-rgw-fix-24603

rgw-multisite: fix endless loop in RGWBucketShardIncrementalSyncCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #22725 into master

* refs/pull/22725/head:
qa/workunits/suites/blogbench.sh: use correct dir name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #22656 into master

* refs/pull/22656/head:
doc: update experimental features - snapshots
doc: highlight the flags

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22698 from cbodley/wip-async-shared-mutex

common: reintroduce async SharedMutex

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #22724 from ceph/rm-23941

ceph-volume: error on commands that need ceph.conf to operate

Reviewed-by: Alfredo Deza <adeza@redhat.com>

build: link ceph-osd with common statically. Enforce IE TLS model.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/shared_cache: add lockless SharedLRU

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: extract LockMutex and LockPolicy out

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/shared_cache: remove less and hash template parameters

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22710 from tchaikov/wip-config-without-lock

common: add lockless `md_config_t`

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22712 from tspmelo/wip-package-version

mgr/dashboard: fix the version of all frontend dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22734 from s0nea/wip-update-map

githubmap, organizationmap: cleanup and add/update contributors/affiliation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #20297 from tanghaodong25/rdma-cm

msg/async/rdma: add iWARP RDMA protocol support

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: error on commands that need ceph.conf to operate

We had been ignoring the failure to load ceph.conf for all subcommands
but most of them require that a ceph.conf be present. This changes that
so only commands that do not need ceph.conf ignore the failure to find
it.

Fixes: http://tracker.ceph.com/issues/23941
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

organizationmap: add openATTIC team affiliation

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

organizationmap: fix typo

Signed-off-by: Tatjana Dehler <tdehler@suse.com>