bluestore/NVMEDevice.cc: fix NVMEManager thread halt

When enable SPDK in Ceph and start up Ceph development cluster, met
NVMEManager thread halt.

On aarch64 platform, the log as below:

Starting SPDK v18.04.1 / DPDK 18.05.0 initialization...
[ DPDK EAL parameters: nvme-device-manager -c 0x1 -m 2048 --file-prefix=spdk_pid16987 ]
EAL: Detected 46 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/spdk_pid16987/mp_socket
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 8086:953 spdk_nvme
EAL:   using IOMMU type 1 (Type 1)
^C

The reason is that pthread_cond_destroy() cannot destroy the active
condition_variable parameter.

Also on x86 debug builds we get the following error messages due to
probe_queue_lock still being active during ~NVMEManager:

/home/ubuntu/ceph/src/common/mutex_debug.h: 114: FAILED ceph_assert(r == 0)
 ceph version 14.0.1-1862-g403622b (403622be721a460f3dff2d84f6bfc628f5026704) nautilus (dev)

The change fixes the issue.

Fixes: http://tracker.ceph.com/issues/37720
Signed-off-by: tone.zhang <tone.zhang@arm.com>
Signed-off-by: Steve Capper <steve.capper@arm.com>

Merge PR #25622 into master

* refs/pull/25622/head:
Objecter.cc: make op resend logic more readable

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25360 into master

* refs/pull/25360/head:
qa/workunits/mon/pg_autoscaler: clean up pools afterwards
qa/suites/rados/singletone/all/pg-autoscaler: whitelist health warnings
qa/tasks/ceph: wait for splits/merges before final scrub
mon/OSDMonitor: be tidy with target_size_ratio and pre-nautilus code
mgr/pg_autoscaler: simplify conditions
qa/suites/rados: add simple pg-autoscaler test
qa/workunits/cephtool/test.sh: pg_autoscale_mode=off while testing pg_num etc
doc/rados/operations: document autoscaler and its health warnings
mgr/pg_autoscaler: add pg autoscaler module
pybind/mgr/mgr_util: move format_ helpers out of status module
mon/OSDMonitor: accept optional target_size_{bytes,ratio} to 'osd pool create'
mon/OSDMonitor: remove max_split_count configurable
osd/osd_types: pool_opts_t: int -> int64_t
osd/osd_types: pool_opts: fix whitespace
osd/osd_types: pool_opts_t: make encoding feature-dependent
mgr/devicehealth: pg_num_min 1 for device_health_metrics pool
mon/OSDMonitor: accept optional pg_num_min to 'osd pool create'
mon/OSDMonitor: apply osd_pool_default_pg_autoscale_mode to new pools
pybind/mgr/mgr_module: some accessors
mon/MgrMonitor: enable progress module by default
osd/osd_types: add pool pg_autoscale_mode, pg_num_min, target_size_{bytes,ratio} properties
osdc/Objecter: revise get_latest_version locking
os/memstore: ignore OP_COLL_SET_BITS
qa: generalise REQUIRE_MEMSTORE
mgr: drop GIL in get_config
mon: add 'size' arg to `osd pool create`
mon: use pg_num_target for checks during creation
mgr: revise locking in getter paths
common/options: add `mon_target_pg_per_osd`
mgr: expose OSDMap.pool_raw_used_rate

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/workunits/mon/pg_autoscaler: clean up pools afterwards

This avoids a huge pg merge from 100s to 4, which takes a long time and
makes the teuthology scrub cleanup time out.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singletone/all/pg-autoscaler: whitelist health warnings

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: wait for splits/merges before final scrub

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25056 from fghaas/doc-rgw-openstack

doc: Improve OpenStack integration and multitenancy docs for radosgw

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Maxime Guyot <maxime@root314.com>

Merge pull request #25610 from jtlayton/wip-orchestrator

mgr/orchestrator: fix _list_services display

Merge PR #25614 into master

* refs/pull/25614/head:
cls/rgw: update rgw_cls_usage_log_trim_op encode version

Reviewed-by: Sage Weil <sage@redhat.com>

Improve OpenStack integration and multitenancy docs for radosgw

Add explanatory information on:

* "rgw swift account in url" (including the Swift account in the Swift
  API url and Keystone endpoint)
* "rgw swift versioning enabled" (enabling Swift object versioning)
* "rgw s3 auth use keystone" (enabling S3 authentication against
  Keystone)
* "rgw keystone implicit tenants" (multi-tenancy via Keystone, including
  its implications for the Swift and S3 APIs)

Fixes: http://tracker.ceph.com/issues/36765
Signed-off-by: Florian Haas <florian@citynetwork.eu>

Merge pull request #25629 from tchaikov/wip-37707

install-deps.sh: disable centos-sclo-rh-source

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25628 from rhcs-dashboard/masthead-changes

mgr/dashboard: Changed background color of Masthead to brand gray

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/orchestrator: fix _list_services display

I see this error when using "ceph orchestrator service ls":

Error EINVAL: Traceback (most recent call last):
  File "/usr/lib64/ceph/mgr/orchestrator_cli/module.py", line 318, in handle_command
    return self._handle_command(inbuf, cmd)
  File "/usr/lib64/ceph/mgr/orchestrator_cli/module.py", line 330, in _handle_command
    return self._list_services(cmd)
  File "/usr/lib64/ceph/mgr/orchestrator_cli/module.py", line 165, in _list_services
    s.config_location))
AttributeError: 'ServiceDescription' object has no attribute 'config_location'

The config_locations field should be rados_config_location.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

install-deps.sh: skip unavailable repos

as centos-sclo-rh-source leads us to 404 at this moment. and we are not
using the source repo for building ceph. so we can just skip any
unavailable repo.

Fixes: http://tracker.ceph.com/issues/37707
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

mgr/dashboard: Changed background color of Masthead to brand gray

Fixes: http://tracker.ceph.com/issues/35690
Signed-off-by: Neha Gupta <gnehapk@gmail.com>

mon/OSDMonitor: be tidy with target_size_ratio and pre-nautilus code

Signed-off-by: Sage Weil <sage@redhat.com>

Objecter.cc: make op resend logic more readable

  make op resend logic more readable
  according to discussion at https://github.com/ceph/ceph/pull/25243

Signed-off-by: Song Shun <song.shun3@zte.com.cn>

mgr/pg_autoscaler: simplify conditions

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25470 into master

* refs/pull/25470/head:
mon: CephBool -> CephChoices in command map for pre-nautilus mons
mon/MonCommand: populate 'availability' field for compat

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/suites/rados: add simple pg-autoscaler test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: pg_autoscale_mode=off while testing pg_num etc

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations: document autoscaler and its health warnings

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/pg_autoscaler: add pg autoscaler module

Suggest or make changes to pool pg_num based on either current
utilization or administrator-provided target_size_{bytes,ratio}
expected utilization.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/mgr_util: move format_ helpers out of status module

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: accept optional target_size_{bytes,ratio} to 'osd pool create'

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: remove max_split_count configurable

This isn't really relevant or useful now that the mgr is throttling the
actual pg_num adjustment based on pg_num_target, % misplaced, etc.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pool_opts_t: int -> int64_t

Encode int32_t for pre-nautilus maps.

Fix up accessors.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pool_opts: fix whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pool_opts_t: make encoding feature-dependent

No functional change (yet).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/devicehealth: pg_num_min 1 for device_health_metrics pool

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: accept optional pg_num_min to 'osd pool create'

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: apply osd_pool_default_pg_autoscale_mode to new pools

Default to 'warn' for now.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/mgr_module: some accessors

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MgrMonitor: enable progress module by default

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add pool pg_autoscale_mode, pg_num_min, target_size_{bytes,ratio} properties

These will direct PG autoscaling.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: revise get_latest_version locking

Avoid holding Objecter lock while calling back
into completion.

Signed-off-by: John Spray <john.spray@redhat.com>

os/memstore: ignore OP_COLL_SET_BITS

...rather than crashing out on it.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: generalise REQUIRE_MEMSTORE

Move it up into CephTestCase so that mgr tests can
use it too, and pick it up in vstart_runner.py so
that these tests will work neatly there.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: drop GIL in get_config

Signed-off-by: John Spray <john.spray@redhat.com>

mon: add 'size' arg to `osd pool create`

Take advantage of keyword arguments to extend
what we can do in a single command during pool creation.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: use pg_num_target for checks during creation

This way, someone creating pools can proceed
as long as they've decreased the pg_num_target
of other pools, even if the adjustment hasn't
fully completed yet.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: revise locking in getter paths

There were many places where with_* methods
were blocking on locks while holding the GIL.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #25613 into master

* refs/pull/25613/head:
qa/suites/upgrade/mimic-x: add missing .qa

Reviewed-by: João Eduardo Luís <joao@suse.de>

common/options: add `mon_target_pg_per_osd`

This is the partner to mon_max_pg_per_osd, where
this is a more conservative target for PG auto adjustment,
leaving some breathing room for situations where we
might temporarily exceed our target PG count (but not
want to exceed our maximum PG count)

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: expose OSDMap.pool_raw_used_rate

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #25190 into master

* refs/pull/25190/head:
mgr/prometheus: adjust to new 'df' fields
mon/Monitor: fix newline between df section
doc: update docs for new ceph df output
mon/PGMap: break down RAW usage by device class
mon/PGMap: tweak df headers
mon/PGMap: GLOBAL -> RAW STORAGE in 'df' output
mon/PGMap: dump_fs_stats -> dump_cluster_stats

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25615 into master

* refs/pull/25615/head:
common/options: fix bad default

Reviewed-by: Neha Ojha <nojha@redhat.com>

common/options: fix bad default

Got the fields wrong.

Fixes ff75a0b4f21eb14b11702ab2df82ed594f650fcd

Signed-off-by: Sage Weil <sage@redhat.com>

cls/rgw: update rgw_cls_usage_log_trim_op encode version

commit 7b17da691f3bee8118ac69a7419519017030a4c9 added a 'bucket' field
to this op without bumping the encode version, and is causing failures
on upgrade

Fixes: http://tracker.ceph.com/issues/37703
Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/suites/upgrade/mimic-x: add missing .qa

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25596 into master

* refs/pull/25596/head:
qa/suites/upgrade: fix wrt librados3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25486 into master

* refs/pull/25486/head:
common: enhance debug macro

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #25456 into master

* refs/pull/25456/head:
common/options: set mon or mgr service on these options
common/options: more mon options
common/options: kill old mon_max_pgmap_epochs option
common/options: mon options
mon: remove dead option mon_pg_min_inactive
mon: mon_osd_pool_ec_fast_read -> osd_pool_default_ec_fast_read
common/options: document mon options
common/options: messenger options
common/options: document misc
common/options: compression options
common/options: document monitor log options

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #25563 from rhcs-dashboard/bugfix-read-write-chart

mgr/dashboard: avoid blank content in Read/Write Card

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #25110 into master

* refs/pull/25110/head:
client: fix failure in quota size limitation when using samba

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25174 into master

* refs/pull/25174/head:
mds: fix bug filelock stuck at LOCK_XSYN leading client can't read data

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25481 into master

* refs/pull/25481/head:
mon: mark REMOVE_SNAPS messages as no_reply

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25548 from dzafman/wip-37653

osd: fix shard_info_wrapper encode

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/upgrade: fix wrt librados3

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25594 from alfredodeza/wip-rm37675

ceph-volume tests.functional fix typo when stopping osd.0 in filestore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.strategies remove unused import

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25483 from dzafman/wip-hashinfo

ceph-objectstore-tool: Dump hashinfo

Reviewed-by: Neha Ojha <nojha@redhat.com>

ceph-volume tests.functional fix typo when stopping osd.0 in filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25377 from aclamk/wip-kv-remove-validate

common/KeyValueDB: Get rid of validate parameter.

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: avoid blank content in Read/Write Card

The card has to show either a chart or 'N/A', but no blank content.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #25489 from rhcs-dashboard/36740-add-info-to-pools-table

mgr/dashboard: Add info to Pools table

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mon: CephBool -> CephChoices in command map for pre-nautilus mons

If pre-nautilus mons are in quorum, send out a pre-nautilus compatible
set of commands.  Although nautilus mons can do this same translation on
the fly for pre-nautilus clients, pre-nautilus mons can't, so do it for
all mons.  This means the new CephBool behavior will only start working
once you upgrade all mons to nautilus.

Fixes: http://tracker.ceph.com/issues/37583
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25405 into master

* refs/pull/25405/head:
msg/async: truly use first addr if multiple incoming addrs provided

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25421 from xiexingguo/wip-37507

os/bluestore: add boundary check for cache-autotune related settings

Merge pull request #25422 from rzarzynski/wip-bl-drop-at_buffer_headtail

common: drop at_buffer_{head,tail} from buffer::ptr.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25474 from tchaikov/wip-update-seastar

cmake,seastar: pick up latest seastar

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #25564 into master

* refs/pull/25564/head:
qa/rados/upgrade: align thrashing with upgrade suite, don't import/export pgs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25468 from rjfd/wip-orch-service-descriptor

mgr/orchestrator: added useful attributes to ServiceDescription

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Add info to Pools table

- Column 'Placement Groups' renamed to 'Pg Status':
It shows PG states.
- Created 'CephSharedModule' for shared services within ceph module.
- Created PgCategoryService & PgCategory model (logic encapsulation).
- Color consistency: PG chart (landing page) and cell text color are similar.

Fixes: https://tracker.ceph.com/issues/36740
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

fmt: remove fmt submodule

since seastar contains fmt 5.2.1 as a submodule. there is no need
to have our own copy.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: update to accommodate seastar changes

- seastar now works with fmt 5.2.1
- seastar now contains fmt as a submodule, so use it instead of
  using our own submodule
- seastar now create include directory under gen/include
- we use std::option<> when using seastar, so enable
  Seastar_STD_OPTIONAL_VARIANT_STRINGVIEW accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup latest seastar

we have quite a few hacks to get seastar's cmake to play with the distro
we support, and since seastar's cmake now is completely rewritten. we
can now use the upstream's cmake without having to maintain our patches.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #25573 into master

* refs/pull/25573/head:
osd: kill obsolete comments

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23362 from luhp/master

common: fix a racing in PerfCounters::perf_counter_data_any_d::read_avg

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: added service status attributes to service descriptor

Signed-off-by: Ricardo Dias <rdias@suse.com>

osd: kill obsolete comments

I added these comments a few years ago.
Since bluestore can read things that aren't committed and
https://github.com/ceph/ceph/pull/20177 should have made it work for
filestore too, this shouldn't matter any more.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25488 from trociny/wip-osd-perf-query-rbd-regex

mgr: improvements for dynamic osd perf counters

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25509 from trociny/wip-37596

librbd: disable image mirroring when moving to trash

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

include/rados: s/requires/req/

requires is a keyword in C++20. which is close =)

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25462 from Songweibin/wip-rbd-invalid-pointer

 librbd: misc fix potential invalid pointer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25568 from cbodley/wip-rgw-pubsub-const-realm

rgw: fix compilation after pubsub conflict

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: pubsub uses const RGWRealm

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused get_tier_config()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23298 from yehudasa/wip-rgw-ps-3

rgw: pub-sub

qa/workunits/rbd: add trash move/restore mirror test

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: disable image mirroring when moving to trash

And enable when restoring if mirror pool mode set.

Fixes: https://tracker.ceph.com/issues/37596
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25357 from cbodley/wip-rgw-reshard-yield

rgw: add optional_yield to block_while_resharding()

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

mgr/osd_perf_query: generalize rbd regex for journal and data pool objects

And process collected stats accordingly.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr: include only capturing groups into osd perf query subkeys

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/osd_perf_query: fix regression

(introduced when adding ALL_SUBKEYS_QUERY)

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/prometheus: fix get option methods names

(which has been renamed recently)

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25412 from cbodley/wip-rgw-const-zone

rgw: zone service only provides const access to its data

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/rados/upgrade: align thrashing with upgrade suite, don't import/export pgs

Don't import/export between versions

Fixes: http://tracker.ceph.com/issues/37665
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25549 from batrick/obsolete-comment

mds: remove obsolete comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #25550 from badone/wip-doc-remove-capability-reset-command

doc/user-management: Remove obsolete reset caps command

Reviewed-by: Jos Collin <jcollin@redhat.com>