mon: update metadata and smart commands to use get_raw_devices

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: add get_raw_devices helper

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: fix BlkDev::get_devid when we got a devname, not fd

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: return optional error string from get_device_id

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: refactor to add block_device_get_metrics returning json

Refactor block_device_run_smartctl into a helper that returns JSON.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25620 into master

* refs/pull/25620/head:
os/bluestore: more robust handling for lack of per-pool stats cases.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25632 into master

* refs/pull/25632/head:
test: add a json tester
ceph_json: parser sets data on all types when parsing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22800 from wjwithagen/wjw-ctest-timeout

cmake: allow setting of the CTest timeout during building.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25661 from joscollin/wip-doc-changing-monitors

doc: fix typo in add-or-rm-mons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25639 from oliveiradan/wip-krb_pr25566_cleanup_oliveiradan

auth/krb: Fix Kerberos build warnings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25537 from jecluis/dev/wip-mgr-module-error

mgr: make module error message more descriptive

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25595 from rzarzynski/wip-ec-drop_bl_crosstalk_dependency

ec: lrc doesn't depend on crosstalks between bufferlists anymore.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25503 from ceph/rbd-mirror-ceph-base

spec: requires ceph base instead of common

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: fix typo in add-or-rm-mons

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #25655 from gregsfortytwo/wip-rbd-snap-doc

doc: Clean up the snapshot consistency note

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #25577 into master

* refs/pull/25577/head:
test: allow ceph_kvstorebench to be build on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25508 into master

* refs/pull/25508/head:
kv/rocksdb_cache: Remove C style struct hack undefined behavior

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25236 into master

* refs/pull/25236/head:
ceph-volume: Add venv to vstart cmake target
mgr/orchestrator: Add test orchestrator

Reviewed-by: Jeff Layton <jlayton@redhat.com>

doc: Clean up the snapshot consistency note

The old note made it sound like we weren't crash-consistent, and had a
confusing section about needing to clone before mounting.
You *do* need to clone the snapshot before mounting it, but that has
nothing to do fs freezing or consistency.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

auth/krb: Fix Kerberos build warnings

Signed-off-by: Daniel Oliveira <doliveira@suse.com>

Merge pull request #25207 from cyx1231st/wip-crimson-msgr-dispatch-events

crimson/net: encapsulate protocol implementations with states (remaining part)

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: let conn directly verify the authorizer with dispatcher

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: remove the messenger gate

There is no need to use messenger gate if SocketConnection themselves
are properly gated.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: encapsulate protocol implementations with accept/connect

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: no open connections in accepting_conns

It should an atomic operation to move the connection from
accepting_conns to connections when switch to the open state.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: don't execute_open() if exception is thrown

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: encapsulate protocol implementations with open state

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: remove unecessary future dependencies for accept/dispatch

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: dispatch events in SocketConnection

* move dispatch(), and exception handling logics in accept() and
  connect() from SocketMessenger into SocketConnection, so we can manage
  the state transition in the same class and at the same abstraction
  level.
* gate the dangling futures in SocketConnection, because the
  connection's smart_ptr won't be hold by messenger any more during
  exception handling.
* don't return close() inside SocketConnection to prevent recursive
  gating -- dead lock.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #25647 from tchaikov/wip-ftbfs-aarch64-gcc-8.1

test/crimson: fix FTBFS of unittest_seastar_perfcounters on arm64

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25641 from tchaikov/cmake-rbd-tracing

cmake: let rbd_api depend on librbd-tp

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25340 from sebastian-philipp/orchestrator-json-output

mgr/orchestrator: Add JSON output to CLI commands

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge PR #25630 into master

* refs/pull/25630/head:
osd/OSDMap: cancel mapping if target osd is out

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25598 into master

* refs/pull/25598/head:
common: avoid bl::get_current_ptr()-driven ref counting in denc.h.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25585 from SUSE/wip-remove-cython-subinterpreter-check

cmake,deb,rpm: remove cython 0.29's subinterpreter check, re-enable build with cython 0.29+

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson: fix FTBFS of unittest_seastar_perfcounters on arm64

this should address the GCC bug which causes following failure:

/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/crimson/test_perfcounters.cc:35:9:
internal compiler error: in tsubst_decomp_names, at cp/pt.c:16537
         for (const auto &[path, perf_counter_ref] : by_path) {
         ^~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: Add venv to vstart cmake target

Adds new add_custom_target `ceph-volume-venv-setup`
that creates a virtual environment, and
calls `python setup.py develop`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Add test orchestrator

1. To be able to run the cli without an external orchestrator.
2. Run the CLI in Teuthology.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cmake: let rbd_api depend on librbd-tp

whoever includes tracing/librbd.h should depends on librbd-tp.

this fix the FTBFS of:

ceph/src/librbd/librbd.cc:50:10: fatal error: tracing/librbd.h: No such
file or directory
 #include "tracing/librbd.h"
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
src/librbd/CMakeFiles/rbd_api.dir/build.make:62: recipe for target
'src/librbd/CMakeFiles/rbd_api.dir/librbd.cc.o' failed

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #25622 into master

* refs/pull/25622/head:
Objecter.cc: make op resend logic more readable

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25360 into master

* refs/pull/25360/head:
qa/workunits/mon/pg_autoscaler: clean up pools afterwards
qa/suites/rados/singletone/all/pg-autoscaler: whitelist health warnings
qa/tasks/ceph: wait for splits/merges before final scrub
mon/OSDMonitor: be tidy with target_size_ratio and pre-nautilus code
mgr/pg_autoscaler: simplify conditions
qa/suites/rados: add simple pg-autoscaler test
qa/workunits/cephtool/test.sh: pg_autoscale_mode=off while testing pg_num etc
doc/rados/operations: document autoscaler and its health warnings
mgr/pg_autoscaler: add pg autoscaler module
pybind/mgr/mgr_util: move format_ helpers out of status module
mon/OSDMonitor: accept optional target_size_{bytes,ratio} to 'osd pool create'
mon/OSDMonitor: remove max_split_count configurable
osd/osd_types: pool_opts_t: int -> int64_t
osd/osd_types: pool_opts: fix whitespace
osd/osd_types: pool_opts_t: make encoding feature-dependent
mgr/devicehealth: pg_num_min 1 for device_health_metrics pool
mon/OSDMonitor: accept optional pg_num_min to 'osd pool create'
mon/OSDMonitor: apply osd_pool_default_pg_autoscale_mode to new pools
pybind/mgr/mgr_module: some accessors
mon/MgrMonitor: enable progress module by default
osd/osd_types: add pool pg_autoscale_mode, pg_num_min, target_size_{bytes,ratio} properties
osdc/Objecter: revise get_latest_version locking
os/memstore: ignore OP_COLL_SET_BITS
qa: generalise REQUIRE_MEMSTORE
mgr: drop GIL in get_config
mon: add 'size' arg to `osd pool create`
mon: use pg_num_target for checks during creation
mgr: revise locking in getter paths
common/options: add `mon_target_pg_per_osd`
mgr: expose OSDMap.pool_raw_used_rate

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/workunits/mon/pg_autoscaler: clean up pools afterwards

This avoids a huge pg merge from 100s to 4, which takes a long time and
makes the teuthology scrub cleanup time out.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singletone/all/pg-autoscaler: whitelist health warnings

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: wait for splits/merges before final scrub

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25056 from fghaas/doc-rgw-openstack

doc: Improve OpenStack integration and multitenancy docs for radosgw

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Maxime Guyot <maxime@root314.com>

Merge pull request #25610 from jtlayton/wip-orchestrator

mgr/orchestrator: fix _list_services display

Merge PR #25614 into master

* refs/pull/25614/head:
cls/rgw: update rgw_cls_usage_log_trim_op encode version

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: more robust handling for lack of per-pool stats cases.

Fixes: https://tracker.ceph.com/issues/37652
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Improve OpenStack integration and multitenancy docs for radosgw

Add explanatory information on:

* "rgw swift account in url" (including the Swift account in the Swift
  API url and Keystone endpoint)
* "rgw swift versioning enabled" (enabling Swift object versioning)
* "rgw s3 auth use keystone" (enabling S3 authentication against
  Keystone)
* "rgw keystone implicit tenants" (multi-tenancy via Keystone, including
  its implications for the Swift and S3 APIs)

Fixes: http://tracker.ceph.com/issues/36765
Signed-off-by: Florian Haas <florian@citynetwork.eu>

Merge pull request #25629 from tchaikov/wip-37707

install-deps.sh: disable centos-sclo-rh-source

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25628 from rhcs-dashboard/masthead-changes

mgr/dashboard: Changed background color of Masthead to brand gray

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

test: add a json tester

specifically test issue #37706

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: parser sets data on all types when parsing

Fixes: http://tracker.ceph.com/issues/37706
This was changed in e47a228a868, but breaks existing usage.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/orchestrator: fix _list_services display

I see this error when using "ceph orchestrator service ls":

Error EINVAL: Traceback (most recent call last):
  File "/usr/lib64/ceph/mgr/orchestrator_cli/module.py", line 318, in handle_command
    return self._handle_command(inbuf, cmd)
  File "/usr/lib64/ceph/mgr/orchestrator_cli/module.py", line 330, in _handle_command
    return self._list_services(cmd)
  File "/usr/lib64/ceph/mgr/orchestrator_cli/module.py", line 165, in _list_services
    s.config_location))
AttributeError: 'ServiceDescription' object has no attribute 'config_location'

The config_locations field should be rados_config_location.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

osd/OSDMap: cancel mapping if target osd is out

Suppose we have a bad pg_upmap_item, say:
```
pg_upmap_items 2.0 [0,8,5,4,6,12]
```
which maps osd.6 to osd.12 that is currently marked as out.

It turns out **maybe_remove_pg_upmaps** can not handle the
above case well because **_apply_upmap** will silently discard
any bad mappings whenver they try to target at some current
__out__ OSDs. So if you call **pg_to_raw_up(2.0)**, you'll probably
get something like: ```up [8,4,6] ``` (e.g., the last mapping pair 6->12
is simply ignored by **_apply_upmap**).

Make **clean_pg_upmaps** do the tidy-up check instead, since it
already has __bare__ access to those pg_upmaps and pg_upmap_items.

Fixes: http://tracker.ceph.com/issues/37501
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: ningtao <ningtao@sangfor.com.cn>

install-deps.sh: skip unavailable repos

as centos-sclo-rh-source leads us to 404 at this moment. and we are not
using the source repo for building ceph. so we can just skip any
unavailable repo.

Fixes: http://tracker.ceph.com/issues/37707
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

mgr/dashboard: Changed background color of Masthead to brand gray

Fixes: http://tracker.ceph.com/issues/35690
Signed-off-by: Neha Gupta <gnehapk@gmail.com>

mon/OSDMonitor: be tidy with target_size_ratio and pre-nautilus code

Signed-off-by: Sage Weil <sage@redhat.com>

Objecter.cc: make op resend logic more readable

  make op resend logic more readable
  according to discussion at https://github.com/ceph/ceph/pull/25243

Signed-off-by: Song Shun <song.shun3@zte.com.cn>

mgr/pg_autoscaler: simplify conditions

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25470 into master

* refs/pull/25470/head:
mon: CephBool -> CephChoices in command map for pre-nautilus mons
mon/MonCommand: populate 'availability' field for compat

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/suites/rados: add simple pg-autoscaler test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: pg_autoscale_mode=off while testing pg_num etc

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations: document autoscaler and its health warnings

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/pg_autoscaler: add pg autoscaler module

Suggest or make changes to pool pg_num based on either current
utilization or administrator-provided target_size_{bytes,ratio}
expected utilization.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/mgr_util: move format_ helpers out of status module

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: accept optional target_size_{bytes,ratio} to 'osd pool create'

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: remove max_split_count configurable

This isn't really relevant or useful now that the mgr is throttling the
actual pg_num adjustment based on pg_num_target, % misplaced, etc.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pool_opts_t: int -> int64_t

Encode int32_t for pre-nautilus maps.

Fix up accessors.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pool_opts: fix whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pool_opts_t: make encoding feature-dependent

No functional change (yet).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/devicehealth: pg_num_min 1 for device_health_metrics pool

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: accept optional pg_num_min to 'osd pool create'

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: apply osd_pool_default_pg_autoscale_mode to new pools

Default to 'warn' for now.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/mgr_module: some accessors

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MgrMonitor: enable progress module by default

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add pool pg_autoscale_mode, pg_num_min, target_size_{bytes,ratio} properties

These will direct PG autoscaling.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: revise get_latest_version locking

Avoid holding Objecter lock while calling back
into completion.

Signed-off-by: John Spray <john.spray@redhat.com>

os/memstore: ignore OP_COLL_SET_BITS

...rather than crashing out on it.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: generalise REQUIRE_MEMSTORE

Move it up into CephTestCase so that mgr tests can
use it too, and pick it up in vstart_runner.py so
that these tests will work neatly there.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: drop GIL in get_config

Signed-off-by: John Spray <john.spray@redhat.com>

mon: add 'size' arg to `osd pool create`

Take advantage of keyword arguments to extend
what we can do in a single command during pool creation.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: use pg_num_target for checks during creation

This way, someone creating pools can proceed
as long as they've decreased the pg_num_target
of other pools, even if the adjustment hasn't
fully completed yet.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: revise locking in getter paths

There were many places where with_* methods
were blocking on locks while holding the GIL.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #25613 into master

* refs/pull/25613/head:
qa/suites/upgrade/mimic-x: add missing .qa

Reviewed-by: João Eduardo Luís <joao@suse.de>

common/options: add `mon_target_pg_per_osd`

This is the partner to mon_max_pg_per_osd, where
this is a more conservative target for PG auto adjustment,
leaving some breathing room for situations where we
might temporarily exceed our target PG count (but not
want to exceed our maximum PG count)

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: expose OSDMap.pool_raw_used_rate

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #25190 into master

* refs/pull/25190/head:
mgr/prometheus: adjust to new 'df' fields
mon/Monitor: fix newline between df section
doc: update docs for new ceph df output
mon/PGMap: break down RAW usage by device class
mon/PGMap: tweak df headers
mon/PGMap: GLOBAL -> RAW STORAGE in 'df' output
mon/PGMap: dump_fs_stats -> dump_cluster_stats

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25615 into master

* refs/pull/25615/head:
common/options: fix bad default

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/orchestrator: Add JSON output to CLI commands

`device ls`, `service ls` and `service status` now support `--format json`

Plus some minor refactoring:

* `s/node/host/`
* Merged `_list_services` and `_service_status`.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

common/options: fix bad default

Got the fields wrong.

Fixes ff75a0b4f21eb14b11702ab2df82ed594f650fcd

Signed-off-by: Sage Weil <sage@redhat.com>

cls/rgw: update rgw_cls_usage_log_trim_op encode version

commit 7b17da691f3bee8118ac69a7419519017030a4c9 added a 'bucket' field
to this op without bumping the encode version, and is causing failures
on upgrade

Fixes: http://tracker.ceph.com/issues/37703
Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/suites/upgrade/mimic-x: add missing .qa

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25596 into master

* refs/pull/25596/head:
qa/suites/upgrade: fix wrt librados3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25486 into master

* refs/pull/25486/head:
common: enhance debug macro

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #25456 into master

* refs/pull/25456/head:
common/options: set mon or mgr service on these options
common/options: more mon options
common/options: kill old mon_max_pgmap_epochs option
common/options: mon options
mon: remove dead option mon_pg_min_inactive
mon: mon_osd_pool_ec_fast_read -> osd_pool_default_ec_fast_read
common/options: document mon options
common/options: messenger options
common/options: document misc
common/options: compression options
common/options: document monitor log options

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #25563 from rhcs-dashboard/bugfix-read-write-chart

mgr/dashboard: avoid blank content in Read/Write Card

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #25110 into master

* refs/pull/25110/head:
client: fix failure in quota size limitation when using samba

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25174 into master

* refs/pull/25174/head:
mds: fix bug filelock stuck at LOCK_XSYN leading client can't read data

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25481 into master

* refs/pull/25481/head:
mon: mark REMOVE_SNAPS messages as no_reply

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>