mon/OSDMonitor: add 'osd numa-status' command

Signed-off-by: Sage Weil <sage@redhat.com>

osd: report numa node for network interface(s)

Signed-off-by: Sage Weil <sage@redhat.com>

common/pick_address: get numa node info for a interface

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: report numa node info for storage

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: report numa info for each device

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: allow get_devices() to run without bluestore started

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: query numa info

Signed-off-by: Sage Weil <sage@redhat.com>

common/numa: cpu_set helpers and numa helpers

Note: the mask variants assume num_cpus is a factor of 4.  Not sure
what local_cpus looks like when that is not the case... we'll use the
local_cpulist style instead.

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: drop debug statements

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25455 into master

* refs/pull/25455/head:
mgr/diskprediction Document guide add predictor accuracy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25210 from dillaman/wip-dashboard-rbd-mirroring

dashboard: support configuring block mirroring pools and peers

Merge pull request #25770 from votdev/fix_telemetry_issues

mgr/telemetry: Fix various issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25783 from neha-ojha/wip-25672

common/blkdev.h: use std::string

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Rick Chen <rick.chen@prophetstor.com>

Merge pull request #25758 from xiexingguo/wip-async-prefetch

msg/async: improve read-prefetch logic

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25760 from cxytt/bug-fix

mon: shutdown messenger early to avoid accessing deleted logger

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common/blkdev.h: use std::string

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #24159 into master

* refs/pull/24159/head:
mon/PGMap: show pg state age instead of stamp

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #25623 into master

* refs/pull/25623/head:
common/ceph_time: 'mo' for month
common/options: use new parse_timespan
common/ceph_time: add parse_timespan
common/config_proxy: pass err_ss through on set_val
common/ceph_time: add exact_timespan_str

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25756 into master

* refs/pull/25756/head:
vstart.sh: force setting dashboard options before first mgr starts
mon/ConfigMonitor: add --force flag to 'config set'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25763 into master

* refs/pull/25763/head:
ceph_argparse: make command ceph acceph SIGINT.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25672 into master

* refs/pull/25672/head:
osd: OSD device smart data include additional nvme data
common/blkdev: add missing get_device_id impl
os/bluestore,filestore: use get_raw_devices
osd: update metadata and smart code to report get_device_id errors
mon: update metadata and smart commands to use get_raw_devices
common/blkdev: add get_raw_devices helper
common/blkdev: fix BlkDev::get_devid when we got a devname, not fd
common/blkdev: return optional error string from get_device_id
common/blkdev: refactor to add block_device_get_metrics returning json

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #25593 into master

* refs/pull/25593/head:
mds: create heartbeat grace config option

Reviewed-by: Zheng Yan <zyan@redhat.com>

osd: OSD device smart data include additional nvme data

Add nvme addition data into the deveh health data. That use nvme tool
and command syntax "nvme <vendor> smart-log-add <dev> -json". The nvme
json output append in the dev smart "nvme_smart_health_information_add_log".

- made run_smartctl static/private
- changed get_metrics to take a const string, not c str

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>
Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: add missing get_device_id impl

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore,filestore: use get_raw_devices

There is a minor change here: before the devices list would include
e.g. 'sda2' and 'sda', and now it will only include 'sda'.

Fixes: http://tracker.ceph.com/issues/37542
Signed-off-by: Sage Weil <sage@redhat.com>

osd: update metadata and smart code to report get_device_id errors

Signed-off-by: Sage Weil <sage@redhat.com>

mon: update metadata and smart commands to use get_raw_devices

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: add get_raw_devices helper

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25674 from alfredodeza/wip-rm37442

ceph-volume normalize comma to dot for string to int conversions

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24635 into master

* refs/pull/24635/head:
test: add scrub control command tests
mds: scrub abort/pause/resume/status control commands
test: switch using "scrub start" tell interface to initiate scrub
mds: introduce "scrub start" tell interface to initiate scrub
mds: dump scrub formatted output when context completion
mds: generate random scrub tag when empty
mds: introduce C_ExecAndReply context completion class
mds: use CInode::scrub_is_in_progress() wherever required

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #25473 from theanalyst/rgw/version-copy-fix

rgw: fix versioned obj copy generating tags

mon: shutdown messenger early to avoid accessing deleted logger

In the monitor shutdown process, the MSG thread exits after the logger is released,
causing the null pointer to be accessed. So move the logger release to the MSG thread after it exits

Fixes: http://tracker.ceph.com/issues/37780
Signed-off-by: ningtao <ningtao@sangfor.com.cn>

common/ceph_time: 'mo' for month

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25761 from tchaikov/wip-qa-health-schema

mgr/dashboard: updated API test to reflect changes in ModuleInfo

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25736 into master

* refs/pull/25736/head:
common/options: document some osd/rados options
unittest_osdmap: feed options as defaults
mon/OSDMonitor: allow osd_pool_default_pgp_num to be 0

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25766 from tchaikov/wip-rgw-readdir-flags

pybind/rgw: pass the flags to callback function

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #25635 from rhcs-dashboard/36740-pool-stats-disabled-by-default

mgr/dashboard: pool stats not returned by default

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/telemetry: Fix various issues

- Set module config type to ``str``, otherwise default values are ``'None'`` (str) instead of ``None`` which will cause ``init_module_config`` to fail.
- RuntimeError does not format strings.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #25528 into master

* refs/pull/25528/head:
mds: log evicted clients to clog/dbg

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #24845 from pritha-srivastava/wip-rgw-assume-role-fixes

rgw: Minor fixes to AssumeRole for boto compliance.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #25180 from pritha-srivastava/wip-rgw-user-policy-fixes

rgw: Fixes to permission evaluation related to user policies.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25659 from Linbing/wip-fixup-read-deadlock

os/bluestore: fixup access a destroy cond cause deadlock or undefine behavior

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25673 into master

* refs/pull/25673/head:
mon/Monitor: quorum_age in JSON output as int, not string

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/options: use new parse_timespan

It's more complete and robust, and lives in ceph_time.h where it probably
belongs.

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_time: add parse_timespan

Make it understand units like 'hours', 'minute', 'm', 'w', etc.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: force setting dashboard options before first mgr starts

These options are unknown before the mgr first starts and registers them.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: add --force flag to 'config set'

Add a force flag to override option and/or value checks.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/mgr: add "module_options" to module_info_schema

it was introduced in 280c23ba. so update the schema test accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rgw: pass the flags to callback function

before this change, the `flags` parameter passed to `LibRGWFS.readdir()`
will be dropped on the floor and ignored.
after this change, it will be passed to the specified callback function.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25580 from cyx1231st/wip-crimson-msgr-logging

crimson/net: fix addresses during banner exchange

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: fix address learning during banner exchange

* Don't store my_addr in `Connection`, because my_addr can be learned
  and thus changed.
* Support nonce in SocketMessenger.
* Always set nonce when set_myaddr().
* Add learned_addr() for SocketMessenger.
* Add side_t and socket_port to show the real connecting
  ports of the SocketConnection.
* Fix bannder exchange logic for addresses, including nonce, type, ip,
  port, socket_port for my_addr and peer_addr.
* Add more detailed logging prefixes for SocketConnection.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: add logic_name to indicate the msgr role for debugging

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: add basic loggings for SocketConnection

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

ceph_argparse: make command ceph acceph SIGINT.

If no --connect-timeout point and there is no ceph.conf, the command
'ceph -s' can't stop by ctrl+c.
This introduced by commit 4d8fc26c8
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #25757 from gregsfortytwo/wip-doc-put-warning

doc: warn about how 'rados put' works in the manpage

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25706 from joscollin/wip-build-doc-librados

librados: fix admin/build-doc warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25734 from tchaikov/wip-mgr-default-options

mgr: define option defaults for MgrStandbyModule as well

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25735 from wjwithagen/wjw-fix-WRITE_LIFE

os/bluestore: Only use WRITE_LIFE_ when available

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25746 from tchaikov/wip-pg-autoscale-mode

osd/OSDMap: set pg_autoscale_mode with setting from conf

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25588 from tchaikov/wip-auth-silence-gcc-warning

cls/rbd: init local var with known value

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

qa/mgr: extract module_info_schema and reused it

both standby mgr and active mgr have this sub-schema. so extract it out
and reuse it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/rbd: init local var with known value

DirectoryState does not have an "invalid" enum so far, since it's
defined using `enum class`, init a value of this type with a known value
could be a better choice even it is always initialized before being read.

this silences the GCC warning of:

src/cls/rbd/cls_rbd.cc:3147:3: warning: ‘on_disk_directory_state’ may be
used uninitialized in this function [-Wmaybe-uninitialized]
   if (directory_state != on_disk_directory_state) {
   ^~

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/PGMap: show pg state age instead of stamp

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25687 from tchaikov/wip-pg-stats

osd/PrimaryLogPG: do not count failed read in delta_stats

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: warn about how 'rados put' works in the manpage

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #25705 from joscollin/wip-doc-common

doc: Fix Create a Cluster url in Running Multiple Clusters

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #25450 from ivancich/wip-rgw-category-cleanup

rgw: clean-up -- use enum class for stats category

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

rgw: clean-up -- remove unnessary "struct"s

In C++ "struct" is unnecessary when referencing a defined
struct. Clean up all uses in src/cls/rgw source files.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: clean-up use enum class for stats category

The stats entries for rgw buckets has a category, which used a
combination of uint8_t and enum RGWObjClass. Clean this up by
converting RGWObjClass to an enum class and using that
throughout. This provides type safety and better code clarity. Also,
add some source code documentation.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

ceph-volume tests.util ensure ints and strings with commas can be converted to ints

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume test verify strings with a comma can be converted to ints

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util normalize comma with dot for str-to-int conversion

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24676 from TheJJ/cryptsetup-allow-discards

ceph-volume: enable device discards

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

common/options: document some osd/rados options

Many of these are actually on the mon.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_osdmap: feed options as defaults

We can't change some of these at runtime due to FLAG_CREATE.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: set pg_autoscale_mode with setting from conf

* update build_simple_optioned() to set pg_autoscale_mode with the setting
  read from conf, otherwise it will be a random value in heap.
* update cli test accordingly, otherwise we will have

/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/osdmaptool/create-racks.t:
failed
---
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/osdmaptool/create-racks.t
+++
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/osdmaptool/create-racks.t.err
@@ -796,7 +796,7 @@
   nearfull_ratio 0
   min_compat_client jewel

-  pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 15296 pgp_num 15296 last_change 0 flags hashpspool
stripe_width 0 application rbd
+  pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 15296 pgp_num 15296 autoscale_mode ??? last_change 0
flags hashpspool stripe_width 0 application rbd

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25441 from theanalyst/decode-refcount-types

cls: refcount: add obj_refcount to ceph-dencoder

Reviewed-By: Casey Bodley <cbodley@redhat.com>

ceph-volume: enable device discards

When using SSDs as encrypted OSD device, discards do not pass the
encryption layer. This option activates discard requests.

Signed-off-by: Jonas Jelten <jj@stusta.net>

mgr: define option defaults for MgrStandbyModule as well

in 0f814f38, we are using self.MODULE_OPTION_DEFAULTS as a fallback, but
the MgrStandbyModule does not have it defined. in this change,

* apply 0f814f38 to MgrStandbyModule
* apply 0228bd79 to MgrStandbyModule

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: add scrub control command tests

Fixes: http://tracker.ceph.com/issues/12282
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: scrub abort/pause/resume/status control commands

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: switch using "scrub start" tell interface to initiate scrub

... and fixup doc too.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: introduce "scrub start" tell interface to initiate scrub

This should be preferred over the traditional asok interface
which would probably get deprecated (soon).

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: dump scrub formatted output when context completion

Include scrub tag as part of the output and move the
formatting in context completion to support scrub opeation
triggered via tell interface (introduced later).

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: generate random scrub tag when empty

With this, scrub operations are tagged with a random
uuid if tag is unspecified. This also helps to show
in-progress scrub operations via "scrub status" command
(introduced in later commits).

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: introduce C_ExecAndReply context completion class

Tell commands that need to asynchronous reply back can
subclass C_ExecAndReply() and implement exec() virtual
function to support asynchronous execution of the command
(via finisher thread).

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: use CInode::scrub_is_in_progress() wherever required

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #25541 from theanalyst/wip-es-log-fixes

rgw: sync module: avoid printing attrs of objects in log

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #25731 from liewegas/wip-37775

osd: reliably send pg_created messages to the mon

Reviewed-by: Kefu Chai <kchai@redhat.com>

librados: fix admin/build-doc warning

Fixed warning: librados.h:321: warning: Member compressed_bytes_alloc
(variable) of class rados_pool_stat_t is not documented.

Signed-off-by: Jos Collin <jcollin@redhat.com>

mon/OSDMonitor: allow osd_pool_default_pgp_num to be 0

If it's 0, use the pg_num value.  This lets you adjust the
osd_pool_default_pg_num option without also adjusting the pgp_num one.

Signed-off-by: Sage Weil <sage@redhat.com>

bluestore: Only use WRITE_LIFE_ when available

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

common/blkdev: fix BlkDev::get_devid when we got a devname, not fd

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: return optional error string from get_device_id

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: refactor to add block_device_get_metrics returning json

Refactor block_device_run_smartctl into a helper that returns JSON.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/Monitor: quorum_age in JSON output as int, not string

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25732 from liewegas/wip-37776

qa/workunits/rados/test_health_warnings: prevent out osds

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25726 from tchaikov/wip-37751

osd: unlock osd_lock when tweaking osd settings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25733 from tchaikov/wip-37719-workaround

librbd: workaround an ICE of GCC

Reviewed-by: Sage Weil <sage@redhat.com>

crimson: workaround an ICE of GCC

this change works around the FTBFS on arm64:

/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/crimson/common/config_proxy.h:74:13:
internal compiler error: in tsubst_decomp_names, at cp/pt.c:16537
             for (auto& [obs, keys] : rev_obs) {
             ^~~
Please submit a full bug report,
with preprocessed source if appropriate.

it seems that this issue is a dup of
https://bugzilla.redhat.com/show_bug.cgi?id=1639019 .

Signed-off-by: Kefu Chai <kchai@redhat.com>