mgr/prometheus: provide RBD stats via osd dynamic perf counters

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25427 from tspmelo/wip-silent-xi18n

mgr/dashboard: Disable showing xi18n's progress

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25434 from majianpeng/osd-getpg-reserve

osd/OSD: preallocate for _get_pgs/_get_pgids to avoid reallocate.

Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25420 from majianpeng/buffer-repeated-condtion

common/buffer: remove repeated condtion-check.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25424 from rzarzynski/wip-bl-c_str_tracker

common: hint bufferlist's buffer_track_c_str accordingly.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25439 from p-na/pna-wip-cleanup

mgr/dashboard: Remove unused code

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #25425 into master

* refs/pull/25425/head:
test/common/blkdev-udevadm-info-samples: add a few test cases
ceph-volume: Micron SSDs don't include vendor name in ID_SERIAL
common/blkdev: micron SSDs don't include vendor name in ID_SERIAL
ceph-volume: update get_device_id to match in-tree implementation
common/blkdev: improve device_id generation

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #25312 from falcon78921/wip-doc-37451

doc: broken link on troubleshooting-mon page

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: broken link on troubleshooting-mon page
Fixes: https://tracker.ceph.com/issues/37451
This fix involves adding :ref: labels to the add-or-rm-mons.rst
page. I also added :ref: labels for other headings within
add-or-rm-mons.rst (for future reference).

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #25410 from dragonylffly/rb

tools/rados: fix memory leak in error path

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25393 from Aran85/fix-agent-state

osd: avoid using null agent_state

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25370 from majianpeng/sharedlru-addfunc

common/shared_cache: bumps it to the front of the LRU if key existed.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25275 into master

* refs/pull/25275/head:
osdc/Journaler: make try_read_entry store data in contiguous memory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25270 into master

* refs/pull/25270/head:
mds: use uniform write handler for PQ

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24991 into master

* refs/pull/24991/head:
mds: disallow dumping huge caches to formatter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24940 into master

* refs/pull/24940/head:
qa: add test for getfattr ceph.dir.pin
client: support getfattr ceph.dir.pin extended attribute

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24748 into master

* refs/pull/24748/head:
qa: use 6h timeout for pjd test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25347 from smithfarm/wip-37485

build/ops: rpm: make ceph-grafana-dashboards own its directories

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #25414 from ivancich/wip-rgw-user-stats-mismatch

rgw: fix bad user stats on versioned bucket after reshard

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

test/common/blkdev-udevadm-info-samples: add a few test cases

The erwan1 one unfortunately has no model information :(.  The other two
work with the current implementation, although erwan.v1.sdb has
"Shared_PERC8" for the model, and I suspect the Shared_ prefix is not
part of the real model?

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: Micron SSDs don't include vendor name in ID_SERIAL

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: micron SSDs don't include vendor name in ID_SERIAL

According to Ryan Meredith <rmeredith@micron.com>, the NVMes begin with
MTFD, but their other devices use the standard Micron_$model_$serial.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: update get_device_id to match in-tree implementation

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: improve device_id generation

A few interesting cases:

1. autriche (my desktop)

$ sudo udevadm info /dev/nvme0n1 | grep ID_
E: ID_PART_TABLE_TYPE=gpt
E: ID_PART_TABLE_UUID=c83d5616-676b-4667-bcf3-c82fd4fc7e64
E: ID_SERIAL=Samsung SSD 960 EVO 250GB_S3ESNX0J958081E
E: ID_SERIAL_SHORT=S3ESNX0J958081E

- no ID_MODEL or ID_VENDOR
-> use ID_SERIAL

2. my dev box

gnit:~ (master) 09:09 AM $ udevadm info /dev/nvme0n1 | grep ID_
E: ID_FS_TYPE=xfs
E: ID_FS_USAGE=filesystem
E: ID_FS_UUID=860d4503-9c9d-4c24-af09-4266b7717a5c
E: ID_FS_UUID_ENC=860d4503-9c9d-4c24-af09-4266b7717a5c
E: ID_MODEL=INTEL SSDPEDMD400G4
E: ID_PATH=pci-0000:82:00.0-nvme-1
E: ID_PATH_TAG=pci-0000_82_00_0-nvme-1
E: ID_SERIAL=INTEL SSDPEDMD400G4_CVFT520200G7400BGN
E: ID_SERIAL_SHORT=CVFT520200G7400BGN
E: ID_WWN=nvme.8086-43564654353230323030473734303042474e-494e54454c205353445045444d443430304734-00000001

- no ID_VENDOR
-> ID_MODEL + ID_SERIAL_SHORT

gnit:~ (master) 09:12 AM $ udevadm info /dev/sda | grep ID_
E: ID_ATA=1
E: ID_ATA_DOWNLOAD_MICROCODE=1
E: ID_ATA_FEATURE_SET_HPA=1
E: ID_ATA_FEATURE_SET_HPA_ENABLED=1
E: ID_ATA_FEATURE_SET_PM=1
E: ID_ATA_FEATURE_SET_PM_ENABLED=1
E: ID_ATA_FEATURE_SET_SECURITY=1
E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=2
E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=2
E: ID_ATA_FEATURE_SET_SECURITY_FROZEN=1
E: ID_ATA_FEATURE_SET_SMART=1
E: ID_ATA_FEATURE_SET_SMART_ENABLED=1
E: ID_ATA_ROTATION_RATE_RPM=0
E: ID_ATA_SATA=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN1=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN2=1
E: ID_ATA_WRITE_CACHE=1
E: ID_ATA_WRITE_CACHE_ENABLED=1
E: ID_BUS=ata
E: ID_MODEL=INTEL_SSDSC2BB240G4
E: ID_MODEL_ENC=INTEL\x20SSDSC2BB240G4\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_PART_TABLE_TYPE=dos
E: ID_PART_TABLE_UUID=bb35118c
E: ID_PATH=pci-0000:00:1f.2-ata-1
E: ID_PATH_TAG=pci-0000_00_1f_2-ata-1
E: ID_REVISION=D2010355
E: ID_SERIAL=INTEL_SSDSC2BB240G4_BTWL3414034J240NGN
E: ID_SERIAL_SHORT=BTWL3414034J240NGN
E: ID_TYPE=disk
E: ID_WWN=0x55cd2e404b4e47d8
E: ID_WWN_WITH_EXTENSION=0x55cd2e404b4e47d8

- no ID_VENDOR
-> ID_MODEL + ID_SERIAL_SHORT

3. mira lab machine (old areca controller in JBOD mode, I think)

root@mira055:~# udevadm info /dev/sdb | grep ID_
E: ID_BUS=scsi
E: ID_MODEL=HUS724040ALA640
E: ID_MODEL_ENC=HUS724040ALA640\x20
E: ID_PART_TABLE_TYPE=gpt
E: ID_PART_TABLE_UUID=957b2db6-de5c-46cb-a672-243fa12d55b2
E: ID_PATH=pci-0000:01:00.0-scsi-0:0:0:1
E: ID_PATH_TAG=pci-0000_01_00_0-scsi-0_0_0_1
E: ID_REVISION=R001
E: ID_SCSI=1
E: ID_SCSI_SERIAL=PN1334PBH5JMJS
E: ID_SERIAL=2001b4d2058da3a00
E: ID_SERIAL_SHORT=001b4d2058da3a00
E: ID_TYPE=disk
E: ID_VENDOR=HGST
E: ID_VENDOR_ENC=HGST\x20\x20\x20\x20

- ID_VENDOR and ID_MODEL
- ID_MODEL doesn't include vendor name + _
- ID_SERIAL and _SHORT are junk
- ID_SCSI_SERIAL has the serial number!  wth
-> ID_VENDOR + ID_MODEL + ID_SCSI_SHORT

Added a bunch of udevadm output samples.

So,

- if ID_VENDOR + ID_MODEL + ID_SCSI_SERIAL are present, use them.
- if ID_MODEL + ID_SERIAL_SHORT are present, use them
- if ID_SERIAL is present, use it.
- fail

Signed-off-by: Sage Weil <sage@redhat.com>

build/ops: rpm: make ceph-grafana-dashboards own its directories on SUSE

This is only required to get the spec file to build in the OpenSUSE
Build Service (OBS). Also, this change could potentially make the
package impossible to install together with grafana (if the latter
owns the same directories with different ownership/permissions).
Therefore, make the change specific to SUSE.

Fixes: http://tracker.ceph.com/issues/37485
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Tim Serong <tserong@suse.com>

mgr/dashboard: Remove unused code

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #25415 from dillaman/wip-rbd-cli-integration

test/cli-integration/rbd: added new parent image attributes

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25234 from jan--f/prometheus-add-iface-objstore-metadata

mgr/prometheus: add interface and objectstore to osd metadata

osd/OSD: preallocate for _get_pgs/_get_pgids to avoid reallocate.

num_pgs is the most accurate value fore reserve when call _get_pgs &
_get_pgids.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #19454 from ifed01/wip-ifed-pool-sizes

osd: per-pool osd stats collection

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25243 into master

* refs/pull/25243/head:
osdc/Objecter: update op_target_t::paused in _calc_target

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25399 into master

* refs/pull/25399/head:
osd: requeue waiting peering events from deleted slots in unprime_split_children

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25428 from trociny/wip-open-flags

librbd: corrected usage of ImageState::open flag parameter

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25430 from cbodley/wip-pr-22539

rgw: rgwgc:process coredump in some special case

rgw: rgwgc:process coredump in some special case。 Gc processes obja, objb, objc in order and pool of objb is deleted (obja and objc is in the same pool and pool exits). RGW will coredump as ctx->io_ctx_impl is an empty point during delete objc.
Fixes:http://tracker.ceph.com/issues/23199

Signed-off-by: zhaokun <develop@hikdata.com>

common: eradicate buffer_track_c_str in bufferlist.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #17789 from shashalu/fix-bucket-link-dm

rgw: fix version bucket stats

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #21622 from tianshan/misc_es_fix

rgw: some fix for es sync

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #24162 from zhangsw/cleanup-rgw-acl_op

rgw: remove repetitive conditional statement in RGWHandler_REST_Obj_S3

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24683 from linuxbox2/wip-v2-url-meta

rgw: handle S3 version 2 pre-signed urls with meta-data

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25142 from ivancich/wip-rgw-reshard-cleanup

rgw: reshard clean-up and associated commits

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25406 from taoCH/wip-librgw-sync-user-regularly

rgw_file: user info never synced since librgw init

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

librbd: corrected usage of ImageState::open flag parameter

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Disable showing xi18n's progress

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25423 from p-na/wip-pna-fix-pool-edit

mgr/dashboard: Fix renaming of pools

Reviewed-by: Laura Paduano <lpaduano@suse.com>

mon/PGMap: Partially unify 'ceph pg stat' output with 'ceph df'

Global stats parameter (like 'available' or 'total') are dumped under the
same names now).

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

pybind/diskprediction: rename raw_bytes_used field reference.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

pybind/mgr/telegraf: rename raw_bytes_used field reference.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

qa/mgr: rename raw_bytes_used field reference.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd: dump legacy (calculated for now) params in osd_stat_t::dump.

This is to ensure backward compatibility with python modules that use
them.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

PendingReleaseNotes: update according to statistics refactoring

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr: rename ceph df stat columns for both influx and prometheus.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

qa: replace raw_bytes_used field access in QA test cases

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mon: rearrange columns in ceph df output

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/test: extend BluestoreStatFS test with pool_statfs verification.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: add fsck/repair coverage for per-pool stats

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

librados: fix a way rados approximate per-pool net amount.
In fact that's just a workaround for now as we need proper raw_used_rate
present at librados.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/mon/PGMap: fix the test case

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd: remove assert on inconsistent total/available values from osd_stat_t encode

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd,os,mon: extend 'ceph df' report to provide both USED and RAW_USED
totals where USED is space allocated for object keeping while RAW_USED
also includes space allocated/reserved for internal purposes, e.g.
DB/WAL or journal.

Signed-off-by: Igor Fedotv <ifedotov@suse.com>

os/bluestore: add debug option to disable per pool statistics collection

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test: add test coverage for rados df reports

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mon/pgmap: make pool id type consistent for statistics operations

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

rados,mon: improve ceph and rados df reports.

Fixes: https://tracker.ceph.com/issues/22159
Fixes: https://tracker.ceph.com/issues/20870
This adds  more parameters to estimate allocation granularity overhead and compression
effectiveness. RAW_USED column has been removed. USED column reflects
amount of space allocated for a pool at all replicas. Including
allocaction granularity overhead and taking ompression savings and
object content gaps into account.

Following columns were appended:
* STORED - approximation of bytes users've actually stored in a
pool, i.e netto data amount without compression, allocation and
other overheads (not applicable for 'rados df').
* USED COMPR - amount of bytes allocated to store compressed data, i.e.
compressed data plus allocation/replication overhead.
* UNDER COMPR - amount of data(icluding repicated one) passed through
compression and stored in that form.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd,mon: start using per-pool statistics reported from OSD.

Notable changes:
* pool statistics collection has been changed to track actual allocated
  store space instead of previous object logical sizes summation.
* 'ceph/rados df' report total space(over all replicas) allocated for a pool under "USED" column.
* "RAW USED" column removed from 'ceph df detail' report

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os: extend ObjectStore interface with per-pool statistics access method

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: add per-pool statistics collection

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #25208 from cbodley/wip-rgw-getobj-throttle

rgw: adapt AioThrottle for RGWGetObj
yay!

Merge PR #25345 into master

* refs/pull/25345/head:
qa/suites: fix bluestore links
qa/objectstore: bluestore -> bluestore-{bitmap,stupid}

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25378 into master

* refs/pull/25378/head:
mgr/hello: add serve() method

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #25382 into master

* refs/pull/25382/head:
mon/OSDMonitor: include host, container info for 'osd find' command

Reviewed-by: Dan Mick <dmick@redhat.com>

common: hint bufferlist's buffer_track_c_str accordingly.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #25301 from wjwithagen/wjw-fix-dtor-pair

rbd: workaround for llvm linker problem, avoid std:pair dtor

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25265 from trociny/wip-rbd-fuse-namespace

rbd-fuse: namespace support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24968 from s0nea/wip-dashboard-osd-recv-speed-profiles

mgr/dashboard: add profiles to set cluster's rebuild performance

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Fix renaming of pools

The validation fails to allow dots in pool names and as such, it wasn't
allowed to create or rename pools with dots in their names.

Fixes: https://tracker.ceph.com/issues/37534
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

tools/rados: rename misleading variables

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

tools/rados: remove misleading comments

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

tools/rados: fix memory leak in error path

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

Merge pull request #25409 from tchaikov/wip-public-iterator-traits

buffer: mark the iterator traits "public"

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

common/buffer: remove repeated condtion-check.

In func push_back(), it already check buffe::ptr::length().

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #25266 from trociny/wip-24608

rbd-ggate: support namespaces

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

buffer: mark the iterator traits "public"

the iterator traits should be accessible from non-dervative classes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25230 from runsisi/wip-cleanup-objecter

osdc/Objecter: no need null pointer check for op->session anymore

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #23550 into master

* refs/pull/23550/head:
auth: Kerberos authentication

test/cli-integration/rbd: added new parent image attributes

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix bad user stats on versioned bucket after reshard

User stats should only count bucket index entries that refer to actual
data. So only count entries with categories MAIN or MULTIMETA.

Note: the full solution is provided by the combination of the fixes in
PR 25333 (https://github.com/ceph/ceph/pull/25333) and these changes.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #25166 from kungf/journal_remove_full

journal: allow remove set when jounal pool is full

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25175 from trociny/wip-36710

librbd: restart io if migration parent gone

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25260 from trociny/wip-24609

rbd-nbd: support namespaces

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25371 from trociny/wip-osd-query-subkey-types

 osd: support more dynamic perf query subkey types

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25151 from trociny/wip-osd-perf-query-limit

mgr: add limit param to osd perf query

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25379 from tchaikov/wip-37508

librbd: reset snaps in rbd_snap_list()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites: fix bluestore links

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24882 from rzarzynski/wip-bl-hypercombined

common: hypercombined bufferlist

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

rgw: store ref to librados::IoCtx in rgw::AioResult

the IoCtx has to outlive its aio completions

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add const overload of RGWSI_RADOS::Obj::get_ref()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rgw_rados_ref contains rgw_raw_obj

rgw_rados_ref duplicated each of the fields from rgw_raw_obj

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use AioThrottle for Object::Read::iterate() ops

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: clean up iterate_obj() callbacks

Signed-off-by: Casey Bodley <cbodley@redhat.com>