build/ops: rpm: make ceph-grafana-dashboards own its directories on SUSE

This is only required to get the spec file to build in the OpenSUSE
Build Service (OBS). Also, this change could potentially make the
package impossible to install together with grafana (if the latter
owns the same directories with different ownership/permissions).
Therefore, make the change specific to SUSE.

Fixes: http://tracker.ceph.com/issues/37485
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #25415 from dillaman/wip-rbd-cli-integration

test/cli-integration/rbd: added new parent image attributes

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25234 from jan--f/prometheus-add-iface-objstore-metadata

mgr/prometheus: add interface and objectstore to osd metadata

Merge pull request #19454 from ifed01/wip-ifed-pool-sizes

osd: per-pool osd stats collection

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25243 into master

* refs/pull/25243/head:
osdc/Objecter: update op_target_t::paused in _calc_target

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25399 into master

* refs/pull/25399/head:
osd: requeue waiting peering events from deleted slots in unprime_split_children

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25428 from trociny/wip-open-flags

librbd: corrected usage of ImageState::open flag parameter

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25430 from cbodley/wip-pr-22539

rgw: rgwgc:process coredump in some special case

rgw: rgwgc:process coredump in some special case。 Gc processes obja, objb, objc in order and pool of objb is deleted (obja and objc is in the same pool and pool exits). RGW will coredump as ctx->io_ctx_impl is an empty point during delete objc.
Fixes:http://tracker.ceph.com/issues/23199

Signed-off-by: zhaokun <develop@hikdata.com>

Merge pull request #17789 from shashalu/fix-bucket-link-dm

rgw: fix version bucket stats

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #21622 from tianshan/misc_es_fix

rgw: some fix for es sync

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #24162 from zhangsw/cleanup-rgw-acl_op

rgw: remove repetitive conditional statement in RGWHandler_REST_Obj_S3

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24683 from linuxbox2/wip-v2-url-meta

rgw: handle S3 version 2 pre-signed urls with meta-data

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25142 from ivancich/wip-rgw-reshard-cleanup

rgw: reshard clean-up and associated commits

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25406 from taoCH/wip-librgw-sync-user-regularly

rgw_file: user info never synced since librgw init

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

librbd: corrected usage of ImageState::open flag parameter

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25423 from p-na/wip-pna-fix-pool-edit

mgr/dashboard: Fix renaming of pools

Reviewed-by: Laura Paduano <lpaduano@suse.com>

mon/PGMap: Partially unify 'ceph pg stat' output with 'ceph df'

Global stats parameter (like 'available' or 'total') are dumped under the
same names now).

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

pybind/diskprediction: rename raw_bytes_used field reference.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

pybind/mgr/telegraf: rename raw_bytes_used field reference.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

qa/mgr: rename raw_bytes_used field reference.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd: dump legacy (calculated for now) params in osd_stat_t::dump.

This is to ensure backward compatibility with python modules that use
them.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

PendingReleaseNotes: update according to statistics refactoring

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr: rename ceph df stat columns for both influx and prometheus.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

qa: replace raw_bytes_used field access in QA test cases

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mon: rearrange columns in ceph df output

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/test: extend BluestoreStatFS test with pool_statfs verification.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: add fsck/repair coverage for per-pool stats

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

librados: fix a way rados approximate per-pool net amount.
In fact that's just a workaround for now as we need proper raw_used_rate
present at librados.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/mon/PGMap: fix the test case

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd: remove assert on inconsistent total/available values from osd_stat_t encode

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd,os,mon: extend 'ceph df' report to provide both USED and RAW_USED
totals where USED is space allocated for object keeping while RAW_USED
also includes space allocated/reserved for internal purposes, e.g.
DB/WAL or journal.

Signed-off-by: Igor Fedotv <ifedotov@suse.com>

os/bluestore: add debug option to disable per pool statistics collection

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test: add test coverage for rados df reports

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mon/pgmap: make pool id type consistent for statistics operations

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

rados,mon: improve ceph and rados df reports.

Fixes: https://tracker.ceph.com/issues/22159
Fixes: https://tracker.ceph.com/issues/20870
This adds  more parameters to estimate allocation granularity overhead and compression
effectiveness. RAW_USED column has been removed. USED column reflects
amount of space allocated for a pool at all replicas. Including
allocaction granularity overhead and taking ompression savings and
object content gaps into account.

Following columns were appended:
* STORED - approximation of bytes users've actually stored in a
pool, i.e netto data amount without compression, allocation and
other overheads (not applicable for 'rados df').
* USED COMPR - amount of bytes allocated to store compressed data, i.e.
compressed data plus allocation/replication overhead.
* UNDER COMPR - amount of data(icluding repicated one) passed through
compression and stored in that form.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd,mon: start using per-pool statistics reported from OSD.

Notable changes:
* pool statistics collection has been changed to track actual allocated
  store space instead of previous object logical sizes summation.
* 'ceph/rados df' report total space(over all replicas) allocated for a pool under "USED" column.
* "RAW USED" column removed from 'ceph df detail' report

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os: extend ObjectStore interface with per-pool statistics access method

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: add per-pool statistics collection

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #25208 from cbodley/wip-rgw-getobj-throttle

rgw: adapt AioThrottle for RGWGetObj
yay!

Merge PR #25345 into master

* refs/pull/25345/head:
qa/suites: fix bluestore links
qa/objectstore: bluestore -> bluestore-{bitmap,stupid}

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25378 into master

* refs/pull/25378/head:
mgr/hello: add serve() method

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #25382 into master

* refs/pull/25382/head:
mon/OSDMonitor: include host, container info for 'osd find' command

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #25301 from wjwithagen/wjw-fix-dtor-pair

rbd: workaround for llvm linker problem, avoid std:pair dtor

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25265 from trociny/wip-rbd-fuse-namespace

rbd-fuse: namespace support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24968 from s0nea/wip-dashboard-osd-recv-speed-profiles

mgr/dashboard: add profiles to set cluster's rebuild performance

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Fix renaming of pools

The validation fails to allow dots in pool names and as such, it wasn't
allowed to create or rename pools with dots in their names.

Fixes: https://tracker.ceph.com/issues/37534
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #25409 from tchaikov/wip-public-iterator-traits

buffer: mark the iterator traits "public"

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #25266 from trociny/wip-24608

rbd-ggate: support namespaces

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

buffer: mark the iterator traits "public"

the iterator traits should be accessible from non-dervative classes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25230 from runsisi/wip-cleanup-objecter

osdc/Objecter: no need null pointer check for op->session anymore

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #23550 into master

* refs/pull/23550/head:
auth: Kerberos authentication

test/cli-integration/rbd: added new parent image attributes

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25166 from kungf/journal_remove_full

journal: allow remove set when jounal pool is full

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25175 from trociny/wip-36710

librbd: restart io if migration parent gone

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25260 from trociny/wip-24609

rbd-nbd: support namespaces

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25371 from trociny/wip-osd-query-subkey-types

 osd: support more dynamic perf query subkey types

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25151 from trociny/wip-osd-perf-query-limit

mgr: add limit param to osd perf query

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25379 from tchaikov/wip-37508

librbd: reset snaps in rbd_snap_list()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites: fix bluestore links

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24882 from rzarzynski/wip-bl-hypercombined

common: hypercombined bufferlist

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

rgw: store ref to librados::IoCtx in rgw::AioResult

the IoCtx has to outlive its aio completions

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add const overload of RGWSI_RADOS::Obj::get_ref()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rgw_rados_ref contains rgw_raw_obj

rgw_rados_ref duplicated each of the fields from rgw_raw_obj

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use AioThrottle for Object::Read::iterate() ops

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: clean up iterate_obj() callbacks

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add append_atomic_test() overload for RGWObjState

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add id to distinguish between Aio results

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Aio reads return bufferlist in result

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move aio throttles to namespace rgw

they're useful outside of putobj

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25365 from xiexingguo/wip-37493

osd/OSDMap: fix upmap mis-killing for erasure-coded PGs

Merge pull request #25262 from aclamk/wip-kvstore-tool-dump

ceph-kvstore-tool: dump fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25383 from dmick/wip-container-name

common/util.cc: add CONTAINER_NAME processing for metadata

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25400 from dillaman/wip-ceph-cli

ceph: do not raise StopIteration within generator

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: workaround for llvm linker proplem, avoid std:pair dtor

When linking with llvm linker on FreeBSD:
```
/usr/bin/ld.lld: error: undefined symbol: std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool>::~pair()
>>> referenced by Image.cc:349 (/home/jenkins/workspace/ceph-master/src/librbd/api/Image.cc:349)
>>>               Image.cc.o:(librbd::api::Image<librbd::ImageCtx>::list_children(librbd::ImageCtx*, cls::rbd::ParentImageSpec const&, std::__1::vector<librbd::linked_image_spec_t, std::__1::allocator<librbd::linked_image_spec_t> >*)) in archive ../../../lib/librbd_internal.a
```

Which is probably because the compiler calls a dtor on the just created pair.
But the FreeBSD linker does not find a dtor in the any of the libraries.
This rewrite build and links without the linker error.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/dashboard: show config option description via cd-helper

Fixes: https://tracker.ceph.com/issues/37406
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: add OSD recovery speed form

Add a form to set the OSD recovery speed. There are three predefined
priorities available: 'Low', 'Default' and 'High'.
By setting the priority the config options 'osd_max_backfills',
'osd_recovery_max_active', 'osd_recovery_max_single_start',
'osd_recovery_sleep' will be updated with predefined values.
It's also possible to set custom values.

Fixes: https://tracker.ceph.com/issues/35540
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25392 from majianpeng/async-rdma

msg/async/rdma: Delete duplicate header file

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file: user info never synced since librgw init

Fixed: https://tracker.ceph.com/issues/37527

Signed-off-by: Tao Chen <chentao@umcloud.com>

qa/objectstore: bluestore -> bluestore-{bitmap,stupid}

The new default is bitmap, so we were testing bitmap twice.  Instead,
explicitly call out stupid and bitmap cases so a future default change
won't break coverage.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph: do not raise StopIteration within generator

This is implicit when a generator exits and under Python 3
it results in a "StopIteration" exception being raised when
"ceph --help" is run.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #24129 into master

* refs/pull/24129/head:
common: drop ::length() from other implementations of the OpQueue.
common, osd: kill OpQueue::length() to minimize inter-core traffic.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge PR #25388 into master

* refs/pull/25388/head:
mon/OSDMonitor: do not no_reply on ready_to_merge message

Reviewed-by: João Eduardo Luís <joao@suse.de>

osd: requeue waiting peering events from deleted slots in unprime_split_children

These child slots we are deleting may have queued events that should be
requeued and reexamined.  For example, a pg query may be waiting for the
split and then discarded, instead of being reexamined and processed via
handle_pg_query_nopg().

Fixes: http://tracker.ceph.com/issues/37525
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/osd_perf_query: example how limit can be used

Signed-off-by: Mykola Golub <mgolub@suse.com>

osd: make use of osd perf query's limit param

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr: add limit param to osd perf query

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25192 from ProgVal/doc-rbd-features

rbd: Document new RBD feature flags and version support

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: add config option bulk_set

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25359 from zmc/wip-dashboard-fix-teuth-runner

mgr/dashboard: Fix PYTHONPATH for test runner

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25231 from rhcs-dashboard/37371-logs-links-permissions

mgr/dashboard: 'Logs' links permission in Landing Page

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #25322 from Devp00l/wip-issue-37468

mgr/dashboard: Document custom RESTController endpoints

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

msg/async/rdma: Delete duplicate header file

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rbd: bench snapshot if it was specified

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/osd_perf_query: add query to test all supported subkeys

Signed-off-by: Mykola Golub <mgolub@suse.com>

osd: support more dynamic perf query subkey types

Signed-off-by: Mykola Golub <mgolub@suse.com>

common: drop the doubly-linked-list compatibility of bl::buffers_t.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: switch bl::buffers_t to a singly-linked list. Drop bi::list.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

bluestore/tools: Add option 'dump' to ceph-kvstore-tool to print both key and values.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #25335 from ifed01/wip-ifed-fix-enospc

os/bluestore: fix and unify log output on allocation failure

Reviewed-by: Kefu Chai <kchai@redhat.com>