mgr/prometheus: remove cmake tests

Temporary removal of the cmake test integration

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/prometheus: update promtool testcase location

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

monitoring/prometheus: Add cmake integration

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/prometheus: add test cases and validation using tox

Focus all tests inside a tests directory, and use pytest/tox to
perform validation of the overall content. tox tests also use
promtool if available to provide rule checks and unittest runs.

In addition to these checks a validate_rules script provides the
format, and content checks against all rules - which is also
called via tox (but can be run independently too)

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/prometheus: track individual healthchecks as metrics

This patch creates a health history object maintained in
the modules kvstore.  The history and current health
checks are used to create a metric per healthcheck whilst
also providing a history feature. Two new commands are added:
ceph healthcheck history ls
ceph healthcheck history clear

In addition to the new commands, the additional metrics
have been used to update the prometheus alerts

Fixes: https://tracker.ceph.com/issues/52638
Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #43146 from jianwei1216/fix-partial-recovery-become-whole-object-recovery-for-master

osd: fix partial recovery become whole object recovery after restart osd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42465 from tchaikov/wip-ceph-volume

debian, rpm: split ceph-volume into a separated package

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #43019 from rhcs-dashboard/fix-52028-master

mgr/dashboard: Incorrect MTU mismatch warning

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: p-se <NOT@FOUND>

Merge pull request #43115 from mgfritch/cephadm-rm-repo-no-podman

cephadm: skip podman check during `rm-repo`

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43129 from cyx1231st/wip-seastore-optimize-onode

crimson/os/seastore: minimize the static size of onode_layout_t

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43106 from rzarzynski/wip-crimson-msgr-bind

crimson: add support for ms_bind_retry_{delay,count}

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43157 from liu-chunmei/seastore-binary-search

crimson/seastore:: add binary search for lba and omap iterator search

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: minimize the static size of onode_layout_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson: exclude oid when encode/decode object_info_t to/from object store

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore:: add binary search for lba and omap iterator search

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #43158 from athanatos/sjust/wip-seastore-omap

crimson/os/seastore: SeaStore::get_omap_iterator should seek_to_first

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43161 from rzarzynski/wip-crimson-fix-no-bs

crimson/os: allow to build crimson when WITH_BLUESTORE=OFF

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rpm, debian: drop xmlstarlet from build deps

xmlstarlet is not used anymore while performing "make check", so drop it
from the build dependency list.

it is still used by the teuthology test though, so keep it as a
dependency of ceph-test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec: remove parted from ceph-base deps

parted is only used by ceph-volume

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec, debian: move deps from base to ceph-volume

this change makes util-linux, xfsprogs and e2fsprogs runtime deps of ceph-volume

ceph-volume uses blkid and lsblk, which are in turn packaged by
util-linux.

util-linux were added as build dependency to fulfill the needs of
ceph-disk. and we tested ceph-disk as part of "make check", since
ceph-disk was dropped, there is no need to have util-linux as
build dependency anymore.

the same applies to e2fsprogs and xfsprogs

see also a99177834120e7a2c4592054f6a8b8736e0ffb92

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian/control: make cryptsetup-bin a ceph-volume runtime dep

was originally added by 6a45b8ed041514816d09b2808d2d390eadbd0dd9 and
cfcaceac44d6b7b7c55e81d0bfb05f4893f3b1d0 as a dependency of ceph-disk.
although, ceph-disk was replaced by ceph-volume, cryptsetup-bin is still
used.

but it should be a runtime dependency of ceph-volume. and it is not a
build dependency of ceph, so it's removed from the build dependency list
as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: make cryptsetup a ceph-volume runtime dep.

cryptsetup is solely used by ceph-volume, so it should be a runtime
dependency of ceph-volume, not ceph-base.

and remove it fomr build-dependency list, as we don't use this tool when
building ceph.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: split ceph-volume into a separated package

ceph-volume is a tool implemented in pure python, so it would be better
to make it a architecture independent package for better
maintainability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: split ceph-volume into a separated package

ceph-volume is a tool implemented in pure python, so it would be better
to make it a architecture independent package for better
maintainability.

in this change

* ceph-volume is extracted out into a separated package
* ceph-volume depends on ceph-osd, as it deploys it and relies on
  an already-installed ceph-osd in the system.
* ceph-osd recommends ceph-volume. as ceph-osd can be used as a
  standalone package. but ceph-volume enhances it. also, to ensure
  the existing users to get ceph-volume installed along with
  ceph-osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,deb: make parted runtime dependency of ceph-osd

we don't run parted when building ceph, instead, it is a runtime
dependency of ceph-volume, which is packaged in ceph-osd.

so in this change

* parted is removed from build dependency list
* parted is added as a runtime depedency of ceph-osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: add the preamble to futurized_store.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: allow to build crimson when WITH_BLUESTORE=OFF.

This lets to ensure nobody is accidentally linking with the alienized
version of `common`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: switch try_bind() to bind().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: bind() respects now ms_bind_retry_{delay,count}.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: make try_bind() aware about address_not_available.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: decouple listen_ertr from bind_ertr.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: decouple listen_ertr from FixedCPUServerSocket.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43109 from guits/guits-cv-24044

ceph-volume: util/prepare fix osd_id_available()

crimson/os/seastore: SeaStore::get_omap_iterator should seek_to_first

Add debugging as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart.sh: default to msgr2 for crimson

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #43123 from liu-chunmei/lba_metrics_iter

crimson/seastore: add lba tree metrics for lba tree profiling

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: add lba tree metrics for lba tree profiling

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #43152 from athanatos/sjust/wip-simplify-delayed

crimson/sesatore: simplify epm related transaction structures, fix some bugs

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/transaction: add trace debugging for write_set

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent: fix unused variable warning

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: fix delayed alloc addr in rewrite_logical_extent

Setting nlextent's to the addr of the original extent violates invariants
in the intrusive_set we're using for ExtentIndex and generally breaks the
assumption in cache that paddrs are unique until reuse.  This means we'll
do an extra update_mapping for now -- we can refactor SegmentCleaner to
avoid it later.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/scanner: fix budget_used ref in scan_valid_records

Fixes: https://tracker.ceph.com/issues/52582
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: more get_extent debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/extent_placement_manager: clarify else branch delayed_alloc_or_ool_write

Also shorten long line.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: replace fresh_block_list with inline/ool_block_list

With this change, Transaction::write_set is the union of mutually exclusive
lists
- delayed_alloc_list, inline_block_list, and mutated_block_list prior to
  calling ExtentPlacementManager::delayed_alloc_or_ool_write

and

- inline_block_list, ool_block_list, and mutated_block_list after
  calling ExtentPlacementManager::delayed_alloc_or_ool_write

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #43136 into master

* refs/pull/43136/head:
qa/tasks/kubeadm: change calico encap to IPIPCrossSubnet
qa/suites/orch/rook/smoke: add host networking to matrix
qa/tasks/rook: fix shadowing of config arg in rook_cluster()

Reviewed-by: Joseph Sawaya <jsawaya@redhat.com>

Merge pull request #42417 from rzarzynski/wip-bl-optimize-emptybptrs

common/bl, tests: optimize carriage handling in bufferlist::c_str()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/kubeadm: change calico encap to IPIPCrossSubnet

Signed-off-by: Sage Weil <sage@newdream.net>

crimson/os/seastore/cached_extent: assert !parent_index in ExtentIndex::insert

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/so/seastore: clarify ExtentIndex clear()

ExtentIndex::clear() needs to handle clearing parent_index.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: remove delayed_set

delayed_set isn't necessary.  Delayed extents have distinct paddr segments
already, so it's simpler to have a single index for all pending extents.

Signed-off-by: Samuel Just <sjust@redhat.com>

tests/bl: verify the carriage handling in bufferlist::c_str().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/bl: abstract from _carriage checking in c_str().

This is intended to enchance readability and sligthly
generalize the previous, `_carriage`-aware optimization.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/bl: c_str() considers _carriage now.

It could be that `_carriage` points on empty bptr which is
also stored in the `_buffers` container -- this happens
when we'are trying to preserve the appendable space to not
waste memory and avoid new allocation.
Before the change this approach was imposing unncessary
rebuilding on `c_str()`. Now we take into consideration
the special case when a bufferlist has the extra, empty
bptr at its end.

Fixes: https://tracker.ceph.com/issues/51725
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-volume: util/prepare fix osd_id_available()

The current check only allows to request an OSD id that exists but
marked as 'destroyed'.
With this small fix, we can now use `--osd-id` with an id that doesn't
exist at all.

Fixes: https://tracker.ceph.com/issues/50880
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #43141 from sebastian-philipp/doc-orch-rm

doc/cephadm: Removing a service

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #42998 into master

* refs/pull/42998/head:
qa/tasks/rook: add testing for host label add/rm
mgr/rook: host add/rm label in rook orchestrator

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #43147 from krunerge/fix_typo

doc/dev/crimson/poseidonstore: fix typo

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43063 from guits/guits-trailing-spaces

ceph-volume: remove trailing spaces in code

Merge pull request #42951 from ronen-fr/wip-ronenf-pg-dtor

osd/scrub: destruct the scrubber shortly before the PG is destructed

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>-
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/dev/crimson/poseidonstore: fix typo

Signed-off-by: krunerge <krunerge@tencent.com>

osd: fix partial recovery become whole object recovery after restart osd

support SERVER_OCTOPUS feature for pg_missing_item::encode()

Fixes: https://tracker.ceph.com/issues/52583
Signed-off-by: Jianwei Zhang <jianwei1216@qq.com>

Merge pull request #42555 from hualongfeng/retire_error_change

librbd/cache/pwl/ssd: remove correct m_blocks_to_log_entries entry

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43038 from majianpeng/librbd-pwl-exclusive-lock

librbd: require exclusive lock for reads if pwl cache is enabled

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd: remove correct m_blocks_to_log_entries entry

When retiring, m_blocks_to_log_entries doesn't remove
corresponding write_entry (should be `*it` not `entry`)
that will be retired. It leads to read error. And
there should also consider discard entries.

Fixes: https://tracker.ceph.com/issues/52579
Signed-off-by: Feng Hualong <hualong.feng@intel.com>

Merge pull request #43143 from sebastian-philipp/further-reading

doc/cephadm: Add lots of links to other chapters

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41869 from ronen-fr/wip-ronenf-fmt-scrub

common, osd: add fmtlib formatting of some OSD types

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43056 from Matan-B/wip-matanb-scrub-passkey

osd: Replacing friend declarations for passkey interface

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42684 from ronen-fr/wip-ronenf-scrub-token

osd/scrub: tag replica scrub messages to identify stale events

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

doc/cephadm: Add lots of links to other chapters

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: Removing a service

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43134 from ceph/debhelpermerged

debian/control: dh-systemd is part of debhelper now

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

debian/control: dh-systemd is part of debhelper now

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=822670

Signed-off-by: David Galloway <dgallowa@redhat.com>

qa/tasks/rook: add testing for host label add/rm

This commit adds simple tests that check if the orchestrator
can add and remove labels from nodes.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

cephadm: skip podman check during `rm-repo`

allow the `rm-repo` command to succeed when podman is not installed

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43104 from ceph/e2e-tests-documentation

doc/dev: specify location of e2e-tests script

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #43053 from a16bitsysop/fsevents

mgr/dashboard: use -f for npm ci to skip fsevents error

Reviewed-by: a16bitsysop <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #42984 from majianpeng/pwl-ssd-race-bug

librbd/cache/pwl/ssd: fix a race between get_cache_bl() and remove_cache_bl()

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42883 from majianpeng/pwl-ssd-calc-allocated-bug

librbd/cache/pwl: fix m_bytes_{allocated,cached} calculation on reopen

Reviewed-by: Yin Congmin <congmin.yin@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43124 from xxhdx1985126/wip-om-log-fix

crimson/os/seastore: fix some debug outputs in omap manager

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore: fix some debug outputs in omap manager

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43108 from xxhdx1985126/wip-epm-roll-leak

crimson/os/seastore: fix epm rolling_segment leak when the transaction is interrupted

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43118 from rzarzynski/wip-crimson-assert-v2-on-bind

crimson/net: assert the address is v2 on attempt to bind.

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/rook: host add/rm label in rook orchestrator

This commit adds the functionality for adding/removing labels
to hosts using the rook orchestrator and orch host label add/rm.
The labels are added to the kubernetes node object as kubernetes
labels prefixed by "ceph-label/".

This commit also changes ceph orch host ls to display only the ceph
labels.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

doc/dev: specify location of e2e-tests script

I was just going through these steps myself, and I found it difficult to locate and run `run-frontend-e2e-tests.sh`. Specifying the location of it might make it clearer to others where the correct script is located.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #42912 from dsavineau/ceph_volume_lvm_active_no_systemd

ceph-volume: fix lvm activate --all --no-systemd

Merge pull request #43013 from rzarzynski/wip-msg-const-features_supported

msg: make Policy::features_supported static and constant.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42853 from sseshasa/wip-fix-vstart-mon-permissions

mon/MonCap: Update osd profile to allow cmd to set iops capacity on mon db

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/net: assert the address is v2 on attempt to bind.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43014 from dsavineau/ceph_volume_lvm_active_id_fsid_mandatory

ceph-volume: fix lvm activate arguments

Merge pull request #43079 from petrutlucian94/hv_addr_doc

doc/rbd: describe Hyper-V disk addressing limitations

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43086 from idryomov/wip-rbd-validate-pool-async

librbd: fix pool validation lockup

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #43107 from SUSE/wip-fix-52553

pybind/rados: don't close watch in dealloc if already closed

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #43042 from dsavineau/ceph_volume_lvm_migrate_args

ceph-volume: fix lvm migrate without args

crimson/os/seastore: fix epm rolling_segment leak when the transaction is interrupted

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: better epm output

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

osd: Replacing friend declerations for passkey interface

removing friend decleration from PrimaryLogScrub and ReplicaReservations

Signed-off-by: Matan Breizman <Matan.Brz@gmail.com>

pybind/rados: don't close watch in dealloc if already closed

Fixes: https://tracker.ceph.com/issues/52553
Signed-off-by: Tim Serong <tserong@suse.com>

crimson/net: s/do_bind/do_listen/ in SocketMessenger.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: simplify error handling in FixedCPUServerSocket::listen().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>