Merge pull request #37287 from pcuzner/dashboard-goals

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #37878 into master

* refs/pull/37878/head:
doc/mgr/orchestrator: Update about "{mds, rgw} add" status in rook

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge PR #37732 into master

* refs/pull/37732/head:
osdc: restart read on truncate/discard

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37898 from lxbsz/nbd_reexpandmeta

rbd-nbd: reexpand the conf meta in child process

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37547 from ifed01/wip-ifed-fix-bmap-alloc

os/bluestore: fix segfault on out-of-bound offset provided to  claim_…

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #37879 from tchaikov/wip-seastar-v6

crimson: use Seastar API v6

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #37894 from dillaman/wip-48035

librbd: global config overrides do not apply to in-use images

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37907 from athanatos/sjust/wip-fix-init-weak

crimson/os/seastore/.../lba_btree_node_impl: hold reference to *this during lookup

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: hold reference to *this during lookup

4f2f4f modified TransactionManager::mount() to use a weak_transaction
while calling init_cached_extents.  This is fine, but lookup() needs
to hold a reference to *this until the child lookup completes in order
to ensure residence in the lba pinning set.

Signed-off-by: Samuel Just <sjust@redhat.com>

doc/dev/dashboard: typo fix

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

doc/dev/dashboard: link designs to dev guide

Add a feature design section to the dashboard dev
guide, linking to the individual docs under dov/dev/dashboard

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

doc/dev/dashboard: Document key UI design principles

Document which defines the high level design principles
for the Ceph Dashboard UI

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #37590 from pcuzner/prometheus-slow-ops

mgr/prometheus: Add SLOW_OPS healthcheck as a metric

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: add doc to describe relationship to prometheus

The healthcheck for slow_ops is used by the mgr/prometheus
module so it's important the alert generated matches the
format expected in mgr/prometheus.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/prometheus: Add healthcheck metric for SLOW_OPS

SLOW_OPS is triggered by op tracker, and generates a health
alert but healthchecks do not create metrics for prometheus to
use as alert triggers. This change adds SLOW_OPS metric, and
provides a simple means to extend to other relevant health
checks in the future

If the extract of the value from the health check message fails
we log an error and remove the metric from the metric set. In
addition the metric description has changed to better reflect
the scenarios where SLOW_OPS can be triggered.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #36626 from lixiaoy1/rbd_image_discard

librbd/rwl: discard cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37870 from xxhdx1985126/wip-crimson-recovery-bug-fixes

crimson: recovery-related bug fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37910 from tchaikov/wip-crimson-osd-send-pg-temp

crimson/osd: merge pg_temp_wanted before sending them

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #37656 from yuvalif/allow_lua_to_set_metadata

rgw/lua: allow setting metadata via lua

crimson/osd: merge pg_temp_wanted before sending them

there is chance that new pg_temp_wanted is added when we are sending
them in ShardServices::send_pg_temp(), but the pg temp are already
collected into the messages before seastar::parallel_for_each(). so,
we cannot move all elements from pg_temp_wanted to pg_temp_pending after
sending the messages, that'd also move the unsent pg temps to
pg_temp_pending as well. so we have to do this before do the async call,

this issue was root caused by Xuehan Xu <xxhdx1985126@gmail.com>.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: drop ShardServices::_sent_pg_temp()

there is no need to define a wrapper for moving pg_temp_wanted to
pg_temp_pending. and it hurts the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: report statfs data

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: rename heartbeat_timer to tick_timer

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: make Gated handle gate_close_exception

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: do not check HAVE_STDLIB_MAP_SPLICING

seastar always uses GCC-9 and up, so there is no need to check for
the existence of map::merge() before using it anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: clean up pending_txn when actingset is changed

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: fix pg info statistics error during recovery

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: make I/O reqs wait for object recovery if it's degraded or backfilling

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #37327 from kamoltat/wip-mgr-progress-global-event

mgr/progress: Global Recovery Event in ceph -s

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37598 from ronen-fr/scrubstore-fix-rf

osd: fix signatures of get_store_errors() and friends

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>

Merge pull request #37751 from changchengx/nasm_avx512

common: use nasm to build isa-l and isa-l_crypto to prepare for AVX512

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37773 from sunnyku/wip-f33-pyfix

do_cmake.sh: use python-3.9 with fedora version 33

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37789 from trociny/wip-47827

mgr/rbd_support: create mirror snapshots asynchronously

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw/lua: use lua exceptions instead of ceph_asserts

also use lua assert() in uni tests instead of print()
fix bug when accessing user id in an ACL grant

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #37482 from rhcs-dashboard/datatable_click_prevent

mgr/dashboard: Prevent table items from getting selected while expanding

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #35769 from Codom/master

mgr/dashboard: Added Versioning to the REST API

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

rgw/lua: allow setting metadata via lua

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #37788 from tchaikov/wip-zbd

rpm,cmake: s/WITH_LIBZBD/WITH_ZBD/ and enable ZBD on demand

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>

Merge pull request #37888 from tchaikov/wip-crimson-gtest

crimson/gtest_seastar: do not keep a copy of argv

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #37891 from tchaikov/wip-cmake-alienstore

cmake: build bluestore-tp before crimson-alienstore

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #37893 from tchaikov/wip-wunused-variable

osd/osd_types: mark unused variable [[maybe_unused]]

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rbd-nbd: reexpand the conf meta in child process

Fixed: https://tracker.ceph.com/issues/48046
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #37864 from trociny/wip-48020

mgr/rbd_support: store global schedule without localized prefix

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37880 from trociny/wip-rbd-finisher-cancel

librbd: relax requirements on finisher canceled callback

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/rbd_support: create mirror snapshots asynchronously

To scale up with number of images.

Fixes: https://tracker.ceph.com/issues/47827
Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/rbd: add async mirror image get mode and info methods

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add missing declaration of rbd_aio_mirror_image_get_mode

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/rbd: add image async open and close methods

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: refresh full global config when applying metadata

The ConfigProxy contains a point-in-time copy of the global config
that is dynamically updated in CephContext::_conf. Upon an image
refresh, pull the latest version of the global config from the
CephContext and apply it to the config stored within the ImageCtx.

Fixes: https://tracker.ceph.com/issues/48035
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/config_proxy: added 'get_config_values' method

Expose the ability to retrieve the ConfigValues from an existing
ConfigProxy object. When combined with the newly exposed
'set_config_values' method, it will allow the safe bulk-updating
of configuration options. This also removes the original private
static version of 'get_config_values'.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37886 from tchaikov/wip-osd-scheduler-move

osd/scheduler: rely on copy ellision to move return val

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #34866 from nickjanus/issue-45193

rgw-admin: fixes BucketInfo for missing buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37657 from batrick/rgw-lua-fixes

rgw: lua refactor

rbd/tool: add image-cache invalidate

Add an rbd command image-cache invalidate to discard RWL cache.

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd/internal: update invalidate_cache

Except invalidating object cache, invalidate image writeback cache.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

librbd/cache: discard writeback cache

Add an interface in ImageCache to discard writeback cache.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

pybind/rbd: async API for creating mirror snapshots

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: async API for creating mirror snapshots

Signed-off-by: Mykola Golub <mgolub@suse.com>

common/config_proxy: expose the 'set_config_values' method

This method was previously only available for seastart builds, but
librbd will also use it to control the timing of updates to the global
config.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37884 from dillaman/wip-48032

qa/workunits/rbd: fix permission issue when removing mirror peer

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37890 from tchaikov/wip-crimson-os-cleanup

crimson/os: fix couple issues reported by clang

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #36739 from pritha-srivastava/wip-rgw-sts-expire-session-token

rgw/sts: adding code for "aws:TokenIssueTime" to be used

Merge pull request #35612 from pritha-srivastava/wip-sts-role-min-duration

rgw/sts: Adding a configurable rgw_sts_min_session_duration

osd/scheduler: rely on copy ellision to move return val

C++14 enforces copy ellision in this case.

also silences warning of

src/osd/scheduler/mClockScheduler.cc:122:21: warning: redundant move in return statement [-Wredundant-move]
  122 |     return std::move(ret);
      |            ~~~~~~~~~^~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/osd_types: mark unused variable [[maybe_unused]]

silences warning like:

rc/osd/osd_types.h:5482:18: warning: unused variable '_' [-Wunused-variable]
     auto [iter, _] = ref_delta.try_emplace(hoid, 0);
                  ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: put "{" at a new line

so the function body is not cluttered with the function signature.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: use plain seastar::future in lookup_range()

otherwise GCC enters a dead loop when compiling this method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: drop variadic future support from errorator

this allows us to bump up Seastar_API_LEVEL to 6 which completely ditch
variadic future from Seastar.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build bluestore-tp before crimson-alienstore

to address the FTBFS of

src/os/bluestore/BlueStore.cc:52:10: fatal error: tracing/bluestore.h: No such file or directory
   52 | #include "tracing/bluestore.h"
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.

tracing/bluestore.h is created by the "bluestore-tp" target, there
is a good chance that we build crimson when WITH_LTTNG is enabled,
let's ensure that tracing/bluestore.h is ready before compiling
alienstore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: capture local bindings using capture variable list

structured binding does not define variables, and the implicit
capture-by-copy `=` does not help in this case, we have to define a
capture with an initializer to create a *variable* enclosed by
the closure explicitly.

this address the error like:

src/crimson/os/seastore/segment_cleaner.cc:315:5: error: reference to local binding 'addr' declared in enclosing lambda expression
                  addr);
                  ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not move temporay variable

copy ellision ensures that copy ctor is not called in this case

silences warning like:

mson/os/seastore/lba_manager/btree/lba_btree_node_impl.cc:262:5: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
    std::move(begin()),
    ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not capture unused variable

silences warning like:

btree/extentmap_btree_node.h:100:8: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
      [this, ec, len] (auto &extents) {
       ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37887 from tchaikov/wip-seastar

seastar: pick up changes to colocate libc allocator and seastar alloc…

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/gtest_seastar: consume argv before passing them to seastar

so we can pass gtest options to the test without annoying seastar app.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/gtest_seastar: do not keep a copy of argv

argv is always available in the whole life cycle of the application, so
there is no need to keep a copy of it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pick up changes to colocate libc allocator and seastar allocator

this allows us to use seastar's builtin allocator along with the libc
allocator used by alien threads.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove yasm since it's been replaced by nasm

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: use nasm to replace yasm to build isa-l

1. use nasm to replace yasm to build isa-l
2. use the defined MACRO for nasm to replace all the defined MACRO for
   yasm

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

install-deps.sh: install nasm for isa-l library

nasm support build isa-l:AVX512 algorithm implementation while yasm
doens't support it. Install nasm assembler to build isa-l
refer to: https://github.com/yasm/yasm/issues/101

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common: use isa-l's crc32 to implement fast crc32 on IA arch

1. Accoring to the commit history, crc32c_intel_fast_asm.s is copied from
isa-l's crc32_iscsi_00.asm. Let's use isa-l's crc32 to replace the old
one.
2. For nasm, it doesn't recongized "rip". Change all the address to be
rip relative by adding "default rel".

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: wrapper nasm as assembler to deal with some options

1. unify some options, such as "-I -isystem"
2. ignore some known options, such as "-W*"
3. ignore some unknown options, such as '-fPIC"

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: check nasm and define related MACRO

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: define cmake macro to check nasm assembler

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: set HAVE_LIBZBD before creating "acconfig.h"

`acconfig.h` is generated using

configure_file(
  ${CMAKE_SOURCE_DIR}/src/include/config-h.in.cmake
  ${CMAKE_BINARY_DIR}/include/acconfig.h
)

in `config-h.in.cmake`, the cmake variable of `HAVE_LIBZBD` is checked.
so we need to ensure that this variable is visible from this
`configure_file()` statement.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: pass -DWITH_ZBD=ON to cmake if zbd is enabled

this is a leftover of fd1b02a341e3afba32776e35ec7c4bbe883f9b06

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/WITH_LIBZBD/WITH_LIBZBD/

fix the regression introduced by
d53638630631cc6596a1238228332e7579318415

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37777 from chenerqi/fix-modified-dentries-count

mds: fix count error of modified dentries

cmake: extract common in one place for further check

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: check whether yasm support AVX512

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common: rename YASM MACRO to hint support SIMD

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common: rename variable name to improve readability

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: refine yasm support x86_64 and AVX2 info

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: more quiet when checking for yasm

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

yasm-wrapper: refine yasm option usage strategy in yasm-wrapper

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

yasm-wrapper: remove redundant option

"-fPIC" is covered by "-f"

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

yasm-wrapper: remove not existed options

commit "0cf627b4" has removed lowmem_builder condition.
ggc-min* not exist now.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #37636 from MahatiC/wip-init-ssd

librbd/cache: init functionality for SSD Cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37680 from orozery/object-dispatch-aligned-crypto

librbd: crypto alignment support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37871 from cyx1231st/wip-seastore-misc-fixes

crimson/seastore: misc fixes in transaction manager

crimson/seastore: misc fixes in transaction manager

Merge pull request #37877 from dang/wip-dang-cond-writeback

rgw: Fix broken merge in #37774