qa/tasks: support explicit disk configuration for qemu task

The 'disks' key will now be treated as a dictionary where all previous
global settings can be individually applied. Additionally, a disk can be
pre-created and provided for use by qemu.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add new 'migration prepare --import-only' CLI optional

The '--import-only' optional can be combined with either a
'--source-spec' or '--source-spec-path' optional to define
the source for the read-only import.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: show migration source spec in image status

When using the non-legacy mode image migration, the 'rbd status'
command will now show the source-spec for the migration if the
source pool id is invalid.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: ensure the help printer doesn't print past the end of the line

When long command names and long optional names are combined,
it's possible for the help text to be printed beyond the 80
character limit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: replace direct reads from ImageRequest with ImageDispatchSpec

Read requests will always need to go via the image dispatch layer to
ensure migrations are properly handled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: allow native v2 images to be opened by name

When in import-only mode, allow the source image to be opened by
name in case it's a V2 image format w/o its image id specified in
the JSON.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: close image if failed to open source

To remain consistent between the native which will automatically close
the ImageCtx if it fails to open the image, the raw format should also
close the ImageCtx.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: initialize default source image layout

For backing source formats that don't care about the low-level
RBD layout details, provide safe default values for the layout
to ensure that striping logic will still function.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix potential recursive lock of ImageCtx::image_lock

If the "list_snaps" IO call returns immediately, it can result
in an attempted recursive lock starting from CopyupRequest.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: migration execute should allow "EXECUTING" state

If the "execute" action was interrupted, we need to be able to
restart it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: read image size when opening migration raw format

The migration prepare operation will need to know the size of
the source image so that it can create the destination image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: fix issue with prepare-import and native-format

When preparing an import, the native format can not expect to have
the child image opened since it hasn't been created yet.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added new 'migration_prepare_import' API methods

These related methods accept a JSON-encoded source-spec that
describes how to read from the source image. The migration is
a one-way operation, so children/groups/etc won't be updated
to point to the new destination image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: store migration progress on both source+destination images

When migrating from an import-only source-spec like an external
file, it will be important to also store the progress on the
destination image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch to migration source-spec builder

Remove all the hard-coded references to the NativeFormat
handler and allow the RawFormat+FileStream handlers to be
instantiated.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch migration native format to use source-spec JSON object

Remove the dependency on the MigrationInfo struct and allow the native
format to directly pull parameters from the JSON-encoded object
structure.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: tweak native-format migration source-spec

Renamed "format" property to "type" and always include the original
"image_id".

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/api: ImageCtx must be closed instead of deleted

On error, the destination image context will need to be properly
closed instead of just deleted.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36626 from lixiaoy1/rbd_image_discard

librbd/rwl: discard cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37870 from xxhdx1985126/wip-crimson-recovery-bug-fixes

crimson: recovery-related bug fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37910 from tchaikov/wip-crimson-osd-send-pg-temp

crimson/osd: merge pg_temp_wanted before sending them

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #37656 from yuvalif/allow_lua_to_set_metadata

rgw/lua: allow setting metadata via lua

crimson/osd: merge pg_temp_wanted before sending them

there is chance that new pg_temp_wanted is added when we are sending
them in ShardServices::send_pg_temp(), but the pg temp are already
collected into the messages before seastar::parallel_for_each(). so,
we cannot move all elements from pg_temp_wanted to pg_temp_pending after
sending the messages, that'd also move the unsent pg temps to
pg_temp_pending as well. so we have to do this before do the async call,

this issue was root caused by Xuehan Xu <xxhdx1985126@gmail.com>.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: drop ShardServices::_sent_pg_temp()

there is no need to define a wrapper for moving pg_temp_wanted to
pg_temp_pending. and it hurts the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: report statfs data

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: rename heartbeat_timer to tick_timer

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: make Gated handle gate_close_exception

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: do not check HAVE_STDLIB_MAP_SPLICING

seastar always uses GCC-9 and up, so there is no need to check for
the existence of map::merge() before using it anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: clean up pending_txn when actingset is changed

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: fix pg info statistics error during recovery

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: make I/O reqs wait for object recovery if it's degraded or backfilling

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #37327 from kamoltat/wip-mgr-progress-global-event

mgr/progress: Global Recovery Event in ceph -s

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37598 from ronen-fr/scrubstore-fix-rf

osd: fix signatures of get_store_errors() and friends

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>

Merge pull request #37751 from changchengx/nasm_avx512

common: use nasm to build isa-l and isa-l_crypto to prepare for AVX512

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37773 from sunnyku/wip-f33-pyfix

do_cmake.sh: use python-3.9 with fedora version 33

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37789 from trociny/wip-47827

mgr/rbd_support: create mirror snapshots asynchronously

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw/lua: use lua exceptions instead of ceph_asserts

also use lua assert() in uni tests instead of print()
fix bug when accessing user id in an ACL grant

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #37482 from rhcs-dashboard/datatable_click_prevent

mgr/dashboard: Prevent table items from getting selected while expanding

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #35769 from Codom/master

mgr/dashboard: Added Versioning to the REST API

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

rgw/lua: allow setting metadata via lua

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #37788 from tchaikov/wip-zbd

rpm,cmake: s/WITH_LIBZBD/WITH_ZBD/ and enable ZBD on demand

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>

Merge pull request #37888 from tchaikov/wip-crimson-gtest

crimson/gtest_seastar: do not keep a copy of argv

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #37891 from tchaikov/wip-cmake-alienstore

cmake: build bluestore-tp before crimson-alienstore

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #37893 from tchaikov/wip-wunused-variable

osd/osd_types: mark unused variable [[maybe_unused]]

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37864 from trociny/wip-48020

mgr/rbd_support: store global schedule without localized prefix

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37880 from trociny/wip-rbd-finisher-cancel

librbd: relax requirements on finisher canceled callback

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/rbd_support: create mirror snapshots asynchronously

To scale up with number of images.

Fixes: https://tracker.ceph.com/issues/47827
Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/rbd: add async mirror image get mode and info methods

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add missing declaration of rbd_aio_mirror_image_get_mode

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/rbd: add image async open and close methods

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37886 from tchaikov/wip-osd-scheduler-move

osd/scheduler: rely on copy ellision to move return val

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #34866 from nickjanus/issue-45193

rgw-admin: fixes BucketInfo for missing buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37657 from batrick/rgw-lua-fixes

rgw: lua refactor

rbd/tool: add image-cache invalidate

Add an rbd command image-cache invalidate to discard RWL cache.

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd/internal: update invalidate_cache

Except invalidating object cache, invalidate image writeback cache.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

librbd/cache: discard writeback cache

Add an interface in ImageCache to discard writeback cache.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

pybind/rbd: async API for creating mirror snapshots

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: async API for creating mirror snapshots

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37884 from dillaman/wip-48032

qa/workunits/rbd: fix permission issue when removing mirror peer

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37890 from tchaikov/wip-crimson-os-cleanup

crimson/os: fix couple issues reported by clang

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #36739 from pritha-srivastava/wip-rgw-sts-expire-session-token

rgw/sts: adding code for "aws:TokenIssueTime" to be used

Merge pull request #35612 from pritha-srivastava/wip-sts-role-min-duration

rgw/sts: Adding a configurable rgw_sts_min_session_duration

osd/scheduler: rely on copy ellision to move return val

C++14 enforces copy ellision in this case.

also silences warning of

src/osd/scheduler/mClockScheduler.cc:122:21: warning: redundant move in return statement [-Wredundant-move]
  122 |     return std::move(ret);
      |            ~~~~~~~~~^~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/osd_types: mark unused variable [[maybe_unused]]

silences warning like:

rc/osd/osd_types.h:5482:18: warning: unused variable '_' [-Wunused-variable]
     auto [iter, _] = ref_delta.try_emplace(hoid, 0);
                  ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build bluestore-tp before crimson-alienstore

to address the FTBFS of

src/os/bluestore/BlueStore.cc:52:10: fatal error: tracing/bluestore.h: No such file or directory
   52 | #include "tracing/bluestore.h"
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.

tracing/bluestore.h is created by the "bluestore-tp" target, there
is a good chance that we build crimson when WITH_LTTNG is enabled,
let's ensure that tracing/bluestore.h is ready before compiling
alienstore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: capture local bindings using capture variable list

structured binding does not define variables, and the implicit
capture-by-copy `=` does not help in this case, we have to define a
capture with an initializer to create a *variable* enclosed by
the closure explicitly.

this address the error like:

src/crimson/os/seastore/segment_cleaner.cc:315:5: error: reference to local binding 'addr' declared in enclosing lambda expression
                  addr);
                  ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not move temporay variable

copy ellision ensures that copy ctor is not called in this case

silences warning like:

mson/os/seastore/lba_manager/btree/lba_btree_node_impl.cc:262:5: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
    std::move(begin()),
    ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not capture unused variable

silences warning like:

btree/extentmap_btree_node.h:100:8: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
      [this, ec, len] (auto &extents) {
       ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37887 from tchaikov/wip-seastar

seastar: pick up changes to colocate libc allocator and seastar alloc…

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/gtest_seastar: consume argv before passing them to seastar

so we can pass gtest options to the test without annoying seastar app.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/gtest_seastar: do not keep a copy of argv

argv is always available in the whole life cycle of the application, so
there is no need to keep a copy of it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pick up changes to colocate libc allocator and seastar allocator

this allows us to use seastar's builtin allocator along with the libc
allocator used by alien threads.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove yasm since it's been replaced by nasm

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: use nasm to replace yasm to build isa-l

1. use nasm to replace yasm to build isa-l
2. use the defined MACRO for nasm to replace all the defined MACRO for
   yasm

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

install-deps.sh: install nasm for isa-l library

nasm support build isa-l:AVX512 algorithm implementation while yasm
doens't support it. Install nasm assembler to build isa-l
refer to: https://github.com/yasm/yasm/issues/101

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common: use isa-l's crc32 to implement fast crc32 on IA arch

1. Accoring to the commit history, crc32c_intel_fast_asm.s is copied from
isa-l's crc32_iscsi_00.asm. Let's use isa-l's crc32 to replace the old
one.
2. For nasm, it doesn't recongized "rip". Change all the address to be
rip relative by adding "default rel".

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: wrapper nasm as assembler to deal with some options

1. unify some options, such as "-I -isystem"
2. ignore some known options, such as "-W*"
3. ignore some unknown options, such as '-fPIC"

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: check nasm and define related MACRO

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: define cmake macro to check nasm assembler

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: set HAVE_LIBZBD before creating "acconfig.h"

`acconfig.h` is generated using

configure_file(
  ${CMAKE_SOURCE_DIR}/src/include/config-h.in.cmake
  ${CMAKE_BINARY_DIR}/include/acconfig.h
)

in `config-h.in.cmake`, the cmake variable of `HAVE_LIBZBD` is checked.
so we need to ensure that this variable is visible from this
`configure_file()` statement.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: pass -DWITH_ZBD=ON to cmake if zbd is enabled

this is a leftover of fd1b02a341e3afba32776e35ec7c4bbe883f9b06

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/WITH_LIBZBD/WITH_LIBZBD/

fix the regression introduced by
d53638630631cc6596a1238228332e7579318415

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37777 from chenerqi/fix-modified-dentries-count

mds: fix count error of modified dentries

cmake: extract common in one place for further check

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: check whether yasm support AVX512

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common: rename YASM MACRO to hint support SIMD

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common: rename variable name to improve readability

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: refine yasm support x86_64 and AVX2 info

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cmake: more quiet when checking for yasm

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

yasm-wrapper: refine yasm option usage strategy in yasm-wrapper

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

yasm-wrapper: remove redundant option

"-fPIC" is covered by "-f"

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

yasm-wrapper: remove not existed options

commit "0cf627b4" has removed lowmem_builder condition.
ggc-min* not exist now.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #37636 from MahatiC/wip-init-ssd

librbd/cache: init functionality for SSD Cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37680 from orozery/object-dispatch-aligned-crypto

librbd: crypto alignment support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37871 from cyx1231st/wip-seastore-misc-fixes

crimson/seastore: misc fixes in transaction manager

crimson/seastore: misc fixes in transaction manager

Merge pull request #37877 from dang/wip-dang-cond-writeback

rgw: Fix broken merge in #37774

qa/workunits/rbd: fix permission issue when removing mirror peer

Fixes: https://tracker.ceph.com/issues/48032
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

RGW - Fix broken merge in #37774

I messed up the merge in #37774 and somehow forgot to push a change.
This resulted it the flag being set on the wrong branch.  This fixes
that.

RHBZ#1845501

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #37865 from neha-ojha/wip-mon-fixes

qa: miscellaneous mon test fixes

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: fix signatures of get_store_errors() and friends

The 'store' parameter was ignored by ScrubStore-related get_*_errors()
functions, and is now removed.

The functions are now marked 'const'. That required tagging the caching
member Store::backend as 'mutable'. While 'mutable' is an 'eyesore',
here is one of the rare cases where its use is justified. Following
https://isocpp.org/wiki/faq/const-correctness:
"When methods change the physical but not logical state, the method should
generally be marked as const since it really is an inspector-method."
(The text then continues and specifically prescribes 'mutable' for these situations.)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>