osd/PGLog: do not use unique_ptr explicitly

* do not use unique_ptr<> explicitly, use `seastar::do_with()` for
  better readability
* use seastar::do_until() instead of seastar::repeat() for better
  readability. plain boolean is simpler than
  `seastar::stop_iteration::yes`
* do not capture variables using FuturizedStoreLogReader if we could
  pass them by instant parameters.
* rename "start()" to "read()". as "read" is more specific in this
  context.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PGLog: move PGLog::read_log_and_missing_crimson() into .cc

to speed up the compilation. and for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PGLog: drop template parameter of FuturizedStoreLogReader

there is no need for the generic type at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use seastar:do_until() when appropriate

a plain boolean is easier to understand than
`seastar::stop_iteration::yes`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark CyanStore::CyanOmapIterator methods final

this class is not intended to be inherited by classes, and these methods
are "final", so mark them "final".

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark methods pure virtual

if the default implementation does not make sense in any case, just mark
it pure virtual.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: let OmapIterator mutators return future<>

instead of returning future<int> where the int represents an errno which
is not checked by any callers, we should just return a future<>, we
could erroratorize it later on, but returning an int is not the right
way to handle this, and it is not consistent with how we handle other
errors elsewhere in crimson.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37944 from tchaikov/wip-crimson-recovery-cleanup

crimson/osd: extract read_omap_for_push_op()

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: extract read_omap_for_push_op()

extract read_omap_for_push_op() out of build_push_op() for better
readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: correct the function names in logging messages

it was a leftover of fbfc53c7081ea0b99fbc519d7fac9f5637cf5425

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #37881 into master

* refs/pull/37881/head:
doc/mgr/orchestrator: Update about nfs add in rook
mgr/rook: Refactor nfs add/apply methods
mgr/rook: Udpate to mypy py3 type style in _create(), _update() and apply()
mgr/rook: Remove unused 'current' argument from update_func method

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge PR #37825 into master

* refs/pull/37825/head:
doc: document MDS cache configuration
test: add DecayCounter test

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #37923 into master

* refs/pull/37923/head:
client: increment file position on _read_sync near eof

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #37737 from votdev/issue_47926_orch_status

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37757 from votdev/issue_47884_display_time

mgr/dashboard: displaying time in human-readable format

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37827 from dillaman/wip-librbd-migration-2

librbd: added API and CLI support for source-spec import migrations

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/mgr/orchestrator: Update about nfs add in rook

"nfs add" is no longer supported in rook. Instead "apply nfs" should be used to
deploy nfs-ganesha daemons.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/rook: Refactor nfs add/apply methods

Fixes: https://tracker.ceph.com/issues/47923
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/rook: Udpate to mypy py3 type style in _create(), _update() and apply()

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/rook: Remove unused 'current' argument from update_func method

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #37829 from liu-chunmei/seastore-extentmap-fix

crimson/seastore: add Extentmap node type in cache.cc

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/man: add new RBD migration prepare source-spec optionals

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suite/rbd: simple migration import test case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks: support explicit disk configuration for qemu task

The 'disks' key will now be treated as a dictionary where all previous
global settings can be individually applied. Additionally, a disk can be
pre-created and provided for use by qemu.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add new 'migration prepare --import-only' CLI optional

The '--import-only' optional can be combined with either a
'--source-spec' or '--source-spec-path' optional to define
the source for the read-only import.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37872 from theanalyst/doc/releases/v14.2.13

doc: releases: v14.2.13 release notes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

client: increment file position on _read_sync near eof

Fixes: https://tracker.ceph.com/issues/48076
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/seastore: add replay test for extent map tree

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #37287 from pcuzner/dashboard-goals

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #37878 into master

* refs/pull/37878/head:
doc/mgr/orchestrator: Update about "{mds, rgw} add" status in rook

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge PR #37732 into master

* refs/pull/37732/head:
osdc: restart read on truncate/discard

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: releases/nautilus: add more changelog entries & fix date

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #37898 from lxbsz/nbd_reexpandmeta

rbd-nbd: reexpand the conf meta in child process

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37547 from ifed01/wip-ifed-fix-bmap-alloc

os/bluestore: fix segfault on out-of-bound offset provided to  claim_…

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #37879 from tchaikov/wip-seastar-v6

crimson: use Seastar API v6

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #37894 from dillaman/wip-48035

librbd: global config overrides do not apply to in-use images

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37907 from athanatos/sjust/wip-fix-init-weak

crimson/os/seastore/.../lba_btree_node_impl: hold reference to *this during lookup

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: hold reference to *this during lookup

4f2f4f modified TransactionManager::mount() to use a weak_transaction
while calling init_cached_extents.  This is fine, but lookup() needs
to hold a reference to *this until the child lookup completes in order
to ensure residence in the lba pinning set.

Signed-off-by: Samuel Just <sjust@redhat.com>

doc/dev/dashboard: typo fix

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

doc/dev/dashboard: link designs to dev guide

Add a feature design section to the dashboard dev
guide, linking to the individual docs under dov/dev/dashboard

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

doc/dev/dashboard: Document key UI design principles

Document which defines the high level design principles
for the Ceph Dashboard UI

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #37590 from pcuzner/prometheus-slow-ops

mgr/prometheus: Add SLOW_OPS healthcheck as a metric

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: add doc to describe relationship to prometheus

The healthcheck for slow_ops is used by the mgr/prometheus
module so it's important the alert generated matches the
format expected in mgr/prometheus.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/prometheus: Add healthcheck metric for SLOW_OPS

SLOW_OPS is triggered by op tracker, and generates a health
alert but healthchecks do not create metrics for prometheus to
use as alert triggers. This change adds SLOW_OPS metric, and
provides a simple means to extend to other relevant health
checks in the future

If the extract of the value from the health check message fails
we log an error and remove the metric from the metric set. In
addition the metric description has changed to better reflect
the scenarios where SLOW_OPS can be triggered.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

rbd: show migration source spec in image status

When using the non-legacy mode image migration, the 'rbd status'
command will now show the source-spec for the migration if the
source pool id is invalid.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: ensure the help printer doesn't print past the end of the line

When long command names and long optional names are combined,
it's possible for the help text to be printed beyond the 80
character limit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: replace direct reads from ImageRequest with ImageDispatchSpec

Read requests will always need to go via the image dispatch layer to
ensure migrations are properly handled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: allow native v2 images to be opened by name

When in import-only mode, allow the source image to be opened by
name in case it's a V2 image format w/o its image id specified in
the JSON.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: close image if failed to open source

To remain consistent between the native which will automatically close
the ImageCtx if it fails to open the image, the raw format should also
close the ImageCtx.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: initialize default source image layout

For backing source formats that don't care about the low-level
RBD layout details, provide safe default values for the layout
to ensure that striping logic will still function.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix potential recursive lock of ImageCtx::image_lock

If the "list_snaps" IO call returns immediately, it can result
in an attempted recursive lock starting from CopyupRequest.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: migration execute should allow "EXECUTING" state

If the "execute" action was interrupted, we need to be able to
restart it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: read image size when opening migration raw format

The migration prepare operation will need to know the size of
the source image so that it can create the destination image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: fix issue with prepare-import and native-format

When preparing an import, the native format can not expect to have
the child image opened since it hasn't been created yet.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added new 'migration_prepare_import' API methods

These related methods accept a JSON-encoded source-spec that
describes how to read from the source image. The migration is
a one-way operation, so children/groups/etc won't be updated
to point to the new destination image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: store migration progress on both source+destination images

When migrating from an import-only source-spec like an external
file, it will be important to also store the progress on the
destination image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch to migration source-spec builder

Remove all the hard-coded references to the NativeFormat
handler and allow the RawFormat+FileStream handlers to be
instantiated.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch migration native format to use source-spec JSON object

Remove the dependency on the MigrationInfo struct and allow the native
format to directly pull parameters from the JSON-encoded object
structure.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: tweak native-format migration source-spec

Renamed "format" property to "type" and always include the original
"image_id".

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/api: ImageCtx must be closed instead of deleted

On error, the destination image context will need to be properly
closed instead of just deleted.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36626 from lixiaoy1/rbd_image_discard

librbd/rwl: discard cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37870 from xxhdx1985126/wip-crimson-recovery-bug-fixes

crimson: recovery-related bug fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37910 from tchaikov/wip-crimson-osd-send-pg-temp

crimson/osd: merge pg_temp_wanted before sending them

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #37656 from yuvalif/allow_lua_to_set_metadata

rgw/lua: allow setting metadata via lua

mgr/dashboard: displaying time in human-readable format

Fixes: https://tracker.ceph.com/issues/47884
Signed-off-by: Volker Theile <vtheile@suse.com>

crimson/osd: merge pg_temp_wanted before sending them

there is chance that new pg_temp_wanted is added when we are sending
them in ShardServices::send_pg_temp(), but the pg temp are already
collected into the messages before seastar::parallel_for_each(). so,
we cannot move all elements from pg_temp_wanted to pg_temp_pending after
sending the messages, that'd also move the unsent pg temps to
pg_temp_pending as well. so we have to do this before do the async call,

this issue was root caused by Xuehan Xu <xxhdx1985126@gmail.com>.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: drop ShardServices::_sent_pg_temp()

there is no need to define a wrapper for moving pg_temp_wanted to
pg_temp_pending. and it hurts the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: report statfs data

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: rename heartbeat_timer to tick_timer

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: make Gated handle gate_close_exception

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: do not check HAVE_STDLIB_MAP_SPLICING

seastar always uses GCC-9 and up, so there is no need to check for
the existence of map::merge() before using it anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: clean up pending_txn when actingset is changed

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: fix pg info statistics error during recovery

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: make I/O reqs wait for object recovery if it's degraded or backfilling

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #37327 from kamoltat/wip-mgr-progress-global-event

mgr/progress: Global Recovery Event in ceph -s

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37598 from ronen-fr/scrubstore-fix-rf

osd: fix signatures of get_store_errors() and friends

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>

Merge pull request #37751 from changchengx/nasm_avx512

common: use nasm to build isa-l and isa-l_crypto to prepare for AVX512

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37773 from sunnyku/wip-f33-pyfix

do_cmake.sh: use python-3.9 with fedora version 33

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37789 from trociny/wip-47827

mgr/rbd_support: create mirror snapshots asynchronously

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw/lua: use lua exceptions instead of ceph_asserts

also use lua assert() in uni tests instead of print()
fix bug when accessing user id in an ACL grant

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #37482 from rhcs-dashboard/datatable_click_prevent

mgr/dashboard: Prevent table items from getting selected while expanding

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #35769 from Codom/master

mgr/dashboard: Added Versioning to the REST API

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

rgw/lua: allow setting metadata via lua

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #37788 from tchaikov/wip-zbd

rpm,cmake: s/WITH_LIBZBD/WITH_ZBD/ and enable ZBD on demand

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>

Merge pull request #37888 from tchaikov/wip-crimson-gtest

crimson/gtest_seastar: do not keep a copy of argv

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #37891 from tchaikov/wip-cmake-alienstore

cmake: build bluestore-tp before crimson-alienstore

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #37893 from tchaikov/wip-wunused-variable

osd/osd_types: mark unused variable [[maybe_unused]]

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rbd-nbd: reexpand the conf meta in child process

Fixed: https://tracker.ceph.com/issues/48046
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #37864 from trociny/wip-48020

mgr/rbd_support: store global schedule without localized prefix

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37880 from trociny/wip-rbd-finisher-cancel

librbd: relax requirements on finisher canceled callback

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/rbd_support: create mirror snapshots asynchronously

To scale up with number of images.

Fixes: https://tracker.ceph.com/issues/47827
Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/rbd: add async mirror image get mode and info methods

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add missing declaration of rbd_aio_mirror_image_get_mode

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/rbd: add image async open and close methods

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: refresh full global config when applying metadata

The ConfigProxy contains a point-in-time copy of the global config
that is dynamically updated in CephContext::_conf. Upon an image
refresh, pull the latest version of the global config from the
CephContext and apply it to the config stored within the ImageCtx.

Fixes: https://tracker.ceph.com/issues/48035
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/config_proxy: added 'get_config_values' method

Expose the ability to retrieve the ConfigValues from an existing
ConfigProxy object. When combined with the newly exposed
'set_config_values' method, it will allow the safe bulk-updating
of configuration options. This also removes the original private
static version of 'get_config_values'.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37886 from tchaikov/wip-osd-scheduler-move

osd/scheduler: rely on copy ellision to move return val

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #34866 from nickjanus/issue-45193

rgw-admin: fixes BucketInfo for missing buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37657 from batrick/rgw-lua-fixes

rgw: lua refactor

rbd/tool: add image-cache invalidate

Add an rbd command image-cache invalidate to discard RWL cache.

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>