crimson/onode-staged-tree: implement a ghobject_t constructor

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: remove unused logic in stages/node_stage_layout.h

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: rename NodeExtentT to NodeExtentAccessorT

Also renamed file name and added class comments.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: remove state_t in NodeExtentT

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: use Seastar API v6

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: rename ns_oid_view_t::test_append()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement string_view_masked_t

Use string_view_masked_t to consolidate a string_key_view_t with type, a
string/string_view, or a masked string. And leverage the consistant
class to do compare, print, convert and append operations.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement DeltaRecorder

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: integrate TMTestState

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add option IS_DUMMY_SYNC to tests

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix non-debug build warning

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement benchmark tool

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: get statistics from tree

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add assertions to split test

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: proper logging and dump

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: sizing related fixes and cleanup

* assert filled size matches during dump;
* assert estimate_split_size matches the left node size after split;
* related fixes to pass the added assertions;
* use node_offset_t where possible;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: test split at end

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add asynchronous dummy backend

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: disable cursor validation in track_split()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: random insert/split test

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: imlement fair locate split strategy

Also make adjustments in unit tests.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix extra_size during locate_split()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup, introduce explicit INDEX_LAST

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: clarify the node sizing considerations

Discuss relationships between NODE_BLOCK_SIZE, ns-oid string size limit,
onode size limit, the strategy of locate_split(), and the effect of
"double split".

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup recursively_locate_split()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix insert_stage calculation

Fix incorrect insert_stage calculation when insert into the front of
leaf node.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add more validations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix trim related issue when insert front

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix target split size calculation

Also adjust the test layouts because the split point may change.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: encapsulate TestTree for leaf split test

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: clean up insert/split logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix errors about misaligned address access

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: misc cleanups in UT

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix incorrect string cstr

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: integrate seastore read/write_node_root()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: implement read/write_onode_root()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: rename Cache::get_root()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement Btree::Cursor::get_ghobj()

Internally, get key_view_t when lookup into the leaf node.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup stage.h

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup the end tree_cursor_t cstr

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: integrate ghobject_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: version based p_value invalidation in tree_cursor_t;

When the node content is modified, deprecate the out-dated p_value cache
in tree_cursor_t by a node version increment.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: decouple Node and Layout classes

* Implement has-a relationship between Node and NodeImpl classes;
* Implement NodeLayoutT for templated Layout implementations;
* Reorganize cross-node logic into Node classes;
* Update tests with the new dependencies;
* MISC related cleanup;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add initial onode staged-fltree

Support basic insert and lookup operations, tested by an isolated dummy
backend, and also a solution to integrate with seastore
TransactionManager and LogicalCachedExtent.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38213 from petrutlucian94/rbd_exit

rbd: minor rbd-wnbd improvements

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38254 from neha-ojha/wip-48299

doc/releases/nautilus.rst: add release notes for 14.2.15

Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38144 from anthonyeleven/anthonyeleven/docs-46365

doc/dev/developer_guide: prefer Latinate verbs to English phrasal verbs

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38260 from bk201/wip-48338

mgr/dashboard: fix tox py3 environment error

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38253 from cbodley/wip-rgw-gc-disable-defer

rgw: temporarily disable calls to defer_gc() in RGWGetObj

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

doc/releases/nautilus.rst: add release notes for 14.2.15

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38142 from theanalyst/doc/releases/v14.2.14

doc: release notes for octopus & nautilus backports

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #38247 from dillaman/wip-librbd-test-rename

test/librbd: rename tests to match proper namespaces

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc: releases: v15.2.6 Octopus release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
Co-authored-by: Ilya Dryomov <idryomov@gmail.com>

doc: releases: v14.2.14 Nautilus release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
Co-authored-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #38261 from tchaikov/wip-45919

pybind/mgr/dashboard/run-backend-api-tests: print more log from mgr

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #38217 from ideepika/wip-fix-lua-failure

cmake: require Lua 5.3 and up

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38204 from lnsyyj/wip-iouring

blk/kernel/io_uring: do not guard liburing backend with __x86_64__ anymore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38256 from badone/wip-dont-build-io_uring.c-every-time

cmake: Stop 'configure' running every time

Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard/run-backend-api-tests: print more log from mgr

in hope to more context when debugging https://tracker.ceph.com/issues/45919

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix tox py3 environment error

Add `ceph_argparse` module to the base environment.

Fixes: https://tracker.ceph.com/issues/48338
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

cmake: Stop 'configure' running every time

Currently the configure script is run on 'liburing_ext' not only for the
initial build but for subsequent incremental builds.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38243 from idryomov/wip-relax-preauth-asserts-again

msg/async/ProtocolV2: allow rxbuf/txbuf get bigger in testing, again

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38252 from idryomov/wip-resurrect-log-compression

qa/tasks/ceph: resurrect log compression

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph: resurrect log compression

Commit 953662555840 ("qa/tasks/ceph: use Cluster.sh() and Remote.sh()
when appropriate") dropped run.wait(), which waits for all given
processes to exit.  This resulted in errors like

  INFO:teuthology.orchestra.run.smithi107.stderr:tar: ./objectstore_tool..log: file changed as we read it
  INFO:teuthology.orchestra.run.smithi107.stderr:tar: ./ceph-client.admin.175125.log: File removed before we read it

as the task moved on to archiving semi-corrupted and uncompressed logs,
filling up the lab cluster.

Revert that hunk, as Cluster.sh() is useless here -- we don't need
stdout or stderr, but very much need parallel execution and wait for
the compression to finish.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rgw: temporarily disable calls to defer_gc() in RGWGetObj

cls_rgw_gc_queue_update_entry() is known to cause data loss when called
on objects that have not actually been scheduled for garbage collection

RGWGetObj is the only caller, and uses defer_gc() when reads are taking
a long time compared to rgw_gc_obj_min_wait. if an object has since been
deleted and submitted for garbage collection, this allows RGWGetObj to
defer that gc until the entire read completes

by disabling these calls to defer_gc(), very long reads (longer than 1hr,
with default configuration) may fail if the object gets deleted, and a
retry will result in a 404 Not Found error as expected

Fixes: https://tracker.ceph.com/issues/47866
Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/dev/developer_guide: prefer Latinate verbs to English phrasal verbs

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>
Fixes: https://tracker.ceph.com/issues/46365

test/librbd: rename tests to match proper namespaces

Some of the new crypto tests use a generic test name that
doesn't include the full namespace of the class under test.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37875 from orozery/librbd-luks-format

librbd: add LUKS support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38201 from lixiaoy1/minor_fix_parent_cache

librbd/plugin: delete field m_image_ctx in ParentCache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: add LUKS support

This commit introduces internal (not yet part of the api) librbd functions for:
1. formating an RBD image in LUKS format
2. parsing an RBD image in LUKS format

 The actual implementation of the LUKS format is done via libcryptsetup, which is added as a new dependency.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38089 from anthonyeleven/anthonyeleven/docs-47976

doc/mgr: Fix msgr2 TCP port in a couple of examples

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38180 from anthonyeleven/anthonyeleven/docs-23442

doc/rados/operations: Add description of new auth profiles

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38229 from wjwithagen/wjw-fix-librbd-HttpStream.cc

librbd: Fix variable declaration in capture list in lambda

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/async/ProtocolV2: allow rxbuf/txbuf get bigger in testing, again

With CEPHX_V2 authorizer challenges brought back in commit
4a82c72e3bdd, these need to be bumped again, as two authorizers
(without and then with the challenge) are transmitted and signed
instead of one (without the challenge).  See commit 94953dd9398a
("msg/async/ProtocolV2: allow rxbuf/txbuf get bigger in testing")
for details.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #38207 into master

* refs/pull/38207/head:
ceph-volume: fix test_setup_device_device_name_is_none

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #38092 from anthonyeleven/anthonyeleven/docs-mixed-device-trick

doc/rados/operations: enhance primary affinity section

Reviewed-by: Zac Dover <zac.dover@gmail.com>

cmake: require Lua 5.3 and up

fixes: https://tracker.ceph.com/issues/48312
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #38181 from anthonyeleven/anthonyeleven/docs-23443

doc: object -> file -> disk is wrong for bluestore

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #34506 from ktdreyer/doc-rm-sysv-init

doc: remove SysV init reference in manual deployment walkthrough

Reviewed-by: Zac Dover <zac.dover@gmail.com>

rbd: add "help" command to rbd-wnbd

At the moment, "rbd-wnbd" says that "help" is an unsupported command,
only "rbd-wnbd -h" works. This isn't very user friendly.

Although "rbd-wnbd" isn't supposed to be used directly in most cases,
users might want to retrieve device specific arguments that can be
passed using "rbd device -o".

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #38228 from ivancich/wip-rgw-gc-defer-should-not-enqueue

rgw: during GC defer, prevent new GC enqueue

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #38231 from anthonyeleven/anthonyeleven/docs-40579

doc/rados/operations: POOL_NEAR_FULL on OSD_NEAR_FULL

Reviewed-by: Kefu Chai <kchai@redhat.com>

blk/kernel/io_uring: do not guard liburing backend with __x86_64__ anymore

Signed-off-by: JiangYu <lnsyyj@hotmail.com>
Let the liburing library to ensure the support of the iouring system call back-end CPU instruction set.

doc/rados/operations: POOL_NEAR_FULL on OSD_NEAR_FULL

Fixes: https://tracker.ceph.com/issues/40579
Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados/operations: enhance primary affinity section

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #38090 from anthonyeleven/anthonyeleven/docs-47339

doc/releases: correct spelling

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37900 from jhonxue/master

os/bluestore/StupidAllocator: choose the right bin when bdev_block_size changed

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38161 from petrutlucian94/msg_timeout

msg: use timeout when initiating connection

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38070 from dvanders/dvanders_trimto

mon/OSDMonitor, mon/PaxosService: additional debug outputs when trimming

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38148 from jhonxue/wip-fix-bluestore-avl-allocator

os/bluestore: do not call _block_picker() again if already searched from start()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38128 from tchaikov/wip-mon-add-space-in-log

mon/OSDMonitor: add a space after function name in logging message

Reviewed-by: Joao Eduardo Luis <joao@suse.com>

doc/releases: correct spelling

Fixes: https://tracker.ceph.com/issues/47339
Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #38227 from tchaikov/wip-store-test

test/store_test: pass raw Formatter pointer as expected

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

librbd: Fix variable declaration in capture list in lambda

Fixes: https://github.com/ceph/ceph/pull/38000
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rgw: during GC defer, prevent new GC enqueue

With the new queue-based GC code, when a GC defer operation is
performed, it adds an "urgent" record to prevent GC from removing
objects that are still being read. It does not check whether the
objects are on the GC queue or not and that's OK for the urgent
record.

The code *also* adds a new GC entry to the queue to cause GC to occur
at a later time. This would be incorrect if there was no GC entry to
begin with, however. In such a case this would cause GC to delete tail
objects when no user-initiated remove has happend. In other words a
READ could cause a DELETE of tail objects and therefore data loss.

This fix prevents such a new GC entry from being enqueued, thus
preventing the data loss in this rare case. There is a new risk that
tail object orphans to be created, but as an immediate fix to prevent
data loss, this is appropriate and it is a rare event. A follow-on PR
that will handle these cases is likely.

This PR adds a level 0 log entry as a way to potentially confirm this
case is being triggered in real-world cases. In time, this log entry
should be deleted.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

test/store_test: pass raw Formatter pointer as expected

this addresses the FTBFS regression introduced by
f153a7031e56534723c0fc891869902c58acb473

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38196 from dillaman/wip-48296

pybind/mgr/rbd_support: delay creation of progress module events

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38223 from Rethan/fix_formatter_pointer

test: free Formatter pointers

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/: use unique_ptr for some formatter whose pointers were not released.

some code under test use formatter to output, but Formatter *f
was not freed like elsewere. So use unique_ptr to avoid this.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

Merge pull request #38220 from neha-ojha/wip-fix-label-2

.github/labeler.yml: fix pybind and common rules

Reviewed-by: Josh Durgin <jdurgin@redhat.com>