Merge pull request #38247 from dillaman/wip-librbd-test-rename

test/librbd: rename tests to match proper namespaces

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38261 from tchaikov/wip-45919

pybind/mgr/dashboard/run-backend-api-tests: print more log from mgr

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #38217 from ideepika/wip-fix-lua-failure

cmake: require Lua 5.3 and up

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38204 from lnsyyj/wip-iouring

blk/kernel/io_uring: do not guard liburing backend with __x86_64__ anymore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38256 from badone/wip-dont-build-io_uring.c-every-time

cmake: Stop 'configure' running every time

Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard/run-backend-api-tests: print more log from mgr

in hope to more context when debugging https://tracker.ceph.com/issues/45919

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: Stop 'configure' running every time

Currently the configure script is run on 'liburing_ext' not only for the
initial build but for subsequent incremental builds.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38243 from idryomov/wip-relax-preauth-asserts-again

msg/async/ProtocolV2: allow rxbuf/txbuf get bigger in testing, again

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38252 from idryomov/wip-resurrect-log-compression

qa/tasks/ceph: resurrect log compression

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph: resurrect log compression

Commit 953662555840 ("qa/tasks/ceph: use Cluster.sh() and Remote.sh()
when appropriate") dropped run.wait(), which waits for all given
processes to exit.  This resulted in errors like

  INFO:teuthology.orchestra.run.smithi107.stderr:tar: ./objectstore_tool..log: file changed as we read it
  INFO:teuthology.orchestra.run.smithi107.stderr:tar: ./ceph-client.admin.175125.log: File removed before we read it

as the task moved on to archiving semi-corrupted and uncompressed logs,
filling up the lab cluster.

Revert that hunk, as Cluster.sh() is useless here -- we don't need
stdout or stderr, but very much need parallel execution and wait for
the compression to finish.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: rename tests to match proper namespaces

Some of the new crypto tests use a generic test name that
doesn't include the full namespace of the class under test.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37875 from orozery/librbd-luks-format

librbd: add LUKS support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38201 from lixiaoy1/minor_fix_parent_cache

librbd/plugin: delete field m_image_ctx in ParentCache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: add LUKS support

This commit introduces internal (not yet part of the api) librbd functions for:
1. formating an RBD image in LUKS format
2. parsing an RBD image in LUKS format

 The actual implementation of the LUKS format is done via libcryptsetup, which is added as a new dependency.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38089 from anthonyeleven/anthonyeleven/docs-47976

doc/mgr: Fix msgr2 TCP port in a couple of examples

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38180 from anthonyeleven/anthonyeleven/docs-23442

doc/rados/operations: Add description of new auth profiles

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38229 from wjwithagen/wjw-fix-librbd-HttpStream.cc

librbd: Fix variable declaration in capture list in lambda

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/async/ProtocolV2: allow rxbuf/txbuf get bigger in testing, again

With CEPHX_V2 authorizer challenges brought back in commit
4a82c72e3bdd, these need to be bumped again, as two authorizers
(without and then with the challenge) are transmitted and signed
instead of one (without the challenge).  See commit 94953dd9398a
("msg/async/ProtocolV2: allow rxbuf/txbuf get bigger in testing")
for details.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #38207 into master

* refs/pull/38207/head:
ceph-volume: fix test_setup_device_device_name_is_none

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #38092 from anthonyeleven/anthonyeleven/docs-mixed-device-trick

doc/rados/operations: enhance primary affinity section

Reviewed-by: Zac Dover <zac.dover@gmail.com>

cmake: require Lua 5.3 and up

fixes: https://tracker.ceph.com/issues/48312
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #38181 from anthonyeleven/anthonyeleven/docs-23443

doc: object -> file -> disk is wrong for bluestore

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #34506 from ktdreyer/doc-rm-sysv-init

doc: remove SysV init reference in manual deployment walkthrough

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38228 from ivancich/wip-rgw-gc-defer-should-not-enqueue

rgw: during GC defer, prevent new GC enqueue

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #38231 from anthonyeleven/anthonyeleven/docs-40579

doc/rados/operations: POOL_NEAR_FULL on OSD_NEAR_FULL

Reviewed-by: Kefu Chai <kchai@redhat.com>

blk/kernel/io_uring: do not guard liburing backend with __x86_64__ anymore

Signed-off-by: JiangYu <lnsyyj@hotmail.com>
Let the liburing library to ensure the support of the iouring system call back-end CPU instruction set.

doc/rados/operations: POOL_NEAR_FULL on OSD_NEAR_FULL

Fixes: https://tracker.ceph.com/issues/40579
Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados/operations: enhance primary affinity section

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #38090 from anthonyeleven/anthonyeleven/docs-47339

doc/releases: correct spelling

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37900 from jhonxue/master

os/bluestore/StupidAllocator: choose the right bin when bdev_block_size changed

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38161 from petrutlucian94/msg_timeout

msg: use timeout when initiating connection

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38070 from dvanders/dvanders_trimto

mon/OSDMonitor, mon/PaxosService: additional debug outputs when trimming

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38148 from jhonxue/wip-fix-bluestore-avl-allocator

os/bluestore: do not call _block_picker() again if already searched from start()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38128 from tchaikov/wip-mon-add-space-in-log

mon/OSDMonitor: add a space after function name in logging message

Reviewed-by: Joao Eduardo Luis <joao@suse.com>

doc/releases: correct spelling

Fixes: https://tracker.ceph.com/issues/47339
Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #38227 from tchaikov/wip-store-test

test/store_test: pass raw Formatter pointer as expected

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

librbd: Fix variable declaration in capture list in lambda

Fixes: https://github.com/ceph/ceph/pull/38000
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rgw: during GC defer, prevent new GC enqueue

With the new queue-based GC code, when a GC defer operation is
performed, it adds an "urgent" record to prevent GC from removing
objects that are still being read. It does not check whether the
objects are on the GC queue or not and that's OK for the urgent
record.

The code *also* adds a new GC entry to the queue to cause GC to occur
at a later time. This would be incorrect if there was no GC entry to
begin with, however. In such a case this would cause GC to delete tail
objects when no user-initiated remove has happend. In other words a
READ could cause a DELETE of tail objects and therefore data loss.

This fix prevents such a new GC entry from being enqueued, thus
preventing the data loss in this rare case. There is a new risk that
tail object orphans to be created, but as an immediate fix to prevent
data loss, this is appropriate and it is a rare event. A follow-on PR
that will handle these cases is likely.

This PR adds a level 0 log entry as a way to potentially confirm this
case is being triggered in real-world cases. In time, this log entry
should be deleted.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

test/store_test: pass raw Formatter pointer as expected

this addresses the FTBFS regression introduced by
f153a7031e56534723c0fc891869902c58acb473

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38196 from dillaman/wip-48296

pybind/mgr/rbd_support: delay creation of progress module events

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38223 from Rethan/fix_formatter_pointer

test: free Formatter pointers

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/: use unique_ptr for some formatter whose pointers were not released.

some code under test use formatter to output, but Formatter *f
was not freed like elsewere. So use unique_ptr to avoid this.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

Merge pull request #38220 from neha-ojha/wip-fix-label-2

.github/labeler.yml: fix pybind and common rules

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

.github/labeler.yml: fix pybind and common rules

Noticed it didn't work in https://github.com/ceph/ceph/pull/38206

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38193 from dang/wip-dang-zipper-9

RGW - Zipper 9: Write path

Zipper - ObjectProcessor

Fix the ObjectProcessor classes to use Zipper APIs

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #37734 from matthewoliver/rgw_swift_dlo_fix

rgw: Check all shards for user manifest parts

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #38062 from s0nea/wip-dashboard-new-password

mgr/dashboard: amend the look & feel of the login password form

Reviewed-by: nizamial09 <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37740 from s0nea/wip-dashboard-ceph-input-validation

mgr/dashboard: validate username while creation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37719 from tspmelo/wip-rbd-invalid-name

mgr/dashboard: RBD: Disable actions when name is not valid

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38000 from dillaman/wip-librbd-migration-3

librbd: http-based migration source

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38214 from rzarzynski/wip-crimson-pg-drop-undefined-declarations

crimson: drop two undefined method declarations from PG

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38216 from rzarzynski/wip-crimson-no-pg-in-opsexecuter

crimson: make OpsExecuter unaware about the giant PG class

Reviewed-by: Kefu Chai <kchai@redhat.com>

Remove copy from ceph::static_ptr

The copy functionality of ceph::static_ptr is unused, and it means that
nothing containing a std::unique_ptr (or any other non-copyable type)
can be put into a static_ptr.

Remove the copy functionality, so that unique_ptr can be used in
static_ptr.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Assorted cleanups

- Move cluste stat into public header, allowing more zipper cleanup
- Swift versioning
- Implement a MPSerializer for Zipper.
- Add Lifecycle APIs to Zipper.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - WriteOp

Add WriteOp to the Zipper API, and use it.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #38122 from lxbsz/rbdmap_init

rbd: drop initd-based systems for rbdmap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37576 from p-se/doc-cephadm-custom-monitoring-templates

mgr/cephadm: document setting custom monitoring templates for configuration files

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #37696 from p-se/cephadm-monitoring-grafana-tls

doc/cepham: instructions for custom Grafana TLS certs

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38210 from rzarzynski/wip-crimson-fix-warning-in-with_locked_obc

crimson: fix a warning in PG::with_lock_obc().

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: OpsExecuter stores only const reference to MOSDOp.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: make OpsExecuter unaware about PG.

`PG` is heavy-weight class with many responsibilities.
Exposing it to lower-layer may suggest there is far more
coupling between them than in reality.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: drop the declared but undefined PG::do_pgnls().

`PgOpsExecuter` takes care; it's not not a responsibility
of `PG` anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: drop the declared but undefined PG::do_osd_op().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: comment on why OpsExecuter needs PG.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38061 from tchaikov/wip-48012

mgr/dashboard: use http://docs.ceph.com/en/${release}/ for the domain…

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

crimson: fix a warning in PG::with_lock_obc().

Without the fix, on a build with `NDEBUG` set,
GCC complains:

```
crimson/osd/pg.cc:869:35: warning: control reaches end of non-void function [-Wreturn-type]
  869 |   const hobject_t oid = get_oid(*m);
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd/plugin: delete field m_image_ctx

librbd::plugin::ParentCache may be shared by more than one images.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #38177 from jhonxue/fix-test-objectstore-allocator

test/objectstore: handle -ENOSPC returned by allocate()

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: fix test_setup_device_device_name_is_none

Let's call this function by using the same syntax than other tests.
This will make it work with py2 in nautilus branch.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

test/objectstore: handle -ENOSPC returned by allocate()

Fixes: https://tracker.ceph.com/issues/48288
Signed-off-by: Xue Yantao <jhonxue@tencent.com>

doc/rados/operations: Add description of new auth profiles

doc/mgr: Fix msgr2 TCP port in a couple of examples

Fixes: https://tracker.ceph.com/issues/47976
Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

doc: object -> file -> disk is wrong for bluestore

Address tracker 23443

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>
doc: object -> file -> disk is wrong for bluestore

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

mgr/dashboard: use http://docs.ceph.com/en/${release}/ for the domain of the docs

* use the URL of RTD, where the locale is encoded in it.
* s/http/https/
* add test accordingly

Fixes: https://tracker.ceph.com/issues/48012
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38179 from tchaikov/wip-github-labeler-manpage

github/labeler: label manpages changes with related component labels

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38192 from rzarzynski/wip-crimson-backfill-improve-debugs

crimson: improve debugs in the backfill code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38191 from rzarzynski/wip-crimson-fix-typo-in-errorator

crimson: fix a typo in errorator-related comment.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38197 from dillaman/wip-github-rbd-label

github/labeler: add rbd label to pybind/mgr/rbd_support

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38153 from tchaikov/wip-cmake-silence-warning

cmake: set advice.detachedHead=false when cloning liburing

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38169 from ktdreyer/cephadm-noarch

rpm: cephadm package is noarch

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

github/labeler: label manpages changes with related component labels

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38183 from tchaikov/wip-doc-peering-diagram

doc: build peering state diagram using sphinx

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: drop initd-based systems for rbdmap

Ceph versions jewel and above only support systemd.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #33750 from petrutlucian94/windows.10

rbd: add Windows RBD support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

github/labeler: add rbd label to pybind/mgr/rbd_support

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #38147 into master

* refs/pull/38147/head:
ceph-volume: cover devices.lvm.prepare.setup_device
ceph-volume: fix filestore/dmcrypt activate

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

pybind/mgr/rbd_support: delay creation of progress event

Create the progress module event upon receipt of the first
progress callback from the librbd API. This will help to ensure
that all prereqs have been validated for retryable errors like
scheduling an image to be removed while it still has attached
cloned children.

Fixes: https://tracker.ceph.com/issues/48296
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/mgr/rbd_support: include retryable error message in task

If a retryable error occurs, include the reason message in the
task JSON structure that is exposed via the "rbd task list"
MGR command.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/mgr/rbd_support: add linear backoff to retried tasks

When a task fails but will be retried, linearly increase the
delay for the next retry up to a maximum of 5 minutes.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

crimson: fix a debug in BackfillState::Enqueuing::Enqueuing.

Printed messages were messed up due to the erroneous comma.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: add a debug in update_on_peers() of the backfill code.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: fix a typo in errorator-related comment.

It was introduced in dca575992b34611d20f50c5db47e4b533cd7dc69.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37586 from cbodley/wip-rgw-bilog-layout

rgw: add bucket index log layout structure

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #37914 from IlsooByun/rgw_invalid_resp

rgw: fix invalid payload issue when serving s3website error page

Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #38182 from orozery/librbd-crypto-image-dispatch-fixes

librbd: fix crypto prepare_copyup to use new remap_extents api

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume: cover devices.lvm.prepare.setup_device

Add some unit tests to cover setup_device() in devices.lvm.prepare

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: fix filestore/dmcrypt activate

The uuid set for tags['ceph.journal_uuid'] should point to its
corresponding lv_uuid instead of the uuid generated for the lv_name.

The variable name 'uuid' used so far was probably too confusing so let's
change it to make it more clear.

Closes: https://tracker.ceph.com/issues/48271
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

doc/script/gen_state_diagram: wrap lines which is over 80 chars

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/script/gen_state_diagram: use itertools.cycle() for cycling in colors

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>