test/store_test: pass raw Formatter pointer as expected

this addresses the FTBFS regression introduced by
f153a7031e56534723c0fc891869902c58acb473

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38196 from dillaman/wip-48296

pybind/mgr/rbd_support: delay creation of progress module events

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38223 from Rethan/fix_formatter_pointer

test: free Formatter pointers

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/: use unique_ptr for some formatter whose pointers were not released.

some code under test use formatter to output, but Formatter *f
was not freed like elsewere. So use unique_ptr to avoid this.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

Merge pull request #38220 from neha-ojha/wip-fix-label-2

.github/labeler.yml: fix pybind and common rules

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

.github/labeler.yml: fix pybind and common rules

Noticed it didn't work in https://github.com/ceph/ceph/pull/38206

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38193 from dang/wip-dang-zipper-9

RGW - Zipper 9: Write path

Zipper - ObjectProcessor

Fix the ObjectProcessor classes to use Zipper APIs

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #37734 from matthewoliver/rgw_swift_dlo_fix

rgw: Check all shards for user manifest parts

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #38062 from s0nea/wip-dashboard-new-password

mgr/dashboard: amend the look & feel of the login password form

Reviewed-by: nizamial09 <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37740 from s0nea/wip-dashboard-ceph-input-validation

mgr/dashboard: validate username while creation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37719 from tspmelo/wip-rbd-invalid-name

mgr/dashboard: RBD: Disable actions when name is not valid

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38000 from dillaman/wip-librbd-migration-3

librbd: http-based migration source

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38214 from rzarzynski/wip-crimson-pg-drop-undefined-declarations

crimson: drop two undefined method declarations from PG

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38216 from rzarzynski/wip-crimson-no-pg-in-opsexecuter

crimson: make OpsExecuter unaware about the giant PG class

Reviewed-by: Kefu Chai <kchai@redhat.com>

Remove copy from ceph::static_ptr

The copy functionality of ceph::static_ptr is unused, and it means that
nothing containing a std::unique_ptr (or any other non-copyable type)
can be put into a static_ptr.

Remove the copy functionality, so that unique_ptr can be used in
static_ptr.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Assorted cleanups

- Move cluste stat into public header, allowing more zipper cleanup
- Swift versioning
- Implement a MPSerializer for Zipper.
- Add Lifecycle APIs to Zipper.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - WriteOp

Add WriteOp to the Zipper API, and use it.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #38122 from lxbsz/rbdmap_init

rbd: drop initd-based systems for rbdmap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37576 from p-se/doc-cephadm-custom-monitoring-templates

mgr/cephadm: document setting custom monitoring templates for configuration files

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #37696 from p-se/cephadm-monitoring-grafana-tls

doc/cepham: instructions for custom Grafana TLS certs

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38210 from rzarzynski/wip-crimson-fix-warning-in-with_locked_obc

crimson: fix a warning in PG::with_lock_obc().

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: OpsExecuter stores only const reference to MOSDOp.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: make OpsExecuter unaware about PG.

`PG` is heavy-weight class with many responsibilities.
Exposing it to lower-layer may suggest there is far more
coupling between them than in reality.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: drop the declared but undefined PG::do_pgnls().

`PgOpsExecuter` takes care; it's not not a responsibility
of `PG` anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: drop the declared but undefined PG::do_osd_op().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: comment on why OpsExecuter needs PG.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38061 from tchaikov/wip-48012

mgr/dashboard: use http://docs.ceph.com/en/${release}/ for the domain…

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

crimson: fix a warning in PG::with_lock_obc().

Without the fix, on a build with `NDEBUG` set,
GCC complains:

```
crimson/osd/pg.cc:869:35: warning: control reaches end of non-void function [-Wreturn-type]
  869 |   const hobject_t oid = get_oid(*m);
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38177 from jhonxue/fix-test-objectstore-allocator

test/objectstore: handle -ENOSPC returned by allocate()

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/objectstore: handle -ENOSPC returned by allocate()

Fixes: https://tracker.ceph.com/issues/48288
Signed-off-by: Xue Yantao <jhonxue@tencent.com>

mgr/dashboard: use http://docs.ceph.com/en/${release}/ for the domain of the docs

* use the URL of RTD, where the locale is encoded in it.
* s/http/https/
* add test accordingly

Fixes: https://tracker.ceph.com/issues/48012
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38179 from tchaikov/wip-github-labeler-manpage

github/labeler: label manpages changes with related component labels

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38192 from rzarzynski/wip-crimson-backfill-improve-debugs

crimson: improve debugs in the backfill code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38191 from rzarzynski/wip-crimson-fix-typo-in-errorator

crimson: fix a typo in errorator-related comment.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38197 from dillaman/wip-github-rbd-label

github/labeler: add rbd label to pybind/mgr/rbd_support

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38153 from tchaikov/wip-cmake-silence-warning

cmake: set advice.detachedHead=false when cloning liburing

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38169 from ktdreyer/cephadm-noarch

rpm: cephadm package is noarch

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

github/labeler: label manpages changes with related component labels

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38183 from tchaikov/wip-doc-peering-diagram

doc: build peering state diagram using sphinx

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: drop initd-based systems for rbdmap

Ceph versions jewel and above only support systemd.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #33750 from petrutlucian94/windows.10

rbd: add Windows RBD support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

github/labeler: add rbd label to pybind/mgr/rbd_support

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #38147 into master

* refs/pull/38147/head:
ceph-volume: cover devices.lvm.prepare.setup_device
ceph-volume: fix filestore/dmcrypt activate

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

pybind/mgr/rbd_support: delay creation of progress event

Create the progress module event upon receipt of the first
progress callback from the librbd API. This will help to ensure
that all prereqs have been validated for retryable errors like
scheduling an image to be removed while it still has attached
cloned children.

Fixes: https://tracker.ceph.com/issues/48296
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/mgr/rbd_support: include retryable error message in task

If a retryable error occurs, include the reason message in the
task JSON structure that is exposed via the "rbd task list"
MGR command.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/mgr/rbd_support: add linear backoff to retried tasks

When a task fails but will be retried, linearly increase the
delay for the next retry up to a maximum of 5 minutes.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

crimson: fix a debug in BackfillState::Enqueuing::Enqueuing.

Printed messages were messed up due to the erroneous comma.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: add a debug in update_on_peers() of the backfill code.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: fix a typo in errorator-related comment.

It was introduced in dca575992b34611d20f50c5db47e4b533cd7dc69.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37586 from cbodley/wip-rgw-bilog-layout

rgw: add bucket index log layout structure

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #37914 from IlsooByun/rgw_invalid_resp

rgw: fix invalid payload issue when serving s3website error page

Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #38182 from orozery/librbd-crypto-image-dispatch-fixes

librbd: fix crypto prepare_copyup to use new remap_extents api

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume: cover devices.lvm.prepare.setup_device

Add some unit tests to cover setup_device() in devices.lvm.prepare

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: fix filestore/dmcrypt activate

The uuid set for tags['ceph.journal_uuid'] should point to its
corresponding lv_uuid instead of the uuid generated for the lv_name.

The variable name 'uuid' used so far was probably too confusing so let's
change it to make it more clear.

Closes: https://tracker.ceph.com/issues/48271
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

doc/script/gen_state_diagram: wrap lines which is over 80 chars

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/script/gen_state_diagram: use itertools.cycle() for cycling in colors

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38158 from rhcs-dashboard/fix-labeler-master

github/labeler: disable sync-labels

Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: neha-ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: pass graphviz options in conf.py

easier to maintain the options of sphinx extensions in a single place.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: generate peering graph in conf.py

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/script/gen_state_diagram: add "output" parameter to emit_dot()

this allows us to reuse this python script as a module.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/script/gen_state_diagram: only run if called as a script

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38146 from ljishen/peering_state_model

doc: increase visibility of the peering state diagram

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: fix crypto prepare_copyup to use new remap_extents api

This commit changes the crypto prepare_copyup implementation to correctly
translate object extents to image extents, via the remap_extents api.
Additionally, we fix relevant unit tests that call this api to correctly expect these calls.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

doc: increase visibility of the peering state diagram

Signed-off-by: Jianshen Liu <jliu120@ucsc.edu>

Merge pull request #38001 from badone/wip-powertools-repo-name-change

install-deps.sh: Make powertools repo case insensitive

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38172 from zdover23/wip-doc-dev-unit-test-list-update

doc/dev: update list in unit test section

Reviewed-By: Nathan Cutler <ncutler@suse.com>

Merge pull request #38114 from orozery/librbd-crypto-prepare-copyup

librbd: add encryption support for copyup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37935 from orozery/librbd-image-address-remap

librbd: add crypto image dispatch layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38133 from zdover23/wip-doc-dev-unit-tests-1-of-7

doc/dev: update unit test 1 of 7

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38134 from zdover23/wip-doc-dev-unit-tests-2-of-7

doc/dev: update unit tests 2 of 7

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #38126 into master

* refs/pull/38126/head:
doc/cephfs/health-messages: add dot between mds identifier and it's name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38174 from neha-ojha/wip-fix-label

.github/labeler.yml: fix bluestore rule, add label for tests and classify core tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37607 from pcuzner/docs-cephadm-host-maintenance

doc/dev/cephadm: Doc defining the design for host maintenance

librbd: add encryption support for copyup

This commit implements the prepare_copyup api by the crypto object dispatch layer.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

.github/labeler.yml: add label for tests and classify core tests

Signed-off-by: Neha Ojha <nojha@redhat.com>

.github/labeler.yml: fix bluestore rule

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/dev: update list in unit test section

This commit removes an unwanted extra
nested list layer from a list.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38150 from zdover23/wip-doc-dev-unit-tests-3-of-7

doc/dev: update unit test declaration

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #38054 from dzafman/wip-test-fixes

test: Fix osd-scrub-test.sh and ver-health.sh tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

rpm: cephadm package is noarch

The cephadm package contains an architecture-independent Python script,
empty directories, and an empty authorized_keys file. There are no
architecture-dependent files here, so we can use a single noarch RPM
across all host architectures.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #38132 from ceph/rhel_8_distros

qa/distros: add RHEL 8.3 and use RHEL 8.3 as default

Merge PR #34552 into master

* refs/pull/34552/head:
tests: add snap schedule tests
snap-schedule: reject invalid retention specs
mgr/snap_schedule: log snapshot name on creation
pybind / snap_schedule: sanely dump schedules and retention specs
pybind/snap_schedule: config option to allow minute granularity snaps
pybind / snap_schedule: remove 'SNAP_SCHED_TESTING' check

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38107 from ceph/wip-mgr-progress-fix-48217

qa/mgr/test_progress: add _get_osd_in_out_events to account for osd marked in/out events

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/dev: update unit test declaration

This commit updates the text in the "How
Unit Tests Are Declared" section of
tests-unit-tests.rst. This commit breaks
long sentences into shorter sentences and
breaks a sentence into a bulleted list that
might reduce the reader's cognitive load.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

common: add trailing newline before parsing cfg

Ceph fails to parse config files that lack a trailing newline.
This mainly affects Windows, where text editors won't add one by
default.

This issue has been addressed by a previous commit [1] but that
code path is no longer reached [2][3], so we need to go down a level.

[1] 3b590314ac49b955b2123f03f4c4417e5534ec98
[2] http://paste.openstack.org/raw/800153/
[3] https://github.com/ceph/ceph/commit/e7dcc403b20922ffeeca114899f90a385bf25f0e

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #38129 from jan--f/add-ceph-volume-autolabel

github: add ceph-volume autolabel

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

github/labeler: disable sync-labels

Yaml syntax cleaned too.

Fixes: https://github.com/ceph/ceph/pull/38107#issuecomment-729300615
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

github: add ceph-volume autolabel

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37606 from ifed01/wip-ifed-fix-mempool-and-others

os/bluestore: a bunch of minor fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38152 from tchaikov/wip-crimson-clang-cleanup

crimson/os: do not capture unused variables

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

compat,msg: improve Windows socket checks

win_socketpair can fail with EADDRINUSE under load, which will
lead to an unhandled exception/crash as per this commit [1].

This change adds a retry, also ensuring that the right error code
gets propagated (the one returned by WSAGetLastError() instead of
the generic SOCKET_ERROR).

While at it, we're fixing the "win_socketpair" indentation and
addressing the SOCKET to int casts.

[1] https://github.com/ceph/ceph/commit/633805060a1002c86570fe50d099e0c1e223e2d7

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

rbd: Use the right device type on Windows

"wnbd" will be the only supported device type on Windows, for now.
We'll update the "rbd" wrapper accordingly.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

rbd: allow mounting images on Windows

This change will allow mapping rbd images on Windows, leveraging the
WNBD[1] Virtual Storport Miniport driver [2].

The behavior and CLI is similar to the Linux rbd-nbd, with a few
notable differences:

* device paths cannot be requested. The disk number and path will
  be picked by Windows. If a device path is provided by the user
  when mapping an image, it will be used as an identifier, which
  can also be used when unmapping the image.
* the "show" command was added, which describes a specific mapping.
  This can be used for retrieving the disk path.
* the "service" command was added, allowing rbd-wnbd to run as a
  Windows service. All mappings are currently perisistent, being
  recreated when the service stops, unless explicitly unmapped.
  The service disconnects the mappings when being stopped.
* the "list" command also includes a "status" column.

The purpose of the "service" mode is to ensure that mappings survive
reboots and that the Windows service start order can be adjusted so
that rbd images can be mapped before starting services that may depend
on it, such as VMMS.

The mapped images can either be consumed by the host directly or exposed
to Hyper-V VMs.

While at it, we'll skip building rbd-mirror as it's quite unlikely that
this daemon is going to be used on Windows for now.

[1] https://github.com/cloudbase/wnbd
[2] https://docs.microsoft.com/en-us/windows-hardware/drivers/storage/overview-of-storage-virtual-miniport-drivers

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Signed-off-by: Alin Gabriel Serdean <aserdean@cloudbasesolutions.com>

rbd: add missing FileStream::get_size

At the moment, a few methods will be stubbed if
BOOST_ASIO_HAS_POSIX_STREAM_DESCRIPTOR is undefined.

The issue is that FileStream::get_size is undefined, so we're
getting link issues on Windows.

In the future, we might consider using asio::windows::stream_handle
on Windows.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

build: disable stack protection on Windows

Passing "-fstack-protector-strong" doesn't seem to work with Mingw,
complaining about undefied "__stack_chk_fail". For this reason,
we'll disable it for now.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

rbd: Update Windows readme

This change updates the Windows readme, describing rbd-wnbd
configuration and usage.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

rbd: fix import image path parsing on Windows

When importing an image, the rbd command uses only the file name
and expects "/" to be used as a separator. On Windows, it will
use the entire path as image name since the path separator is not
the same.

This change updates it so that the "\\" path separator can be
properly handled as well.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

common: move Windows files to a separate folder

We ended up with quite a few files having the "_win32" suffix.
It's probably better if we move them to a separate folder and drop
the suffix.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #38022 from tchaikov/wip-crimson-with-lock

crimson/os: use with_lock() to guard obc

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>