]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
2 years agodoc/glossary: add "Scrubbing" 50701/head
Zac Dover [Mon, 27 Mar 2023 20:48:32 +0000 (06:48 +1000)]
doc/glossary: add "Scrubbing"

Add "Scrubbing" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 4a66819da48cee430ca74830e1a6f63685b6b240)

2 years agoMerge pull request #50484 from cbodley/wip-58932-reef
Yuri Weinstein [Mon, 27 Mar 2023 19:51:00 +0000 (12:51 -0700)]
Merge pull request #50484 from cbodley/wip-58932-reef

reef: rgw/admin: 'bucket stats' displays non-empty time

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2 years agoMerge pull request #50688 from weirdwiz/wip-59175-reef
Ilya Dryomov [Mon, 27 Mar 2023 16:41:54 +0000 (18:41 +0200)]
Merge pull request #50688 from weirdwiz/wip-59175-reef

reef: rbd-mirror: switch to labeled perf counters

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2 years agorbd-mirror: add and rename perf counters for journal and snapshot mirroring 50688/head
Divyansh Kamboj [Fri, 17 Mar 2023 09:19:14 +0000 (14:49 +0530)]
rbd-mirror: add and rename perf counters for journal and snapshot mirroring

This commit renames the existing snapshot and journal based performance
counters in the rbd-mirror daemon to better reflect their purpose.
Additionally, new perf counters have been added to provide more detailed
information about the synchronization of snapshots between the source
and destination clusters.

This commit also switches to use labels instead of having the image spec
in the name of the counter.

Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>
(cherry picked from commit 1a1477b9fd7f3fcf7eb66c8a768476bea1322d1d)

2 years agoMerge pull request #50677 from zdover23/wip-doc-2023-03-27-backport-50675-to-reef
Anthony D'Atri [Sun, 26 Mar 2023 19:33:04 +0000 (15:33 -0400)]
Merge pull request #50677 from zdover23/wip-doc-2023-03-27-backport-50675-to-reef

reef: doc/rados: clean up ops/bluestore-migration.rst

2 years agodoc/rados: clean up ops/bluestore-migration.rst 50677/head
Zac Dover [Sun, 26 Mar 2023 15:03:58 +0000 (01:03 +1000)]
doc/rados: clean up ops/bluestore-migration.rst

Clean up internal links, fix the numbering of a procedure, and implement
Anthony D'Atri's suggestions in
https://github.com/ceph/ceph/pull/50487 and
https://github.com/ceph/ceph/pull/50488.

https://tracker.ceph.com/issues/58485

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit b28be76d0f9a5d78943fe3c583a234d0b50356f1)

2 years agoMerge pull request #50656 from petrutlucian94/wip-59161-reef
Ilya Dryomov [Sun, 26 Mar 2023 19:00:26 +0000 (21:00 +0200)]
Merge pull request #50656 from petrutlucian94/wip-59161-reef

reef: rbd-wnbd: optionally handle wnbd adapter restart events

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2 years agoMerge pull request #50671 from zdover23/wip-doc-2023-03-26-backport-50654-to-reef
colemitchell [Sat, 25 Mar 2023 19:56:08 +0000 (15:56 -0400)]
Merge pull request #50671 from zdover23/wip-doc-2023-03-26-backport-50654-to-reef

reef: doc/glossary: add "User"

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
2 years agodoc/glossary: add "User" 50671/head
Zac Dover [Fri, 24 Mar 2023 10:56:51 +0000 (20:56 +1000)]
doc/glossary: add "User"

Add "User" to glossary.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit fd6bfaf3fe6337511dd7ab344c2936f22a5d43bf)

2 years agoMerge pull request #50661 from zdover23/wip-doc-2023-03-25-backport-50660-to-reef
Anthony D'Atri [Fri, 24 Mar 2023 18:52:38 +0000 (14:52 -0400)]
Merge pull request #50661 from zdover23/wip-doc-2023-03-25-backport-50660-to-reef

reef: doc/operations: fix prompt in bluestore-migration

2 years agodoc/operations: fix prompt in bluestore-migration 50661/head
Zac Dover [Fri, 24 Mar 2023 17:47:10 +0000 (03:47 +1000)]
doc/operations: fix prompt in bluestore-migration

Fix a single prompt in bluestore-migration.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 5e54641aec134188c8001e212ea66c1a15d5945c)

2 years agoMerge pull request #50629 from mkogan1/wip-59126-reef
Yuri Weinstein [Fri, 24 Mar 2023 16:40:43 +0000 (09:40 -0700)]
Merge pull request #50629 from mkogan1/wip-59126-reef

reef: rgw : fix python script using s3cmd with error code 403

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2 years agoMerge pull request #50603 from mkogan1/wip-59004-reef
Yuri Weinstein [Fri, 24 Mar 2023 16:40:13 +0000 (09:40 -0700)]
Merge pull request #50603 from mkogan1/wip-59004-reef

reef: qa: d3n: fix test failure when "inline_data": false

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2 years agoMerge pull request #50577 from cbodley/wip-qa-reef-ubuntu22
Yuri Weinstein [Fri, 24 Mar 2023 16:39:40 +0000 (09:39 -0700)]
Merge pull request #50577 from cbodley/wip-qa-reef-ubuntu22

reef: qa/distros: add ubuntu 22 as supported distro

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
2 years agoMerge pull request #50509 from ivancich/wip-59055-reef
Yuri Weinstein [Fri, 24 Mar 2023 16:38:30 +0000 (09:38 -0700)]
Merge pull request #50509 from ivancich/wip-59055-reef

reef: rgw: experimental support for restoring a lost bucket index

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
2 years agorbd-wnbd: consistently use negative error codes in rbd-wnbd 50656/head
Lucian Petrut [Thu, 23 Mar 2023 08:24:01 +0000 (08:24 +0000)]
rbd-wnbd: consistently use negative error codes in rbd-wnbd

The rbd-wnbd iterators return positive errors, which is why
in certain cases we may end up with both positive and negative
error codes.

This change ensures that we'll consistently use negative
error codes.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit 98a7aff741eb95bcc98df625d6558d726711a5fb)

2 years agocommon, rbd-wnbd: bump Windows log level
Lucian Petrut [Wed, 7 Dec 2022 13:18:48 +0000 (15:18 +0200)]
common, rbd-wnbd: bump Windows log level

We're increasing the log level for certain Windows operational log
messages.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit 3d8afc00215146fbc0226c2c6ccc7cb5dd3a2bb5)

2 years agorbd-wnbd: optionally handle wnbd adapter restart events
Lucian Petrut [Wed, 16 Nov 2022 11:24:34 +0000 (13:24 +0200)]
rbd-wnbd: optionally handle wnbd adapter restart events

The WNBD adapter may be reset in certain situations (e.g. driver
upgrade, MS WHQL tests, etc).

We're going to monitor the WNBD adapter using WMI[1] events, restarting
the rbd-wnbd disk mappings whenever necessary. Adapter monitoring can be
enabled by passing the --adapter-monitoring-enabled flag to the service.

This feature is optional for the following reasons:

* it's mainly used during development / driver certification
* we had to use a relatively small polling interval, which might imply
  additional resource usage. WMI quotas also have to be considered.

While at it, we're updating two lambdas that are submitted to thread pools,
avoiding default reference capturing and explicitly specifying the variables
that get copied.

[1] https://learn.microsoft.com/en-us/windows/win32/wmisdk/wmi-start-page

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit 0c25ca656430b4d5280ab7d033495ad8ce4e776c)

2 years agocommon: add win32/wstring.h
Lucian Petrut [Wed, 16 Nov 2022 08:50:14 +0000 (10:50 +0200)]
common: add win32/wstring.h

Windows APIs heavily use wchar. ceph-dokan and rbd-wnbd
have some duplicated helpers that convert wstrings to/from
utf8 strings.

To avoid duplication and allow reusing those helpers, we're moving
them to common/win32/wstring.h.

We're using the "win32" subfolder because it's unlikely that this
will ever be used on other platforms.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit a47caa156a4e667d6b9ff7236cc9148ba37ea772)

2 years agoMerge pull request #50640 from zdover23/wip-doc-2023-03-23-backport-50634-to-reef
Anthony D'Atri [Thu, 23 Mar 2023 13:19:37 +0000 (09:19 -0400)]
Merge pull request #50640 from zdover23/wip-doc-2023-03-23-backport-50634-to-reef

reef: doc/rados: edit user-management.rst (1 of x)

2 years agodoc/rados: edit user-management.rst (1 of x) 50640/head
Zac Dover [Thu, 23 Mar 2023 02:21:06 +0000 (12:21 +1000)]
doc/rados: edit user-management.rst (1 of x)

Edit doc/rados/operations/user-management.rst. (1 of x)

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 3f90958909815a287279eccb22193ba5d67cfc1d)

2 years agorgw : fix python script using s3cmd with error code 403 w/https 50629/head
Mark Kogan [Tue, 21 Mar 2023 16:30:12 +0000 (18:30 +0200)]
rgw : fix python script using s3cmd with error code 403 w/https

Fixes: https://tracker.ceph.com/issues/54104
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit 2cb94411801201b53789b3b9b509abccb63c4b1f)

2 years agorgw : fix python script using s3cmd with error code 403
cao.leilc [Mon, 12 Dec 2022 01:50:20 +0000 (09:50 +0800)]
rgw : fix python script using s3cmd with error code 403

Fixes: https://tracker.ceph.com/issues/54104
Signed-off-by: caolei <cao.leilc@inspur.com>
(cherry picked from commit a10baddb82b6febfb1765ccb3d3c2d007b1f2aee)

2 years agoMerge pull request #50618 from zdover23/wip-doc-2023-03-22-backport-50606-to-reef
Anthony D'Atri [Wed, 22 Mar 2023 03:34:03 +0000 (23:34 -0400)]
Merge pull request #50618 from zdover23/wip-doc-2023-03-22-backport-50606-to-reef

reef: doc/rados: line-edit erasure-code.rst

2 years agodoc/rados: line-edit erasure-code.rst 50618/head
Zac Dover [Tue, 21 Mar 2023 12:27:15 +0000 (22:27 +1000)]
doc/rados: line-edit erasure-code.rst

Line-edit doc/rados/operations/erasure-code.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 2ff9965e02756636f8d33fd96b7c909f2fca7414)

2 years agoMerge pull request #50572 from petrutlucian94/wip-59098-reef
Ilya Dryomov [Tue, 21 Mar 2023 13:19:40 +0000 (14:19 +0100)]
Merge pull request #50572 from petrutlucian94/wip-59098-reef

reef: common: avoid redefining clock type on Windows

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2 years agoreef: qa: d3n: fix test failure when "inline_data": false 50603/head
Mark Kogan [Mon, 6 Feb 2023 12:05:51 +0000 (14:05 +0200)]
reef: qa: d3n: fix test failure when "inline_data": false

backport tracker: https://tracker.ceph.com/issues/59004
backport of https://github.com/ceph/ceph/pull/50003
parent tracker: https://tracker.ceph.com/issues/58552

Fixes: https://tracker.ceph.com/issues/58552
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit f474c7fdbe937a07451ecef246e43e3f46b7010d)

2 years agoMerge pull request #50589 from zdover23/wip-doc-2023-03-20-backport-50488-to-reef
Anthony D'Atri [Mon, 20 Mar 2023 04:47:29 +0000 (00:47 -0400)]
Merge pull request #50589 from zdover23/wip-doc-2023-03-20-backport-50488-to-reef

reef: doc/rados: edit operations/bs-migration (2 of x)

2 years agoMerge pull request #50586 from zdover23/wip-doc-2023-03-20-backport-50487-to-reef
Anthony D'Atri [Mon, 20 Mar 2023 04:45:18 +0000 (00:45 -0400)]
Merge pull request #50586 from zdover23/wip-doc-2023-03-20-backport-50487-to-reef

reef: doc/rados: edit operations/bs-migration (1 of x)

2 years agodoc/rados: edit operations/bs-migration (2 of x) 50589/head
Zac Dover [Sun, 12 Mar 2023 01:17:03 +0000 (11:17 +1000)]
doc/rados: edit operations/bs-migration (2 of x)

Disambiguate and improve the English language in
doc/rados/operations/bluestore-migration.rst up to but not including the
section called "Whole Host Replacement".

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ca803a24c64059023733e21d755edb9c6c973ecf)

2 years agodoc/rados: edit operations/bs-migration (1 of x) 50586/head
Zac Dover [Sun, 12 Mar 2023 01:17:03 +0000 (11:17 +1000)]
doc/rados: edit operations/bs-migration (1 of x)

Disambiguate and improve the English language in
doc/rados/operations/bluestore-migration.rst up to but not including the
section called "Whole Host Replacement".

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 96166b463b1f8800e59f80faebfba1208248725e)

2 years agoMerge pull request #50581 from zdover23/wip-doc-2023-03-19-backport-50579-to-reef
Anthony D'Atri [Sun, 19 Mar 2023 12:35:00 +0000 (08:35 -0400)]
Merge pull request #50581 from zdover23/wip-doc-2023-03-19-backport-50579-to-reef

reef: doc/rados: line edit mon-lookup-dns top matter

2 years agodoc/rados: line edit mon-lookup-dns top matter 50581/head
Zac Dover [Sat, 18 Mar 2023 05:27:54 +0000 (15:27 +1000)]
doc/rados: line edit mon-lookup-dns top matter

Improve the syntax of the top matter of
doc/rados/operations/mon-lookup-dns.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ed5919964bd914c118f556baf3c517e5e35df277)

2 years agoqa/workunits/rbd: use bionic version of qemu-iotests for jammy 50577/head
Ilya Dryomov [Wed, 15 Mar 2023 16:12:36 +0000 (17:12 +0100)]
qa/workunits/rbd: use bionic version of qemu-iotests for jammy

Same as in commit 2de2146c300c ("qa/workunits/rbd: use bionic version
of qemu-iotests for focal").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 3b1610997a3f36c7ab4a7721cd266ec0f14bc107)

2 years agoqa/distros/supported-random: update ubuntu_latest.yaml and add ubuntu_20.04.yaml
Casey Bodley [Wed, 8 Mar 2023 15:34:56 +0000 (10:34 -0500)]
qa/distros/supported-random: update ubuntu_latest.yaml and add ubuntu_20.04.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 62e520cbad7a2beff130a4459abcb0fb7d762da0)

2 years agoqa/distros/supported: update ubuntu_latest.yaml and add ubuntu_20.04.yaml
Casey Bodley [Wed, 8 Mar 2023 15:31:42 +0000 (10:31 -0500)]
qa/distros/supported: update ubuntu_latest.yaml and add ubuntu_20.04.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 929172bd718bdc45f796ad52c64dd42c1bc70a9a)

2 years agoqa/distros/all: add ubuntu_latest.yaml symlink to ubuntu_22.04.yaml
Casey Bodley [Wed, 8 Mar 2023 15:28:28 +0000 (10:28 -0500)]
qa/distros/all: add ubuntu_latest.yaml symlink to ubuntu_22.04.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 31846942edd4f10c5250fdedcf69c18d535e43f2)

2 years agoMerge pull request #50565 from zdover23/wip-doc-2023-03-17-backport-50563-to-reef
zdover23 [Fri, 17 Mar 2023 14:24:56 +0000 (00:24 +1000)]
Merge pull request #50565 from zdover23/wip-doc-2023-03-17-backport-50563-to-reef

reef: doc/rados: line-edit upmap.rst

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
2 years agocommon: avoid redefining clock type on Windows 50572/head
Lucian Petrut [Thu, 16 Mar 2023 14:25:50 +0000 (14:25 +0000)]
common: avoid redefining clock type on Windows

mingw >= 8.0.1 defines CLOCK_REALTIME_COARSE, so we'll avoid
overriding it if already set.

Clock precision [1]:

mingw < 8.0.1:
  * CLOCK_REALTIME: ~10-55ms (GetSystemTimeAsFileTime)
mingw >= 8.0.1:
  * CLOCK_REALTIME: <1us (GetSystemTimePreciseAsFileTime)
  * CLOCK_REALTIME_COARSE: ~10-55ms (GetSystemTimeAsFileTime)

* CLOCK_MONOTONIC: <1us if TSC is usable, ~10-55ms otherwise
                   (QueryPerformanceCounter)

[1] https://github.com/mirror/mingw-w64/commit/dcd990ed423381cf35702df9495d44f1979ebe50

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit 489fd858479a79de75f6327d5cca40c289b53f35)

2 years agodoc/rados: line-edit upmap.rst 50565/head
Zac Dover [Thu, 16 Mar 2023 17:41:08 +0000 (03:41 +1000)]
doc/rados: line-edit upmap.rst

Edit all of doc/rados/upmap.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit a4a681846499c8d0041a6632e7e31fa83e22ed0e)

2 years agoMerge pull request #50541 from rhcs-dashboard/wip-59072-reef
Nizamudeen A [Thu, 16 Mar 2023 06:28:45 +0000 (11:58 +0530)]
Merge pull request #50541 from rhcs-dashboard/wip-59072-reef

reef: mgr/dashboard: fix prometheus api issues on landing page v3

Reviewed-by: Pegonzal <NOT@FOUND>
2 years agoMerge pull request #50532 from zdover23/wip-doc-2023-03-15-backport-50390-to-reef
zdover23 [Thu, 16 Mar 2023 01:30:32 +0000 (11:30 +1000)]
Merge pull request #50532 from zdover23/wip-doc-2023-03-15-backport-50390-to-reef

reef: doc/crimson: add SeaStore when using vstart.

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
2 years agomgr/dashboard: improve the logic on the alert viewer 50541/head
Nizamudeen A [Wed, 22 Feb 2023 16:52:56 +0000 (22:22 +0530)]
mgr/dashboard: improve the logic on the alert viewer

only fetch the alerts when you toggle the alert viewer. don't fetch it
every 5 second cause its going to cause a heavy toll on the api.

Fixes: https://tracker.ceph.com/issues/58867
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 6cb0679aaf28b5f128ca3cae5f962cac50d084af)

2 years agomgr/dashboard: fix prometheus api error on landing page v3
Nizamudeen A [Wed, 22 Feb 2023 07:33:34 +0000 (13:03 +0530)]
mgr/dashboard: fix prometheus api error on landing page v3

When no prometheus is configured in the cluster, it gives out error
while polling to the prometheus endpoint. so a proper check needs to be
added there.

Also some improvements to the exisiting behaviour

Fixes: https://tracker.ceph.com/issues/58867
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 2d80e57dd4a60cbf52074d982ba3ebd4471a14ca)

2 years agoMerge pull request #50525 from epuertat/wip-59050-reef
Ernesto Puerta [Tue, 14 Mar 2023 20:17:18 +0000 (21:17 +0100)]
Merge pull request #50525 from epuertat/wip-59050-reef

reef: mgr/dashboard: force TLS 1.3

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
2 years agodoc/crimson: add SeaStore when using vstart. 50532/head
Rongqi Sun [Mon, 6 Mar 2023 06:01:51 +0000 (14:01 +0800)]
doc/crimson: add SeaStore when using vstart.

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>
(cherry picked from commit 055bb2bd5364e66c6e7f87760b9c409f2b8c7935)

2 years agomgr/dashboard: force TLS 1.3 50525/head
Ernesto Puerta [Mon, 13 Mar 2023 10:03:37 +0000 (11:03 +0100)]
mgr/dashboard: force TLS 1.3

Fixes: https://tracker.ceph.com/issues/58942
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit cd8946631b222dfff1727083d75dfe2dd825d1a4)

2 years agoMerge pull request #50460 from rhcs-dashboard/custom-image-kcli-bootstrap-reef
Nizamudeen A [Tue, 14 Mar 2023 07:07:47 +0000 (12:37 +0530)]
Merge pull request #50460 from rhcs-dashboard/custom-image-kcli-bootstrap-reef

reef: mgr/dashboard: custom image for kcli bootstrap script

Reviewed-by: Pegonzal <NOT@FOUND>
2 years agoMerge pull request #50517 from zdover23/wip-doc-2023-03-14-backport-50500-to-reef
Anthony D'Atri [Tue, 14 Mar 2023 02:26:17 +0000 (22:26 -0400)]
Merge pull request #50517 from zdover23/wip-doc-2023-03-14-backport-50500-to-reef

reef: doc/dev/release-checklist: add another step for ceph-build

2 years agodoc/dev/release-checklist: add another step for ceph-build 50517/head
Josh Durgin [Mon, 13 Mar 2023 14:45:07 +0000 (07:45 -0700)]
doc/dev/release-checklist: add another step for ceph-build

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 4fcad7161fa094af210229b6f15cded63781080e)

2 years agoMerge pull request #50464 from ljflores/wip-tracker-58496-reef-backport
Laura Flores [Mon, 13 Mar 2023 20:16:33 +0000 (15:16 -0500)]
Merge pull request #50464 from ljflores/wip-tracker-58496-reef-backport

reef: osd/scrub: only call publish_stats_to_osd() where allowed and needed

2 years agorgw: add rgw-restore-bucket-index 50509/head
J. Eric Ivancich [Wed, 8 Mar 2023 21:51:17 +0000 (16:51 -0500)]
rgw: add rgw-restore-bucket-index

Adds an experimental script that allows a bucket index of a
non-versioned bucket to be restored by applying `radosgw-admin object
reindex ...` to all objects in the specified bucket. The objects in
the bucket are determined by scanning the data pool for head objects
containing the bucket's marker.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 6ea11b1305c77f9141f2dd8093ac56b579f250be)

2 years agoMerge pull request #50424 from ljflores/wip-tracker-58925-reef
Laura Flores [Mon, 13 Mar 2023 20:16:08 +0000 (15:16 -0500)]
Merge pull request #50424 from ljflores/wip-tracker-58925-reef

reef: qa: suppress memory leak in rocksdb

2 years agorgw: add an "object reindex" subcommand to radosgw-admin
J. Eric Ivancich [Thu, 2 Mar 2023 04:41:12 +0000 (23:41 -0500)]
rgw: add an "object reindex" subcommand to radosgw-admin

This subcommand will take a bucket and an object (or a list of objects
in a file) and add those objects to the bucket's index. It does so by
preparing the bucket index entry update and then allowing the
so-called "dir suggest" mechansim, which is triggered by listing the
bucket, to completing the bucket index entry. This mechanism is
triggered by listing the bucket.  Using this mechanism has the
advantage of doing this lazily that both allows the reindex operation
to run more quickly and distributes the workload over time.

This subcommand can help restore lost bucket index entries.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit be0583c3082b9f924fcdbe4c81d540ad08bcca24)

2 years agorgw/sysobj: read() can query size/mtime 50484/head
Casey Bodley [Tue, 7 Mar 2023 22:28:54 +0000 (17:28 -0500)]
rgw/sysobj: read() can query size/mtime

rgw_get_system_obj() stopped calling stat() before read() in
90aec61298fc378e1733bc76f0d3f95ce8ec135a, which left the optional
'real_time *pmtime' argument uninitialized

when requested, read() will add a stat op to initialize size/mtime

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 20ab42758c8ba0cef867752be3b215064d44a6f2)

2 years agorgw/admin: 'bucket stats' displays non-empty mtime
Casey Bodley [Tue, 7 Mar 2023 22:49:44 +0000 (17:49 -0500)]
rgw/admin: 'bucket stats' displays non-empty mtime

Fixes: https://tracker.ceph.com/issues/58932
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e9720ddbe9ae033e14c2f3def51a06a4e204d4c8)

2 years agoMerge pull request #50414 from zdover23/wip-doc-2023-03-07-backport-50384-to-reef
zdover23 [Sat, 11 Mar 2023 04:22:45 +0000 (14:22 +1000)]
Merge pull request #50414 from zdover23/wip-doc-2023-03-07-backport-50384-to-reef

reef: doc/radosgw: Redd up s3select.rst as followup to #49411

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
2 years agoMerge pull request #50476 from zdover23/wip-doc-2023-03-11-backport-50470-to-reef
Anthony D'Atri [Fri, 10 Mar 2023 21:35:25 +0000 (16:35 -0500)]
Merge pull request #50476 from zdover23/wip-doc-2023-03-11-backport-50470-to-reef

reef: doc/index: remove "uniquely" from landing page

2 years agodoc/index: remove "uniquely" from landing page 50476/head
Zac Dover [Fri, 10 Mar 2023 10:35:18 +0000 (20:35 +1000)]
doc/index: remove "uniquely" from landing page

Change "Ceph uniquely delivers[...]" to "Ceph delivers[...]"

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6e5be7ad66aa76cb798c52c87e34f8fe079cda5c)

2 years agoMerge pull request #50452 from zdover23/wip-doc-2023-03-09-backport-50439-to-reef
zdover23 [Fri, 10 Mar 2023 07:08:44 +0000 (17:08 +1000)]
Merge pull request #50452 from zdover23/wip-doc-2023-03-09-backport-50439-to-reef

reef: doc/rados/operations: Improve wording, capitalization, formatting

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agoosd/scrub: only call publish_stats_to_osd() where allowed and needed 50464/head
Ronen Friedman [Sat, 11 Feb 2023 19:16:32 +0000 (21:16 +0200)]
osd/scrub: only call publish_stats_to_osd() where allowed and needed

This patch removes several unnecessary calls to publish_stats_to_osd() in
the clear_pgscrub_state pathway used during interval changes. We didn't
actually have anything to publish and we can't validly call publish_stats_to_osd()
when we aren't an active primary anyway (which we can't be during an interval
change).

Fixes: https://tracker.ceph.com/issues/58496
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
(cherry picked from commit aeff19db99ab53fbaec7eb530e35d578e6254e4c)

2 years agomgr/dashboard: custom image for kcli bootstrap script 50460/head
Nizamudeen A [Thu, 9 Mar 2023 11:51:44 +0000 (17:21 +0530)]
mgr/dashboard: custom image for kcli bootstrap script

the stable branches like quincy pulls from the quay.io/ceph/ceph:v17 to
bootstrap the ceph cluster in test environments. This will cause issues
because the branches are changing constantly but the image is not. So
using the quay.ceph.io repo to bring the cluster in test environment.

Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit be8f1e7f4a81dbf04692446edcc429d93d59e09d)

2 years agoMerge pull request #50387 from zdover23/wip-doc-2023-03-06-backport-50386-to-reef
zdover23 [Thu, 9 Mar 2023 10:09:47 +0000 (20:09 +1000)]
Merge pull request #50387 from zdover23/wip-doc-2023-03-06-backport-50386-to-reef

reef: doc/rgw - fix grammar in table in s3.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agodoc/rados/operations: Improve wording, capitalizatiopn, formatting 50452/head
Anthony D'Atri [Wed, 8 Mar 2023 12:29:55 +0000 (07:29 -0500)]
doc/rados/operations: Improve wording, capitalizatiopn, formatting

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 5f2060c0829992b31a0d17e23d7b1d2fb2280ac0)

2 years agoMerge pull request #50431 from zdover23/wip-doc-2023-03-08-backport-50430-to-reef
zdover23 [Wed, 8 Mar 2023 04:01:03 +0000 (14:01 +1000)]
Merge pull request #50431 from zdover23/wip-doc-2023-03-08-backport-50430-to-reef

reef: doc/install: update index.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agodoc/install: update index.rst 50431/head
Zac Dover [Wed, 8 Mar 2023 01:52:12 +0000 (11:52 +1000)]
doc/install: update index.rst

Update index.rst by making minor grammar improvements. This file was
long overdue for a backport to Reef, Quincy, and Pacific, so this commit
was a good way to pass a human eyeball over the text before making those
backports.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 82d748ff590cfd31fefddc094b10622d090b054f)

2 years agoMerge pull request #50417 from soumyakoduri/wip-skoduri-reef
Casey Bodley [Tue, 7 Mar 2023 13:45:47 +0000 (08:45 -0500)]
Merge pull request #50417 from soumyakoduri/wip-skoduri-reef

[Reef] rgw/tests: Reset endpoint before creating additional zones

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2 years agorgw/tests: Reset endpoint before creating additional zones 50417/head
Soumya Koduri [Mon, 6 Mar 2023 14:23:08 +0000 (19:53 +0530)]
rgw/tests: Reset endpoint before creating additional zones

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Fixes: https://tracker.ceph.com/issues/58822
2 years agodoc/radosgw: Redd up s3select.rst as followup to #49411 50414/head
Anthony D'Atri [Sun, 5 Mar 2023 16:31:36 +0000 (11:31 -0500)]
doc/radosgw: Redd up s3select.rst as followup to #49411

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 78b42ee0d27995a322cdd3998fb9726e490007a0)

2 years agoMerge pull request #50399 from zdover23/wip-doc-2023-03-07-backport-50398-to-reef
zdover23 [Mon, 6 Mar 2023 23:13:47 +0000 (09:13 +1000)]
Merge pull request #50399 from zdover23/wip-doc-2023-03-07-backport-50398-to-reef

reef: doc/dev: add full stop to sentence in basic-wo

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agoMerge pull request #50374 from cbodley/wip-rgw-reef-s3select-json
Casey Bodley [Mon, 6 Mar 2023 22:11:02 +0000 (17:11 -0500)]
Merge pull request #50374 from cbodley/wip-rgw-reef-s3select-json

reef: json integration: identifying new data-source and execute json flow.

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
2 years agoqa: suppress memory leak in rocksdb 50424/head
Laura Flores [Mon, 6 Mar 2023 21:46:31 +0000 (21:46 +0000)]
qa: suppress memory leak in rocksdb

Fixes: https://tracker.ceph.com/issues/58925
Signed-off-by: Laura Flores <lflores@redhat.com>
(cherry picked from commit 7d6390dfbdadaf05a9b0cd2fd9f0fc5ac253832b)

2 years agoMerge pull request #50358 from cbodley/wip-qa-rgw-reef-tests
Casey Bodley [Mon, 6 Mar 2023 22:05:09 +0000 (17:05 -0500)]
Merge pull request #50358 from cbodley/wip-qa-rgw-reef-tests

reef: qa/rgw: point all test repos at new ceph-reef branch

Reviewed-by: Ali Maredia <amaredia@redhat.com>
2 years agoMerge pull request #50376 from cbodley/wip-58891-reef
Casey Bodley [Mon, 6 Mar 2023 19:44:13 +0000 (14:44 -0500)]
Merge pull request #50376 from cbodley/wip-58891-reef

reef: rgw/reshard: treat old RGWBucketInfo::num_shards=0 as 1 shard

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2 years agodoc/dev: add full stop to sentence in basic-wo 50399/head
Zac Dover [Mon, 6 Mar 2023 16:15:27 +0000 (02:15 +1000)]
doc/dev: add full stop to sentence in basic-wo

Add a full stop to a sentence in basic-workflow.rst. I looked at this
document and noticed that it wasn't there, and it was just bugging me.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 262b115b7554b7931cc174500eb12ac7a5d67513)

2 years agoMerge pull request #50394 from zdover23/wip-doc-2023-03-06-backport-50382-to-reef
Anthony D'Atri [Mon, 6 Mar 2023 15:04:55 +0000 (10:04 -0500)]
Merge pull request #50394 from zdover23/wip-doc-2023-03-06-backport-50382-to-reef

reef: doc/rgw: fix caption

2 years agodoc/rgw: fix caption 50394/head
Zac Dover [Sun, 5 Mar 2023 06:43:38 +0000 (16:43 +1000)]
doc/rgw: fix caption

Rewrite the third paragraph of the caption to the figure depicting how a
zonegroup works.

Follows https://github.com/ceph/ceph/pull/50316.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 8fa521466023635436305c1af31501bbbe20e857)

2 years agodoc/rgw - fix grammar in table in s3.rst 50387/head
Zac Dover [Mon, 6 Mar 2023 00:09:24 +0000 (10:09 +1000)]
doc/rgw - fix grammar in table in s3.rst

Put an "only" in the correct place. Also, this is a very small change
that will allow me to see if Jenkins is working (it probably isn't).

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 1beab1e8abf59088be628cca972c59202526637d)

2 years agoMerge pull request #50377 from zdover23/wip-doc-2023-03-04-backport-50375-to-reef
Anthony D'Atri [Fri, 3 Mar 2023 23:04:36 +0000 (18:04 -0500)]
Merge pull request #50377 from zdover23/wip-doc-2023-03-04-backport-50375-to-reef

reef: doc/rgw: refine keycloak.rst

2 years agodoc/rgw: refine keycloak.rst 50377/head
Zac Dover [Fri, 3 Mar 2023 19:36:35 +0000 (05:36 +1000)]
doc/rgw: refine keycloak.rst

Full line-edit of doc/radosgw/keycloak.rst. Corrected syntax, grammar,
RST, and broken links.

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 5706ff1e451e6c7b3c3d40d492985377161b0133)

2 years agorgw/reshard: avoid infinite loop if bi_list() returns -ENOENT 50376/head
Casey Bodley [Wed, 1 Mar 2023 22:20:06 +0000 (17:20 -0500)]
rgw/reshard: avoid infinite loop if bi_list() returns -ENOENT

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3e10944ff551e7bbb3d9083cb351ceb53c6b07b9)

2 years agorgw/rados: RGWSI_BucketIndex_RADOS requires layouts, not num_shards
Casey Bodley [Wed, 1 Mar 2023 21:06:14 +0000 (16:06 -0500)]
rgw/rados: RGWSI_BucketIndex_RADOS requires layouts, not num_shards

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 46f89cbae76d68af83a712f42443466e466681a3)

2 years agorgw/rados: use rgw::num_shards() helper
Casey Bodley [Wed, 1 Mar 2023 15:48:30 +0000 (10:48 -0500)]
rgw/rados: use rgw::num_shards() helper

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c8908fdd4cf4bb6dd14787dd7a49f75b32cc53ce)

2 years agorgw/reshard: refuse to reshard to new_num_shards=0
Casey Bodley [Wed, 1 Mar 2023 15:44:31 +0000 (10:44 -0500)]
rgw/reshard: refuse to reshard to new_num_shards=0

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4db197e858a4cedef4ad86af66d9b2cd2f4e4e31)

2 years agorgw/reshard: use the rgw::num_shards() helper
Casey Bodley [Wed, 1 Mar 2023 15:44:09 +0000 (10:44 -0500)]
rgw/reshard: use the rgw::num_shards() helper

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e376080940605e21ba4189597706371bd7a29d9b)

2 years agorgw: rgw::num_shards() returns 1 instead of 0
Casey Bodley [Wed, 1 Mar 2023 15:02:11 +0000 (10:02 -0500)]
rgw: rgw::num_shards() returns 1 instead of 0

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4497ce51c40fe04688d7a80e9b3241fe89e1db5e)

2 years agojson integration: identifying new data-source and execute json flow. 50374/head
galsalomon66 [Tue, 13 Dec 2022 14:31:43 +0000 (16:31 +0200)]
json integration: identifying new data-source and execute json flow.
s3select submodule

integrate the limit-operator into RGW. current changes is for CSV format

integrate the limit-operator with CSV-flow, and JSON-flow. an update of s3select submodule

a fix for the input-serialization-type selection

debug functionality.

fix for error handling

adding the scan-range feature. i.e. enables the user to define the range of processing(text only)

remove the temp variable(the s3select-layer handles the sql-result setting upon JSON flow)

adding documentation. for JSON, SQL limit-operator, scan-range option.
a fix for JSON test framework in s3select module

investigate crash. replace len with it.length()

editorial changes; non-initialized variables

adding validation for offset&length. (it seems that the offset&length causes Invalid reads, in some cases)

changes related to trino integration.(1) syntax issues related to trino-engine statement conbersions(s3select parser handles that) (2).
(2). trino rejects(HIVE_CURSUR_ERROR) some of the s3select responses do to its size, the change is about control the size of the respond.

update JSON documentation

trino changes: response in paging (limitation on size), trino syntax issues.

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
(cherry picked from commit 9f20e75a0b611a9de96e67070ce6078e23f6ba41)

2 years agoMerge pull request #50366 from zdover23/wip-doc-2023-03-03-backport-50364-to-reef
Anthony D'Atri [Fri, 3 Mar 2023 15:55:32 +0000 (10:55 -0500)]
Merge pull request #50366 from zdover23/wip-doc-2023-03-03-backport-50364-to-reef

reef: doc/cephadm: Redd up compatibility.rst

2 years agodoc/cephadm: Redd up compatibility.rst 50366/head
Anthony D'Atri [Fri, 3 Mar 2023 07:44:26 +0000 (02:44 -0500)]
doc/cephadm: Redd up compatibility.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit d608dec43b84c9a06990458219dda769829e05f8)

2 years agoMerge pull request #50361 from zdover23/wip-doc-2023-03-03-backport-50354-to-reef
Anthony D'Atri [Fri, 3 Mar 2023 06:48:42 +0000 (01:48 -0500)]
Merge pull request #50361 from zdover23/wip-doc-2023-03-03-backport-50354-to-reef

reef: doc/rgw: clean zone-sync.svg

2 years agoMerge pull request #50355 from zdover23/wip-doc-2023-03-03-backport-50351-to-reef
zdover23 [Fri, 3 Mar 2023 05:05:38 +0000 (15:05 +1000)]
Merge pull request #50355 from zdover23/wip-doc-2023-03-03-backport-50351-to-reef

reef: doc/radosgw: format admonitions

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agodoc/rgw: clean zone-sync.svg 50361/head
Zac Dover [Thu, 2 Mar 2023 19:24:00 +0000 (05:24 +1000)]
doc/rgw: clean zone-sync.svg

Refine zone-sync.svg (fix kerning and use fonts that render properly).

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 064fdc9d9b4ad3393bfb5069d2f97dc622aaf12d)

2 years agoqa/rgw: point all test repos at new ceph-reef branch 50358/head
Casey Bodley [Thu, 2 Mar 2023 23:30:26 +0000 (18:30 -0500)]
qa/rgw: point all test repos at new ceph-reef branch

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2 years agoqa/smoke: use rgw/s3tests-branch.yaml
Casey Bodley [Thu, 2 Mar 2023 23:29:44 +0000 (18:29 -0500)]
qa/smoke: use rgw/s3tests-branch.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2 years agodoc/radosgw: format admonitions 50355/head
Zac Dover [Thu, 2 Mar 2023 18:04:30 +0000 (04:04 +1000)]
doc/radosgw: format admonitions

Break up the text of two similar admonitions into three paragraphs (in
each of the two instances). This makes the content of the admonition
much easier to read at a glance.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 4db68f5711f48b190f0135a3a7a6df8dfcf5abfa)

2 years agoMerge pull request #50338 from zdover23/wip-doc-2023-03-02-backport-50335-to-reef
zdover23 [Thu, 2 Mar 2023 03:59:45 +0000 (13:59 +1000)]
Merge pull request #50338 from zdover23/wip-doc-2023-03-02-backport-50335-to-reef

reef: doc/cephadm: update cephadm compatability and stability page

Reviewed-by: Adam King <adking@redhat.com>
2 years agodoc/cephadm: update cephadm compatability and stability page 50338/head
Adam King [Wed, 1 Mar 2023 21:10:41 +0000 (16:10 -0500)]
doc/cephadm: update cephadm compatability and stability page

This page is very out of date. This commit probably doesn't
cover everything there is to say about stability and compatability
in cephadm, but it at least gets it noticeably closer to reality

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 383f46aa79e7a6f5571405a352ad023983560625)

2 years agoMerge pull request #50316 from zdover23/wip-doc-2023-03-01-radosgw-multisite-diagram...
zdover23 [Wed, 1 Mar 2023 18:04:25 +0000 (04:04 +1000)]
Merge pull request #50316 from zdover23/wip-doc-2023-03-01-radosgw-multisite-diagram-object-replication-text-revision

doc/rgw: improve diagram caption

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agodoc/rgw: improve diagram caption 50316/head
Zac Dover [Wed, 1 Mar 2023 07:36:18 +0000 (17:36 +1000)]
doc/rgw: improve diagram caption

Improve the syntax and semantics of the caption of the diagram
"Replication of Object Data Between Zones" in doc/radosgw/multisite.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
2 years agoMerge pull request #50256 from yuvalif/wip-yuval-zipper-notifications
Yuval Lifshitz [Wed, 1 Mar 2023 14:13:55 +0000 (16:13 +0200)]
Merge pull request #50256 from yuvalif/wip-yuval-zipper-notifications

rgw/notifications: add bucket notification configuration to zipper

Reviewed-By: dang, cbodley