]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
3 years agorgw/CloudTransition: Update object metadata and bi post cloud tranistion
Soumya Koduri [Tue, 28 Jul 2020 20:27:04 +0000 (01:57 +0530)]
rgw/CloudTransition: Update object metadata and bi post cloud tranistion

After transitioning the object to cloud, following updates are done
to the existing object.

* In bi entry, change object category to CloudTiered
* Update cloud-tier details (like endpoint, keys etc) in Object Manifest
* Mark the tail objects expired to be deleted by gc

TODO:
* Update all the cloud config details including multiparts
* Check if any other object metadata needs to be changed
* Optimize to avoid using read_op again to read attrs.
* Check for mtime to resolve conflicts when multiple zones try to transition obj

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
3 years agorgw/CloudTransition: Tier objects to remote cloud
Soumya Koduri [Wed, 23 Dec 2020 05:44:53 +0000 (11:14 +0530)]
rgw/CloudTransition: Tier objects to remote cloud

If the storage class configured is of cloud, transition
the objects to remote endpoint configured.

In case the object size is >mulitpart size limit (say 5M),
upload the object into multiparts.

As part of transition, map rgw attributes to http attrs,
including ACLs.

A new attribute (x-amz-meta-source: rgw) is added to denote
that the object is transitioned from RGW source.

Added two new options to tier-config to configure multipart size -
* multipart_sync_threshold - determines the limit of object size,
when exceeded transitioned in multiparts
* multipart_min_part_size - the minimum size of the multipart upload part

Default values for both the options is 32M and minimum value supported
is 5M.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
3 years agorgw/CloudTransition: Add new options to configure tier endpoint
Soumya Koduri [Sat, 11 Jul 2020 17:05:04 +0000 (22:35 +0530)]
rgw/CloudTransition: Add new options to configure tier endpoint

As mentioned in https://docs.google.com/document/d/1IoeITPCF64A5W-UA-9Y3Vp2oSfz3xVQHu31GTu3u3Ug/edit,
the tier storage class will be configured at zonegroup level.

So the existing CLI "radosgw-admin zonegroup placement add  <id> --storage-class <class>" will be
used to add tier storage classes as well but with extra tier-config options mentioned below -

--tier-type : "cloud"
--tier-config : [<key,value>,]

These tier options are already defined to configure cloud sync module which are being reused here.

TODO:
* Add multipart options (if any , like part size, threshold)
* Document
* Test upgrade/downgrade

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
3 years agoMerge PR #43929 into master
Sage Weil [Wed, 17 Nov 2021 22:05:23 +0000 (17:05 -0500)]
Merge PR #43929 into master

* refs/pull/43929/head:
qa/suites/orch/cephadm: verify /var/log/ceph/$fsid ownership
cephadm: only make_log_dir for ceph daemons

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
3 years agoMerge PR #43934 into master
Sage Weil [Wed, 17 Nov 2021 22:04:57 +0000 (17:04 -0500)]
Merge PR #43934 into master

* refs/pull/43934/head:
qa/suites/rados/dashboard: use single-container-host.yaml
qa/distros: add single-container-host.yaml
qa/suites: use distros/container-hosts/
qa/distros/container-hosts: add 8.stream + crun
qa/distros/container-hosts: add collection of container targets

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
3 years agoMerge pull request #43433 from soumyakoduri/wip-skoduri-dbstore-lc
Casey Bodley [Wed, 17 Nov 2021 17:39:19 +0000 (12:39 -0500)]
Merge pull request #43433 from soumyakoduri/wip-skoduri-dbstore-lc

rgw/dbstore: handle lc related state

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
3 years agoMerge pull request #43921 from aclamk/wip-aclamk-fix-omap-upgrade-fix
Neha Ojha [Wed, 17 Nov 2021 17:35:19 +0000 (09:35 -0800)]
Merge pull request #43921 from aclamk/wip-aclamk-fix-omap-upgrade-fix

 BlueStore: Omap upgrade to per-pg fix fix

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
3 years agoos/bluestore: Fix omap upgrade to per-pg scheme 43921/head
Adam Kupczyk [Sat, 13 Nov 2021 10:28:18 +0000 (11:28 +0100)]
os/bluestore: Fix omap upgrade to per-pg scheme

This is fix to regression introduced by fix to omap upgrade: https://github.com/ceph/ceph/pull/43687
The problem was that we always skipped first omap entry.
This worked fine with objects having omap header key.
For objects without header key we skipped first actual omap key.

Fixes: https://tracker.ceph.com/issues/53260
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
3 years agoos/bluestore: Add more legacy -> per PG upgrade tests
Adam Kupczyk [Sat, 13 Nov 2021 09:41:14 +0000 (10:41 +0100)]
os/bluestore: Add more legacy -> per PG upgrade tests

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
3 years agoMerge pull request #43977 from xxhdx1985126/wip-53273
Samuel Just [Wed, 17 Nov 2021 15:34:47 +0000 (07:34 -0800)]
Merge pull request #43977 from xxhdx1985126/wip-53273

crimson/os/seastore/lba_manager: do full merge if the donor node is *AT* its minimum capacity

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
3 years agoqa/suites/rados/dashboard: use single-container-host.yaml 43934/head
Sage Weil [Tue, 16 Nov 2021 15:57:01 +0000 (09:57 -0600)]
qa/suites/rados/dashboard: use single-container-host.yaml

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agoqa/distros: add single-container-host.yaml
Sage Weil [Tue, 16 Nov 2021 15:59:25 +0000 (09:59 -0600)]
qa/distros: add single-container-host.yaml

This is a single, possibly preferred, os + container runtime
combination.

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agoqa/suites: use distros/container-hosts/
Sage Weil [Tue, 16 Nov 2021 15:53:28 +0000 (09:53 -0600)]
qa/suites: use distros/container-hosts/

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agoMerge pull request #43095 from sebastian-philipp/_check_for_moved_osds
Sebastian Wagner [Wed, 17 Nov 2021 14:09:06 +0000 (15:09 +0100)]
Merge pull request #43095 from sebastian-philipp/_check_for_moved_osds

mgr/cephadm: Add _check_for_moved_osds

Reviewed-by: Adam King <adking@redhat.com>
3 years agoMerge pull request #42906 from sebastian-philipp/rm-ceph-orch-osd-i
Sebastian Wagner [Wed, 17 Nov 2021 10:18:51 +0000 (11:18 +0100)]
Merge pull request #42906 from sebastian-philipp/rm-ceph-orch-osd-i

mgr/orch: deprecated `ceph orch apply osd -i <osd_spec_file>`

Reviewed-by: Adam King <adking@redhat.com>
3 years agoMerge pull request #43670 from jtlayton/wip-53004
Venky Shankar [Wed, 17 Nov 2021 06:14:41 +0000 (11:44 +0530)]
Merge pull request #43670 from jtlayton/wip-53004

client: flesh out doc comments for struct ceph_client_callback_args

Reviewed-by: Venky Shankar <vshankar@redhat.com>
3 years agocrimson/os/seastore/lba_manager: do full merge if the donor node is *AT* its minimum... 43977/head
Xuehan Xu [Wed, 17 Nov 2021 01:10:59 +0000 (09:10 +0800)]
crimson/os/seastore/lba_manager: do full merge if the donor node is *AT* its minimum capacity

Fixes: https://tracker.ceph.com/issues/53273
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
3 years agocrimson/os/seastore/lba_manager: better debug outputs for lba node merge
Xuehan Xu [Wed, 17 Nov 2021 01:09:41 +0000 (09:09 +0800)]
crimson/os/seastore/lba_manager: better debug outputs for lba node merge

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
3 years agoMerge pull request #43868 from rhcs-dashboard/fix-53209-master
Ernesto Puerta [Wed, 17 Nov 2021 00:03:15 +0000 (01:03 +0100)]
Merge pull request #43868 from rhcs-dashboard/fix-53209-master

mgr/dashboard: Device health status is not getting listed under hosts section

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
3 years agoMerge pull request #43803 from Matan-B/wip-matanb-pgstats-support
Samuel Just [Tue, 16 Nov 2021 23:10:44 +0000 (15:10 -0800)]
Merge pull request #43803 from Matan-B/wip-matanb-pgstats-support

crimson/osd: Fixing empty PG stats issue

Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #43669 from rhcs-dashboard/grafana-unit-tests
Ernesto Puerta [Tue, 16 Nov 2021 22:30:10 +0000 (23:30 +0100)]
Merge pull request #43669 from rhcs-dashboard/grafana-unit-tests

monitoring/grafana: Grafana query tester

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
3 years agoMerge pull request #43909 from ivancich/wip-small-code-fixups
J. Eric Ivancich [Tue, 16 Nov 2021 19:19:44 +0000 (14:19 -0500)]
Merge pull request #43909 from ivancich/wip-small-code-fixups

rgw: small code fix-ups

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
3 years agoMerge pull request #43932 from gregsfortytwo/wip-53258-dump-disallowed
Gregory Farnum [Tue, 16 Nov 2021 18:22:00 +0000 (10:22 -0800)]
Merge pull request #43932 from gregsfortytwo/wip-53258-dump-disallowed

mon: MonMap: display disallowed_leaders whenever they're set

Reviewed-by: Samuel Just <sjust@redhat.com
Reviewed-by: Laura Flores <lflores@redhat.com>
3 years agoMerge pull request #43899 from gregsfortytwo/wip-53237-stretch-kernels
Gregory Farnum [Tue, 16 Nov 2021 18:20:28 +0000 (10:20 -0800)]
Merge pull request #43899 from gregsfortytwo/wip-53237-stretch-kernels

mon: MonMap: do not increase mon_info_t's compatv in stretch mode, really

Reviewed-by: Samuel Just <sjust@redhat.com
Reviewed-by: Laura Flores <lflores@redhat.com>
3 years agoMerge pull request #43818 from ifed01/wip-ifed-fix-vol-select
Yuri Weinstein [Tue, 16 Nov 2021 17:02:53 +0000 (09:02 -0800)]
Merge pull request #43818 from ifed01/wip-ifed-fix-vol-select

os/bluestore: do not select absent device in volume selector

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
3 years agoMerge pull request #43913 from ivancich/wip-fix-bucket-removal
J. Eric Ivancich [Tue, 16 Nov 2021 16:49:22 +0000 (11:49 -0500)]
Merge pull request #43913 from ivancich/wip-fix-bucket-removal

rgw: remove prefix & delim params for bucket removal & mp upload abort

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 years agoMerge pull request #43908 from ivancich/wip-fix-bi-put-shard-calc
J. Eric Ivancich [Tue, 16 Nov 2021 16:47:53 +0000 (11:47 -0500)]
Merge pull request #43908 from ivancich/wip-fix-bi-put-shard-calc

rgw: fix `bi put` not using right bucket index shard

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 years agoMerge pull request #42880 from Kalkulus1/patch-1
Casey Bodley [Tue, 16 Nov 2021 16:01:00 +0000 (11:01 -0500)]
Merge pull request #42880 from Kalkulus1/patch-1

doc: adds a way to allow users set_contents_from_filename for rgw

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 years agoqa/distros/container-hosts: add 8.stream + crun
Sage Weil [Tue, 16 Nov 2021 15:49:35 +0000 (09:49 -0600)]
qa/distros/container-hosts: add 8.stream + crun

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agoqa/distros/container-hosts: add collection of container targets
Sage Weil [Tue, 16 Nov 2021 15:49:14 +0000 (09:49 -0600)]
qa/distros/container-hosts: add collection of container targets

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agoMerge pull request #43912 from ronen-fr/wip-rf-test-dump
Ronen Friedman [Tue, 16 Nov 2021 15:58:07 +0000 (17:58 +0200)]
Merge pull request #43912 from ronen-fr/wip-rf-test-dump

qa/standalone: fix osd-scrub-dump following changes to 'pg dump pgs' output

Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #43854 from cbodley/wip-53199
Casey Bodley [Tue, 16 Nov 2021 15:28:44 +0000 (10:28 -0500)]
Merge pull request #43854 from cbodley/wip-53199

cls/rgw: index cancelation still cleans up remove_objs

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
3 years agoMerge pull request #43300 from cfsnyder/wip-52730-fix-dbwal-slot-size-calc
Guillaume Abrioux [Tue, 16 Nov 2021 14:15:28 +0000 (15:15 +0100)]
Merge pull request #43300 from cfsnyder/wip-52730-fix-dbwal-slot-size-calc

ceph-volume: fix bug with miscalculation of required db/wal slot size for VGs with multiple PVs

3 years agoMerge pull request #43927 from guits/fix_raw_activate
Sébastien Han [Tue, 16 Nov 2021 13:11:10 +0000 (14:11 +0100)]
Merge pull request #43927 from guits/fix_raw_activate

ceph-volume: follow-up on PR #42727

3 years agocrimson/osd: Fixing empty PG stats issue 43803/head
Matan Breizman [Tue, 26 Oct 2021 11:47:44 +0000 (11:47 +0000)]
crimson/osd: Fixing empty PG stats issue

Signed-off-by: Matan Breizman <Matan.Brz@gmail.com>
3 years agoMerge pull request #43780 from mgfritch/nfs-rgw-v4
Sebastian Wagner [Tue, 16 Nov 2021 10:38:03 +0000 (11:38 +0100)]
Merge pull request #43780 from mgfritch/nfs-rgw-v4

doc/radosgw/nfs: add note about NFSv3 deprecation

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Sage Weil <sage@newdream.net>
Reviewed-by: Varsha Rao <rvarsha016@gmail.com>
3 years agoMerge pull request #43933 from ceph/dependabot/pip/src/ceph-volume/plugin/zfs/pip...
Sebastian Wagner [Tue, 16 Nov 2021 09:39:37 +0000 (10:39 +0100)]
Merge pull request #43933 from ceph/dependabot/pip/src/ceph-volume/plugin/zfs/pip-21.1

build(deps-dev): bump pip from 19.2 to 21.1 in /src/ceph-volume/plugin/zfs

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
3 years agomonitoring/grafana: Grafana query tester 43669/head
Pere Diaz Bou [Tue, 26 Oct 2021 12:42:17 +0000 (14:42 +0200)]
monitoring/grafana: Grafana query tester

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
3 years agoMerge pull request #43938 from neha-ojha/wip-quick-fix
Neha Ojha [Tue, 16 Nov 2021 00:09:30 +0000 (16:09 -0800)]
Merge pull request #43938 from neha-ojha/wip-quick-fix

qa/suites/upgrade/octopus-x: set bluestore fsck quick fix on mount to true

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
3 years agoqa/suites/upgrade/octopus-x: set bluestore fsck quick fix on mount to true 43938/head
Neha Ojha [Thu, 28 Oct 2021 21:02:48 +0000 (21:02 +0000)]
qa/suites/upgrade/octopus-x: set bluestore fsck quick fix on mount to true

First step towards catching bugs like https://tracker.ceph.com/issues/53062.

Signed-off-by: Neha Ojha <nojha@redhat.com>
3 years agomon: MonMap: do not increase mon_info_t's compatv in stretch mode, for real 43899/head
Greg Farnum [Thu, 11 Nov 2021 20:20:11 +0000 (20:20 +0000)]
mon: MonMap: do not increase mon_info_t's compatv in stretch mode, for real

This was supposed to be fixed a year ago in commit
2e3643647bfbe955b54c62c8aaf114744dedb86e, but it set compat_v to 4 instead of all
the way back to 1 as it should have.

Our testing for stretch mode in these areas is just not very thorough -- the
kernel only supports compat_v 1 and apparently nobody's noticed the issue
since then? :/
As the prior commit says, you can't set locations without being gated on a
server feature bit, so simply cancelling this enforcement is completely safe.

Fixes: https://tracker.ceph.com/issues/53237
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
3 years agoMerge pull request #43928 from josephsawaya/futurize-futurized-store
Samuel Just [Mon, 15 Nov 2021 20:17:33 +0000 (12:17 -0800)]
Merge pull request #43928 from josephsawaya/futurize-futurized-store

crimson: futurize FuturizedStore::create()

Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #43344 from Matan-B/wip-matanb-crimson-clog
Matan [Mon, 15 Nov 2021 19:37:49 +0000 (21:37 +0200)]
Merge pull request #43344 from Matan-B/wip-matanb-crimson-clog

crimson/osd: Implementing cluster logging

3 years agoMerge pull request #43761 from cbodley/wip-beast-timeout-alloc
Casey Bodley [Mon, 15 Nov 2021 18:31:18 +0000 (13:31 -0500)]
Merge pull request #43761 from cbodley/wip-beast-timeout-alloc

rgw/beast: optimizations for request timeout

Reviewed-by: Mark Kogan <mkogan@redhat.com>
3 years agorgw/rados: index transactions pass remove_objs to cancel() too 43854/head
Casey Bodley [Tue, 9 Nov 2021 03:13:39 +0000 (22:13 -0500)]
rgw/rados: index transactions pass remove_objs to cancel() too

whenever an index transaction uses remove_objs for complete(), it also
needs to pass them for cancel() to avoid leaking index entries

Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 years agocls/rgw: index cancelation still cleans up remove_objs
Casey Bodley [Tue, 9 Nov 2021 02:24:52 +0000 (21:24 -0500)]
cls/rgw: index cancelation still cleans up remove_objs

when multipart uploads complete their final bucket index transaction,
they pass the list of part objects in 'remove_objs' for bulk removal -
the part objects, along with their bucket stats, get replaced by the
head object

but if CompleteMultipart races with another upload, the head object
write will fail with ECANCELED and the bucket index transaction gets
canceled with CLS_RGW_OP_CANCEL. these canceled uploads still need to
clean up their 'remove_objs', but cancelation was returning too early.
as a result, these bucket index entries get orphaned and leave the
bucket stats inconsistent

this commit reworks rgw_bucket_complete_op() so that CLS_RGW_OP_CANCEL
is handled the same way as OP_ADD and OP_DEL, so always runs the loop to
clean up 'remove_objs'

Fixes: https://tracker.ceph.com/issues/53199
Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 years agocls/rgw: add complete_remove_obj() helper for remove_objs
Casey Bodley [Tue, 9 Nov 2021 02:21:29 +0000 (21:21 -0500)]
cls/rgw: add complete_remove_obj() helper for remove_objs

Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 years agobuild(deps-dev): bump pip in /src/ceph-volume/plugin/zfs 43933/head
dependabot[bot] [Mon, 15 Nov 2021 17:48:13 +0000 (17:48 +0000)]
build(deps-dev): bump pip in /src/ceph-volume/plugin/zfs

Bumps [pip](https://github.com/pypa/pip) from 19.2 to 21.1.
- [Release notes](https://github.com/pypa/pip/releases)
- [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst)
- [Commits](https://github.com/pypa/pip/compare/19.2...21.1)

---
updated-dependencies:
- dependency-name: pip
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
3 years agocls/rgw: helpers take const input params
Casey Bodley [Tue, 9 Nov 2021 02:19:41 +0000 (21:19 -0500)]
cls/rgw: helpers take const input params

Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 years agocrimosn/osd: Implementing cluster logging 43344/head
Matan Breizman [Sat, 25 Sep 2021 13:17:37 +0000 (13:17 +0000)]
crimosn/osd: Implementing cluster logging

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
3 years agoMerge pull request #43903 from rhcs-dashboard/edit-service-feature
Ernesto Puerta [Mon, 15 Nov 2021 16:25:33 +0000 (17:25 +0100)]
Merge pull request #43903 from rhcs-dashboard/edit-service-feature

mgr/dashboard: Edit a service feature

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
3 years agoMerge PR #43894 into master
Sage Weil [Mon, 15 Nov 2021 16:13:11 +0000 (11:13 -0500)]
Merge PR #43894 into master

* refs/pull/43894/head:
qa/suites/orch/cephadm: verify that 'orch ls' reports OSDs properly
mgr/cephadm: show unmanaged OSDs under 'osd' service

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
3 years agorgwi/beast: stream timer with duration 0 disables timeout 43761/head
Mark Kogan [Mon, 15 Nov 2021 15:50:49 +0000 (15:50 +0000)]
rgwi/beast: stream timer with duration 0 disables timeout

fixes all S3 operations failing with:
`2021-11-15T15:46:05.992+0000 7ffee17fa700 20 failed to read header: Bad file descriptor`
when `--rgw_frontends="beast port=8000 request_timeout_ms=0"`

Signed-off-by: Mark Kogan <mkogan@redhat.com>
3 years agoqa/suites/orch/cephadm: verify /var/log/ceph/$fsid ownership 43929/head
Sage Weil [Mon, 15 Nov 2021 15:46:01 +0000 (10:46 -0500)]
qa/suites/orch/cephadm: verify /var/log/ceph/$fsid ownership

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agocephadm: only make_log_dir for ceph daemons
Sage Weil [Mon, 15 Nov 2021 15:43:32 +0000 (10:43 -0500)]
cephadm: only make_log_dir for ceph daemons

For non-ceph daemons, (1) they don't log to /var/log/ceph, and (2) the
container image isn't a ceph image, which means the uid/gid extraction
won't have the correct uid/gid and we'll end up with a log directory that
ceph daemons no longer have write permissions for.

Fixes: https://tracker.ceph.com/issues/53257
Signed-off-by: Sage Weil <sage@newdream.net>
3 years agocrimson: futurize FuturizedStore::create() 43928/head
Joseph Sawaya [Mon, 15 Nov 2021 15:02:23 +0000 (15:02 +0000)]
crimson: futurize FuturizedStore::create()

This commit changes the FuturizedStore::create() function to return a
seastar::future containing its original return value.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>
3 years agoceph-volume: follow-up on PR #42727 43927/head
Guillaume Abrioux [Mon, 15 Nov 2021 14:34:48 +0000 (15:34 +0100)]
ceph-volume: follow-up on PR #42727

PR #42727 introduced a regression in `ceph-volume raw activate`

since it dropped `nargs='+'` from the argument `--device`, the variable
is no longer a list but a string.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
3 years agoMerge pull request #42110 from joscollin/wip-doc-cephfs-top-misc-fixes
Venky Shankar [Mon, 15 Nov 2021 06:20:22 +0000 (11:50 +0530)]
Merge pull request #42110 from joscollin/wip-doc-cephfs-top-misc-fixes

cephfs-top: fix Refresh interval

Reviewed-by: Venky Shankar <vshankar@redhat.com>
3 years agoMerge pull request #42046 from CongMinYin/align-entry-bit
Deepika Upadhyay [Sat, 13 Nov 2021 11:57:27 +0000 (17:27 +0530)]
Merge pull request #42046 from CongMinYin/align-entry-bit

librbd/cache/pwl/ssd: make log entry 64 bit and add ssd version control

Reviewed-by: Mykola Golub <mykola.golub@clyso.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
3 years agoMerge pull request #43837 from majianpeng/librbd-fix-reorder-problem-between-process_...
Deepika Upadhyay [Sat, 13 Nov 2021 11:56:24 +0000 (17:26 +0530)]
Merge pull request #43837 from majianpeng/librbd-fix-reorder-problem-between-process_writeback_dirty_entries

librbd/cache/pwl: fix reorder issue between func process_writeback_dirty_entries

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
3 years agorgw: remove prefix & delim params for bucket removal & mp upload abort 43913/head
J. Eric Ivancich [Fri, 12 Nov 2021 22:24:35 +0000 (17:24 -0500)]
rgw: remove prefix & delim params for bucket removal & mp upload abort

The calls to remove a bucket had parameters to specify a prefix and
delimiter, which does not make sense. This was precipitated due to some
existing Swift protocol logic, but buckets are removed irrespective of
prefix and delimiter. So the functions and calls are adjusted to
remove those parameters. Additionally, those same parameters were
removed for aborting incomplete multipart uploads.

Additionally a bug is fixed in which during bucket removal, multipart
uploads were only removed if the prefix was non-empty.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
3 years agomon: MonMap: display disallowed_leaders whenever they're set 43932/head
Greg Farnum [Fri, 12 Nov 2021 23:05:02 +0000 (23:05 +0000)]
mon: MonMap: display disallowed_leaders whenever they're set

In c59a6f89465e3933631afa2ba92e8c1ae1c31c06, I erroneously changed
the CLI display output so it would only dump disallowed_leaders in
stretch mode. But they can also be set in connectivity or disallow
election modes and we want users to be able to see them then as well.

Fixes: https://tracker.ceph.com/issues/53258
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
3 years agoMerge pull request #43773 from linuxbox2/wip-rgwnfs-crash
Casey Bodley [Fri, 12 Nov 2021 20:57:37 +0000 (15:57 -0500)]
Merge pull request #43773 from linuxbox2/wip-rgwnfs-crash

librgw: treat empty root path as "/" on mount

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
3 years agoMerge pull request #43563 from Zhiwei-Dai/wip-fix-mutex-unlock-twice
Casey Bodley [Fri, 12 Nov 2021 20:56:18 +0000 (15:56 -0500)]
Merge pull request #43563 from Zhiwei-Dai/wip-fix-mutex-unlock-twice

common: avoid pthread_mutex_unlock twice

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
3 years agoMerge pull request #43829 from cbodley/wip-rgw-auth-uninitialized-optional
Casey Bodley [Fri, 12 Nov 2021 20:53:10 +0000 (15:53 -0500)]
Merge pull request #43829 from cbodley/wip-rgw-auth-uninitialized-optional

rgw/auth: perm_mask uses std::optional

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
3 years agoMerge pull request #43103 from ivancich/wip-none-stats-fix
Casey Bodley [Fri, 12 Nov 2021 20:52:46 +0000 (15:52 -0500)]
Merge pull request #43103 from ivancich/wip-none-stats-fix

rgw: fix rgw.none statistics

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 years agoqa/standalone: fix scrub/osd-scrub-dump following changes to 'pg dump pgs' output 43912/head
Ronen Friedman [Fri, 12 Nov 2021 18:43:41 +0000 (18:43 +0000)]
qa/standalone: fix scrub/osd-scrub-dump following changes to 'pg dump pgs' output

Make osd-scrub-dump test ignore the 'scrubbing' that might be late to disappear
from the modified (PR #43403) 'pg dump' output.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
3 years agoMerge pull request #43877 from rhcs-dashboard/fix-spelling-network-address
Ernesto Puerta [Fri, 12 Nov 2021 18:10:23 +0000 (19:10 +0100)]
Merge pull request #43877 from rhcs-dashboard/fix-spelling-network-address

mgr/dashboard: Spelling mistake in host-form Network address field

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
3 years agorgw: make some logging easier to read 43909/head
J. Eric Ivancich [Thu, 11 Nov 2021 22:20:55 +0000 (17:20 -0500)]
rgw: make some logging easier to read

While __PRETTY_FUNCTION__ includes more information, it can clutter
the logs. So this reverts some uses of __PRETTY_FUNCTION__ back to
__func__.

I'm thinking that a strategy going forward is for the function entry
logging to use __PRETTY_FUNCTION__ to disambiguate overloaded
functions, but all others in the function simply to use __func__.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
3 years agorgw: a few coding fix-ups
J. Eric Ivancich [Thu, 11 Nov 2021 16:13:01 +0000 (11:13 -0500)]
rgw: a few coding fix-ups

Adds some code documentation and nudges code towards ceph coding
standards.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
3 years agorgw: add ability to easily display ListParams
J. Eric Ivancich [Thu, 11 Nov 2021 16:10:17 +0000 (11:10 -0500)]
rgw: add ability to easily display ListParams

During debugging it can be useful to see all the contents of
rgw::sal::Bucket::ListParams. This allows the structure to be dumped
to an output stream in human-readable format.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
3 years agorgw: fix `bi put` not using right bucket index shard 43908/head
J. Eric Ivancich [Thu, 11 Nov 2021 22:20:24 +0000 (17:20 -0500)]
rgw: fix `bi put` not using right bucket index shard

When `radosgw-admin bi put` adds an entry for an incomplete multipart
upload, the bucket index shard is not calculated correctly. It should
be based on the name of the ultimate object. However the calculation
was including the added organizational suffixes as well. This corrects
that.

NOTE: When entries are not put in the correct index shard, unordered
listing becomes unreliable, perhaps causing entries to be skipped or
infinite loops to form.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
3 years agoMerge PR #43896 into master
Patrick Donnelly [Fri, 12 Nov 2021 15:20:05 +0000 (10:20 -0500)]
Merge PR #43896 into master

* refs/pull/43896/head:
doc: document subvolume (group) pins

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
3 years agorgw/beast: reference count Connections for timeout_handler
Casey Bodley [Thu, 11 Nov 2021 17:01:06 +0000 (12:01 -0500)]
rgw/beast: reference count Connections for timeout_handler

resolves a use-after-free in the timeout_handler, where a timeout fires
and schedules the timeout_handler for execution, but the coroutine exits
and destroys the socket before asio executes the timeout_handler

timeout_handler now holds a reference on the Connection to extend its
lifetime

now that the Connection is allocated on the heap, we can include the
parse_buffer in this memory instead of allocating it separately

Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 years agorgw/beast: replace beast::tcp_stream with manual timeouts
Casey Bodley [Sat, 30 Oct 2021 23:47:02 +0000 (19:47 -0400)]
rgw/beast: replace beast::tcp_stream with manual timeouts

remove the beast::tcp_stream wrapper from the socket, and track timeouts
manually with a timeout_timer. this timer uses ceph's coarse_mono_clock
which is cheaper to sample than std::chrono::steady_clock

Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 years agorgw/beast: use explicit executor type for tcp socket and stream
Casey Bodley [Mon, 1 Nov 2021 17:18:54 +0000 (13:18 -0400)]
rgw/beast: use explicit executor type for tcp socket and stream

Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 years agospawn: use explicit strand executor
Casey Bodley [Mon, 1 Nov 2021 17:14:16 +0000 (13:14 -0400)]
spawn: use explicit strand executor

the default spawn::yield_context uses the polymorphic boost::asio::executor
to support any executor type

rgw's beast frontend always uses the same executor type for these
coroutines, so we can use that type directly to avoid the overhead of
type erasure and virtual function calls

Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 years agorgw: clean up WITH_RADOSGW_BEAST_OPENSSL
Casey Bodley [Sat, 30 Oct 2021 23:45:45 +0000 (19:45 -0400)]
rgw: clean up WITH_RADOSGW_BEAST_OPENSSL

the #ifdef was covering more includes than it should have

Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 years agoqa/suites/rbd/persistent-writeback-cache: add test case 42046/head
Yin Congmin [Fri, 12 Nov 2021 08:54:31 +0000 (16:54 +0800)]
qa/suites/rbd/persistent-writeback-cache: add test case

Add the test case which size is 8GB, So that some problems that occur
only in test scenarios above 4GB may be found in this test. For example,
the variables of 32-bit may be unexpected value when it operates with
a 64 bit value.

Signed-off-by: Yin Congmin <congmin.yin@intel.com>
3 years agomgr/dashboard: Edit a service feature 43903/head
Nizamudeen A [Fri, 12 Nov 2021 08:14:51 +0000 (13:44 +0530)]
mgr/dashboard: Edit a service feature

Fixes: https://tracker.ceph.com/issues/53077
Signed-off-by: Nizamudeen A <nia@redhat.com>
3 years agoMerge pull request #43570 from ljflores/wip-pg-stats
Neha Ojha [Thu, 11 Nov 2021 19:51:59 +0000 (11:51 -0800)]
Merge pull request #43570 from ljflores/wip-pg-stats

mgr/telemetry: modify stats_per_pool and add stats_per_pg

Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>
3 years agoqa/suites/orch/cephadm: verify that 'orch ls' reports OSDs properly 43894/head
Sage Weil [Thu, 11 Nov 2021 17:59:57 +0000 (12:59 -0500)]
qa/suites/orch/cephadm: verify that 'orch ls' reports OSDs properly

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agomgr/cephadm: show unmanaged OSDs under 'osd' service
Sage Weil [Thu, 11 Nov 2021 16:42:32 +0000 (11:42 -0500)]
mgr/cephadm: show unmanaged OSDs under 'osd' service

1- If the unit.meta file service_name = osd.NNN (which is true for lots of
OSDs deployed on older version of cephadm) then ignore the field entirely.

2- If an OSD has not service_name (see above) then show it under the 'osd'
service (instead of 'osd.unmanaged').

Sample 'ceph orch ls' output with a drivegroup + unmanaged OSD:

NAME      PORTS  RUNNING  REFRESHED  AGE  PLACEMENT
...
osd                    1  85s ago    -    <unmanaged>
osd.hdds               5  85s ago    2s   *
...

Fixes: https://tracker.ceph.com/issues/53235
Signed-off-by: Sage Weil <sage@newdream.net>
3 years agoMerge pull request #43587 from zenomri/wip-omri-tracer-opentelemetry
Yuval Lifshitz [Thu, 11 Nov 2021 17:40:12 +0000 (19:40 +0200)]
Merge pull request #43587 from zenomri/wip-omri-tracer-opentelemetry

common/tracer: Tracer implementation using opentelemetry sdk

3 years agodoc: document subvolume (group) pins 43896/head
Patrick Donnelly [Thu, 11 Nov 2021 17:30:02 +0000 (12:30 -0500)]
doc: document subvolume (group) pins

Fixes: https://tracker.ceph.com/issues/53236
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agoMerge pull request #43464 from rsommer/wip-prometheus-standby-behaviour
Ernesto Puerta [Thu, 11 Nov 2021 16:36:30 +0000 (17:36 +0100)]
Merge pull request #43464 from rsommer/wip-prometheus-standby-behaviour

mgr/prometheus: Make prometheus standby behaviour configurable

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
3 years agoMerge pull request #43874 from liewegas/qa-podman-add-stream
Sebastian Wagner [Thu, 11 Nov 2021 15:43:34 +0000 (16:43 +0100)]
Merge pull request #43874 from liewegas/qa-podman-add-stream

qa/suites/orch/cephadm: add 8.stream + container_tools

Reviewed-by: Adam King <adking@redhat.com>
3 years agoMerge PR #43046 into master
Sage Weil [Thu, 11 Nov 2021 15:31:22 +0000 (10:31 -0500)]
Merge PR #43046 into master

* refs/pull/43046/head:
mgr/rook: get running pods, auth rm, better error checking for orch nfs
qa/tasks/rook: add apply nfs to rook qa task
mgr/rook: prevent creation of NFS clusters not in .nfs rados pool
mgr/rook, mgr/nfs: update rook orchestrator to create and use .nfs pool

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Varsha Rao <rvarsha016@gmail.com>
3 years agoMerge PR #43851 into master
Patrick Donnelly [Thu, 11 Nov 2021 15:17:56 +0000 (10:17 -0500)]
Merge PR #43851 into master

* refs/pull/43851/head:
mds/FSMap: allow upgrade when no MDS is "in"

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
3 years agoMerge PR #43800 into master
Patrick Donnelly [Thu, 11 Nov 2021 15:14:50 +0000 (10:14 -0500)]
Merge PR #43800 into master

* refs/pull/43800/head:
pybind/mgr/cephadm: disable FSMap sanity checks during MDS upgrade
mds/FSMap: assign v16.2.4 compat to pre-v16.2.5 standby daemons

Reviewed-by: Jeff Layton <jlayton@redhat.com>
3 years agocephfs-top, doc: fix Refresh interval 42110/head
Jos Collin [Tue, 22 Jun 2021 11:09:22 +0000 (16:39 +0530)]
cephfs-top, doc: fix Refresh interval

Signed-off-by: Jos Collin <jcollin@redhat.com>
3 years agorgw/dbstore: LC APIs definition 43433/head
Soumya Koduri [Tue, 19 Oct 2021 17:57:56 +0000 (23:27 +0530)]
rgw/dbstore: LC APIs definition

Implement methods to add & fetch LCEntries in dbstore.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
3 years agorgw/dbstore: APIs to handle lc state
Soumya Koduri [Mon, 11 Oct 2021 12:55:02 +0000 (18:25 +0530)]
rgw/dbstore: APIs to handle lc state

Add placeholder APIs to create and store LC related state.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
3 years agomgr/prometheus: Make standby discoverable 43464/head
Roland Sommer [Fri, 8 Oct 2021 06:40:26 +0000 (08:40 +0200)]
mgr/prometheus: Make standby discoverable

Enable config settings to modify standby's behaviour on the index page
This makes the standby discoverable by reverse proxy or loadbalancer
setups. Testing for the empty response of the '/metrics' endpoint would
trigger metric collection on the active manager instance.

The newly added configuration options settings standby_behaviour and
standby_error_status_code are documented and flagged as runtime, as
modifying both settings has an immediate effect (no restart required).

Co-authored-by: Ernesto Puerta <37327689+epuertat@users.noreply.github.com>
Signed-off-by: Roland Sommer <rol@ndsommer.de>
Fixes: https://tracker.ceph.com/issues/53229
3 years agoMerge pull request #43521 from ronen-fr/wip-rf-scrub-noscrub
Ronen Friedman [Thu, 11 Nov 2021 07:25:41 +0000 (09:25 +0200)]
Merge pull request #43521 from ronen-fr/wip-rf-scrub-noscrub

osd/scrub: fix the handling of deep-scrub when noscrub is set

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
3 years agoMerge pull request #43860 from rzarzynski/wip-crimson-msgr-drop-extra-clientident... 43884/head
Samuel Just [Thu, 11 Nov 2021 00:04:55 +0000 (16:04 -0800)]
Merge pull request #43860 from rzarzynski/wip-crimson-msgr-drop-extra-clientident-check

crimson/net: drop crimson-specific check for the addr in ClientIdentFrame

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #43783 from pcuzner/alert-fixes
Sebastian Wagner [Wed, 10 Nov 2021 21:32:51 +0000 (22:32 +0100)]
Merge pull request #43783 from pcuzner/alert-fixes

mgr/prometheus: Update rule format and enhance SNMP support

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
3 years agoMerge pull request #43845 from rhcs-dashboard/add-mfa-ids-in-rgw-user-details
Ernesto Puerta [Wed, 10 Nov 2021 20:52:11 +0000 (21:52 +0100)]
Merge pull request #43845 from rhcs-dashboard/add-mfa-ids-in-rgw-user-details

mgr/dashboard: include mfa_ids in rgw user-details section

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
3 years agoMerge pull request #43411 from ljflores/wip-mgr-command-cleanup
Laura Flores [Wed, 10 Nov 2021 20:09:51 +0000 (14:09 -0600)]
Merge pull request #43411 from ljflores/wip-mgr-command-cleanup

mon: simplify 'mgr module ls' output

3 years agolibrgw: treat empty root path as "/" on mount 43773/head
Matt Benjamin [Tue, 2 Nov 2021 13:47:31 +0000 (09:47 -0400)]
librgw: treat empty root path as "/" on mount

This prevents an invalid access (and possible crash) when path is
given as "" in ganesha.conf.

Fixes: https://tracker.ceph.com/issues/53030
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>