]> git.apps.os.sepia.ceph.com Git - ceph-ci.git/log
ceph-ci.git
3 years agomds: add truncate size handling support for fscrypt
Xiubo Li [Thu, 14 Oct 2021 04:45:20 +0000 (12:45 +0800)]
mds: add truncate size handling support for fscrypt

The kclient will only send truncate requests with the modified and
encrypted last block contents when new size is smaller and is not
aligned to CEPH_FSCRYPT_BLOCK_SIZE, which is 4KB for now.

Or if the Fx caps is issued and the new size is larger the kclient
will buffer the truncating. Or it will send truncate requests wihtout
the last block filled.

When the fscrypt is enabled and when truncating with a smaller size,
both the old size and new size in the truncate request will always
be rounded up to CEPH_FSCRYPT_BLOCK_SIZE, which is 4K for now, in
kclient. For example if truncating a file size from 3KB to 2KB, the
MDS will always get old_size == new_size == 4KB. So we need to check
whether there has last block data passed together with the truncate
request to make sure whether truncating to a smaller size.

The kclinet will send it's 'change_attr' along with the truncate req,
and the MDS will compare it with the one in CInode just after the MDS
successfully xlockes the CInode's filelock, if they are different
that means it's possibly some clients have update the file or have
dirty caps just before MDS xlockes the CInode's filelock. We will let
the kclient retry it by returning a -EAGAIN errno.

Then the MDS will write the last block to OSD and then truncate
the size as normal.

Currently the last block contents will be journaled together with
the project inode only and it will be cleared after the truncate
being finished, and won't make it persistent together with the
CInode:inode_t in the metadata pool.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
3 years agoclient: userland clients are only allowed to read if fscrypt is enabled
Xiubo Li [Fri, 26 Nov 2021 02:23:25 +0000 (10:23 +0800)]
client: userland clients are only allowed to read if fscrypt is enabled

For now the userland clients hasn't support the fscrypt yet, they
will be only allowed to read the encrypted files.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
3 years agoclient: trust the fscrypt_file as the inode size and truncate caches
Xiubo Li [Fri, 26 Nov 2021 02:20:12 +0000 (10:20 +0800)]
client: trust the fscrypt_file as the inode size and truncate caches

Signed-off-by: Xiubo Li <xiubli@redhat.com>
3 years agomds: move the class MDCacheIOContext to MDCache.h
Xiubo Li [Thu, 14 Oct 2021 04:45:20 +0000 (12:45 +0800)]
mds: move the class MDCacheIOContext to MDCache.h

Signed-off-by: Xiubo Li <xiubli@redhat.com>
3 years agomds: add mds_fscrypt_last_block_max_size option support
Xiubo Li [Thu, 25 Nov 2021 00:52:37 +0000 (08:52 +0800)]
mds: add mds_fscrypt_last_block_max_size option support

The maximum size of the last block without the header along with
a truncate request when the fscrypt is enabled. Default value is
4KB.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
3 years agomds: send the fscrypt_auth/fscrypt_file to clients in MClientCaps
Xiubo Li [Wed, 24 Nov 2021 02:19:21 +0000 (10:19 +0800)]
mds: send the fscrypt_auth/fscrypt_file to clients in MClientCaps

The clients will trust and need fscrypt_file field to truncate the
pagecaches and update the i_size.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
3 years agomds: don't allow changing layout on encrypted files/directories
Luis Henriques [Thu, 12 Aug 2021 12:46:20 +0000 (13:46 +0100)]
mds: don't allow changing layout on encrypted files/directories

Encryption is currently only supported on files/directories with layouts
where stripe_count=1.  Forbid changing layouts when encryption is involved.

Signed-off-by: Luis Henriques <lhenriques@suse.de>
3 years agomds: encode fscrypt_auth and fscrypt_file in appropriate mds locks
Jeff Layton [Mon, 21 Jun 2021 20:00:03 +0000 (16:00 -0400)]
mds: encode fscrypt_auth and fscrypt_file in appropriate mds locks

Signed-off-by: Jeff Layton <jlayton@redhat.com>
3 years agotest: add fscrypt attribute testcases
Jeff Layton [Mon, 10 May 2021 13:19:10 +0000 (09:19 -0400)]
test: add fscrypt attribute testcases

Create a file and set both fscrypt_auth and fscrypt_file on it. Verify
that they are set, remount and verify it again.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
3 years agoclient: add support for fscrypt_auth and fscrypt_file fields
Jeff Layton [Tue, 11 May 2021 15:26:38 +0000 (11:26 -0400)]
client: add support for fscrypt_auth and fscrypt_file fields

We don't really expect the userland client to use these (at least not
initially) but plumb in new vxattrs for fetching and setting these
fields. The new vxattr just fetches whatever is in the in-core inode,
and setting it issues a setattr under the hood.

Also, ensure that we update these fields on cap updates and client
requests.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
3 years agomds: allow setattr to change fscrypt_auth/file
Jeff Layton [Wed, 5 May 2021 12:54:16 +0000 (08:54 -0400)]
mds: allow setattr to change fscrypt_auth/file

Add new mask bits for those fields and update them if a setattr request
comes in with them.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
3 years agomds: populate fscrypt_auth/_file on inode creation
Jeff Layton [Wed, 5 May 2021 12:18:36 +0000 (08:18 -0400)]
mds: populate fscrypt_auth/_file on inode creation

Signed-off-by: Jeff Layton <jlayton@redhat.com>
3 years agoclient: send fscrypt_auth and fscrypt_file in MClientRequest
Jeff Layton [Thu, 15 Apr 2021 17:04:38 +0000 (13:04 -0400)]
client: send fscrypt_auth and fscrypt_file in MClientRequest

...and update the fscrypt_file field on setattr changes for size.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
3 years agomds: add fscrypt_auth and fscrypt_file fields to MClientCaps
Jeff Layton [Wed, 14 Apr 2021 15:04:15 +0000 (11:04 -0400)]
mds: add fscrypt_auth and fscrypt_file fields to MClientCaps

...and update those fields when the appropriate caps are
acquired/released/flushed.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
3 years agomds: encode fscrypt_auth and fscrypt_file in InodeStat
Jeff Layton [Mon, 21 Jun 2021 19:59:18 +0000 (15:59 -0400)]
mds: encode fscrypt_auth and fscrypt_file in InodeStat

Signed-off-by: Jeff Layton <jlayton@redhat.com>
3 years agomds: add fscrypt opaque field to inode_t encoding
Jeff Layton [Thu, 15 Apr 2021 18:37:58 +0000 (14:37 -0400)]
mds: add fscrypt opaque field to inode_t encoding

Add fscrypt_auth and fscrypt_file attributes to the inode_t encoding.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
3 years agomds: convert fscrypt flag to two opaque fields
Jeff Layton [Thu, 29 Apr 2021 17:33:53 +0000 (13:33 -0400)]
mds: convert fscrypt flag to two opaque fields

A flag isn't sufficient as we can't reasonably use an xattr to store the
context. Switch the fscrypt fields to two vectors of opaque bytes, one
governed by AUTH caps and the other by FILE.

Also remove the special handling for encryption.ctx xattr, since we
won't be using that going forward anyway.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
3 years agocommon: add encode and decode routines for opaque vectors of bytes
Jeff Layton [Tue, 4 May 2021 18:22:43 +0000 (14:22 -0400)]
common: add encode and decode routines for opaque vectors of bytes

Signed-off-by: Jeff Layton <jlayton@redhat.com>
3 years agoMerge pull request #43286 from lxbsz/improve_setattr
Venky Shankar [Thu, 13 Jan 2022 12:53:27 +0000 (18:23 +0530)]
Merge pull request #43286 from lxbsz/improve_setattr

client: buffer the truncate if we have the Fx caps

Reviewed-by: Venky Shankar <vshankar@redhat.com>
3 years agoMerge pull request #44229 from lxbsz/mds-buffix
Venky Shankar [Thu, 13 Jan 2022 12:46:13 +0000 (18:16 +0530)]
Merge pull request #44229 from lxbsz/mds-buffix

mds: remove the duplicated or incorrect respond

Reviewed-by: Venky Shankar <vshankar@redhat.com>
3 years agoMerge pull request #44397 from lxbsz/wip-53726
Venky Shankar [Thu, 13 Jan 2022 12:45:24 +0000 (18:15 +0530)]
Merge pull request #44397 from lxbsz/wip-53726

mds: dump tree '/' when the path is empty

Reviewed-by: Venky Shankar <vshankar@redhat.com>
3 years agoMerge pull request #44422 from lxbsz/wip-51705
Venky Shankar [Thu, 13 Jan 2022 12:44:14 +0000 (18:14 +0530)]
Merge pull request #44422 from lxbsz/wip-51705

qa: do not use any time related suffix for *_op_timeouts

Reviewed-by: Venky Shankar <vshankar@redhat.com>
3 years agoMerge pull request #43995 from TRYTOBE8TME/wip-rgw-kafka-teuth-cleanup
Yuval Lifshitz [Thu, 13 Jan 2022 09:57:03 +0000 (11:57 +0200)]
Merge pull request #43995 from TRYTOBE8TME/wip-rgw-kafka-teuth-cleanup

qa/tasks: Checking for kafka cleanup

3 years agoMerge pull request #43299 from markhpc/wip-age-binning-rebase-20210923
Yuri Weinstein [Thu, 13 Jan 2022 00:54:23 +0000 (16:54 -0800)]
Merge pull request #43299 from markhpc/wip-age-binning-rebase-20210923

common/PriorityCache: Updated Implementation of Cache Age Binning

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
3 years agoMerge pull request #43494 from majianpeng/enable-test-librbd-BlockGuard
Ilya Dryomov [Wed, 12 Jan 2022 20:50:00 +0000 (21:50 +0100)]
Merge pull request #43494 from majianpeng/enable-test-librbd-BlockGuard

test/librbd: re-enable BlockGuard test

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
3 years agoMerge pull request #44478 from cyx1231st/wip-crimson-improve-log-3
Samuel Just [Wed, 12 Jan 2022 20:27:14 +0000 (12:27 -0800)]
Merge pull request #44478 from cyx1231st/wip-crimson-improve-log-3

crimson/os/seastore/../segment_manager: improve logs and validations

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
3 years agoMerge pull request #44536 from yuvalif/wip-yuval-dynamic-reshard
Casey Bodley [Wed, 12 Jan 2022 19:14:41 +0000 (14:14 -0500)]
Merge pull request #44536 from yuvalif/wip-yuval-dynamic-reshard

rgw: fix dynamic reshard happening during user stats sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 years agoMerge pull request #39440 from pdvian/wip-warn-filestore-osds
Yuri Weinstein [Wed, 12 Jan 2022 16:49:02 +0000 (08:49 -0800)]
Merge pull request #39440 from pdvian/wip-warn-filestore-osds

mon/OSDMonitor, osd: Add warning on filestore deprecation and force use of wpq scheduler for filestore OSDs

Reviewed-by: Neha Ojha <nojha@redhat.com>
3 years agoMerge pull request #41778 from felixhuettner/fix_subdir_name
Daniel Gryniewicz [Wed, 12 Jan 2022 14:22:04 +0000 (09:22 -0500)]
Merge pull request #41778 from felixhuettner/fix_subdir_name

rgw: RGWSwiftWebsiteHandler::is_web_dir checks empty subdir_name

3 years agoMerge pull request #38234 from inspur-wyq/wip-copy-obj-check-size
Daniel Gryniewicz [Wed, 12 Jan 2022 14:21:48 +0000 (09:21 -0500)]
Merge pull request #38234 from inspur-wyq/wip-copy-obj-check-size

rgw : check the object size when copy obj

3 years agoMerge pull request #38532 from Rjerk/wip-empty-tagset
Daniel Gryniewicz [Wed, 12 Jan 2022 14:21:35 +0000 (09:21 -0500)]
Merge pull request #38532 from Rjerk/wip-empty-tagset

rgw: an empty tagset is allowed by S3

3 years agoMerge pull request #40573 from Huber-ming/rgw_admin
Daniel Gryniewicz [Wed, 12 Jan 2022 14:21:15 +0000 (09:21 -0500)]
Merge pull request #40573 from Huber-ming/rgw_admin

rgw: delete abbreviation for option "--new-uid"

3 years agoMerge pull request #40575 from Huber-ming/rgw_admin-f
Daniel Gryniewicz [Wed, 12 Jan 2022 14:20:49 +0000 (09:20 -0500)]
Merge pull request #40575 from Huber-ming/rgw_admin-f

radosgw-admin: delete the abbreviation of option "--infile"

3 years agoMerge pull request #44500 from idryomov/wip-rbd-test-group-leak
Ilya Dryomov [Wed, 12 Jan 2022 11:56:09 +0000 (12:56 +0100)]
Merge pull request #44500 from idryomov/wip-rbd-test-group-leak

test/librbd: fix group_info.name leaks in TestGroup.add_image

Reviewed-by: Mykola Golub <mgolub@suse.com>
3 years agoMerge pull request #44490 from liu-chunmei/crimson-fix-aligned
Liu-Chunmei [Wed, 12 Jan 2022 06:05:22 +0000 (22:05 -0800)]
Merge pull request #44490 from liu-chunmei/crimson-fix-aligned

crimson: fix assert_aligned(size) in trim_data_reservation

reviewed by: Samuel Just <sjust@redhat.com> , Yingxin <yingxin.cheng@intel.com>

3 years agocrimson/os/seastore/../segment_manager: add more validations
Yingxin Cheng [Thu, 6 Jan 2022 08:03:20 +0000 (16:03 +0800)]
crimson/os/seastore/../segment_manager: add more validations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore/../segment_manager: consolidate logs with structured level and...
Yingxin Cheng [Thu, 6 Jan 2022 08:01:28 +0000 (16:01 +0800)]
crimson/os/seastore/../segment_manager: consolidate logs with structured level and format

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore/../segment_manager: cleanup device_id usage
Yingxin Cheng [Thu, 6 Jan 2022 02:26:20 +0000 (10:26 +0800)]
crimson/os/seastore/../segment_manager: cleanup device_id usage

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore/../segment_manager: pretty print data structures
Yingxin Cheng [Wed, 5 Jan 2022 14:56:13 +0000 (22:56 +0800)]
crimson/os/seastore/../segment_manager: pretty print data structures

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore/../segment_manager: convert to seastore logging
Yingxin Cheng [Wed, 5 Jan 2022 05:32:46 +0000 (13:32 +0800)]
crimson/os/seastore/../segment_manager: convert to seastore logging

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore/../segment_manager: suppress compile warning about unused logger
Yingxin Cheng [Wed, 5 Jan 2022 03:22:09 +0000 (11:22 +0800)]
crimson/os/seastore/../segment_manager: suppress compile warning about unused logger

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agoMerge pull request #44532 from rzarzynski/wip-crimson-fix-test-runner
Yingxin [Wed, 12 Jan 2022 01:23:53 +0000 (09:23 +0800)]
Merge pull request #44532 from rzarzynski/wip-crimson-fix-test-runner

test/crimson: fix a race condition in SeastarRunner

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #43593 from ljflores/wip-rocksdb
Neha Ojha [Wed, 12 Jan 2022 00:31:41 +0000 (16:31 -0800)]
Merge pull request #43593 from ljflores/wip-rocksdb

mgr: expose rocksdb version number for use in telemetry

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
3 years agocrimson: fix assert_aligned(size) in trim_data_reservation
chunmei-liu [Tue, 11 Jan 2022 03:44:43 +0000 (19:44 -0800)]
crimson: fix assert_aligned(size) in trim_data_reservation

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
signed-off-by: Samuel Just <sjust@redhat.com>

3 years agomgr/telemetry: add the rocksdb version number to telemetry
Laura Flores [Tue, 4 Jan 2022 22:54:33 +0000 (22:54 +0000)]
mgr/telemetry: add the rocksdb version number to telemetry

Capturing the RocksDB version number in Telemetry would allow us to check that users are using the appropriate RocksDB version for their Ceph cluster. For instance, if a user is working in a Pacific cluster, but their RocksDB version is meant for Nautilus, that might be a problem.

It is strucured as "rocksdb_stats" --> "version" in anticipation of more stats that can will be added under "rocksdb_stats".

Signed-off-by: Laura Flores <lflores@redhat.com>
3 years agomgr: expose rocksdb version number in the mgr module
Laura Flores [Wed, 22 Dec 2021 23:09:25 +0000 (23:09 +0000)]
mgr: expose rocksdb version number in the mgr module

It is only necessary here to link the rocksdb include directory
since the mgr simply needs access to the rocksdb version numbers.

Signed-off-by: Laura Flores <lflores@redhat.com>
Co-authored-by: Kefu Chai <tchaikov@gmail.com>
Co-authored-by: Adam Kupczyk <akupczyk@redhat.com>
3 years agoMerge pull request #43794 from aclamk/wip-bluefs-fine-grain-locking-4
Neha Ojha [Tue, 11 Jan 2022 21:59:16 +0000 (13:59 -0800)]
Merge pull request #43794 from aclamk/wip-bluefs-fine-grain-locking-4

os/bluestore: BlueFS fine grain locking

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
3 years agocommon/PriorityCache: Implement Cache Age Binning
Mark Nelson [Tue, 13 Oct 2020 23:24:18 +0000 (23:24 +0000)]
common/PriorityCache: Implement Cache Age Binning

Signed-off-by: Mark Nelson <mnelson@redhat.com>
3 years agoMerge pull request #44537 from pdvian/wip-typo-fix
Casey Bodley [Tue, 11 Jan 2022 20:50:34 +0000 (15:50 -0500)]
Merge pull request #44537 from pdvian/wip-typo-fix

rgw: Fix typo error for stat_remote_obj

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 years agoMerge pull request #44496 from idryomov/wip-diff-iterate-parent-fix
Ilya Dryomov [Tue, 11 Jan 2022 19:28:19 +0000 (20:28 +0100)]
Merge pull request #44496 from idryomov/wip-diff-iterate-parent-fix

librbd: restore diff-iterate include_parent functionality in fast-diff mode

Reviewed-by: Mykola Golub <mgolub@suse.com>
3 years agorgw: Fix typo error for stat_remote_obj
Prashant D [Tue, 11 Jan 2022 18:58:20 +0000 (13:58 -0500)]
rgw: Fix typo error for stat_remote_obj

Signed-off-by: Prashant D <pdhange@redhat.com>
3 years agorgw: fix dynamic reshard happening during user stats sync
Yuval Lifshitz [Mon, 10 Jan 2022 16:55:50 +0000 (18:55 +0200)]
rgw: fix dynamic reshard happening during user stats sync

this is a regression caused by the following commit:
https://github.com/ceph/ceph/commit/72d1a363263cf707d022ee756122236ba175cda2

to verify use the following instructions:
https://gist.github.com/yuvalif/d526c0a3a4c5b245b9e951a6c5a10517

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
3 years agoMerge pull request #44483 from idryomov/wip-diff-iterate-offset-fix
Ilya Dryomov [Tue, 11 Jan 2022 17:59:12 +0000 (18:59 +0100)]
Merge pull request #44483 from idryomov/wip-diff-iterate-offset-fix

librbd: diff-iterate reports incorrect offsets in fast-diff mode

Reviewed-by: Mykola Golub <mgolub@suse.com>
3 years agoMerge pull request #44355 from sseshasa/wip-fix-enhance-mclock-logging
Yuri Weinstein [Tue, 11 Jan 2022 17:32:12 +0000 (09:32 -0800)]
Merge pull request #44355 from sseshasa/wip-fix-enhance-mclock-logging

osd:  Enhance mclockscheduler logging and formatted dump information.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
3 years agoMerge pull request #44482 from sebastian-philipp/nfs-upgrade-port-conflict
Yuri Weinstein [Tue, 11 Jan 2022 17:31:35 +0000 (09:31 -0800)]
Merge pull request #44482 from sebastian-philipp/nfs-upgrade-port-conflict

mgr/cephadm: nfs migration: avoid port conflicts

Reviewed-by: Adam King adking@redhat.com
3 years agoMerge pull request #36308 from linuxbox2/wip-rgwlc-bcktdel
Matt Benjamin [Tue, 11 Jan 2022 17:17:22 +0000 (12:17 -0500)]
Merge pull request #36308 from linuxbox2/wip-rgwlc-bcktdel

rgwlc:  remove lc entry on bucket delete

3 years agoMerge pull request #42891 from ofriedma/wip-ofriedma-rgw-qos-finale
Casey Bodley [Tue, 11 Jan 2022 16:35:05 +0000 (11:35 -0500)]
Merge pull request #42891 from ofriedma/wip-ofriedma-rgw-qos-finale

rgw: Add rgw rate limiting per user and per bucket

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 years agoMerge pull request #38774 from linuxbox2/wip-doc-replsupport
Casey Bodley [Tue, 11 Jan 2022 16:13:24 +0000 (11:13 -0500)]
Merge pull request #38774 from linuxbox2/wip-doc-replsupport

rgw: document S3 bucket replication support

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 years agoqa/tasks: Checking for kafka cleanup
Kalpesh Pandya [Thu, 11 Nov 2021 06:46:16 +0000 (12:16 +0530)]
qa/tasks: Checking for kafka cleanup

Adding a sleep after running ./kafka-server-stop.sh and ./zookeeper-server-stop.sh
scripts so that nothing gets logged into the kafka logs after the sleep time.
And finally killing the process.

This resolves: https://tracker.ceph.com/issues/53220

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>
3 years agotest/crimson: fix a race condition in SeastarRunner
Radoslaw Zarzynski [Tue, 11 Jan 2022 12:09:28 +0000 (12:09 +0000)]
test/crimson: fix a race condition in SeastarRunner

This patch is supposed to fix the following problem:

```
        Start 234: unittest-seastar-errorator
216/258 Test #234: unittest-seastar-errorator ................Child aborted***Exception:   0.95 sec
WARNING: debug mode. Not for benchmarking or production
WARN  2022-01-11 10:01:32,973 [shard 0] seastar - Creation of perf_event based stall detector failed, falling back to posix timer: std::system_error (error system:13, perf_event_open() failed: Permission denied)
unittest-seastar-errorator: ../src/test/crimson/seastar_runner.h:45: int SeastarRunner::init(int, char **): Assertion `begin_signaled == true' failed.
Aborting.
Backtrace:
  0xb3397a
  0x1d19430
  0x1d1905d
  0x1b72332
  0x1b9dd95
  0x1c75fe9
  0x1c76231
  0x1c7605a
  0x7f20b7aee3bf
  /lib/x86_64-linux-gnu/libc.so.6+0x4618a
  /lib/x86_64-linux-gnu/libc.so.6+0x25858
  /lib/x86_64-linux-gnu/libc.so.6+0x25728
  /lib/x86_64-linux-gnu/libc.so.6+0x36f35
  0xc8bb88
  0xc7ec71
  /lib/x86_64-linux-gnu/libc.so.6+0x270b2
  0xafc95d
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agoMerge pull request #43404 from rhcs-dashboard/angular-12-upgradee
Ernesto Puerta [Tue, 11 Jan 2022 12:02:07 +0000 (13:02 +0100)]
Merge pull request #43404 from rhcs-dashboard/angular-12-upgradee

mgr/dashboard: Update Angular version to 12

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
3 years agoMerge pull request #44088 from ceph/feature-48388-cache
Ernesto Puerta [Tue, 11 Jan 2022 11:52:41 +0000 (12:52 +0100)]
Merge pull request #44088 from ceph/feature-48388-cache

mgr: TTL cache implementation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
3 years agoMerge pull request #44294 from rhcs-dashboard/feature-bluestore-onode
Ernesto Puerta [Tue, 11 Jan 2022 10:24:21 +0000 (11:24 +0100)]
Merge pull request #44294 from rhcs-dashboard/feature-bluestore-onode

mgr/dashboard: monitoring:Implement BlueStore onode hit/miss counters into the dashboard

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: neha-ojha <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
3 years agoMerge pull request #44456 from jtlayton/wip-53765
Venky Shankar [Tue, 11 Jan 2022 09:09:16 +0000 (14:39 +0530)]
Merge pull request #44456 from jtlayton/wip-53765

mount.ceph: fix the handling of new-syntax device names

Reviewed-by: Venky Shankar <vshankar@redhat.com>
3 years agoosd: Display scheduler specific info when dumping an OpSchedulerItem
Sridhar Seshasayee [Fri, 17 Dec 2021 12:30:05 +0000 (18:00 +0530)]
osd: Display scheduler specific info when dumping an OpSchedulerItem

Implement logic to dump information relevant to the scheduler type being
employed when dumping details about an OpSchedulerItem. For e.g., the
'priority' field is relevant for the 'wpq' scheduler, but for the
'mclock_scheduler', the 'qos_cost' gives more information during debugging.

A couple of additional fields called 'qos_cost' and 'is_qos_request' are
introduced in OpSchedulerItem class. These are mainly used to facilitate
dumping of relevant information depending on the scheduler type. The
interesting points are when an item is enqueued and dequeued.

For the 'mclock_scheduler', the 'class_id' and the 'qos_cost' fields are
dumped during enqueue and dequeue op respectively. For the 'wpq' scheduler
things remain the same as before.

An additional benefit of this change is to help immediately identify the
type of scheduler being used for a given shard depending on what is dumped
in the debug messages while debugging.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
3 years agoosd: Add debug logs and formatted dumps in the mClockScheduler
Sridhar Seshasayee [Tue, 14 Dec 2021 10:41:14 +0000 (16:11 +0530)]
osd: Add debug logs and formatted dumps in the mClockScheduler

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
3 years agoosd: Change description of 'dump_op_pq_state' command to be more generic
Sridhar Seshasayee [Thu, 6 Jan 2022 09:58:21 +0000 (15:28 +0530)]
osd: Change description of 'dump_op_pq_state' command to be more generic

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
3 years agocommon: Introduce 'mclock' debug subsystem
Sridhar Seshasayee [Thu, 6 Jan 2022 09:56:20 +0000 (15:26 +0530)]
common: Introduce 'mclock' debug subsystem

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
3 years agoMerge pull request #44410 from tchaikov/wip-cmake-cleanup
Samuel Just [Mon, 10 Jan 2022 21:36:09 +0000 (13:36 -0800)]
Merge pull request #44410 from tchaikov/wip-cmake-cleanup

cmake: detect linux/blk/zoned support

Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #44502 from yuvalif/fix-call-to-mrgw
Yuval Lifshitz [Mon, 10 Jan 2022 20:05:22 +0000 (22:05 +0200)]
Merge pull request #44502 from yuvalif/fix-call-to-mrgw

rgw/multisite/test: pass zero (no) ssl port to mrgw.sh

3 years agoMerge pull request #44495 from Matan-B/wip-matanb-crimson-flavor
Matan [Mon, 10 Jan 2022 18:30:18 +0000 (20:30 +0200)]
Merge pull request #44495 from Matan-B/wip-matanb-crimson-flavor

qa/suites: Crimson flavor configuration fix

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: chunmei-liu chunmei.liu@intel.com
3 years agotest/librbd: make diff-iterate clone tests exercise fast-diff mode
Ilya Dryomov [Fri, 7 Jan 2022 12:31:08 +0000 (13:31 +0100)]
test/librbd: make diff-iterate clone tests exercise fast-diff mode

The fast-diff feature wasn't propagated to the clone so these tests
were exercising the slow list_snaps path no matter what RBD_FEATURES
value was supplied to ceph_test_librbd.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
3 years agolibrbd: restore diff-iterate include_parent functionality in fast-diff mode
Ilya Dryomov [Wed, 5 Jan 2022 19:24:40 +0000 (20:24 +0100)]
librbd: restore diff-iterate include_parent functionality in fast-diff mode

Commit 4429ed4f3f4c ("librbd: switch diff iterate API to use new snaps
list dispatch methods") removed the recursive execute() call.  The new
list_snaps method does indeed handle parent diffs internally but it is
not used in fast-diff mode.  Nothing changed there -- we still need to
load the parent object map, calculate parent object_diff_state, etc.

Fixes: https://tracker.ceph.com/issues/53787
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
3 years agolibrbd: stash unmodified include_parent value in DiffContext
Ilya Dryomov [Wed, 5 Jan 2022 18:45:50 +0000 (19:45 +0100)]
librbd: stash unmodified include_parent value in DiffContext

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
3 years agotest/librbd: fix group_info.name leaks in TestGroup.add_image
Ilya Dryomov [Sat, 8 Jan 2022 13:18:32 +0000 (14:18 +0100)]
test/librbd: fix group_info.name leaks in TestGroup.add_image

Introduced in commit c04e8c3374e0 ("test/librbd: add get_group test").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
3 years agodoc: rgw: document S3 bucket replication support
Matt Benjamin [Tue, 5 Jan 2021 20:30:23 +0000 (15:30 -0500)]
doc: rgw: document S3 bucket replication support

Support was added at Octopus.

Fixes: https://tracker.ceph.com/issues/48755
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
3 years agorgw: Add admin ops API for rate limiting
Or Friedmann [Mon, 3 Jan 2022 16:32:16 +0000 (16:32 +0000)]
rgw: Add admin ops API for rate limiting

Add admin ops API for rate limiting and some bug fixes

Signed-off-by: Or Friedmann <ofriedma@redhat.com>
3 years agoMerge pull request #44221 from BenoitKnecht/ceph-mon-device-metrics
Yuri Weinstein [Mon, 10 Jan 2022 15:56:29 +0000 (07:56 -0800)]
Merge pull request #44221 from BenoitKnecht/ceph-mon-device-metrics

mon: Abort device health when device not found

Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>
3 years agoMerge pull request #43362 from ybwang0211/pubsub_settingng_spell
Daniel Gryniewicz [Mon, 10 Jan 2022 15:10:51 +0000 (10:10 -0500)]
Merge pull request #43362 from ybwang0211/pubsub_settingng_spell

rgw:word spelling error in rgw/rgw_pubsub.h file

3 years agoMerge pull request #42115 from sandrobonazzola/patch-2
Sebastian Wagner [Mon, 10 Jan 2022 14:41:32 +0000 (15:41 +0100)]
Merge pull request #42115 from sandrobonazzola/patch-2

doc/install: ensure to open ceph-mon firewall ports

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
3 years agorgw/multisite/test: passzero (no) ssl port to mrgw.sh
Yuval Lifshitz [Sun, 9 Jan 2022 16:27:40 +0000 (18:27 +0200)]
rgw/multisite/test: passzero (no) ssl port to mrgw.sh

in multisite tests script.
this is to fix an issue from: 1f3fbf218faa4859f02917828dbdb48be41a579a

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
3 years agoMerge pull request #44301 from yuvalif/wip-yuval-cloudevents
Yuval Lifshitz [Sun, 9 Jan 2022 07:32:53 +0000 (09:32 +0200)]
Merge pull request #44301 from yuvalif/wip-yuval-cloudevents

rgw/notifications: add cloudevents support to HTTP endpoint

3 years agoMerge pull request #44278 from fengchunsong/dpdk-tso
Kefu Chai [Sat, 8 Jan 2022 11:28:00 +0000 (19:28 +0800)]
Merge pull request #44278 from fengchunsong/dpdk-tso

msg/async/dpdk: Add an option to enable/disable TSO

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
3 years agoMerge pull request #44198 from wjwithagen/fix-bluestore-get-io-size
Kefu Chai [Sat, 8 Jan 2022 04:29:21 +0000 (12:29 +0800)]
Merge pull request #44198 from wjwithagen/fix-bluestore-get-io-size

common/blkdev: add missing stubs for OSes not Linux

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
3 years agoMerge pull request #44402 from inspur-wyq/wip-pybind-mgr
Yuri Weinstein [Fri, 7 Jan 2022 23:36:58 +0000 (15:36 -0800)]
Merge pull request #44402 from inspur-wyq/wip-pybind-mgr

pybind/mgr: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agoMerge pull request #44055 from ifed01/wip-ifed-cosmetics-for-optimize-rm
Yuri Weinstein [Fri, 7 Jan 2022 23:36:37 +0000 (15:36 -0800)]
Merge pull request #44055 from ifed01/wip-ifed-cosmetics-for-optimize-rm

test/store_test, os/bluestore: bunch of minor improvements/cleanups

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
3 years agoMerge pull request #43993 from SMIL-Infra/cmake-systemd-cleanup
Yuri Weinstein [Fri, 7 Jan 2022 23:35:42 +0000 (15:35 -0800)]
Merge pull request #43993 from SMIL-Infra/cmake-systemd-cleanup

cmake: cleanups about systemd and install

Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agoMerge pull request #44423 from 5cs/fix-unshare-blob-test
Igor Fedotov [Fri, 7 Jan 2022 23:27:42 +0000 (02:27 +0300)]
Merge pull request #44423 from 5cs/fix-unshare-blob-test

test: fix bluestore unshare blob test case

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
3 years agoMerge pull request #44416 from yuvalif/wip-yuval-rgw-params-in-multisite
Casey Bodley [Fri, 7 Jan 2022 20:19:44 +0000 (15:19 -0500)]
Merge pull request #44416 from yuvalif/wip-yuval-rgw-params-in-multisite

rgw/multisite/test: allow passing rgw parameter to multisite script

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 years agoMerge pull request #44020 from adk3798/host-ls-scale
Sebastian Wagner [Fri, 7 Jan 2022 10:59:04 +0000 (11:59 +0100)]
Merge pull request #44020 from adk3798/host-ls-scale

mgr/orchestrator: add filtering and count option for orch host ls

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
3 years agorgw/notifications: add cloudevents support to HTTP endpoint
yuval Lifshitz [Mon, 13 Dec 2021 18:56:20 +0000 (20:56 +0200)]
rgw/notifications: add cloudevents support to HTTP endpoint

following the cloudevents HTTP spec:
https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md
and more specifically this aws-s3 spec:
https://github.com/cloudevents/spec/blob/main/cloudevents/adapters/aws-s3.md

Signed-off-by: yuval Lifshitz <ylifshit@redhat.com>
3 years agorgw/common: make "get" methods on RGWHTTPArgs const
yuval Lifshitz [Mon, 13 Dec 2021 16:45:19 +0000 (18:45 +0200)]
rgw/common: make "get" methods on RGWHTTPArgs const

Signed-off-by: yuval Lifshitz <ylifshit@redhat.com>
3 years agoMerge pull request #44412 from cyx1231st/wip-crimson-improve-log
Yingxin [Fri, 7 Jan 2022 07:04:46 +0000 (15:04 +0800)]
Merge pull request #44412 from cyx1231st/wip-crimson-improve-log

crimson: leverage --default-log-level to configure the global log level

Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #44473 from johnbent/patch-1
Laura Flores [Thu, 6 Jan 2022 16:05:09 +0000 (10:05 -0600)]
Merge pull request #44473 from johnbent/patch-1

README.md: Update README.md to add link to tracker.ceph.com

3 years agoMerge pull request #43833 from melissa-kun-li/prevent-cephadm-stray-daemon-tcmurunner
Sebastian Wagner [Thu, 6 Jan 2022 16:00:00 +0000 (17:00 +0100)]
Merge pull request #43833 from melissa-kun-li/prevent-cephadm-stray-daemon-tcmurunner

mgr/cephadm: fix tcmu-runner cephadm_stray_daemon

3 years agoqa/suites: Fix crimson flavor configuration
Matan Breizman [Thu, 6 Jan 2022 13:32:33 +0000 (13:32 +0000)]
qa/suites: Fix crimson flavor configuration

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
3 years agoMerge PR #44054 into master
Sage Weil [Thu, 6 Jan 2022 13:54:45 +0000 (08:54 -0500)]
Merge PR #44054 into master

* refs/pull/44054/head:
doc/rados/operations: document pg_num_max
mgr: set max of 32 pgs for .mgr pool
mgr/dashboard: expect pg_num_max property for pools
mon/OSDMonitor: add option --pg-num_max arg for create pool
mon/OSDMonitor: disallow setting pg_num < min or > max
mgr/pg_autoscaler: apply pg_num_max
mon: add pg_num_max pool property

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agoMerge pull request #44157 from CongMinYin/pwl-small-fixes
Ilya Dryomov [Thu, 6 Jan 2022 13:54:31 +0000 (14:54 +0100)]
Merge pull request #44157 from CongMinYin/pwl-small-fixes

librbd/cache/pwl: code cleanup and some minor fixes

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
3 years agolibrbd: diff-iterate reports incorrect offsets in fast-diff mode
Ilya Dryomov [Tue, 4 Jan 2022 19:38:35 +0000 (20:38 +0100)]
librbd: diff-iterate reports incorrect offsets in fast-diff mode

If rbd_diff_iterate2() is called on an image offset that doesn't
correspond to an object boundary, the callback is invoked with an
incorrect image offset.  For example, assuming a fully allocated
image, a diff request for 806354944~57344 results in offs=807403520,
len=57344, exists=true invocation, which is ahead by 1048576 bytes.
This occurs only in fast-diff mode, for a diff request on an image
with the fast-diff feature disabled or if whole_object parameter is
set to false the invocation is correct.

This bug goes back to the introduction of fast-diff mode in commit
6d5b969d4206 ("librbd: add diff_iterate2 to API").

Fixes: https://tracker.ceph.com/issues/53784
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
3 years agomgr/cephadm: nfs migration: avoid port conflicts
Sebastian Wagner [Thu, 6 Jan 2022 11:35:38 +0000 (12:35 +0100)]
mgr/cephadm: nfs migration: avoid port conflicts

The nfs upgrade renames the nfs spec from `nfs.ganesha-{service_id}`
to `nfs.{service_id}`. Previously we used the orphan-daemon check
to remove the old `nfs.ganesha-{service_id}` daemons. This does not work
as sometimes serve() tries to deploy the new daemons before cleaning up the
old daemons. This results in a port conflict breaking the upgrade.

Fixes: https://tracker.ceph.com/issues/53424
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>