ceph.git
3 years agoMerge pull request #39011 from dillaman/wip-librbd-crypto-fixes-pacific v16.1.0
Mykola Golub [Fri, 22 Jan 2021 08:20:30 +0000 (10:20 +0200)]
Merge pull request #39011 from dillaman/wip-librbd-crypto-fixes-pacific

pacific: librbd: miscellaneous fixes for crypto layer

Reviewed-by: Mykola Golub <mgolub@suse.com>
3 years agoMerge pull request #38986 from dillaman/wip-rbd-error-pacific
Jason Dillaman [Thu, 21 Jan 2021 20:09:26 +0000 (15:09 -0500)]
Merge pull request #38986 from dillaman/wip-rbd-error-pacific

pacific: rbd: deprecation warning should be printed to stderr

Reviewed-by: Mykola Golub <mgolub@suse.com>
3 years agolibrbd: don't restart empty copyups in crypto layer 39011/head
Or Ozeri [Wed, 20 Jan 2021 16:43:47 +0000 (18:43 +0200)]
librbd: don't restart empty copyups in crypto layer

This commit fixes a bug where an empty parent copyup is restarted indefinitely.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
(cherry picked from commit cd6f3ccd3363f40c4fbe1c52bba9cc77e4162950)

3 years agotools/rbd-nbd: change block size to 4K for LUKS2 images
Or Ozeri [Wed, 20 Jan 2021 13:36:10 +0000 (15:36 +0200)]
tools/rbd-nbd: change block size to 4K for LUKS2 images

This commit sets the NBD device block size to 4K for images encrypted using LUKS2 (whose sector size is by default 4K(.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
(cherry picked from commit 988039f6ec9210d13f7bdaee9d73ac009c0ceb91)

3 years agoqa/tasks/rbd: increase image size of encrypted disks
Or Ozeri [Wed, 20 Jan 2021 10:59:06 +0000 (12:59 +0200)]
qa/tasks/rbd: increase image size of encrypted disks

This commit increases the rbd image sizes in used by qemu to support the storage requirements of the luks encryption header.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
(cherry picked from commit 114a7cfb2ea3cd86509588e0d528beb1494b36a0)

3 years agoMerge pull request #38993 from neha-ojha/wip-48906-pacific
Neha Ojha [Wed, 20 Jan 2021 23:39:10 +0000 (15:39 -0800)]
Merge pull request #38993 from neha-ojha/wip-48906-pacific

pacific: osd: Remove override for osd_async_recovery_min_cost for mclock profiles

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
3 years agoosd: Remove override for osd_async_recovery_min_cost for mclock profiles 38993/head
Sridhar Seshasayee [Mon, 18 Jan 2021 08:34:46 +0000 (14:04 +0530)]
osd: Remove override for osd_async_recovery_min_cost for mclock profiles

Overriding osd_async_recovery_min_cost as part of enabling a built-in
mclock profile has the undesirable side effect of peers not choosing
the correct async recovery targets if osds are using mixed schedulers
(this could happen during upgrades or if "debug_random" is set for
osd_op_queue config option). Due to the above, osds get into a
"choose_acting" loop during peering.

The solution is to remove the override of osd_async_recovery_min_cost.

Fixes: https://tracker.ceph.com/issues/48906
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
(cherry picked from commit 922d93f4491207411b4ee9427e49ccf08b0ef648)

3 years agorbd: deprecation warning should be printed to stderr 38986/head
Jason Dillaman [Tue, 19 Jan 2021 16:46:38 +0000 (11:46 -0500)]
rbd: deprecation warning should be printed to stderr

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8af6f8bb5fdd9d01f862cdea399b6e489d147259)

3 years agoMerge pull request #38979 from badone/wip-pacific-valgrind-tracker-48918
Brad Hubbard [Wed, 20 Jan 2021 00:42:01 +0000 (10:42 +1000)]
Merge pull request #38979 from badone/wip-pacific-valgrind-tracker-48918

pacific: osd: initialise m_interval_start

Reviewed-by: Neha Ojha <nojha@redhat.com>
3 years agoosd: initialise m_interval_start 38979/head
Brad Hubbard [Tue, 19 Jan 2021 02:11:45 +0000 (12:11 +1000)]
osd: initialise m_interval_start

Fixes: https://tracker.ceph.com/issues/48918
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 45f36bbfc6c39e4a94529d13fe0c2f74fd1fecd4)

3 years ago:qa/tasks/rbd: test qemu on top of rbd encryption
Or Ozeri [Thu, 14 Jan 2021 15:03:42 +0000 (17:03 +0200)]
:qa/tasks/rbd: test qemu on top of rbd encryption

This commit adds new qemu xfstests workloads that run on top of librbd luks1/luks2 encryption.
This is currently done via nbd, instead of the qemu rbd driver.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
(cherry picked from commit 3754c665a11edba1f0a978527e35fd4de31d8212)

3 years agoceph_release: dev -> rc
Sage Weil [Mon, 18 Jan 2021 15:59:14 +0000 (09:59 -0600)]
ceph_release: dev -> rc

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agoMerge pull request #38920 from sunnyku/wip-ceph-mclock-op
Neha Ojha [Sun, 17 Jan 2021 02:24:13 +0000 (18:24 -0800)]
Merge pull request #38920 from sunnyku/wip-ceph-mclock-op

osd: handle ceph specific config changes for the mclock scheduler

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
3 years agoMerge pull request #38675 from sseshasa/wip-dmclock-config-sets
Neha Ojha [Sun, 17 Jan 2021 02:23:25 +0000 (18:23 -0800)]
Merge pull request #38675 from sseshasa/wip-dmclock-config-sets

osd: Add mclock config sets that implement built-in and custom profiles

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
3 years agoMerge pull request #38933 from wjwithagen/wjw-fix-QCOW
Jason Dillaman [Sat, 16 Jan 2021 22:04:46 +0000 (17:04 -0500)]
Merge pull request #38933 from wjwithagen/wjw-fix-QCOW

librbd: remove inclusion of endian.h

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
3 years agoosd: Set recovery specific Ceph options for mclock profiles to work. 38675/head
Sridhar Seshasayee [Thu, 14 Jan 2021 13:06:04 +0000 (18:36 +0530)]
osd: Set recovery specific Ceph options for mclock profiles to work.

Set and disable relevant recovery specific Ceph options for mclock
profiles to work as expected. Broadly,
 - Set low value for recovery min cost
 - High values for max active recoveries and max backfills
 - Disable recovery sleep{_hdd, _ssd, _hybrid}

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
3 years agoosd: Handle configuration changes to mclock config options
Sridhar Seshasayee [Tue, 12 Jan 2021 11:32:37 +0000 (17:02 +0530)]
osd: Handle configuration changes to mclock config options

Handle configuration changes to the mclock cost per io, the max
capacity options and the mclock profile. Handle the case where the
profile is changed from a built-in profile to the custom profile.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
3 years agoosd: Add mclock profile infrastructure and implement mclock profiles
Sridhar Seshasayee [Mon, 7 Dec 2020 11:09:57 +0000 (16:39 +0530)]
osd: Add mclock profile infrastructure and implement mclock profiles

Define config options to specify the cost per io for an osd (hdd & ssd).
 - osd_mclock_cost_per_io_msec
 - osd_mclock_cost_per_io_msec_hdd
 - osd_mclock_cost_per_io_msec_ssd

Define config options to set max osd capacity (hdd & ssd) to be allocated
between clients of dmclock namely,
 - osd_mclock_max_capacity_iops
 - osd_mclock_max_capacity_iops_hdd
 - osd_mclock_max_capacity_iops_ssd

Define config option "osd_mclock_profile" to specify the built-in profile
to enable.

Also, Set the number of op shards being used in the osd within the mclock
scheduler as well. This is necessary to calculate the per shard limits
within the mclock scheduler.

With the above information, enable the specified mclock profile by
calling the appropriate method to set the profile specific mclock
parameters and Ceph options.

Prior to enqueuing an op, the scheduler performs a calculation to scale
up or down the cost associated for the OpSchedulerItem. This calculation
is done based on the existing item cost, the max osd capacity provided
and an additional cost factor based on underlying device type(hdd/ssd).

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
3 years agoosd: handle ceph specific config changes for the mclock scheduler 38920/head
Sunny Kumar [Fri, 15 Jan 2021 02:53:05 +0000 (02:53 +0000)]
osd: handle ceph specific config changes for the mclock scheduler

The below ceph parameters are set automatically
while enabling the mclock scheduler with a built-in profile.
The user in this case will not be able to modify these
ceph specific config options during runtime.

  a. osd_async_recovery_min_cost
  b. osd_recovery_max_active{_hdd,_ssd}
  c. osd_max_backfills
  d. osd_recovery_sleep{_hdd,_ssd,_hybrid}

If the custom profile is enabled for the mclock scheduler,
the user can modify these parameters.

Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
3 years agoMerge PR #37297 into master
Patrick Donnelly [Sat, 16 Jan 2021 18:40:51 +0000 (10:40 -0800)]
Merge PR #37297 into master

* refs/pull/37297/head:
qa: add new client tests
test: add client tests
client: wire up alternate_name
mds: fix alternate_name durability
mds: add alternate_name feature support for dentries
mds: add static encode/decode helpers for remote in CDentry
mds: add "fscrypt" flag support for inode_t
mds: add new CDentry tags support
include: cleanup filepath cons
mds: add comment on linkage durability
client: constify some RWRef methods
common: add strescape helper

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agolibrbd: remove inclusion of endian.h 38933/head
Willem Jan Withagen [Sat, 16 Jan 2021 13:33:16 +0000 (14:33 +0100)]
librbd: remove inclusion of endian.h

The file is not available on FreeBSD.

Fixes: https://github.com/ceph/ceph/pull/38862
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
3 years agoMerge pull request #38880 from xxhdx1985126/wip-crimson-bug-fix
Kefu Chai [Sat, 16 Jan 2021 06:57:47 +0000 (14:57 +0800)]
Merge pull request #38880 from xxhdx1985126/wip-crimson-bug-fix

crimson/osd: make PGRecovery::start_primary_recovery_ops take objs recovered by UrgentRecovery into account

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agoMerge pull request #38245 from batrick/ceph-status-pg-state-sort
Kefu Chai [Sat, 16 Jan 2021 04:20:15 +0000 (12:20 +0800)]
Merge pull request #38245 from batrick/ceph-status-pg-state-sort

osd: use more efficient CachedStackStringStream

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
3 years agoMerge pull request #37391 from aclamk/wip-kv-onode-cache
Neha Ojha [Sat, 16 Jan 2021 02:36:47 +0000 (18:36 -0800)]
Merge pull request #37391 from aclamk/wip-kv-onode-cache

Add ability to control rocksdb block cache

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
3 years agoMerge pull request #38925 from thomasgoirand/master
Neha Ojha [Sat, 16 Jan 2021 01:40:55 +0000 (17:40 -0800)]
Merge pull request #38925 from thomasgoirand/master

common/ipaddr: Allow binding on lo

Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agoqa: add new client tests 37297/head
Patrick Donnelly [Fri, 15 Jan 2021 04:17:42 +0000 (20:17 -0800)]
qa: add new client tests

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agotest: add client tests
Patrick Donnelly [Mon, 11 Jan 2021 21:39:42 +0000 (13:39 -0800)]
test: add client tests

For alternate_name, so far.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agoclient: wire up alternate_name
Patrick Donnelly [Mon, 11 Jan 2021 21:39:27 +0000 (13:39 -0800)]
client: wire up alternate_name

Here we're exposing a public Client::walk (aka path_walk) so that the
user can inspect dentries (not something normally possible in POSIX).
We're going to skip exposing such an interface in libcephfs since
there's no reason to do that (who would use it?) except for testing.
Instead, a follow-up PR will add Client tests (for the first time, yay!)
that will exercise this code.

Also, ideally we'd also expose alternate_name via readdir results but
that is a bit more complicated since dirents do not normally refer to
external memory. So, just rely on Client::walk for testing for now.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agomds: fix alternate_name durability
Patrick Donnelly [Fri, 15 Jan 2021 03:55:43 +0000 (19:55 -0800)]
mds: fix alternate_name durability

This is a collection of fixes to Xiubo's prior work. Namely:

- Add new mds_alternate_name_max option to limit the size of
  alternate_name. Otherwise a Client could trick the MDS into creating
  an alternate_name of any size!

- Clean up how alternate_name is assigned to CDentry. In the general
  case, this should be assigned as part of creating the dentry. We want
  this value to be immutable for the life of the dentry. Even for the
  very special case of rename(2) where the destination dentry already
  exists.  We explicitly check (after discussion with Jeff) that the
  target dentry alternate_name already matches what the rename RPC is
  giving.

- The MDS is now properly journaling the alternate_name.

- The MDS rejoin phase is properly transmitting each dentry's
  alternate_name. I've discovered that this MMDSCacheRejoin message
  actually wasn't versioned which I've raised in a tracker [1]. In the
  mean time, we'll just bump CEPH_MDS_PROTOCOL as usual.

[1] https://tracker.ceph.com/issues/48886

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agomds: add alternate_name feature support for dentries
Xiubo Li [Mon, 21 Sep 2020 04:55:31 +0000 (12:55 +0800)]
mds: add alternate_name feature support for dentries

This will support the "alternate_name" filename support, and will save
an alternate name for each dentry. This alternate name is not used in
path lookup or for any other usual file system purpose. The name is
simply an added blob of metadata on the dentry that is distributed to
clients so that "long" file names may be supported for clients which
require them. In the case of an fscrypt enhanced kernel mount driver,
the long name may be the cyphertext (exceeding FILENAME_MAX) of a long
file name.

Because this affects only files with long file names, the use of this
feature should be rare but could be common for some unusual
applications.

The client mount should check the CEPHFS_FEATURE_ALTERNATE_NAME feature
bit first to check whether the MDS has support for this feature or not.
The alternate_name is transmitted as part of the message payload in
MClientRequest when setting the alternate_name. The LeaseStat structure
in MClientReply contains the alternate_name.

When executing a metadata mutation RPC, the client will set the
alternate_name (if it exists) as part of the operation. The MDS will
pick that up and set it on the new or mutated dentry.

Fixes: https://tracker.ceph.com/issues/47162
Signed-off-by: Xiubo Li <xiubli@redhat.com>
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agomds: add static encode/decode helpers for remote in CDentry
Xiubo Li [Fri, 20 Nov 2020 02:02:42 +0000 (10:02 +0800)]
mds: add static encode/decode helpers for remote in CDentry

This will unify all the same work in different places.

Fixes: https://tracker.ceph.com/issues/47162
Signed-off-by: Xiubo Li <xiubli@redhat.com>
3 years agomds: add "fscrypt" flag support for inode_t
Xiubo Li [Sun, 27 Sep 2020 03:53:55 +0000 (11:53 +0800)]
mds: add "fscrypt" flag support for inode_t

If the client has set the "encryption.ctx" attribute, the MDS side
will set the "fscrypt" flag to truth, false as default.

The clients could use this flag to get to know whether the current
inodes are under enscrypted or not.

Fixes: https://tracker.ceph.com/issues/47162
Signed-off-by: Xiubo Li <xiubli@redhat.com>
3 years agomds: add new CDentry tags support
Xiubo Li [Wed, 23 Sep 2020 02:54:34 +0000 (10:54 +0800)]
mds: add new CDentry tags support

This will add new tag 'i' for inode and new tag 'l' for remote link
for the CDentry. And at the same time will add one proper dentry
version, which will be helpful to add new features/members in future
for the CDentry.

Fixes: https://tracker.ceph.com/issues/47162
Signed-off-by: Xiubo Li <xiubli@redhat.com>
3 years agoinclude: cleanup filepath cons
Patrick Donnelly [Tue, 12 Jan 2021 03:15:06 +0000 (19:15 -0800)]
include: cleanup filepath cons

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agomds: add comment on linkage durability
Patrick Donnelly [Fri, 15 Jan 2021 03:58:29 +0000 (19:58 -0800)]
mds: add comment on linkage durability

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agoclient: constify some RWRef methods
Patrick Donnelly [Fri, 15 Jan 2021 03:54:20 +0000 (19:54 -0800)]
client: constify some RWRef methods

And make some Client state checks const.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agocommon: add strescape helper
Patrick Donnelly [Fri, 15 Jan 2021 03:54:46 +0000 (19:54 -0800)]
common: add strescape helper

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agoMerge pull request #38730 from ronen-fr/wip-ronenf-scrub-interval
Neha Ojha [Sat, 16 Jan 2021 01:02:19 +0000 (17:02 -0800)]
Merge pull request #38730 from ronen-fr/wip-ronenf-scrub-interval

osd: improved handling of new-interval events by the scrubber

Reviewed-by: David Zafman <dzafman@redhat.com>
3 years agoMerge pull request #38856 from dzafman/wip-48789
Neha Ojha [Sat, 16 Jan 2021 00:27:59 +0000 (16:27 -0800)]
Merge pull request #38856 from dzafman/wip-48789

test: Fix osd-scrub-scaps.sh to handle DB format change

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
3 years agoMerge pull request #38726 from ronen-fr/wip-ronenf-48720
Neha Ojha [Fri, 15 Jan 2021 21:46:30 +0000 (13:46 -0800)]
Merge pull request #38726 from ronen-fr/wip-ronenf-48720

qa/standalone/scrub/osd-recovery-scrub: handle primary change when waiting for scrub

Reviewed-by: David Zafman <dzafman@redhat.com>
3 years agoMerge PR #38160 into master
Patrick Donnelly [Fri, 15 Jan 2021 19:53:02 +0000 (11:53 -0800)]
Merge PR #38160 into master

* refs/pull/38160/head:
doc/mgr/volumes: Document 'fs subvolume authorized_list' cli
qa: Add tests for list auth-ids of a subvolume
mgr/volumes: Update the 'volumes' key to 'subvolumes' in auth metadata file
mgr/volumes: Optionally authorize existing auth-ids
mgr/volumes: Preserve existing caps while authorize/deauthorize auth-id
mgr/volumes: Disallow authorize existing auth_id
mgr/volumes: Add subvolume authorized_list command
mgr/volumes: Add tenant_id option to subvolume authorize
mgr/volumes: Persist auth and subvolume metadata

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agoMerge pull request #38862 from dillaman/wip-librbd-migration-qcow
Mykola Golub [Fri, 15 Jan 2021 18:58:30 +0000 (20:58 +0200)]
Merge pull request #38862 from dillaman/wip-librbd-migration-qcow

librbd/migration: basic QCOW format handler

Reviewed-by: Mykola Golub <mgolub@suse.com>
3 years agotest: Fix osd-scrub-scaps.sh to handle DB format change 38856/head
David Zafman [Tue, 12 Jan 2021 00:00:05 +0000 (16:00 -0800)]
test: Fix osd-scrub-scaps.sh to handle DB format change

Caused by: f9c95fa7fc7b0ee992c0249ff090fa7f751e9719

Fixes: https://tracker.ceph.com/issues/48789
Signed-off-by: David Zafman <dzafman@redhat.com>
3 years agoMerge pull request #38900 from Exotelis/update-i18n
Ernesto Puerta [Fri, 15 Jan 2021 18:07:16 +0000 (19:07 +0100)]
Merge pull request #38900 from Exotelis/update-i18n

mgr/dashboard: Update locale en to en-US to match transifex

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
3 years agoMerge pull request #38722 from tchaikov/wip-mgr-args-ann
Kefu Chai [Fri, 15 Jan 2021 17:51:27 +0000 (01:51 +0800)]
Merge pull request #38722 from tchaikov/wip-mgr-args-ann

mgr: use annotations for argdesc

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
3 years agoMerge pull request #38899 from ronen-fr/wip-ronenf-blue-dangling
Kefu Chai [Fri, 15 Jan 2021 17:49:46 +0000 (01:49 +0800)]
Merge pull request #38899 from ronen-fr/wip-ronenf-blue-dangling

os/bluestore: fixing a dangling pointer

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agoMerge pull request #38902 from ifed01/wip-ifed-fix-deleting
Kefu Chai [Fri, 15 Jan 2021 17:48:30 +0000 (01:48 +0800)]
Merge pull request #38902 from ifed01/wip-ifed-fix-deleting

osd/PG: fix no exit() call for PeeringState::Deleting state

Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agoMerge pull request #38857 from dzafman/wip-48842
Kefu Chai [Fri, 15 Jan 2021 17:46:43 +0000 (01:46 +0800)]
Merge pull request #38857 from dzafman/wip-48842

test: osd-recovery-prio.sh replace sleep with wait for both PGs recovering

Reviewed-by: Neha Ojha <nojha@redhat.com>
3 years agoMerge pull request #38762 from peng-jiaqi/master
Kefu Chai [Fri, 15 Jan 2021 17:44:38 +0000 (01:44 +0800)]
Merge pull request #38762 from peng-jiaqi/master

mgr: fix deadlock in ActivePyModules::get_osdmap()

Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agoMerge pull request #38878 from tchaikov/wip-ceph-time-print
Kefu Chai [Fri, 15 Jan 2021 17:41:04 +0000 (01:41 +0800)]
Merge pull request #38878 from tchaikov/wip-ceph-time-print

common/ceph_time: print {milli,nano}seconds as float

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
3 years agocommon/options: Set pacific+ rocksdb column family options. 37391/head
Mark Nelson [Mon, 11 Jan 2021 15:19:48 +0000 (15:19 +0000)]
common/options: Set pacific+ rocksdb column family options.

Signed-off-by: Mark Nelson <mnelson@redhat.com>
3 years agoBlueStore: Add block_cache logic to column family definition
Adam Kupczyk [Thu, 24 Sep 2020 10:54:46 +0000 (12:54 +0200)]
BlueStore: Add block_cache logic to column family definition

Modified original onode column family cache into generic feature.
Now 2 options are possible for each column family:
1) use generic block cache but apply different BlockBasedTableOptions
2) create separate block cache. it will be applied for all shards of column family

It is done by specifying special option "block_cache" to CF definition:
Example: O(3,0-13)=block_cache={high_ratio=1.000}

"block_cache" accepts all BlockBasedTableOptions options with additions:
-"type" - binned_lru/lru/clock (default: ceph_options.rocksdb_cache_type)
-"size" - e.g.: 100M           (default: ceph_options.rocksdb_cache_size)
-"high_ratio" - e.g.: 0.75     (default: 0.0)
If any of above is set, new block cache is created, otherwise default is used.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
3 years agoMerge pull request #38903 from yuvalif/wip-yuval-fix-48869
Casey Bodley [Fri, 15 Jan 2021 16:23:26 +0000 (11:23 -0500)]
Merge pull request #38903 from yuvalif/wip-yuval-fix-48869

rgw/amqp: fix race condition in amqp manager initialization

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 years agoBlueStore: Add onode column family KV block_cache.
Mark Nelson [Sat, 13 Apr 2019 00:33:59 +0000 (19:33 -0500)]
BlueStore: Add onode column family KV block_cache.

Signed-off-by: Mark Nelson <mnelson@redhat.com>
3 years agodoc/mgr: document CLICommand and friends 38722/head
Kefu Chai [Fri, 15 Jan 2021 03:41:10 +0000 (11:41 +0800)]
doc/mgr: document CLICommand and friends

the annotation and docstring are critical after we use them for
describing the command.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agocommon/ipaddr: Allow binding on lo 38925/head
Thomas Goirand [Fri, 15 Jan 2021 09:50:05 +0000 (10:50 +0100)]
common/ipaddr: Allow binding on lo

Commmit 5cf0fa872231f4eaf8ce6565a04ed675ba5b689b, solves the issue that
the osd can't restart after seting a virtual local loopback IP. However,
this commit also prevents a bgp-to-the-host over unumbered Ipv6
local-link is setup, where OSD typically are bound to the lo interface.

To solve this, this single char patch simply checks against "lo:" to
match only virtual interfaces instead of anything that starts with "lo".

Fixes: https://tracker.ceph.com/issues/48893
Signed-off-by: Thomas Goirand <zigo@debian.org>
3 years agolibrbd/migration: address code review comments 38862/head
Jason Dillaman [Fri, 15 Jan 2021 14:24:31 +0000 (09:24 -0500)]
librbd/migration: address code review comments

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
3 years agoMerge pull request #38815 from sebastian-philipp/cephadm-osd-queue-race
Sebastian Wagner [Fri, 15 Jan 2021 11:26:33 +0000 (12:26 +0100)]
Merge pull request #38815 from sebastian-philipp/cephadm-osd-queue-race

mgr/cephadm: lock multithreaded access to OSDRemovalQueue

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
3 years agoMerge pull request #38865 from s0nea/wip-dashboard-48847
Ernesto Puerta [Fri, 15 Jan 2021 11:07:07 +0000 (12:07 +0100)]
Merge pull request #38865 from s0nea/wip-dashboard-48847

mgr/dashboard: add `--ssl` to `ng serve`

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
3 years agoMerge pull request #38919 from tchaikov/wip-cephadm-yield_fixture
Sebastian Wagner [Fri, 15 Jan 2021 09:38:05 +0000 (10:38 +0100)]
Merge pull request #38919 from tchaikov/wip-cephadm-yield_fixture

mgr/cephadm: s/yield_fixture/fixture/

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
3 years agopybind/mgr/mgr_module: annotate some methods
Kefu Chai [Thu, 14 Jan 2021 17:26:13 +0000 (01:26 +0800)]
pybind/mgr/mgr_module: annotate some methods

as a vehicle to understand the interface

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agomgr/dashboard/awsauth: drop py2 support
Kefu Chai [Thu, 14 Jan 2021 14:19:48 +0000 (22:19 +0800)]
mgr/dashboard/awsauth: drop py2 support

silences the warning from mypy:

dashboard/awsauth.py:42: error: Module 'base64' has no attribute
'encodestring'

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agodoc/_ext: use iterator of tuple to build dict
Kefu Chai [Thu, 14 Jan 2021 12:17:59 +0000 (20:17 +0800)]
doc/_ext: use iterator of tuple to build dict

As suggested by Ernesto Puerta <epuertat@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agomgr/dashboard: disable unsubscriptable-object warning from pylint
Kefu Chai [Thu, 14 Jan 2021 04:53:19 +0000 (12:53 +0800)]
mgr/dashboard: disable unsubscriptable-object warning from pylint

see https://github.com/PyCQA/pylint/issues/3882

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agodoc/_ext: PEP8 compliance
Kefu Chai [Wed, 13 Jan 2021 02:30:04 +0000 (10:30 +0800)]
doc/_ext: PEP8 compliance

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agopybind/mgr/orchestrator: fix typo in comment
Kefu Chai [Tue, 12 Jan 2021 14:49:36 +0000 (22:49 +0800)]
pybind/mgr/orchestrator: fix typo in comment

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agodoc/_ext: do not mock ceph_argparse
Kefu Chai [Tue, 12 Jan 2021 15:05:21 +0000 (23:05 +0800)]
doc/_ext: do not mock ceph_argparse

this module is used for convert argspec to argdesc by
CLICommand.load_func_metadata()

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agodoc/_ext: add python_path option to ceph-mgr-commands
Kefu Chai [Wed, 13 Jan 2021 02:30:46 +0000 (10:30 +0800)]
doc/_ext: add python_path option to ceph-mgr-commands

so we can add more python paths in addition to the one used for finding
mgr modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agodoc/_ext: parse prefix and args for command signature
Kefu Chai [Tue, 12 Jan 2021 14:24:42 +0000 (22:24 +0800)]
doc/_ext: parse prefix and args for command signature

unlike the commands defined by C++, the commands defined by python
now uses "prefix" and "args" properties of elements in COMMAND class
attribute to define their command and arguments. the "cmd" property is
still available for the ceph CLI.

but the ceph_command sphinx extension should now use "prefix" and "args"
for printing out the usage of commands implemented using python. in
this change, the extension is updated to read "prefix" and "args"
properties of command defined by python modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agopybind/mgr/orchestrator: do not pass cmd_args or desc to CLICommand anymore
Kefu Chai [Sun, 27 Dec 2020 14:46:59 +0000 (22:46 +0800)]
pybind/mgr/orchestrator: do not pass cmd_args or desc to CLICommand anymore

they will be attached to the handler function as annotations and docstring

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agopybind/mgr/orchestrator: use enum parameter when appropriate
Kefu Chai [Sun, 27 Dec 2020 14:45:24 +0000 (22:45 +0800)]
pybind/mgr/orchestrator: use enum parameter when appropriate

and move desc as docstring.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agomgr: fix deadlock in ActivePyModules::get_osdmap() 38762/head
peng jiaqi [Tue, 5 Jan 2021 03:15:26 +0000 (11:15 +0800)]
mgr: fix deadlock in ActivePyModules::get_osdmap()

In function "ActivePyModules::get_osdmap()", We do not read or write to
object "ActivePyModules", so it is safe to delete lock
"ActivePyModules::lock", and it can avoid other thread waiting for lock
"ActivePyModules::lock"

Fixes: https://tracker.ceph.com/issues/48852
Signed-off-by: peng jiaqi <peng.jiaqi@zte.com.cn>
3 years agodoc/mgr/volumes: Document 'fs subvolume authorized_list' cli 38160/head
Kotresh HR [Tue, 5 Jan 2021 13:25:32 +0000 (18:55 +0530)]
doc/mgr/volumes: Document 'fs subvolume authorized_list' cli

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
3 years agoqa: Add tests for list auth-ids of a subvolume
Kotresh HR [Wed, 16 Dec 2020 12:49:42 +0000 (18:19 +0530)]
qa: Add tests for list auth-ids of a subvolume

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
3 years agocrimson/osd: make PGRecovery::start_primary_recovery_ops take objs recovered by Urgen... 38880/head
xuxuehan xuxuehan [Fri, 14 Feb 2020 11:52:28 +0000 (19:52 +0800)]
crimson/osd: make PGRecovery::start_primary_recovery_ops take objs recovered by UrgentRecovery into account

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
3 years agopybind/mgr/mgr_module: cast string to enum when collecting kwargs
Kefu Chai [Sun, 27 Dec 2020 14:42:33 +0000 (22:42 +0800)]
pybind/mgr/mgr_module: cast string to enum when collecting kwargs

as the parameters of handlers are properly typed, they are expecting
enum parameter, let's cast string parameter to enum if the callee claims
that it wants an enum.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agopybind/mgr/mgr_module: preserve the signature of wrapped func
Kefu Chai [Wed, 13 Jan 2021 11:28:48 +0000 (19:28 +0800)]
pybind/mgr/mgr_module: preserve the signature of wrapped func

before this change, the annotations of the func wrapped by
CLICheckNonemptyFileInput and returns_command_result are lost after being
wrapped with this decorator. after this change, the parameter annotations
are preserved. these annotations are used for generating the argdesc in a
later commit.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agopybind/mgr/mgr_module: extract load_func_metadata()
Kefu Chai [Sun, 27 Dec 2020 14:41:38 +0000 (22:41 +0800)]
pybind/mgr/mgr_module: extract load_func_metadata()

so it can be reused by Command class.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agomgr/dashboard: annotate command handler
Kefu Chai [Sun, 27 Dec 2020 14:29:15 +0000 (22:29 +0800)]
mgr/dashboard: annotate command handler

and use enum as the argument

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agomgr/cephadm: s/yield_fixture/fixture/ 38919/head
Kefu Chai [Fri, 15 Jan 2021 01:27:26 +0000 (09:27 +0800)]
mgr/cephadm: s/yield_fixture/fixture/

silences pytest warning. it complained:`

1: cephadm/tests/fixtures.py:68
1:   /var/ssd/ceph/src/pybind/mgr/cephadm/tests/fixtures.py:68: PytestDeprecationWarning: @pytest.yield_fixture is deprecated.
1:   Use @pytest.fixture instead; they are the same.
1:     @pytest.yield_fixture()

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agopybind/mgr/cephadm: use desc of command for docstring
Kefu Chai [Sun, 27 Dec 2020 14:26:30 +0000 (22:26 +0800)]
pybind/mgr/cephadm: use desc of command for docstring

in the new command mechinary, the docstring of command handler
is used as the description of the command.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agopybind/mgr/mgr_module: drop arg parameter from CLICommand constructor
Kefu Chai [Sun, 27 Dec 2020 09:11:13 +0000 (17:11 +0800)]
pybind/mgr/mgr_module: drop arg parameter from CLICommand constructor

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agopybind/mgr/snap_schedule: use param annotation for argsdesc
Kefu Chai [Sun, 27 Dec 2020 09:22:51 +0000 (17:22 +0800)]
pybind/mgr/snap_schedule: use param annotation for argsdesc

use parameter annotation for CLICommand args desc, less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agomgr/dashboard: use param annotation for args desc
Kefu Chai [Sun, 27 Dec 2020 04:47:02 +0000 (12:47 +0800)]
mgr/dashboard: use param annotation for args desc

use parameter annotation for CLICommand arg desc

less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agoMerge PR #38896 into master
Patrick Donnelly [Fri, 15 Jan 2021 00:37:16 +0000 (16:37 -0800)]
Merge PR #38896 into master

* refs/pull/38896/head:
cephfs: Remove explicit forward declare CephContext

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agoMerge pull request #38912 from xenago/patch-1
Xie Xingguo [Thu, 14 Jan 2021 23:53:59 +0000 (07:53 +0800)]
Merge pull request #38912 from xenago/patch-1

common/options: Fix typos for osd_recovery_max_active

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
3 years agoMerge pull request #38891 from yuriw/wip-yuriw-octopus-x-master
Yuri Weinstein [Thu, 14 Jan 2021 22:11:33 +0000 (14:11 -0800)]
Merge pull request #38891 from yuriw/wip-yuriw-octopus-x-master

qa/tests: added stress-split-erasure-code-no-cephadm tests

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
3 years agoMerge pull request #38721 from changchengx/pwl
Jason Dillaman [Thu, 14 Jan 2021 22:02:11 +0000 (17:02 -0500)]
Merge pull request #38721 from changchengx/pwl

librbd/cache/pwl: fix parameter type to update argument

Reviewed-by: Xiaoyan Li <xiaoyan.li@intel.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
3 years agocommon/options: Fix typos for osd_recovery_max_active 38912/head
xenago [Thu, 14 Jan 2021 21:27:02 +0000 (16:27 -0500)]
common/options: Fix typos for osd_recovery_max_active

This commit fixes a minor typo in the descriptions of osd_recovery_max_active_hdd and osd_recovery_max_active_ssd.

Signed-off-by: Noah Kruiper <nkruiper@gmail.com>
3 years agoMerge pull request #38907 from rzarzynski/wip-crimson-errorator-handle_exception_type
Kefu Chai [Thu, 14 Jan 2021 17:37:45 +0000 (01:37 +0800)]
Merge pull request #38907 from rzarzynski/wip-crimson-errorator-handle_exception_type

crimson/common: errorated future exposes handle_exception_type().

Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agoMerge PR #38708 into master
Patrick Donnelly [Thu, 14 Jan 2021 17:22:40 +0000 (09:22 -0800)]
Merge PR #38708 into master

* refs/pull/38708/head:
doc: update multiple mds scrub docs

Reviewed-by: Jeff Layton <jlayton@redhat.com>
3 years agoqa: remove invalid characters from octopus-x/parallel 38891/head
Josh Durgin [Thu, 14 Jan 2021 17:04:22 +0000 (12:04 -0500)]
qa: remove invalid characters from octopus-x/parallel

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
3 years agoqa/workunits/rbd: simplify QCOW migration distro check
Jason Dillaman [Thu, 14 Jan 2021 16:52:05 +0000 (11:52 -0500)]
qa/workunits/rbd: simplify QCOW migration distro check

If the qemu-img command fails when attempting to use the QCOW (v1) format,
assume that the distro does not have support for the format enabled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
3 years agoMerge pull request #38870 from orozery/rbd-encryption-pybind
Jason Dillaman [Thu, 14 Jan 2021 16:37:57 +0000 (11:37 -0500)]
Merge pull request #38870 from orozery/rbd-encryption-pybind

librbd: python bindings for encryption api

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
3 years agoqa/tests: fixed yaml format
Yuri Weinstein [Thu, 14 Jan 2021 16:31:40 +0000 (08:31 -0800)]
qa/tests: fixed yaml format

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
3 years agoMerge PR #38846 into master
Patrick Donnelly [Thu, 14 Jan 2021 16:10:19 +0000 (08:10 -0800)]
Merge PR #38846 into master

* refs/pull/38846/head:
*: remove legacy ceph_volume_client.py library

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
3 years agoMerge PR #34858 into master
Patrick Donnelly [Thu, 14 Jan 2021 16:09:01 +0000 (08:09 -0800)]
Merge PR #34858 into master

* refs/pull/34858/head:
mds: do not allow the service to run on Windows
win32*.sh: fix SKIP_TESTS flag
include/win32/win32_errno:h Add missing include
client: Port CephFS client to Windows

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
3 years agoMerge PR #38574 into master
Patrick Donnelly [Thu, 14 Jan 2021 16:08:17 +0000 (08:08 -0800)]
Merge PR #38574 into master

* refs/pull/38574/head:
mds: update defaults for recall configs

Reviewed-by: Jeff Layton <jlayton@redhat.com>
3 years agoMerge PR #38889 into master
Patrick Donnelly [Thu, 14 Jan 2021 16:07:40 +0000 (08:07 -0800)]
Merge PR #38889 into master

* refs/pull/38889/head:
qa: add delays only for osd/mds

Reviewed-by: Jeff Layton <jlayton@redhat.com>
3 years agoMerge PR #38858 into master
Patrick Donnelly [Thu, 14 Jan 2021 15:39:38 +0000 (07:39 -0800)]
Merge PR #38858 into master

* refs/pull/38858/head:
client/libcephfs: Don't let one unmount screw up another mount

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>