]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
5 years agomgr/volumes: asynchronous cloner module
Venky Shankar [Wed, 4 Dec 2019 05:23:48 +0000 (00:23 -0500)]
mgr/volumes: asynchronous cloner module

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 4f09568b012cef24d2075733f9d81064790fe4e6)

5 years agomgr/volumes: purge thread uses new async interface
Venky Shankar [Wed, 4 Dec 2019 04:49:12 +0000 (23:49 -0500)]
mgr/volumes: purge thread uses new async interface

This also makes `_cancel_jobs()` thread safe, which was not the
case earlier (with `_cancel_purge_job()`) -- this also makes the
code simpler by sharing the lock betweent two condition variables.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit f16cc1e8eebb0868a1e07c25f8d8e4656b11b7bf)

5 years agomgr/volumes: fetch oldest clone entry
Venky Shankar [Tue, 3 Dec 2019 06:47:20 +0000 (01:47 -0500)]
mgr/volumes: fetch oldest clone entry

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 7ad14cf09b447fcdbfe76ac739971be8609aa862)

5 years agomgr/volumes: add clone specific commands
Venky Shankar [Mon, 2 Dec 2019 08:11:45 +0000 (03:11 -0500)]
mgr/volumes: add clone specific commands

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit b2145b73d7640851c27ac4bb09fd98e146de953d)

5 years agomgr/volumes: interface for fetching cloned subvolume status
Venky Shankar [Mon, 2 Dec 2019 08:10:36 +0000 (03:10 -0500)]
mgr/volumes: interface for fetching cloned subvolume status

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit fa3c56f55273eca5be0a4a9d9fc93aab3dacda2b)

5 years agomgr/volumes: add protect/unprotect and snap clone interface
Venky Shankar [Mon, 2 Dec 2019 08:08:45 +0000 (03:08 -0500)]
mgr/volumes: add protect/unprotect and snap clone interface

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 8d68f1af3a12ce096e52024aa710e03e0006ba0c)

5 years agomgr/volumes: handle transient subvolume states
Venky Shankar [Mon, 2 Dec 2019 08:00:30 +0000 (03:00 -0500)]
mgr/volumes: handle transient subvolume states

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 461909be2edd2f691263785469e5fc90a82448e8)

5 years agomgr/volumes: interface for creating a cloned subvolume
Venky Shankar [Mon, 2 Dec 2019 07:56:48 +0000 (02:56 -0500)]
mgr/volumes: interface for creating a cloned subvolume

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 7089808bf82321712676803dd4bfdd62cf5fc00e)

5 years agomgr/volumes: get/set property for subvolume mode/uid/gid
Venky Shankar [Thu, 16 Jan 2020 12:07:08 +0000 (07:07 -0500)]
mgr/volumes: get/set property for subvolume mode/uid/gid

This will be required when creating a clone as the clone would
inherit source subvolumes creation mode and uid/gid.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit f02b1e7e546609a4f4287a27b3495a09e287888b)

5 years agomgr/volumes: module to track pending clone operations
Venky Shankar [Mon, 2 Dec 2019 06:56:25 +0000 (01:56 -0500)]
mgr/volumes: module to track pending clone operations

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 53187795838ccee4b470f3a08aeaf6e474975408)

5 years agomgr/volumes: add operation state machine table
Venky Shankar [Mon, 2 Dec 2019 06:27:55 +0000 (01:27 -0500)]
mgr/volumes: add operation state machine table

... and fetch creation state from state machine table.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 46f29bf3b4fbb86aeefc940d47eadd6634b5a1ec)

5 years agomgr/volumes: fail removing subvolume with snapshots
Venky Shankar [Fri, 17 Jan 2020 09:42:58 +0000 (04:42 -0500)]
mgr/volumes: fail removing subvolume with snapshots

Fixes: http://tracker.ceph.com/issues/43645
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit c158a1334207966c9727eb1e9e2eaab756e5e08b)

5 years agomgr/volumes: remove stale subvolume module
Venky Shankar [Wed, 15 Jan 2020 04:46:09 +0000 (23:46 -0500)]
mgr/volumes: remove stale subvolume module

this was lying around post versioning changes.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 5595476af7fe4dc2dfdeae17d5507d1d8eeeb3f5)

5 years agotest: auto-upgrade subvolume test
Venky Shankar [Tue, 17 Dec 2019 05:55:35 +0000 (00:55 -0500)]
test: auto-upgrade subvolume test

Fixes: https://tracker.ceph.com/issues/43349
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 03ee966b6c92b43a4f38f35351baf9ebc0126bbc)

5 years agomgr/volumes: tie everything together to implement versioned subvolumes
Venky Shankar [Wed, 20 Nov 2019 14:11:17 +0000 (09:11 -0500)]
mgr/volumes: tie everything together to implement versioned subvolumes

apart from the new way of provisioning subvolumes, this makes heavy
use of context manager for volumes, groups and subvolumes.

this change classifies volumes, groups and subvolumes to be treated
as filesystem dentries and inodes. a "volume" can be thought as a
dentry with "groups" as it's entries (inodes). likewise, a "group"
is a dentry again with "subvolumes" as entries (inodes). this is
built into the access mechanism as follows:

      with open_volume(...) as fs_handle:
          with open_gorup(fs_handle, ...) as group:
              with open_subvolume(group, ...) as subvolume:
                  # call subvolume object API
                  path = subvolume.getpath()

this way, lot of redundant checks such as verifying if a volume or
group exist before accessing a subvolume is built right into the
access mechanism, plus, an added bonus of simple error handling.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 9b87bd72093eaecda5e2ffc3ed32e482431f9842)

5 years agomgr/volumes: provide subvolume create/remove/open APIs
Venky Shankar [Wed, 20 Nov 2019 14:02:53 +0000 (09:02 -0500)]
mgr/volumes: provide subvolume create/remove/open APIs

create_subvolume() creates a subvolume with the max version known
to the plugin. open_subvolume() performs version discovery by
using loader stub and returns a subvoule object.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 8a649148025620991a67ed761153f18594f5268c)

5 years agomgr/volumes: implement subvolume based on subvolume template
Venky Shankar [Wed, 20 Nov 2019 13:45:44 +0000 (08:45 -0500)]
mgr/volumes: implement subvolume based on subvolume template

subvolume base class implements common routines/helpers and
initializes a metadata manager. later, when v2 subvolume version
is implemented, the metadata manager would be used to persist
subvolume metadata in ceph filesystem. this would allow flexible
metadata management when complex subvolume features are added.

typically, a subvolume would be implemented by subclassing the
subvolume base class and the subvolume template -- instantiating
this would be called a "subvolume object".

with this commit, current subvolume topology is maintained. but
we introduce the concept of subvolume versions. a loader stub
loads available "versions" of subvolumes. right now, the only
available version is v1. since backward compatibility needs to
be maintained for existing subvolumes, the loader API allows
version discovery w/ auto upgradation to the most recent version.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 97170d72b764ed26e93b45e8e2e7ecc2368a2a04)

5 years agomgr/volumes: implement subvolume group based on group template
Venky Shankar [Wed, 20 Nov 2019 13:37:41 +0000 (08:37 -0500)]
mgr/volumes: implement subvolume group based on group template

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 3eccd618ecce13052e31d07437ee2d09ae98e5c4)

5 years agomgr/volumes: implement trash as a subvolume group
Venky Shankar [Wed, 20 Nov 2019 14:09:37 +0000 (09:09 -0500)]
mgr/volumes: implement trash as a subvolume group

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 0e3c48e39b2f366877fb38aa79438b0f7c5ed075)

5 years agomgr/volumes: snapshot util module
Venky Shankar [Wed, 20 Nov 2019 13:36:52 +0000 (08:36 -0500)]
mgr/volumes: snapshot util module

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit b30f0cb18543d7bcd9161b4e39d163425df19a27)

5 years agomgr/volumes: template for implementing groups and subvolumes
Venky Shankar [Wed, 20 Nov 2019 13:33:01 +0000 (08:33 -0500)]
mgr/volumes: template for implementing groups and subvolumes

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 74f349fe1550f4e5a67760582628436a828a99ce)

5 years agomgr/volumes: implement filesystem volume module
Venky Shankar [Wed, 20 Nov 2019 13:41:38 +0000 (08:41 -0500)]
mgr/volumes: implement filesystem volume module

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit f9ae6e38600bada917e9d7c953e736c5c45e7064)

5 years agomgr/volumes: lock module to serialize volume operations
Venky Shankar [Wed, 20 Nov 2019 13:30:44 +0000 (08:30 -0500)]
mgr/volumes: lock module to serialize volume operations

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit bc89d05358404039938d260da1a7f0f6a4953e00)

5 years agomgr/volumes: introduce volume specification module
Venky Shankar [Wed, 20 Nov 2019 13:27:11 +0000 (08:27 -0500)]
mgr/volumes: introduce volume specification module

unlike existing subvolume specification, this is just a
minimal set of globally available configurations. bulk
of other configurations will be moved to the respective
entity modules (subsequent commits).

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 0039b5d90142c9eedd23af3c5dbf2c21bcb56823)

5 years agomgr/volumes: add fs_util helper module
Venky Shankar [Wed, 20 Nov 2019 13:24:41 +0000 (08:24 -0500)]
mgr/volumes: add fs_util helper module

helpers for various filesystem querying routines, utils
for creating/removing filesystem, pool and MDSs.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 6682c7736f4d6462aefd5387ea9f34f97ce28264)

5 years agomgr/volumes: drop obsolete comment in _cmd_fs_volume_create
Jos Collin [Fri, 3 Jan 2020 03:14:44 +0000 (08:44 +0530)]
mgr/volumes: drop obsolete comment in _cmd_fs_volume_create

This is fixed already.
Now the pool names are:
cephfs.<volume name>.meta
cephfs.<volume name>.data

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit ffda5f651106d2d0466a3376d0dab7291720b38c)

5 years agomgr/volumes: cleanup on fs create error
Jos Collin [Thu, 2 Jan 2020 10:18:37 +0000 (15:48 +0530)]
mgr/volumes: cleanup on fs create error

* clean up on fs create error
* drop unnecessary check in create_pool

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 171c37504f71df95ec10fb294229abbc09d36e47)

5 years agomgr/volumes: move up 'confirm' validation
Joshua Schmid [Wed, 20 Nov 2019 13:58:47 +0000 (14:58 +0100)]
mgr/volumes: move up 'confirm' validation

Instead of checking if the --yes-i-really-mean-it
flag was set _after_ removing the MDS daemon, we
need to check if before starting any removal operation.

Fixes: https://tracker.ceph.com/issues/42931
Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit c1db7f8b42f238f5875386a71d30bc34ae29171c)

5 years agomgr/volumes: remove unsed variable
Joshua Schmid [Thu, 21 Nov 2019 11:54:19 +0000 (12:54 +0100)]
mgr/volumes: remove unsed variable

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit 2f705ee69bbad22c1744905276527c3515877caf)

5 years agomgr/volumes: guard volume delete by waiting for pending ops
Venky Shankar [Fri, 11 Oct 2019 07:44:32 +0000 (03:44 -0400)]
mgr/volumes: guard volume delete by waiting for pending ops

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 968f67576e10fcb486819cfe3e5b44586d0d7b84)

5 years agomgr/volumes: cleanup libcephfs handles when stopping
Venky Shankar [Thu, 10 Oct 2019 13:23:10 +0000 (09:23 -0400)]
mgr/volumes: cleanup libcephfs handles when stopping

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 2eb0c503047bdf46b0a884df96819cba8979b624)

5 years agomgr/volumes: refactor dir handle cleanup
Jos Collin [Mon, 14 Oct 2019 04:51:35 +0000 (10:21 +0530)]
mgr/volumes: refactor dir handle cleanup

introduce with statement in rmtree. This change
simplifies the code's handling of directory cleanup.

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 9e27cd1aac1169b2cf547cc4adb38c7ac8409dd1)

5 years agomgr/volumes: cleanup leftovers from earlier purge job implementation
Venky Shankar [Thu, 10 Oct 2019 07:28:24 +0000 (03:28 -0400)]
mgr/volumes: cleanup leftovers from earlier purge job implementation

... which was not fully implemented anyway, so just remove the
boilerplates.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 45c22bdcea4e23d101e27c748536eecf65dc79e7)

5 years agoqa/tasks: Nothing to clean up if the volume was not created
Jos Collin [Fri, 3 Jan 2020 06:38:35 +0000 (12:08 +0530)]
qa/tasks: Nothing to clean up if the volume was not created

There are only 2 cases which needs cleanup:
1. The volume is successfully created
2. The volume is successfully created but create_mds fails

In either case, we could do a 'volume rm'.

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 67e43f466826f121d7649da33954b5b6a5454101)

5 years agoqa/tasks: Fix the volume ls in test_volume_rm
Jos Collin [Tue, 26 Nov 2019 12:24:30 +0000 (17:54 +0530)]
qa/tasks: Fix the volume ls in test_volume_rm

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 691352413e2b8ada25b587c1287a47087c3b9385)

5 years agoqa/tasks: tests for 'fs volume create' and 'fs volume ls'
Jos Collin [Tue, 26 Nov 2019 10:00:58 +0000 (15:30 +0530)]
qa/tasks: tests for 'fs volume create' and 'fs volume ls'

Fixes: https://tracker.ceph.com/issues/42872
Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 5e998bd9b5d0a6d819d87f74b9769a4086b886fe)

5 years agoqa/tasks: remove subvolume, subvolumegroup and their snapshots with --force
Jos Collin [Tue, 15 Oct 2019 12:35:20 +0000 (18:05 +0530)]
qa/tasks: remove subvolume, subvolumegroup and their snapshots with --force

* tests 'fs subvolume rm --force'
* tests 'fs subvolume snapshot rm --force'
* tests 'fs subvolumegroup rm --force'
* tests 'fs subvolumegroup snapshot rm --force'

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 42c135d85a01213b6abf0b3b86d549e025e7325d)

5 years agoqa/tasks: Fix the commands success
Jos Collin [Tue, 10 Sep 2019 11:33:48 +0000 (17:03 +0530)]
qa/tasks: Fix the commands success

* Raised RuntimeException when the commands, which were expected to fail succeed.
* Dropped some commands with --force remove commands, as it is unnecessary.

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit c8f6a25ca4f5c4ec870bfed81d640ea2aa5cc7bb)

5 years agoqa/tasks: Fix raises that doesn't re-raise
Jos Collin [Fri, 6 Sep 2019 06:17:47 +0000 (11:47 +0530)]
qa/tasks: Fix raises that doesn't re-raise

* Fixed raises that doesn't re-raise
* Dropped some commands with --force remove commands, as it is unnecessary.

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 992d8b6a74400133405db69f1da94c98cf47fed6)

5 years agotest: use distinct subvolume/group/snapshot names
Venky Shankar [Tue, 5 Nov 2019 08:37:41 +0000 (03:37 -0500)]
test: use distinct subvolume/group/snapshot names

Fixes: http://tracker.ceph.com/issues/42646
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit b9cff8af8e8f423c8e2b58de2b4da48842223cf3)

5 years agopybind/cephfs: add method that stats symlinks without following
Rishabh Dave [Sat, 13 Jul 2019 05:59:17 +0000 (11:29 +0530)]
pybind/cephfs: add method that stats symlinks without following

Add a new Python binding equivalent to lstat so that information about
the symlink itself can be also obtained, along with other type of files.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 8cce7dad8f8f23f19023d9b68b072ec8026ae04d)

5 years agoMerge pull request #33116 from batrick/i43137
Ramana Raja [Wed, 12 Feb 2020 10:08:08 +0000 (15:38 +0530)]
Merge pull request #33116 from batrick/i43137

nautilus: pybind/mgr/volumes: idle connection drop is not working

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #33220 from yuriw/wip-yuriw-clients-upgrades-nautilus
Yuri Weinstein [Tue, 11 Feb 2020 21:41:31 +0000 (13:41 -0800)]
Merge pull request #33220 from yuriw/wip-yuriw-clients-upgrades-nautilus

qa/tests: added client-upgrade-nautilus suite to be used on octopus …

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoqa/tests: added client-upgrade-nautilus suite to be used on octopus release 33220/head
Yuri Weinstein [Tue, 11 Feb 2020 19:45:06 +0000 (11:45 -0800)]
qa/tests: added client-upgrade-nautilus suite to be used on octopus release

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
5 years agoMerge pull request #32863 from shyukri/wip-43341-nautilus
Jan Fajerski [Tue, 11 Feb 2020 19:39:08 +0000 (20:39 +0100)]
Merge pull request #32863 from shyukri/wip-43341-nautilus

nautilus: ceph-volume: lvm/deactivate: add unit tests, remove --all

5 years agoMerge pull request #33217 from jan--f/wip-32242-notrack-nautilus
Jan Fajerski [Tue, 11 Feb 2020 18:11:58 +0000 (19:11 +0100)]
Merge pull request #33217 from jan--f/wip-32242-notrack-nautilus

nautilus: ceph-volume: add methods to pass filters to pvs, vgs and lvs commands

5 years agoceph-volume: add methods to pass filters to pvs, vgs and lvs commands 33217/head
Rishabh Dave [Thu, 21 Nov 2019 14:33:32 +0000 (20:03 +0530)]
ceph-volume: add methods to pass filters to pvs, vgs and lvs commands

Filters can be passed to these commands by using option '-S'.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit a4f2fcefb80df524c472782e8c1d18b3c2d1c1fc)

5 years agoMerge pull request #32912 from smithfarm/wip-43509-nautilus
Yuri Weinstein [Tue, 11 Feb 2020 16:41:52 +0000 (08:41 -0800)]
Merge pull request #32912 from smithfarm/wip-43509-nautilus

nautilus: mon: print FSMap regardless of file system count

Reviewed-by: Ramana Raja <rraja@redhat.com>
5 years agolvm/deactivate: add unit tests, remove --all 32863/head
Jan Fajerski [Mon, 16 Dec 2019 15:57:58 +0000 (16:57 +0100)]
lvm/deactivate: add unit tests, remove --all

Remove the --all flag until its actually implemented.

Fixes: https://tracker.ceph.com/issues/43330
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit c13901f963d515fd34bd0ff9df67bcf214507b70)

Conflicts:
src/ceph-volume/ceph_volume/devices/lvm/deactivate.py

5 years agoMerge pull request #33209 from jan--f/wip-deactivate-nautilus
Jan Fajerski [Tue, 11 Feb 2020 16:14:12 +0000 (17:14 +0100)]
Merge pull request #33209 from jan--f/wip-deactivate-nautilus

nautilus: ceph-volume: lvm deactivate command

5 years agoMerge pull request #32915 from smithfarm/wip-43628-nautilus
Yuri Weinstein [Tue, 11 Feb 2020 16:03:05 +0000 (08:03 -0800)]
Merge pull request #32915 from smithfarm/wip-43628-nautilus

nautilus: client: disallow changing fuse_default_permissions option at runtime

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
5 years agoMerge pull request #32914 from smithfarm/wip-43624-nautilus
Yuri Weinstein [Tue, 11 Feb 2020 16:02:21 +0000 (08:02 -0800)]
Merge pull request #32914 from smithfarm/wip-43624-nautilus

nautilus: mds: note client features when rejecting client

Reviewed-by: Ramana Raja <rraja@redhat.com>
5 years agoMerge pull request #32913 from smithfarm/wip-43573-nautilus
Yuri Weinstein [Tue, 11 Feb 2020 16:01:45 +0000 (08:01 -0800)]
Merge pull request #32913 from smithfarm/wip-43573-nautilus

nautilus: cephfs-journal-tool: fix crash and usage

Reviewed-by: Ramana Raja <rraja@redhat.com>
5 years agoMerge pull request #32909 from smithfarm/wip-43343-nautilus
Yuri Weinstein [Tue, 11 Feb 2020 16:00:25 +0000 (08:00 -0800)]
Merge pull request #32909 from smithfarm/wip-43343-nautilus

nautilus: mds: fix revoking caps after after stale->resume circle

Reviewed-by: Ramana Raja <rraja@redhat.com>
5 years agoMerge pull request #32602 from batrick/i43558
Yuri Weinstein [Tue, 11 Feb 2020 15:59:47 +0000 (07:59 -0800)]
Merge pull request #32602 from batrick/i43558

nautilus: mds: reject forward scrubs when cluster has multiple active MDS (more than one rank)

Reviewed-by: Ramana Raja <rraja@redhat.com>
5 years agoMerge pull request #32600 from batrick/i43506
Yuri Weinstein [Tue, 11 Feb 2020 15:59:19 +0000 (07:59 -0800)]
Merge pull request #32600 from batrick/i43506

nautilus: MDSMonitor: warn if a new file system is being created with an EC default data pool

Reviewed-by: Ramana Raja <rraja@redhat.com>
5 years agoMerge pull request #30843 from smithfarm/wip-41853-nautilus
Yuri Weinstein [Tue, 11 Feb 2020 15:58:33 +0000 (07:58 -0800)]
Merge pull request #30843 from smithfarm/wip-41853-nautilus

nautilus: mds: reject sessionless messages

Reviewed-by: Ramana Raja <rraja@redhat.com>
5 years agolvm: add deactivate subcommand 33209/head
Jan Fajerski [Wed, 11 Dec 2019 13:54:16 +0000 (14:54 +0100)]
lvm: add deactivate subcommand

This new subcommand unmounts and OSDs tmpfs mount and closes crypt
devices if there are any.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 9797f6b83d727933b1a0daab239f3034675a1d0e)

5 years agoutil/system: add unmount_tmpfs helper
Jan Fajerski [Thu, 12 Dec 2019 08:28:27 +0000 (09:28 +0100)]
util/system: add unmount_tmpfs helper

Thsi unmounts a path if and only if it's a tmpfs mount.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 705ed1110138b1e7c77768f97d5acda4a76d868f)

5 years agoapi/lvm: add get_lv_by_osd_id method
Jan Fajerski [Wed, 11 Dec 2019 12:17:19 +0000 (13:17 +0100)]
api/lvm: add get_lv_by_osd_id method

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 2558b55b0ecbcc83acccbf8412c19f01e8dafb3e)

5 years agoapi/lvm: add deactivate method to Volume class
Jan Fajerski [Wed, 11 Dec 2019 12:17:02 +0000 (13:17 +0100)]
api/lvm: add deactivate method to Volume class

Simply calls lvchange -an to deactivate a logical volume.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 8087600d4853554794b709291a08d97befd07ed1)

5 years agoMerge pull request #32868 from shyukri/wip-42945-nautilus
Jan Fajerski [Tue, 11 Feb 2020 13:23:27 +0000 (14:23 +0100)]
Merge pull request #32868 from shyukri/wip-42945-nautilus

nautilus: ceph-volume: allow raw block devices everywhere

5 years agodoc: update ceph-volume lvm prepare 32868/head
Jan Fajerski [Tue, 5 Nov 2019 12:39:27 +0000 (13:39 +0100)]
doc: update ceph-volume lvm prepare

Add option to pass raw physical devices everywhere, restructure a little
(bluestore section before filestore) and reword a few things.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit f2018d76eb8936aed8643328f0d5f111c87514e4)

5 years agoceph-volume: make lvm report fields into constants
Jan Fajerski [Tue, 5 Nov 2019 07:31:44 +0000 (08:31 +0100)]
ceph-volume: make lvm report fields into constants

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 01a603f6e952d2c5931b89699a86503e281fb376)

5 years agoceph-volume: api/lvm create or reuse a vg
Jan Fajerski [Tue, 22 Oct 2019 12:32:11 +0000 (14:32 +0200)]
ceph-volume: api/lvm create or reuse a vg

This changes create_lv so one can pass the desired device and either a
VG with a name starting with ceph is re-used or a new one is created.
This commit also adds two new lvm primitives, making use of lvm's select
feature. The goal is to eventually avoid keeping a full list of lv's (or
vg's) around and query the lvm system as needed.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit bb4de1a3fc238eaf9f717dc59c6bdf338ef6d657)

5 years agoMerge pull request #33202 from jan--f/wip-43853-nautilus
Jan Fajerski [Tue, 11 Feb 2020 12:08:01 +0000 (13:08 +0100)]
Merge pull request #33202 from jan--f/wip-43853-nautilus

nautilus: ceph-volume/batch: fail on filtered devices when non-interactive

5 years agoMerge pull request #33200 from jan--f/wip-42898-nautilus
Jan Fajerski [Tue, 11 Feb 2020 12:03:59 +0000 (13:03 +0100)]
Merge pull request #33200 from jan--f/wip-42898-nautilus

nautilus: ceph-volume: make get_devices fs location independent

5 years agoMerge pull request #32870 from shyukri/wip-43117-nautilus
Jan Fajerski [Tue, 11 Feb 2020 11:34:53 +0000 (12:34 +0100)]
Merge pull request #32870 from shyukri/wip-43117-nautilus

nautilus: ceph-volume: import mock.mock instead of unittest.mock (py2)

5 years agoMerge pull request #32877 from shyukri/wip-43570-nautilus
Jan Fajerski [Tue, 11 Feb 2020 11:32:01 +0000 (12:32 +0100)]
Merge pull request #32877 from shyukri/wip-43570-nautilus

nautilus: ceph-volume: Dereference symlink in lvm list

5 years agoMerge pull request #32873 from shyukri/wip-43201-nautilus
Jan Fajerski [Tue, 11 Feb 2020 11:31:08 +0000 (12:31 +0100)]
Merge pull request #32873 from shyukri/wip-43201-nautilus

nautilus: ceph-volume: fix the integer overflow

5 years agoMerge pull request #32874 from shyukri/wip-43321-nautilus
Jan Fajerski [Tue, 11 Feb 2020 11:19:50 +0000 (12:19 +0100)]
Merge pull request #32874 from shyukri/wip-43321-nautilus

nautilus: ceph-volume: use correct extents if using db-devices and >1 osds_per_device

5 years agoMerge pull request #32864 from shyukri/wip-43462-nautilus
Jan Fajerski [Tue, 11 Feb 2020 11:09:23 +0000 (12:09 +0100)]
Merge pull request #32864 from shyukri/wip-43462-nautilus

nautilus: ceph-volume/lvm/activate.py: clarify error message: fsid refers to osd_fsid

5 years agoMerge pull request #32860 from shyukri/wip-43281-nautilus
Jan Fajerski [Tue, 11 Feb 2020 11:08:22 +0000 (12:08 +0100)]
Merge pull request #32860 from shyukri/wip-43281-nautilus

nautilus: ceph-volume: util: look for executable in $PATH

5 years agoMerge pull request #31616 from jan--f/wip-42800-nautilus
Jan Fajerski [Tue, 11 Feb 2020 11:07:20 +0000 (12:07 +0100)]
Merge pull request #31616 from jan--f/wip-42800-nautilus

nautilus: ceph-volume: assume msgrV1 for all branches containing mimic

5 years agoceph-volume/batch: fail on filtered devices when non-interactive 33202/head
Jan Fajerski [Tue, 3 Dec 2019 12:44:00 +0000 (13:44 +0100)]
ceph-volume/batch: fail on filtered devices when non-interactive

When batch is called non-interactively and a user explicitly specifies,
say a db-device, this will be filtered when unavailable. This can cause
the resulting OSD to be very different from the users intention
(standalone vs external db when the db-device was filtered). If devices
get filtered in non-interactive mode, ceph-volume should fail.

Fixes: https://tracker.ceph.com/issues/43105
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 2e985053deec6c4cf60c0b85aec3df16cd77ceeb)

5 years agoceph-volume: refactor tests for refactored get_devices 33200/head
Jan Fajerski [Mon, 11 Nov 2019 13:35:43 +0000 (14:35 +0100)]
ceph-volume: refactor tests for refactored get_devices

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 4749f4c59dc4edb1fd689e18e8d9f358f95cb945)

Conflicts:
src/ceph-volume/ceph_volume/tests/conftest.py
        resolved by importing PropertyMock

5 years agoceph-volume: refactor get_devices, don't use os.path.realpath
Jan Fajerski [Fri, 8 Nov 2019 15:36:30 +0000 (16:36 +0100)]
ceph-volume: refactor get_devices, don't use os.path.realpath

Fixes: https://tracker.ceph.com/issues/42777
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit b35e8c485f73b5b65e0be29444eb06172d6df3e3)

5 years agoMerge pull request #32558 from shyukri/wip-43275-nautilus
Jan Fajerski [Tue, 11 Feb 2020 10:40:48 +0000 (11:40 +0100)]
Merge pull request #32558 from shyukri/wip-43275-nautilus

nautilus: ceph-volume/test: patch VolumeGroups

5 years agoMerge pull request #32556 from shyukri/wip-43022-nautilus
Jan Fajerski [Tue, 11 Feb 2020 10:39:40 +0000 (11:39 +0100)]
Merge pull request #32556 from shyukri/wip-43022-nautilus

nautilus: ceph-volume: minor clean-up of "simple scan" subcommand help

5 years agoMerge pull request #32998 from neha-ojha/wip-min-alloc-nautilus
Yuri Weinstein [Mon, 10 Feb 2020 21:46:00 +0000 (13:46 -0800)]
Merge pull request #32998 from neha-ojha/wip-min-alloc-nautilus

nautilus: common/options: bluestore 4k min_alloc_size for SSD

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
5 years agoMerge pull request #32846 from smithfarm/wip-43256-nautilus
Yuri Weinstein [Mon, 10 Feb 2020 21:37:08 +0000 (13:37 -0800)]
Merge pull request #32846 from smithfarm/wip-43256-nautilus

nautilus: common/config: update values when they are removed via mon

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #32997 from neha-ojha/wip-32939-nautilus
Yuri Weinstein [Mon, 10 Feb 2020 20:46:12 +0000 (12:46 -0800)]
Merge pull request #32997 from neha-ojha/wip-32939-nautilus

nautilus: mon/MgrMonitor.cc: add always_on_modules to the output of "ceph mgr module ls"

Reviewed-by: David Zafman <dzafman@redhat.com>
5 years agoMerge pull request #32901 from smithfarm/wip-43631-nautilus
Yuri Weinstein [Mon, 10 Feb 2020 20:44:42 +0000 (12:44 -0800)]
Merge pull request #32901 from smithfarm/wip-43631-nautilus

nautilus: common/util: use ifstream to read from /proc files

Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoMerge pull request #32858 from smithfarm/wip-43473-nautilus
Yuri Weinstein [Mon, 10 Feb 2020 20:43:33 +0000 (12:43 -0800)]
Merge pull request #32858 from smithfarm/wip-43473-nautilus

nautilus: common: fix deadlocky inflight op visiting in OpTracker.

Reviewed-by: Kefu Chai <kchai@redhat.com>
5 years agoMerge pull request #32845 from smithfarm/wip-43245-nautilus
Yuri Weinstein [Mon, 10 Feb 2020 20:41:21 +0000 (12:41 -0800)]
Merge pull request #32845 from smithfarm/wip-43245-nautilus

nautilus: os/bluestore/BlueStore.cc: set priorities for compression stats

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
5 years agoMerge pull request #32774 from dzafman/wip-43726-nautilus
Yuri Weinstein [Mon, 10 Feb 2020 20:38:35 +0000 (12:38 -0800)]
Merge pull request #32774 from dzafman/wip-43726-nautilus

nautilus: test: Fix wait_for_state() to wait for a PG to get into a state

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoceph-volume: Dereference symlink in lvm list 32877/head
Benoît Knecht [Thu, 2 Jan 2020 07:07:21 +0000 (08:07 +0100)]
ceph-volume: Dereference symlink in lvm list

This allows for a symlink to be passed to

```
ceph-volume lvm list <path>
```

which makes it possible to use `/dev/disk/by-path/*` devices, for
instance.

Fixes: https://tracker.ceph.com/issues/43497
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
(cherry picked from commit 09fa3df8a39d361f2af11bded8aab9e0da334d51)

5 years agoceph-volume: use correct extents when using db-devices and >1 osds_per_device 32874/head
Fabian Niepelt [Wed, 11 Dec 2019 13:19:14 +0000 (14:19 +0100)]
ceph-volume: use correct extents when using db-devices and >1 osds_per_device

Actual data size depending on osds_per_device needs to be calculated here. Otherwise, if osds_per_device is greater than 1, ceph-volume will allocate 100% of the device to the first osd and then fail to create the LV for the second because the volume group is already full.

Fixes: https://tracker.ceph.com/issues/39442
Signed-off-by: Fabian Niepelt <f.niepelt@mittwald.de>
(cherry picked from commit ecde6cd619605003706f545b0b5ad4e2e8932f28)

 Conflicts:
src/ceph-volume/ceph_volume/devices/lvm/strategies/bluestore.py

I've removed `data_uuid` since it's not in nautilus already

5 years agoceph-volume: fix the type mismatch, covert the tries and interval to int from string. 32873/head
taodd [Sun, 8 Dec 2019 02:10:59 +0000 (10:10 +0800)]
ceph-volume: fix the type mismatch, covert the tries and interval to int from string.
Fixes: https://tracker.ceph.com/issues/43186
Signed-off-by: dongdong tao <dongdong.tao@canonical.com>
(cherry picked from commit 81ff4be4d13ce034d7a8b921113138a942e3051a)

5 years agoceph-volume: import mock.mock instead of unittest.mock (py2) 32870/head
Jan Fajerski [Fri, 22 Nov 2019 15:57:11 +0000 (16:57 +0100)]
ceph-volume: import mock.mock instead of unittest.mock (py2)

Fixes: bb4de1a3fc238eaf9f717dc59c6bdf338ef6d657
Fixes: https://tracker.ceph.com/issues/42970
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit c1bd09f2591d983f1988e3dadcf3c65c59afb15d)

Conflicts:
src/ceph-volume/ceph_volume/tests/api/test_lvm.py

5 years agolvm/activate.py: clarify error message: fsid refers to osd_fsid 32864/head
Yaniv Kaul [Thu, 19 Dec 2019 14:54:58 +0000 (16:54 +0200)]
lvm/activate.py: clarify error message: fsid refers to osd_fsid

Users complained[1] the error message isn't clear, and they thought
it referred to the cluster fsid instead of the osd_fsid.
Made it clearer.

[1] rook/rook#4547

Fixes: https://tracker.ceph.com/issues/43442
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
(cherry picked from commit ff3ba9200a6b5856fbdf4372fe292a7b94dd2539)

5 years agoceph-volume: util: look for executable in $PATH 32860/head
Shyukri Shyukriev [Thu, 21 Nov 2019 14:04:02 +0000 (15:04 +0100)]
ceph-volume: util: look for executable in $PATH

Fixes: https://tracker.ceph.com/issues/36728
Fallback to predefined paths for backward compatibility.
Alter test involved for partial match in warning

Signed-off-by: Shyukri Shyukriev <shshyukriev@suse.com>
(cherry picked from commit a8577085dc52b0f214d9568c29a9605d1a826a45)

5 years agoMerge pull request #30689 from smithfarm/wip-42120-nautilus
Yuri Weinstein [Sat, 8 Feb 2020 00:27:15 +0000 (16:27 -0800)]
Merge pull request #30689 from smithfarm/wip-42120-nautilus

nautilus: core: osd/OSDMap: health alert for non-power-of-two pg_num

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #32857 from smithfarm/wip-43471-nautilus
Yuri Weinstein [Fri, 7 Feb 2020 22:31:36 +0000 (14:31 -0800)]
Merge pull request #32857 from smithfarm/wip-43471-nautilus

nautilus: osd/PeeringState.cc: don't let num_objects become negative

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #32848 from smithfarm/wip-43346-nautilus
Yuri Weinstein [Fri, 7 Feb 2020 22:31:01 +0000 (14:31 -0800)]
Merge pull request #32848 from smithfarm/wip-43346-nautilus

nautilus: qa/suites/rados/thrash: force normal pg log length with cache tiering

Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #32847 from smithfarm/wip-43319-nautilus
Yuri Weinstein [Fri, 7 Feb 2020 22:30:26 +0000 (14:30 -0800)]
Merge pull request #32847 from smithfarm/wip-43319-nautilus

nautilus: osd/PeeringState.cc: skip peer_purged when discovering all missing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge pull request #32843 from smithfarm/wip-43099-nautilus
Yuri Weinstein [Fri, 7 Feb 2020 22:29:48 +0000 (14:29 -0800)]
Merge pull request #32843 from smithfarm/wip-43099-nautilus

nautilus: osd/OSD: enhance osd numa affinity compatibility

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #32773 from dzafman/wip-43246-nautilus
Yuri Weinstein [Fri, 7 Feb 2020 22:29:08 +0000 (14:29 -0800)]
Merge pull request #32773 from dzafman/wip-43246-nautilus

nautilus: osd: Use physical ratio for nearfull (doesn't include backfill resserve)

Reviewed-by: Neha Ojha <nojha@redhat.com>
5 years agoMerge branch 'nautilus' into wip-42120-nautilus 30689/head
David Zafman [Fri, 7 Feb 2020 22:27:18 +0000 (14:27 -0800)]
Merge branch 'nautilus' into wip-42120-nautilus

5 years agoMerge pull request #32716 from smithfarm/wip-43650-nautilus
Yuri Weinstein [Fri, 7 Feb 2020 22:11:40 +0000 (14:11 -0800)]
Merge pull request #32716 from smithfarm/wip-43650-nautilus

nautilus: osd: Diagnostic logging for upmap cleaning

Reviewed-by: David Zafman <dzafman@redhat.com>