Remove information about the installation of the Zabbix module and link
to a discussion of the reasoning behind Ceph's refusal to support
Zabbix.
John Jasen developed a procedure explaining how to install "Zabbix 2".
This commit removes outdated procedures and explains why those
procedures were removed. Immediately following this explanation, the
text includes an explanation of how to install "Zabbix 2".
Rishabh Dave [Thu, 2 Nov 2023 19:02:06 +0000 (00:32 +0530)]
qa/cephfs: allow finer control on test runs
When working with large group tests (18 in this case), it gets very
tedious to debug and fix tests when all 18 have to be run again for
every mistake. Cheap fix for this to split these 18 tests into several
classes.
But when modification are made to the feature, all these 18 tests needs
to exercised and previous solution forces developer to intitiate all
these test classes to run one by one.
Best of both worlds can be achieved if we split tests into group but
move all these related group to a new file.
Rishabh Dave [Sun, 29 Jan 2023 20:48:54 +0000 (02:18 +0530)]
cephfs: add command "ceph fs swap"
Add a FS command that enables users to swap names of two file systems in
a single PAXOS transaction. Add an option to this command that swaps
FSCIDS along with FS names. This commands also updates the application
pool tags and fails when mirroring is enabled on either or both FSs.
Fixes: https://tracker.ceph.com/issues/58129 Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 717031454243898071bdb254b2cadc7048057903)
Conflicts:
doc/man/8/ceph.rst
Conflict occurred because Ceph man page contians lesser amout of
CephFS commands in it in Reef branch
src/mds/FSMap.h
* Methods like set_fscid() can be cherry-picked as it is but due
to different lines around, patch couldn't be applied as it is.
* Methods like get_fscid(), get_mds_map() are absent in Reef
branch. They have been kept/added during this conflict
resolution.
Conflicts:
* qa/tasks/cephfs/test_admin.py
The commit needs an update so that importing "sleep" from "time"
instead of importing "time" fully works on Reef branch too.
Rishabh Dave [Tue, 18 Apr 2023 14:55:01 +0000 (20:25 +0530)]
qa/cephfs/cap_tester: simplify CapTester and its instantiation
Class CapTester contains two distinct immiscible group of methods: one
that tests MON caps and other that tests MDS caps. When using CapTester
for the former reason the instantiation neither needs mount object and
the path where files for testing will be created nor it needs to run the
method that creates files for testing rw permissions. When using
this class for latter the case is the exact opposite.
Create 2 separate classes for each of these purpose and class that
inherits both of these classes so that instantiating the class becomes
as simple as it can be.
Rishabh Dave [Thu, 6 Apr 2023 09:42:14 +0000 (15:12 +0530)]
qa/cephfs: move few methods such that they can be reused
Move get_mon_cap_from_keyring() and get_fsnmes_from_moncap() from class
CapTester to main namespace of caps_helper.py so that they can be
imported freely and reused by tests.
This method checks if the output of the command "ceph fs ls" for client
ID it receives is same as the output printed for client.admin. Don't do
so, limit the test to only checking if "ceph fs ls --id client.x -k
keyring_file" prints fs name for which client.x has permissions.
Rishabh Dave [Fri, 31 Mar 2023 19:14:52 +0000 (00:44 +0530)]
qa/cephfs: improve caps_helper.CapTester
Improvement #1:
CapTester.write_test_files() not only creates the test file but also
does the following for every mount object it receives in parameters -
* carefully produces the path for the test file as per parameters
received
* generates the unique data for each test file on a CephFS mount
* creates a data structure -- list of lists -- that holds all this
information along with mount object itself for each mount object so
that tests can be conducted at a later point
Untangle this mess of code by splitting this method into 3 separate
methods -
1. To produce the path for test file (as per user's need).
2. To generate the data that will be written into the test file.
3. To actually create the test file on CephFS.
Improvement #2:
Remove the internal data structure used for testing -- self.test_set --
and use separate class attributes to store all the data required for
testing instead of a tuple. This serves two purpose -
One, it makes it easy to manipulate all this data from helper methods
and during debugging session, especially while using a PDB session.
And two, make it impossible to have multiple mounts/multiple "test sets"
within same CapTester instance for the sake of simplicity. Users can
instead create two instances of CapTester instances if needed.
Conflicts:
qa/tasks/cephfs/caps_helper.py
- Conflict occurred because this file went under modification due to
commit 59c9104c54d6.
qa/tasks/cephfs/test_admin.py
- Conflict occurred because code region located around the patch has
changed.
Backport of #56509 in reef didn't include the Table Schema in the
utils.libsonnet because of which the json generation ias failing. This
PR tends to add the Table Schema
Afreen [Wed, 27 Mar 2024 09:35:58 +0000 (15:05 +0530)]
mgr/dashboard:Use advanced fieldset for rbd image
Fixes https://tracker.ceph.com/issues/65025
- using cd-advanced-fieldset component for advanced section
- this will align advanced sections of forms in dashboard
- changed heading of sections of rbd-configuration to h5 to better represent it
as a sub section.
- added unit tests for rbd image config
Afreen [Mon, 25 Mar 2024 08:39:08 +0000 (14:09 +0530)]
mgr/dashboard:Update encryption and tags in bucket form
Fixes https://tracker.ceph.com/issues/65110
- renamed Security to Encryption
- aligned security fieldset with rest of the form using `cd-help-text`
and aligning it with object locking in UX
- aligned security fieldset top use `cd-help-text`
- changed help text of Tags
Rishabh Dave [Fri, 24 Mar 2023 11:47:36 +0000 (17:17 +0530)]
qa/cephfs: simplify some code in test_multifs_auth.py
test_mutlifs_auth.TestMDSCaps._create_client() not only creates a client
but also generate caps strings for the client as per the parameter this
method receives and and then writes the keyring to all remote machines.
This creates confusion when reading code on test methods in TestMDSCaps.
Let's re-arrange this code such that this confusion is avoided.
Rishabh Dave [Thu, 13 Apr 2023 19:08:33 +0000 (00:38 +0530)]
qa/cephfs: don't inherit CephFSTestCase in CapTester
Inheritting CephFSTestCase in CapTester just for methods assertEqual()
and assertIn() from class unittest.TestCase is odd and heavy-weight.
Don't inherit CephFSTestCase and use simple assert instead.
Rishabh Dave [Mon, 13 Mar 2023 10:46:13 +0000 (16:16 +0530)]
qa/cephfs: add more helper methods to caps_helper.py
Add methods that will accept read/write permissions, CephFS names and
CephFS mount point and in return will generate string form of MON, OSD
and MDS caps exactly as it is reported in Ceph keyrings.
Replace similar code in test_multifs_auth.py with calls to these helper
methods.
Afreen [Fri, 1 Mar 2024 07:26:25 +0000 (12:56 +0530)]
mgr/dashboard: Add advanced fieldset component
Fixes https://tracker.ceph.com/issues/65024
- adds a new shared component for displaying advanced fields in /shared
- utilizes that component in bucket form
- placement targets are under advanced fieldset
- minor help text changes in object locking help text
- updated e2e tests
This commit is not a backport but a new commit to fix the code present
on Reef branch but not present in release branches after Reef. At least
some of these issues were exposed during QA done for the backport.
Rishabh Dave [Sat, 24 Jun 2023 04:17:12 +0000 (09:47 +0530)]
AuthMonitor: no need to check permission in MDS caps
For "fs authorize" command, AuthMonitor.cc checks if permissions is "r"
or begins with "rw". This check is redundant now.
AuthMonitor::valid_caps() runs MDSAuthCaps.parse() which now runs same
check for the MDS caps, regardless of the command.
Rishabh Dave [Fri, 9 Jun 2023 18:54:12 +0000 (00:24 +0530)]
MDSAuthCaps: print a special error message for wrong permissions
Permissions mentioned in MDS caps flags can either begin with "r" or
"rw", or can be "*" and "all". But it can't start with or be just "w" or
something else. This is confusing for some CephFS users since MON caps
can be just "w".
Command "ceph fs authorize" complains about this to the user. But other
commands (specifically, "ceph auth add", "ceph auth caps",
"ceph auth get-or-create" and "ceph auth get-or-create-key") don't. Make
these commands too print a helpful message, the way "ceph fs authorize"
command does.
Fixes: https://tracker.ceph.com/issues/61666 Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit f163dd3ef1fd9f05f05fa50eda9993225770d524)
Conflicts:
src/mds/MDSAuthCaps.cc
"std::string" was replace "string" in main but that's not the case
in Reef.
Rishabh Dave [Sat, 24 Jun 2023 17:11:07 +0000 (22:41 +0530)]
qa/ceph_test_case: add a method to negative test Ceph commands
Also, add comments to explain the users the arguments are accepted by
run_ceph_cmd(), get_ceph_cmd_result(), get_ceph_cmd_stdout() and
negtest_ceph_cmd() methods of class RunCephCmd.
Rishabh Dave [Wed, 9 Aug 2023 12:40:32 +0000 (18:10 +0530)]
qa: inherit RunCephCmd in CephTestCase instead of CephFSTestCase
MgrTestCase also needs RunCephCmd. If RunCephCmd is inherited by
CephTestCase, instead of CephFSTestCase, MgrTestCase will automatically
inherit RunCephCmd because it inhertis CephTestCase.
Fixes: https://tracker.ceph.com/issues/62084 Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 4b369cf18ed1391a426ab4ae86da834e9c074f81)
Rishabh Dave [Mon, 27 Mar 2023 06:21:16 +0000 (11:51 +0530)]
qa/cephfs: use run_ceph_cmd() when cmd output is not needed
In filesystem.py and wherever instance of class Filesystem are used, use
run_ceph_cmd() instead of get_ceph_cluster_stdout() when output of Ceph
command is not required.
Rishabh Dave [Mon, 27 Mar 2023 06:09:11 +0000 (11:39 +0530)]
qa/cephfs: add helper methods to filesystem.py
Add run_ceph_cmd(), get_ceph_cmd_stdout() and get_ceph_cmd_result() to
class Filesystem so that running Ceph command is easier. This affects
not only methods inside class Filesystem but also methods elsewhere that
uses instance of class Filesystem to run Ceph commands.
Instead of "self.fs.mon_manager.raw_cluster_cmd()" writing
"self.fs.run_ceph_cmd()" will suffice.
Conflicts:
qa/tasks/cephfs/test_mirroring.py
- Commit e4dd0e41a3a0 was not present on main but it is now
present on main as well as on Reef, which leads to conflict.
- The line located right before one of the patches in this
commit was modified in latest Reef branch, thus creating
conflict when PR branch was rebased on latest Reef.
Rishabh Dave [Thu, 16 Mar 2023 10:02:39 +0000 (15:32 +0530)]
qa/cephfs: add and use get_ceph_cmd_stdout()
Add method get_ceph_cmd_stdout() to class CephFSTestCase so that one
doesn't have to type something as long as
"self.mds_cluster.mon_manager.raw_cluster_cmd()" to execute a
command and get its output. And delete and replace
CephFSTestCase.run_cluster_cmd() too.
Conflicts:
qa/tasks/cephfs/caps_helper.py
- This file is very different in Reef.
qa/tasks/cephfs/test_mirroring.py
- Commit e4dd0e41a3a0 was not present on main but it is now
present on main as well as on Reef, which leads to conflict.
- On Reef branch, the line before that patch in this commit was
thus creating a conflict when the PR branch for this commit
series was rebased on latest Reef.
Rishabh Dave [Thu, 16 Mar 2023 09:41:08 +0000 (15:11 +0530)]
qa/cephfs: add and use run_ceph_cmd()
Instead of writing something as long as
"self.mds_cluster.mon_manager.run_cluster_cmd()" to execute a command,
let's add a helper method to class CephFSTestCase and use it instead.
With this, running a command becomes simple - "self.run_ceph_cmd()".
Conflicts:
qa/tasks/cephfs/test_damage.py
This file is slightly different because this commit c8f8324ee2fae48e8d3c2bbdbf45cc9ffe46fd4c was merged on main and
backported after the commit being cherry-picked here was merged
in main.
Rishabh Dave [Tue, 14 Mar 2023 19:43:56 +0000 (01:13 +0530)]
qa/cephfs: add and use get_ceph_cmd_result()
To run a command and get its return value, instead of typing something
as long as "self.mds_cluster.mon_manager.raw_cluster_cmd_result" add a
hepler method in CephFSTestCase and use it. This makes this task very
simple - "self.get_ceph_cmd_result()".
Also, remove method CephFSTestCase.run_cluster_cmd_result() in favour of
this new method.
Rishabh Dave [Mon, 13 Mar 2023 13:05:50 +0000 (18:35 +0530)]
qa/cephfs: create CephManager instance in CephFSTestCase
To run a Ceph command conveniently, run_cluster_cmd(), raw_cluster_cmd()
or raw_cluster_cmd_result() must be called. These methods are available
in class CephManager which in turn is available only if an instance of
Filesystem, MDSCluster, CephCluster or MgrCluster is initialized. Having
an instance of CephManager in CephFSTestCase will provide easy access to
these methods.
For example, in CephFS tests writing "self.mon_manager.raw_cluser_cmd()"
instead of writing "self.mds_cluster.mon_manager.raw_cluster()" will
suffice.
This commit provides a basis for upcoming commits in this patch series.
With next patches, running Ceph command will be further simplified. Just
writing self.run_ceph_cmd() will suffice for running a CephFS command.
Patrick Donnelly [Tue, 27 Feb 2024 00:44:27 +0000 (19:44 -0500)]
mds: skip sr moves when target is an unlinked dir
A directory in the stray directory cannot have any HEAD inodes with caps so
there is no need to move anything to the snaprealm opened for the unlinked
directory.
Following the parent commit's reproducer, the behavior now looks expectedly like:
Discussions with Dan van der Ster led to the creation of this patch.
Fixes: https://tracker.ceph.com/issues/53192 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> Signed-off-by: Dan van der Ster <dan.vanderster@clyso.com>
(cherry picked from commit c190a3f1633e9282772e5ec54fe10556856a2540)
Patrick Donnelly [Fri, 12 Nov 2021 00:43:27 +0000 (19:43 -0500)]
mds: memoize descendent results during realm splits
This change uses an unordered_map to memoize results of CInode::is_ancestor_of
so that subsequent invocations can skip directory inodes which are already
known to not be a descendent of the target directory.
In the worst case, this unordered_map can grow to the number of inodes in
memory when all inodes are directories and at least one client has a cap for
each inode. However, in general this will not be the case. The size of each
entry in the map will be a 64-bit pointer and bool. The total size will vary
across platforms but we can say that with a conservative estimate of 192 bits /
entry overhead (including the entry linked list pointer in the bucket), the map
will grow to ~24MB / 1M inodes.
The result of this change is not eye-popping but it does have a significant performance advantage.
For an unpatched MDS with 1M inodes with caps in the global snaprealm (with debugging commits preceding this one):
or about 1,338ms. This caused a split of 100k inodes. This takes more time
because directories are actually moved to the snaprealm with a lot of list
twiddling for caps.
or about 840ms. This can be easily done by making a directory in one of the
trees created (see reproducer below).
Reproducing can be done with:
for ((i =0; i < 10; i++)); do (pushd $(mktemp -d -p . ); for ((j = 0; j < 30; ++j)); do mkdir "$j"; pushd "$j"; done; for ((j = 0; j < 10; ++j)); do for ((k = 0; k < 10000; ++k)); do mkdir $j.$k; done & done) & done
to make 1M directories. We put the majority of directories in a 30-deep nesting
to exercise CInode::is_ancestor_of with some worst-case type scenario.
Make sure all debugging configs are disabled for the MDS/clients. Make sure the
client has a cache size to accomodate 1M caps. Make at least one snapshot:
It is not necessary to delete any snapshots to reproduce this behavior. It's
only necessary to have a lot of inodes_with_caps in a snaprealm and effect a
split.
Fixes: https://tracker.ceph.com/issues/53192 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit a0ccb79fa0806792c7ee666c667328a8aeb09e97)
projects / ceph.git / log