Zac Dover [Tue, 14 Feb 2023 11:53:02 +0000 (21:53 +1000)]
doc/rgw: refine "Multi-site Config Ref" (1 of x)
Refine the syntax (and in some cases, the semantics) of the English in
"Multi-site Configuration Reference" in doc/radosgw/multisite.rst. This
is a long section, so more PRs are likely.
https://tracker.ceph.com/issues/58632
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Co-authored-by: Casey Bodley <cbodley@redhat.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 2d45a4e5e087cfd64d9e3cb5f346d24452a8742f)
Zac Dover [Fri, 10 Feb 2023 02:40:01 +0000 (12:40 +1000)]
doc/rgw: refine "Configuring Secondary Zones"
English grammar and syntax pass (and some pragmatics, too, where I had
enough understanding to be helpful that way) for "Configuring Secondary
Zones" in doc/radosgw/multisite.rst.
Add a note on the fact that CephFS extended attributes are not printed
by "getfattr /some/cephfs/dir/ -d -m -" even when the CephFS directory
has some extended attributes set on it.
Zac Dover [Sat, 4 Feb 2023 07:53:32 +0000 (17:53 +1000)]
doc/rgw: refine 1-50 of multisite.rst
Refine the English in the first fifty lines of multisite.rst. This is
part of a project whose goal is the creation of a document that will
impart a good aerial view of RGW to a first-time reader in the minimum
number of words.
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6721196ff129b00d9dea3515fbaaacde8f84eaea)
Prashant D [Tue, 27 Sep 2022 02:40:45 +0000 (22:40 -0400)]
mon: Fix condition to check for ceph version mismatch
MON fails to send MMgrUpdate to the MGR to update
DaemonState with correct metadata information due to
incorrect conditional check. Send MMgrUpdate to MGR
if we observe ceph version mismatch after an election
is over.
Fixes: https://tracker.ceph.com/issues/57678 Signed-off-by: Prashant D <pdhange@redhat.com>
(cherry picked from commit 5dc0d6f7f299cc7af0c243ba67036b075f4b72af)
Zac Dover [Fri, 3 Feb 2023 07:21:54 +0000 (17:21 +1000)]
doc/rgw: refine "Zones" in multisite.rst
Clean up the English under the section called "Zones". Part of a larger
project aimed at giving the reader more understanding earlier in the
documentation of how buckets and zones and zone groups and endpoints
work.
John Mulligan [Fri, 2 Sep 2022 15:36:53 +0000 (11:36 -0400)]
mgr/nfs: add --sectype flag to nfs export create functions
Fixes: https://tracker.ceph.com/issues/57404
Add the sectype argument, also for use as a CLI option, to the functions
that create nfs exports. NFS Ganesha can use the sectype configuration
option to decide what kind of security will be applied/required for
connection to an export.
NOTE: This option is only useful when Ganesha is configured along with
LDAP/Kerberos integration. Configuration of that integration is outside
the scope of these patches.
John Mulligan [Fri, 2 Sep 2022 15:33:43 +0000 (11:33 -0400)]
mgr/nfs: reformat long function calls to one line per arg
Using a hand rolled version of the long function call style similar to
that produced by the "Black" tool, put every argument to the
create_export function on a line of its own. This makes the code easier
to read and, importantly for an upcoming change, easier to add new
arguments.
John Mulligan [Thu, 25 Aug 2022 21:22:03 +0000 (17:22 -0400)]
mgr/nfs: add awareness of the SecType option to ganesha_conf.py
This NFS-Ganesha option may be used to select kerberos integration in the
nfs client connection. If the option is not present in the configuration
dict, the option will not be present in the ganesha config blocks, just
as it was before.
Adam King [Sun, 22 Jan 2023 18:59:51 +0000 (13:59 -0500)]
mgr/cephadm: fix removing offline hosts with ingress daemons
We need to convert the daemon type to service type here
or we'll try to access the "haproxy" or "keepalived" index
of the cephadm_services dictionary, which doesn't have those
keys (but does have an "ingress" key)
Adam King [Mon, 26 Sep 2022 18:02:19 +0000 (14:02 -0400)]
mgr/cephadm: fix handling of mgr upgrades with 3 or more mgrs
Fixes: https://tracker.ceph.com/issues/57675
When daemons are upgraded by cephadm, there are two criteria taken into
account for a daemon to be considered totally upgraded. The first is the
container image the daemon actually has currently. The second is the container
image of the mgr that deployed the daemon. I'll refer to these as a daemon
having the "correct version" and "correct deployed by". For reference,
the correct deployed by needs to be tracked as cephadm may change
something about the unit files it generates between versions and not
making sure daemons are deployed by the current version of cephadm
risks some obscure bugs.
The function _detect_need_upgrade takes a list of daemons and returns
two new lists. The first is all daemons from the input list that
are on the wrong version. The second are all daemons that are on the
right version but deployed by the wrong version. Additionally it returns
a bool to say whether the current active mgr must be upgraded (i.e. it
would belong in either of the two returned lists). Prior to this change,
how it would work is the second list (list of daemons that are on the right
version but have the wrong deployed by version) would simply be added to
the first list if the active mgr does not need to be upgraded. The idea
is that if you are upgrading from X image to Y image, we can only
really "fix" the deployed by version of the daemon if the active mgr
is on the Y version as it will be the one deploying the daemon. So if
the active mgr is not upgraded we can just ignore the daemons that just
have the wrong deployed by version in hte current iteration. All of this is
really only important when the mgr daemons are being upgraded. After all the
mgrs are upgraded any future upgrades of daemons will be done by a mgr on
the new version so deployed by version will always get completed
along with the version of the daemon itself. This system also works fine
for the typical 2 mgr setup.
Imagine mgr A and B on version X deployed by version X being upgraded to
version Y with A as active. First A deploys B with version Y. Now B
has version Y and deployed by version X. A then fails over to B as it
sees it needs to be upgraded. B then upgrades A so A now has version Y
and deployed by version Y. B then fails over to A as it sees it needs
to be upgraded as its deployed by version is still X. Finally, A
redeploys B and both mgrs are fully upgraded and everything is fine.
However, things can get trickier with 3 or more mgrs due to the
fact that cephadm does not control which other mgr takes over after
a failover. Imagine a similar scenario but now you have mgr
A, B, and C. First A will upgrade B and C to Y so they now
are both on version Y with deployed by version X. It then fails
over since it needs to be upgraded and let's say B takes over as
active. B then upgrade A so it now has version Y and deployed by
version Y. However, it will not redeploy C even though it should
as, given it sees that it needs to be upgraded due to its deployed by
version being wrong, it doesn't touch any daemon that just needs its
deployed by version fixed. It then fails over and lets say C takes
over. Since it still has the wrong deployed by version and therefore
thinks that it needs to be upgraded, it won't touch B since that
only needs its deployed by version fixed. It sees that it needs
to be upgraded however so it fails over. Lets say B takes over again.
You can see how we can end up in a loop here where B and C say they
need to be upgraded but never upgrade each other. It seems from what
I've seen that which mgr is picked after a failover isn't totally
random so this type of scenario can actually happen and it can get
stuck here until the user takes some action. The change here is
to, instead of not touching daemons that needs their deployed by version
fixed if the active mgr needs upgrade, only don't touch that list
if the active mgr is on the wrong version. So in our example scenario
B would still have upgraded C the first time around as it would
see it is on the correct version Y and can therefore fix the deployed
by version for C. This is what the check always should have been
but since most of the testing is with 2 mgr daemons and even with
more its by chance you end up in the loop this issue wasn't seen.
Will add that it is also possible to end up in this loop with
only 2 mgr daemons if some amount of manual upgrading of the mgr
daemons is done.
Adam King [Tue, 13 Sep 2022 17:25:02 +0000 (13:25 -0400)]
mgr/cephadm: write client files after applying services
Fixes: https://tracker.ceph.com/issues/56696 Fixes: https://tracker.ceph.com/issues/57462 Fixes: https://tracker.ceph.com/issues/57449
There is a bug described in the linked trackers where client
files are being removed for seemingly no reason. In the cases
of the trackers it's the client keyring or ceph.conf being
removed and it's causing failures in the test scenarios. After
giving the relevant code a thorough look, I couldn't come up
with any potential causes. Since one of the failures
was occurring with some regularity (maybe 1/5 times) in a teuthology
test, I just added a bunch of extra logging and ran through the
tests. That logging included
if self.mgr.manage_etc_ceph_ceph_conf or self.mgr.keys.keys:
client_files = self._calc_client_files()
self.log.error(f'ADMK - all client files {client_files}')
else:
self.log.error('ADMK - no calcing client files')
client_files = {}
around the part of the code that actually figured out what the client
files should be (that code is being removed in this commit).
In theory, this should tell me what client files were calculated
for that iteration or that no client files were calculated at all.
Additionally, there was logging to say when we were actually
writing or removing a client file (all prepended with "ADMK" to
help with searching through the logs). There was more logging
around scheduling, multiple parts of _calc_client_files etc.
You can see an example run with the logging at
http://qa-proxy.ceph.com/teuthology/adking-2022-09-13_11:46:25-fs:workload-wip-adk-removal-debugging2-distro-default-smithi/7030534/teuthology.log
where searching for "ADMK" will show all the new log statements
and can look at the two last commits in
https://github.com/adk3798/ceph/tree/wip-adk-removal-debugging2
to see what all the logging statement that were added are.
As expected, this logigng eventually gave a message about
it removing the relevant files and additionally stated that
the client_files calculated were "{}" as in, no files were
calculated. Given that, I expected to see "ADMK - all client files {}"
or "ADMK - no calcing client files" in the logs not far above that.
But they weren't there.I could not find a single instance within 30 minutes
of the removal of it actually calculating that there should be no client
files (the only instances of that were very early in the test before things
had been setup). In fact, directly leading up to when the files were removed,
the most recent instance of either of the logging statements in the posted
code snippet was about 20 seconds earlier and had correctly
calculated that there should be a number of client files.
Somehow it is deciding that client_files is an empty dict
but I simply cannot find any instance anywhere of it actually
doing so (outside of early on before the client files are setup
where it is correct to say that). The solution proposed in this
commit is simply to calculate the client files right before writing
them in the hope that this consolidation of the functionality will
maybe prevent whatever was setting client_files to an empty dict from
doing so or at least make the debugging process easier. The plan is to
basically, unless someone has a good idea, to run this through testing
we know causes the issue and see if it still happens. If not, I'll
consider it "fixed". Otherwise, hopefully the consolidation makes
debugging easier. Keep in mind so far this issue has only been seen
in long running tests and, as mentioned earlier when talking about
the teuthology a that produces this, doesn't happen every time.
Adam King [Wed, 9 Nov 2022 18:24:25 +0000 (13:24 -0500)]
mgr/cephadm: fix check for if devices have changed
Directly comparing the dicts doesn't work since the
"created" field was added in. We should instead make use
of the existing Devices equality function.
projects / ceph.git / log