Ilya Dryomov [Fri, 26 Jun 2020 20:57:06 +0000 (20:57 +0000)]
msg/async/ProtocolV2: take care of features when replacing the socket
reuse_connection() can be called on exproto in BANNER_CONNECTING
(i.e. without peer_supported_features and with tx/rx_frame_asm set to
msgr2.0), but this state isn't carried over. If the donor connection
is msgr2.1, this leads to repeated connection faults on crc or auth tag
mismatches because we end up assembling 2.0 frames while the peer is
expecting 2.1 frames.
msg/async/ProtocolV2: store supported features instead of required
We aren't interested in peer_required_features anywhere outside
_handle_peer_banner_payload() -- once we know there is no mismatch,
it's all about peer_supported_features.
Ilya Dryomov [Mon, 11 May 2020 12:06:44 +0000 (14:06 +0200)]
msg/async/ProtocolV2: short circuit empty segments and epilogue
In both msgr2.0 and msgr2.1, segments can be empty. In msgr2.1,
epilogue can be empty as well.
Handle both by calling the respective handler function directly
instead of allocating a buffer::ptr_node for an empty buffer and
passing that through READ[_RXBUF].
msg/async/crypto_onwire: implement msgr2.1 nonce format
Move to a 64-bit counter to avoid wrapping and having to reset
the session before the counter repeats. This is in line with NIST
Recommendation for GCM [1]:
"... this Recommendation suggests, but does not require, that
the leading (i.e., leftmost) 32 bits of the IV hold the fixed
field; and that the trailing (i.e., rightmost) 64 bits hold the
invocation field."
See commit bb61e6a5adc3 ("msg/async/ProtocolV2: avoid AES-GCM nonce
reuse vulnerabilities").
Ilya Dryomov [Mon, 4 May 2020 15:52:13 +0000 (17:52 +0200)]
msg/async/frames_v2: rename and clarify FRAME_FLAGS_LATEABRT
Clarify that the frame can be aborted at any point after the
preamble and the first segment are put on the wire. When that
happens, the remaining segments (including the data segment)
may be filled with zeros.
Conflicts:
src/crimson/net/ProtocolV2.cc [ crimson doesn't support
msgr2 in nautilus ]
src/msg/async/frames_v2.h [ context: commit c70f779d12a2
("headers: Make ceph_le member private") not in nautilus ]
Ilya Dryomov [Mon, 4 May 2020 15:42:28 +0000 (17:42 +0200)]
msg/async/frames_v2: rename epilogue structs
In preparation for msgr2,1, rename epilogue structs:
epilogue_plain_block_t to epilogue_crc_rev0_block_t and
epilogue_secure_block_t to epilogue_secure_rev0_block_t
(rev0 stands for revision 0).
Also, get rid of size constants that just disguise the
struct type.
Conflicts:
src/msg/async/ProtocolV2.cc [ context: commit d3ec4c01d17
("msg: Build target 'common' without using namespace in
headers") not in nautilus ]
Start separating frame assembly and disassembly code from
frame sending, receiving and handling code, so that assembly
and disassembly pieces can be unit tested and hopefully also
shared between different messengers (e.g. crimson).
This commit factors out the assembly code from Frame.
msg/async/crypto_onwire: perform decryption in place
OpenSSL supports in-place decryption so we can avoid
allocating potentially multi-megabyte and strictly aligned
buffer for each decryption operation.
ProtocolV2 actually gets the alignment wrong: after
read_frame_segment() allocates with cur_rx_desc.alignment,
handle_read_frame_segment() effectively replaces that with
segment_t::DEFAULT_ALIGNMENT.
Alfonso Martínez [Thu, 23 Jan 2020 10:16:27 +0000 (11:16 +0100)]
ceph.spec.in: fix 'make check' deps for centos8
When running 'FOR_MAKE_CHECK=1 ./install-deps.sh' in CentOS 8
these dependencies were not being installed.
Missing dependencies are provided by
https://copr.fedorainfracloud.org/coprs/ktdreyer/ceph-el8/
Kefu Chai [Tue, 24 Dec 2019 05:17:55 +0000 (13:17 +0800)]
ceph.spec.in: re-enable "make check" deps for el8
this change partially reverts e92cb7a0. as these packages are now
available in AppStream, BaseOS or PowerTools in el8, in this change,
they are re-enabled.
qa/test_exports: fix TestExports failure under new python3 compability changes
self.mount_a.client_remote.sh() returns an 'str' object rather than a StringIO object. Hence the p.stdout.getvalue() produces an error. This commit fixes this and also fix str and byte mismatch as byte and string were the same object in Python2 but this is not the case in Python3.
mgr/volumes: Create subvolume with isolated rados namespace
1. Add --namespace-isolated option to 'subvolume create' command
to create subvolume in a separate RADOS namespace
2. Add "pool_namespace" field to 'subvolume info' command
which displays the rados namespace if set else empty string
Jan Fajerski [Tue, 31 Mar 2020 14:07:45 +0000 (16:07 +0200)]
ceph-volume: add and delete lvm tags in a single lvchange call.
Otherwise we can end up in race-y situations when a concurrent c-v calls
sees only one tag but expects all tags to be present. Say if the
ceph.type tag is present, c-v expects ceph.osd_id to be present. By
setting/deleting tags in bulk, we use lvchange (and lvms internal
locking) as a sync mechanism.
Fixes: https://tracker.ceph.com/issues/44852 Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 20ecc309371e53fda5d6a5b6cf6de6110dbe5497)
Jeff Layton [Fri, 17 Apr 2020 13:55:41 +0000 (09:55 -0400)]
client: add a new inode release request callback
trim_caps() walks the list of caps on the session, and releases
non-auth caps, and attempts to trim dentries until the cache
size is under the max_caps value requested by MDS.
This is fine for FUSE, but doesn't really match the use-case of
nfs-ganesha. Ganesha typically looks up inodes by inode number, not
by dentry. It's quite possible that after a restart, we may have a
ton of outstanding inodes with no dentries associated with them.
Ganesha holds a reference to each inode, so libcephfs can't release
them, and we don't have a way to request that ganesha do so.
Add a new ino_release_callback and finisher. The intent is to allow
libcephfs to "upcall" to the application and request that it release
references to a specific inode.
Jeff Layton [Tue, 28 Apr 2020 18:00:13 +0000 (14:00 -0400)]
test: add a new program for testing ino_release_cb
Create a bunch of files and get their inode numbers. Remount, look them
all up by inode number and hold references. Stop looking up inodes as
soon as we get a callback from libcephfs. If we got the callback, return
success. Fail otherwise.
Since this has the same cluster setup as the other client_trim_caps
testcase, we can piggyback onto that task.
Jeff Layton [Tue, 21 Apr 2020 12:50:54 +0000 (08:50 -0400)]
client: only override umask_cb with non-NULL values
Client::init sets this, but if we later call ll_register_callbacks again
with a new set of function pointers that has umask_cb set to nullptr,
it'll override the value in the cmount.
Only reset umask_cb if the one in args is not nullptr.
Conflicts:
doc/cephfs/administration.rst
- nautilus has "filesystems" where master has "file systems"
- a difference that is not relevant to this backport
Kefu Chai [Mon, 25 May 2020 07:52:04 +0000 (15:52 +0800)]
qa/suites/rgw/tempest: bump up keystone to 17.0.0
Fixes: https://tracker.ceph.com/issues/45692
please note, the change on master also includes changes to use identity
v3. in this change, we only pick the change to use newer keystone to
address the pip install issue
Kefu Chai [Sun, 24 May 2020 15:06:08 +0000 (23:06 +0800)]
qa/tasks/tempest.py: install python3 for tempest test
install python3 for tempest's tox based test otherwise the test would
fail if python3.6 or python3.8 is not found.
this change is not cherry-picked from master, as our python bindings
are now built with python3.6, so python3.6 is always used. it's
supported by tempest's test.
Kefu Chai [Sun, 24 May 2020 14:51:05 +0000 (22:51 +0800)]
qa/tasks/keystone.py: install python3
install python3 for keystone's tox based test otherwise python34 would
be installed, and keystone's tox based test does not support python34.
this change is not cherry-picked from master, as our python bindings are
now built with python3.6, so python3.6 is always used. it's supported by
keystone's test.
Casey Bodley [Mon, 4 May 2020 20:39:59 +0000 (16:39 -0400)]
test/rgw: Cluster.admin() returns output as utf-8 str
fixes the TypeError:
File "rgw_multi/tests.py", line 1107, in test_bucket_sync_disable
check_buckets_sync_status_obj_not_exist(zone, buckets)
File "rgw_multi/tests.py", line 438, in check_buckets_sync_status_obj_not_exist
if log_list.find(':'+bucket+":") >= 0:
TypeError: argument should be integer or bytes-like object, not 'str'
Kefu Chai [Fri, 1 May 2020 08:52:41 +0000 (16:52 +0800)]
qa/tasks/qemu: open file in text mode
in python3, if a file is opened in 'rb' mode, we will have bytes when
reading from it, as a result, we have following failure like:
```
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 32, in nested
vars.append(enter())
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/teuthworker/src/github.com_tchaikov_ceph_nautilus-py3/qa/tasks/qemu.py", line 124, in generate_iso
test_setup = ''.join(f.readlines())
TypeError: sequence item 0: expected str instance, bytes found
```
qa/tasks/rbd: update NamedTemporaryFile usage for python3
- open the file in text mode to avoid 'bytes' objects
- 'bufsize' is now spelled 'buffering' and switching buffering off
with 'buffering=0' is only allowed in binary mode. As we need the
file to be either in the page cache or on disk before we copy it
to the remote, call flush().
1ac34a5ea3d1aca299b02e574b295dd4bf6167f4 is not backported to mimic, but
we are still running upgrade test from mimic. so in mimic, monitor
still complains at seeing too many pgs per osd.
projects / ceph.git / log