Instruct readers to use "mkdir /mnt/cephfs1" to create a mountpoint
before using "ceph-fuse" to mount a filesystem, if "/mnt/cephfs1"
doesn't already exist. cf.
https://github.com/ceph/ceph/pull/56831#discussion_r1561102227
Rongqi Sun [Fri, 12 Apr 2024 06:51:34 +0000 (06:51 +0000)]
bluestore/bluestore_types: check 'it' valid before using
When sanitizer is enabled, unittest_bluestore_types fails as following
```
[ RUN ] sb_info_space_efficient_map_t.basic
=================================================================
==143714==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xffff99f8b7f4 at pc 0xaaaab50bde18 bp 0xffffebefcdb0 sp 0xffffebefcda8
READ of size 8 at 0xffff99f8b7f4 thread T0
#0 0xaaaab50bde14 in sb_info_t::get_sbid() const /root/ceph/src/os/bluestore/bluestore_types.h:1337:30
#1 0xaaaab50a5908 in sb_info_space_efficient_map_t::find(unsigned long) /root/ceph/src/os/bluestore/bluestore_types.h:1385:10
#2 0xaaaab50bd638 in sb_info_space_efficient_map_t::_add(long) /root/ceph/src/os/bluestore/bluestore_types.h:1424:15
#3 0xaaaab50a52bc in sb_info_space_efficient_map_t::add_maybe_stray(unsigned long) /root/ceph/src/os/bluestore/bluestore_types.h:1358:12
#4 0xaaaab4fec03c in sb_info_space_efficient_map_t_basic_Test::TestBody() /root/ceph/src/test/objectstore/test_bluestore_types.cc:113:11
#5 0xaaaab51e9a40 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2605:10
#6 0xaaaab5197040 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2641:14
#7 0xaaaab51488a4 in testing::Test::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:2680:5
#8 0xaaaab514a7e8 in testing::TestInfo::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:2858:11
#9 0xaaaab514bde8 in testing::TestSuite::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:3012:28
#10 0xaaaab5167bac in testing::internal::UnitTestImpl::RunAllTests() /root/ceph/src/googletest/googletest/src/gtest.cc:5723:44
#11 0xaaaab51f3940 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2605:10
#12 0xaaaab519e5d8 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2641:14
#13 0xaaaab5167024 in testing::UnitTest::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:5306:10
#14 0xaaaab50b4d6c in RUN_ALL_TESTS() /root/ceph/src/googletest/googletest/include/gtest/gtest.h:2486:46
#15 0xaaaab50a1080 in main /root/ceph/src/test/objectstore/test_bluestore_types.cc:2847:10
#16 0xffff9d6c73f8 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#17 0xffff9d6c74c8 in __libc_start_main csu/../csu/libc-start.c:392:3
#18 0xaaaab4f3812c in _start (/root/ceph/build/bin/unittest_bluestore_types+0xe4812c) (BuildId: cb75399658026f83a4e89012de8fb02f08f6d239)
0xffff99f8b7f4 is located 0 bytes to the right of 20-byte region [0xffff99f8b7e0,0xffff99f8b7f4)
allocated by thread T0 here:
#0 0xaaaab4fe636c in operator new[](unsigned long) (/root/ceph/build/bin/unittest_bluestore_types+0xef636c) (BuildId: cb75399658026f83a4e89012de8fb02f08f6d239)
#1 0xaaaab50c0d2c in mempool::pool_allocator<(mempool::pool_index_t)11, sb_info_t>::allocate(unsigned long, void*) /root/ceph/src/include/mempool.h:375:33
#2 0xaaaab50c0c0c in std::allocator_traits<mempool::pool_allocator<(mempool::pool_index_t)11, sb_info_t> >::allocate(mempool::pool_allocator<(mempool::pool_index_t)11, sb_info_t>&, unsigned long) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:318:20
#3 0xaaaab50c044c in std::_Vector_base<sb_info_t, mempool::pool_allocator<(mempool::pool_index_t)11, sb_info_t> >::_M_allocate(unsigned long) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:346:20
#4 0xaaaab50bf954 in void std::vector<sb_info_t, mempool::pool_allocator<(mempool::pool_index_t)11, sb_info_t> >::_M_realloc_insert<long&>(__gnu_cxx::__normal_iterator<sb_info_t*, std::vector<sb_info_t, mempool::pool_allocator<(mempool::pool_index_t)11, sb_info_t> > >, long&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:440:33
#5 0xaaaab50be0d8 in sb_info_t& std::vector<sb_info_t, mempool::pool_allocator<(mempool::pool_index_t)11, sb_info_t> >::emplace_back<long&>(long&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:121:4
#6 0xaaaab50bd760 in sb_info_space_efficient_map_t::_add(long) /root/ceph/src/os/bluestore/bluestore_types.h:1429:24
#7 0xaaaab50a5e78 in sb_info_space_efficient_map_t::add_or_adopt(unsigned long) /root/ceph/src/os/bluestore/bluestore_types.h:1361:15
#8 0xaaaab4feb07c in sb_info_space_efficient_map_t_basic_Test::TestBody() /root/ceph/src/test/objectstore/test_bluestore_types.cc:103:11
#9 0xaaaab51e9a40 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2605:10
#10 0xaaaab5197040 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2641:14
#11 0xaaaab51488a4 in testing::Test::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:2680:5
#12 0xaaaab514a7e8 in testing::TestInfo::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:2858:11
#13 0xaaaab514bde8 in testing::TestSuite::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:3012:28
#14 0xaaaab5167bac in testing::internal::UnitTestImpl::RunAllTests() /root/ceph/src/googletest/googletest/src/gtest.cc:5723:44
#15 0xaaaab51f3940 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2605:10
#16 0xaaaab519e5d8 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2641:14
#17 0xaaaab5167024 in testing::UnitTest::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:5306:10
#18 0xaaaab50b4d6c in RUN_ALL_TESTS() /root/ceph/src/googletest/googletest/include/gtest/gtest.h:2486:46
#19 0xaaaab50a1080 in main /root/ceph/src/test/objectstore/test_bluestore_types.cc:2847:10
#20 0xffff9d6c73f8 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#21 0xffff9d6c74c8 in __libc_start_main csu/../csu/libc-start.c:392:3
#22 0xaaaab4f3812c in _start (/root/ceph/build/bin/unittest_bluestore_types+0xe4812c) (BuildId: cb75399658026f83a4e89012de8fb02f08f6d239)
SUMMARY: AddressSanitizer: heap-buffer-overflow /root/ceph/src/os/bluestore/bluestore_types.h:1337:30 in sb_info_t::get_sbid() const
Shadow bytes around the buggy address:
0x200ff33f16a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff33f16b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff33f16c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff33f16d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff33f16e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x200ff33f16f0: fa fa fa fa fa fa fa fa fa fa fa fa 00 00[04]fa
0x200ff33f1700: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff33f1710: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff33f1720: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff33f1730: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff33f1740: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==143714==ABORTING
```
'it' might be invalid, so before using 'it', need to figure validity out
Patrick Donnelly [Thu, 11 Apr 2024 23:00:53 +0000 (19:00 -0400)]
Merge PR #56839 into main
* refs/pull/56839/head:
script/ptl-tool: push branch to shaman for qa runs
script/ptl-tool: add release name to branch with switch
script/ptl-tool: alphabetize arguments
script/ptl-tool: avoid repo specific remotes entirely
script/ptl-tool: improve help text for envvars
Adds move constructor and move assignment operator to the librbd::Image.
Also marks copy ctor/assign op as deleted, and makes them public for better compiler diagnostics.
Patrick Donnelly [Thu, 11 Apr 2024 17:10:59 +0000 (13:10 -0400)]
Merge PR #56835 into main
* refs/pull/56835/head:
script/ptl-tool: create qa trackers for test branches
script/ptl-tool: add switch for debugging
script/ptl-tool: add --stop-at-built flag
When sanitizer is enabled, unittest_mds_quiesce_agent fails as following
```
[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from QuiesceAgentTest
[ RUN ] QuiesceAgentTest.ThreadManagement
[ OK ] QuiesceAgentTest.ThreadManagement (3 ms)
[ RUN ] QuiesceAgentTest.DbUpdates
[ OK ] QuiesceAgentTest.DbUpdates (1 ms)
[ RUN ] QuiesceAgentTest.QuiesceProtocol
[ OK ] QuiesceAgentTest.QuiesceProtocol (3 ms)
[ RUN ] QuiesceAgentTest.DuplicateQuiesceRequest
[ OK ] QuiesceAgentTest.DuplicateQuiesceRequest (2 ms)
[ RUN ] QuiesceAgentTest.TimeoutBeforeComplete
[ OK ] QuiesceAgentTest.TimeoutBeforeComplete (2 ms)
[----------] 5 tests from QuiesceAgentTest (11 ms total)
[----------] Global test environment tear-down
[==========] 5 tests from 1 test suite ran. (11 ms total)
[ PASSED ] 5 tests.
Rongqi Sun [Mon, 1 Apr 2024 07:49:18 +0000 (07:49 +0000)]
test/cephadm: extend timeout limit to fix timeout issue
When run make check(arm64) on omani series servers, eg.
https://jenkins.ceph.com/job/ceph-pull-requests-arm64/54751/.
Shows:
python: timeout after 30 seconds
INFO root:call_wrappers.py:284 Non-zero exit code 124 from /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/cephadm/.tox/py3/bin/python -c for i in range(1000000): print(i, flush=True)
Fixes: https://tracker.ceph.com/issues/65355 Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>
qa/rgw/s3tests: remove 'client.0' from bucket prefix
new sns test cases are using this for topic names, but the '.' is not
allowed there:
> api_params = {'Name': 'test-client.0-n3bdgre5el2jk8v-606'}
> botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the CreateTopic operation: Name must be made up of only uppercase and lowercase ASCII letters, numbers, underscores, and hyphens
rgw/notify: support cross-tenant and cross-account notifications
a bucket's notification configuration may refer to topics from several
different tenants or accounts. when publishing to a given topic, look in
the correct namespace for each topic instead of defaulting to the
requesting user's tenant namespace
Casey Bodley [Tue, 12 Mar 2024 20:26:44 +0000 (16:26 -0400)]
rgw/pubsub: return 404 NotFound instead of NoSuchKey
repurpose the ERR_NOT_FOUND define which was otherwise unused to
customize the error response for sns apis, which return the NotFound
error code instead of NoSuchKey from s3:
Casey Bodley [Tue, 12 Mar 2024 23:08:50 +0000 (19:08 -0400)]
rgw/pubsub: notifications can refer to topics in other accounts/tenants
accounts can use topic policy to grant sns:Publish permissions to other
accounts. the PutBucketNotification op should expect TopicArns from
other accounts. the account name from each TopicArn should be used as
the 'tenant' argument for RGWPubSub's constructor so we look for the
topic in the right namespace
Casey Bodley [Tue, 12 Mar 2024 20:25:58 +0000 (16:25 -0400)]
rgw/pubsub: when present, use account id instead of tenant
RGWPubSub provides topic namespace isolation for tenants by adding
prefixes to rados object names and topic metadata keys. accounts use
this the same way
refactor verify_topic_owner_or_policy() to share the same interface
as similar functions like verify_user/bucket/object_permission()
from rgw_common.cc
in addition to the topic resource policy, this now also consults iam
identity policies like user, group, or role policy
for account users, this now implements cross-account policy evaluation.
this only comes into play for sns:Publish permissions though, because
the topics themselves are scoped to the account
Casey Bodley [Sat, 9 Mar 2024 16:08:17 +0000 (11:08 -0500)]
rgw/pubsub: do init/validation in init_processing()
verify_permission() should do permission checks and nothing else!
admin/system users ignore errors from verify_permission() and go on to
call execute() regardless. that means that execute() can't rely on any
initialization that happened during verify_permission(), at risk of
crashing on admin/system requests. it also means that any permission
checks in execute() won't get overridden for admin/system users,
breaking their superuser access
by moving all parameter validation and initialization into
init_processing(), we can prepare all the state that verify_permission()
will need to do it's thing
Casey Bodley [Tue, 5 Mar 2024 18:57:09 +0000 (13:57 -0500)]
rgw/auth: expose Identity::get_account()
now that all identities store an optional account, expose that to the
rest of rgw with get_account(). this cleans up lots of code that
otherwise has to deal with the rgw_owner variant
Casey Bodley [Fri, 1 Mar 2024 14:36:31 +0000 (09:36 -0500)]
rgw/auth: object ops use new verify_bucket_permission() overload
several object operations like PutObject, DeleteObject, etc were handling
policy evaluation manually instead of using the helper functions like
verify_user/bucket/object_permission(), so were missing the cross-policy
evaluation rules for account users
these now call the new 'custom arn' overload of verify_bucket_permission()
for equivalent functionality
the eval_identity_or_session_policies() function is no longer exposed by
rgw_common.h to prevent other ops from adding new logic that doesn't
handle cross-account access
Casey Bodley [Mon, 4 Mar 2024 21:10:17 +0000 (16:10 -0500)]
rgw/auth: pass user policies into identities
loading user policies in rgw_build_bucket_policies() doesn't work for
PostObj requests because we haven't authenticated yet at that point
instead, auth engines load/parse policies when they load the user info.
policies are passed into the auth identities and applied to req_state
via modify_request_state() similar to how RoleApplier handles role
policy
this also moves the load_iam_identity_policies() into rgw_auth.cc for
use by transform_old_authinfo()
Casey Bodley [Sun, 25 Feb 2024 15:04:44 +0000 (10:04 -0500)]
rgw: make user email matching case-insensitive
handle user emails the same way we do account account emails. store
RGWUserInfo::user_email exactly as the user specified it, but convert
the object name to lower-case for case-insensitive matching
projects / ceph.git / log