rgw/cache: this commit squashes commits for removing CephContext* from
ssd, redis driver and related test files.
rgw/cache: removing CephContext* from ssd driver and redis driver.
It can be accessed in every api using dpp->get_cct(), where dpp
is of type DoutPrefixProvider*.
rgw/test: Remove `cct` parameter from cache driver initialization call
redis driver test file.
rgw/qa: removing cct parameter from ssd driver test file.
Samarah [Tue, 31 Oct 2023 14:21:06 +0000 (14:21 +0000)]
rgw/d4n: this commit squashes the following commits related to LFUDA Policy.
rgw/d4n: Make minor updates, improve `LFUDAPolicy::erase` method, and
fix `localWeight` policy sets
rgw/d4n: Update `LFUDA::find_victim` method, add locks in policy code,
and remove setting of attrs in filter's `put_async` calls
d4n/policy: Update logic for minimum average weight calculation
d4n/policy: Remove iteration in `eviction`
d4n/policy: this commit squashes the following changes to policy and filter driver files.
d4n/policy: Update policy with correct directory usage; create LFUDAEntry struct + entries map; add `optional_yield` to `insert` method
d4n/policy: Add `version` to Entry struct/policy methods and calls in
the D4N filter; add error checking and fix logic in LFUDA code
rgw/cache: this commit squashes following commits related to changes in
redis driver and ssd driver.
rgw/redis: Add `del`, `put_async`, and `calculate_free_space` methods to RedisDriver
rgw/redis: Work on RedisDriver `put_async` and `get_async` methods
rgw/cache: Add error-checking to `SSDDriver::set_attr`
rgw/redis: Remove `calculate_free_space` method from RedisDriver, fix
`free_space` calculations, clean up async code
test/d4n: this commit squashes the following commits related to tests for d4n.
test/redis: Add temporary `boost::redis` unit test program for experimentation
test/rgw: Add/update unit test programs for D4N classes + RedisDriver; switch to `boost::redis` usage
test/d4n: Begin `cpp_redis` removal from D4N filter test program; needs
to be fixed
test/d4n: Temporarily remove D4N filter test program until D4N workflows
are fully implemented
rgw/d4n: this commit squashes the following commits related to
clean up and removal of cpp_redis.
d4n/filter: Add `optional_yield` to CacheDriver calls in D4N Filter and
make minor updates to several filter methods
rgw/d4n: fix compilation issue.
rgw/cache: Add `del` method to CacheDriver and SSDDriver
cmake/d4n: Remove unnecessary D4N lines
rgw: Add `io_context` to D4N Filter and RedisDriver, remove `cpp_redis`
library from RedisDriver, and perform minor cleanup
d4n: Remove `cpp_redis` library from D4N directory and policy; update calls in filter; move Entry struct to base CachePolicy class
build/cpp_redis: Remove `cpp_redis` library
rgw/d4n: including <boost/asio/detached.hpp> wheerever needed.
rgw/d4n : fixes to d4n filter, policy, directory and redis driver files
for compilation errors.
rgw/d4n: copying over read op params of the filter driver to
the read op params of 'next' in prepare() method, to be used
while storing objects to the backend store.
rgw/cache: moving LRU related data structures and interfaces to the Policy Manager.
Replacing the related calls in d4n filter driver with that of the cachepolicy in the
Policy Manager and adding cache block to the block directory
once the block is written to the cache backend and the
policy manager data structure.
rgw/cache: this commit removes indexing from ssd and redis
driver and squashes the following commits.
rgw/cache: modifications to the ssd cache driver after removing
indexing from the cache backend.
rgw/cache: modifications to redis cache driver, after
removing indexing from cache backend.
rgw/cache: modifications to policy driver code, commenting out
some pieces which refer to cache apis. Those apis have been
removed as part of modification to remove indexing from the
cache backend. These need to appropriately re-instated, once
we finalize which module will contain the data structures
related to cache policy.
rgw/cache: this commit squashes commits related to removing
cache_aio_read()/cache_aio_write() from cache api, ssd driver
and redis driver.
rgw/cache: removing cache_aio_read()/cache_aio_write()
interfaces from the Cache API. And moving cache_read_op
from rgw_aio.cc to ssd cache driver.
rgw/cache: changes related to cache_aio_read()/cache_aio_write()
in redis cache driver.
rgw/cache: implementation of put_async method in ssd driver
and redis driver and squashes the following commits.
rgw/cache: implementation of async put. the call
does not take into account throttling for now.
rgw/cache: dummy implementation of put_async in redis
driver to fix compilation error.
rgw/cache: This commit squashes the following commits
related to xattrs in ssd driver.
rgw/cache: code to handle xattrs, which are stored as
file xattrs.
rgw/cache: correcting ssd driver get_attrs.
rgw/cache: fixing set_attrs in ssd cache driver.
Samarah [Thu, 25 May 2023 18:58:26 +0000 (14:58 -0400)]
rgw/d4n: This commit squashes the following commits related to
d4n policy and filter driver.
RGW: Add cache driver base class - add d4n filter driver.
RGW: Update cache and policy files; add RedisDriver into D4N filter
RGW: Add D4N classes and unit testing; update cpp_redis submodule
QA: Add D4N teuthology suite
RGW: Improve D4N readability and structure
RGW: Add base D4N policy and test
RGW: Add GWF policy to D4N
RGW: Add global weight property
RGW: Added D4N namespace
RGW: Update policy driver interface
RGW: Update unit tests
RGW: Add Address struct to all files
RGW: Update D4N names and structure
RGW: Update structure and fix errors
RGW: Add more features to policy driver
RGW: Access local weight in policy code
RGW: Work on D4N workflows
RGW: Add object directory class and remove copy_value method
RGW: Establish object directory in read workflow
RGW: Update cache and policy files; add RedisDriver into D4N filter
RGW: Switch out D4N cache methods with Redis driver methods
RGW: Update RedisDriver to match new CacheDriver structure; define set_attrs method
RGW: Fix D4N read workflow crashes
RGW: Update D4N files to match CacheDriver changes
RGW: fix key_exists method for RedisDriver and clean up rgw_sal_d4n.cc
RGW: Use correct get_block method
RGW: Make CachePolicy a virtual class
RGW: Initialize localWeight if not found and develop find_victim method
RGW: Debugging network failure
RGW: Rebase and debugging network failure
RGW: Update RedisDriver::list_entries and usage in D4N policy driver
RGW: Fix network failure issue; add entries and entry methods
RGW: Update D4N test files to match rebase
RGW: Update D4N policy and RedisDriver with entries
RGW: make localWeight an attribute
rgw/cache: commit to fix compilation issues.
Samarah [Thu, 8 Jun 2023 17:10:58 +0000 (13:10 -0400)]
rgw/cache: This commit squashes the following commits related to list_entries cache api.
RGW: Add list_entries method to CacheDriver and to Redis driver.
RGW: Change CacheDriver::list_entries output type
RGW: Update RedisDriver::list_entries and usage in D4N policy driver
rgw/cache: This commit squashes modifications to base class.
1. Rename namespace from rgw::cal to rgw::cache
2. Remove Partition from the CacheDriver constructor.
3. Removing rgw_cache_driver.cc since all methods
are pure virtual in rgw_cache_driver.h.
4. Moving structures Partition and Entry out of
class CacheDriver. And modifying the signatures of methods
get_current_partition_info() and get_free_space().
5. modifying the cache driver base class.
Kefu Chai [Sun, 31 Mar 2024 01:14:21 +0000 (09:14 +0800)]
test/on_exit: use static variables for on_exit hooks
before this change, we allocate memory chunks using malloc(), but
we never free them. and LeakSanitizer points this out
```
Direct leak of 4 byte(s) in 1 object(s) allocated from:
#0 0x5588bfe532de in malloc (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_on_exit+0xa52de) (BuildId: 7c7a92bf5719592938c5307214bcd9b080bd847f)
#1 0x5588bfe911d7 in func_scope() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/on_exit.cc:33:22
#2 0x5588bfe90804 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/on_exit.cc:64:3
#3 0x7f23081c1d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
Direct leak of 4 byte(s) in 1 object(s) allocated from:
#0 0x5588bfe532de in malloc (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_on_exit+0xa52de) (BuildId: 7c7a92bf5719592938c5307214bcd9b080bd847f)
#1 0x5588bfe91160 in func_scope() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/on_exit.cc:29:22
#2 0x5588bfe90804 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/on_exit.cc:64:3
#3 0x7f23081c1d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
```
in this change, instead of allocating the variables using `malloc()`,
we keep them in static variables, so that they can be accessed by
`OnExitManager` even if it is a static variable.
with this change, the memory leak reports for this source file go away.
Niklas Hambüchen [Sat, 30 Mar 2024 16:42:48 +0000 (17:42 +0100)]
doc/rados/operations: Improve crush_location docs
* Fix incorrect syntax
* Use underscores for config options, like other ceph docs did
* Fix incorrect statement that crush_location_hook adds fiels; it replaces
* Explain `root=default host=HOSTNAME` is not set if `crush_location` is given
* Remove duplication across sections
* Point out that `root=default` is important
in BlueFS.test_shared_alloc, we keep the return value of
`fs.get_perf_counters()`, and deference it after umounting the fs,
but the `PerfCounters*` pointer returned from `fs.get_perf_counters()`
is destroyed in `BlueFS::_shutdown_logger()` which is in turn called
by `BlueFS::umount()`. so ASan points this out:
```
==1662613==ERROR: AddressSanitizer: heap-use-after-free on address 0x6110000b2d80 at pc 0x7f0eefc30644 bp 0x7ffcdbab6430 sp 0x7ffcdbab6428
READ of size 8 at 0x6110000b2d80 thread T0
#0 0x7f0eefc30643 in ceph::common::PerfCounters::get(int) const /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/perf_counters.cc:246:8
#1 0x557595ddfc15 in BlueFS_test_shared_alloc_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1182:3
#2 0x557595eeef66 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#3 0x557595ea8b22 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#4 0x557595e5974c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
#5 0x557595e5b782 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
#6 0x557595e5cdbb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
#7 0x557595e7a248 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
#8 0x557595ef7816 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#9 0x557595eaf5c2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#10 0x557595e795d2 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
#11 0x557595e05370 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
#12 0x557595dfc1f5 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1603:10
#13 0x7f0eed083d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#14 0x7f0eed083e3f in __libc_start_main csu/../csu/libc-start.c:392:3
#15 0x557595cd46a4 in _start (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs+0x2856a4) (BuildId: 5439261504ca3d7549fe9bcda1d17ef6d4d9b644)
0x6110000b2d80 is located 0 bytes inside of 208-byte region [0x6110000b2d80,0x6110000b2e50)
freed by thread T0 here:
#0 0x557595d92b1d in operator delete(void*) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs+0x343b1d) (BuildId: 5439261504ca3d7549fe9bcda1d17ef6d4d9b644)
#1 0x557595f31c43 in BlueFS::_shutdown_logger() /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueFS.cc:462:3
#2 0x557595f54ab5 in BlueFS::umount(bool) /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueFS.cc:1076:3
#3 0x557595ddfbd7 in BlueFS_test_shared_alloc_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1180:6
#4 0x557595eeef66 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#5 0x557595ea8b22 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#6 0x557595e5974c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
#7 0x557595e5b782 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
#8 0x557595e5cdbb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
#9 0x557595e7a248 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
#10 0x557595ef7816 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#11 0x557595eaf5c2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#12 0x557595e795d2 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
#13 0x557595e05370 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
#14 0x557595dfc1f5 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1603:10
#15 0x7f0eed083d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
previously allocated by thread T0 here:
#0 0x557595d922bd in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs+0x3432bd) (BuildId: 5439261504ca3d7549fe9bcda1d17ef6d4d9b644)
#1 0x7f0eefc33180 in ceph::common::PerfCountersBuilder::PerfCountersBuilder(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/perf_counters.cc:537:21
#2 0x557595f30ac9 in BlueFS::_init_logger() /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueFS.cc:221:23
#3 0x557595f42bc6 in BlueFS::mount() /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueFS.cc:977:3
#4 0x557595ddd339 in BlueFS_test_shared_alloc_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1139:3
#5 0x557595eeef66 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#6 0x557595ea8b22 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#7 0x557595e5974c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
#8 0x557595e5b782 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
#9 0x557595e5cdbb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
#10 0x557595e7a248 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
#11 0x557595ef7816 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#12 0x557595eaf5c2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#13 0x557595e795d2 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
#14 0x557595e05370 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
#15 0x557595dfc1f5 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1603:10
#16 0x7f0eed083d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
```
in this change, instead of keeping `logger` across the `umount()` and
`mount()` calls, we get another instance of `logger`, query it for
the perf counter that we are interested, and compare the value
to see if it is unchanged.
John Mulligan [Fri, 29 Mar 2024 18:04:33 +0000 (14:04 -0400)]
ceph.spec.in: remove command-with-macro line
A comment clearly left as a breadcrumb for a node-proxy manpage is
causing (intermittent) build failures. Remove the line and hope
the manpage is added if/when appropriate.
Fixes: 0dd73643649ddc2366e60de4fe6c078b6e112091 Signed-off-by: John Mulligan <jmulligan@redhat.com>
Fixes: f68248944815acbf5fb08ef66f36da6d5baef824
It seems the shaman CentOS8 builders, which serve as the base for container images, aren't current with the ":=" syntax. Amend to support RHEL8, whose default python is python 3.6.8.
projects / ceph.git / log