]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
15 months agod4n: Update block `mtime` in filter and change `creationTime` to string
Samarah [Thu, 9 Nov 2023 21:27:04 +0000 (21:27 +0000)]
d4n: Update block `mtime` in filter and change `creationTime` to string
in directory due to seg faults

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agod4n/directory: Add `ObjectDirectory::update_field` method
Samarah [Thu, 9 Nov 2023 14:28:07 +0000 (14:28 +0000)]
d4n/directory: Add `ObjectDirectory::update_field` method

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agorgw/d4n: this commit squashes the following commits related to LFUDA Policy.
Samarah [Tue, 31 Oct 2023 14:21:06 +0000 (14:21 +0000)]
rgw/d4n: this commit squashes the following commits related to LFUDA Policy.

rgw/d4n: Make minor updates, improve `LFUDAPolicy::erase` method, and
fix `localWeight` policy sets
rgw/d4n: Update `LFUDA::find_victim` method, add locks in policy code,
and remove setting of attrs in filter's `put_async` calls
d4n/policy: Update logic for minimum average weight calculation
d4n/policy: Remove iteration in `eviction`

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agod4n/filter: Change from copying block to object in `copy_obj`
Samarah [Thu, 26 Oct 2023 17:49:02 +0000 (17:49 +0000)]
d4n/filter: Change from copying block to object in `copy_obj`

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agorgw/d4n: this commit squashes the following two commits.
Samarah [Thu, 19 Oct 2023 15:34:20 +0000 (15:34 +0000)]
rgw/d4n: this commit squashes the following two commits.

rgw/d4n: Remove `get_block` from policy and add logic to D4N filter
d4n/policy: Create a min heap of the local weights

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agod4n: Store `optional_yield` in client cb and update index used in
Samarah [Fri, 6 Oct 2023 20:20:43 +0000 (20:20 +0000)]
d4n: Store `optional_yield` in client cb and update index used in
D4N/cache classes

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agod4n/policy: Update policy to use block name
Samarah [Fri, 29 Sep 2023 13:35:57 +0000 (13:35 +0000)]
d4n/policy: Update policy to use block name

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agod4n: Add connection client names for policy and directory
Samarah [Fri, 29 Sep 2023 13:26:50 +0000 (13:26 +0000)]
d4n: Add connection client names for policy and directory

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agorgw: Add `rgw_local_cache_address` yaml config
Samarah [Tue, 26 Sep 2023 20:36:31 +0000 (20:36 +0000)]
rgw: Add `rgw_local_cache_address` yaml config

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agorgw/d4n: Remove `cacheBlock` and commented out read workflow from D4N filter
Samarah [Fri, 22 Sep 2023 18:20:16 +0000 (18:20 +0000)]
rgw/d4n: Remove `cacheBlock` and commented out read workflow from D4N filter

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agod4n/directory: Update directory `exist_key` method to take in
Samarah [Thu, 7 Sep 2023 21:36:28 +0000 (21:36 +0000)]
d4n/directory: Update directory `exist_key` method to take in
block/object; fix endpoint generation; add `remove_host` method

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agod4n/policy: this commit squashes the following changes to policy and filter driver...
Samarah [Tue, 5 Sep 2023 16:53:32 +0000 (16:53 +0000)]
d4n/policy: this commit squashes the following changes to policy and filter driver files.

d4n/policy: Update policy with correct directory usage; create LFUDAEntry struct + entries map; add `optional_yield` to `insert` method
d4n/policy: Add `version` to Entry struct/policy methods and calls in
the D4N filter; add error checking and fix logic in LFUDA code

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agorgw/cache: this commit squashes following commits related to changes in
Samarah [Mon, 31 Jul 2023 17:32:49 +0000 (13:32 -0400)]
rgw/cache: this commit squashes following commits related to changes in
redis driver and ssd driver.

rgw/redis: Add `del`, `put_async`, and `calculate_free_space` methods to RedisDriver
rgw/redis: Work on RedisDriver `put_async` and `get_async` methods
rgw/cache: Add error-checking to `SSDDriver::set_attr`
rgw/redis: Remove `calculate_free_space` method from RedisDriver, fix
`free_space` calculations, clean up async code

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agotest/d4n: this commit squashes the following commits related to tests for d4n.
Samarah [Fri, 28 Jul 2023 13:44:03 +0000 (09:44 -0400)]
test/d4n: this commit squashes the following commits related to tests for d4n.

test/redis: Add temporary `boost::redis` unit test program for experimentation
test/rgw: Add/update unit test programs for D4N classes + RedisDriver; switch to `boost::redis` usage
test/d4n: Begin `cpp_redis` removal from D4N filter test program; needs
to be fixed
test/d4n: Temporarily remove D4N filter test program until D4N workflows
are fully implemented

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agorgw/d4n: this commit squashes the following commits related to
Samarah [Mon, 25 Sep 2023 14:49:43 +0000 (14:49 +0000)]
rgw/d4n: this commit squashes the following commits related to
clean up and removal of cpp_redis.

d4n/filter: Add `optional_yield` to CacheDriver calls in D4N Filter and
make minor updates to several filter methods
rgw/d4n: fix compilation issue.
rgw/cache: Add `del` method to CacheDriver and SSDDriver
cmake/d4n: Remove unnecessary D4N lines
rgw: Add `io_context` to D4N Filter and RedisDriver, remove `cpp_redis`
library from RedisDriver, and perform minor cleanup
d4n: Remove `cpp_redis` library from D4N directory and policy; update calls in filter; move Entry struct to base CachePolicy class
build/cpp_redis: Remove `cpp_redis` library
rgw/d4n: including <boost/asio/detached.hpp> wheerever needed.
rgw/d4n : fixes to d4n filter, policy, directory and redis driver files
for compilation errors.

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Co-authored-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw: Change D4N and RedisDriver `ifndef` directives to `#pragma once`; make integer...
Samarah [Tue, 5 Sep 2023 13:50:43 +0000 (13:50 +0000)]
rgw: Change D4N and RedisDriver `ifndef` directives to `#pragma once`; make integer errors negative; fix filter includes

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agorgw/cache: this commit adds entries to redis driver and squashes the
Samarah [Thu, 9 Nov 2023 16:39:15 +0000 (16:39 +0000)]
rgw/cache: this commit adds entries to redis driver and squashes the
following two commits.

rgw: Update D4N policy and RedisDriver with entries
rgw/d4n: fixing compilation error in d4n policy code.

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
Co-authored-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: fixing update_attrs and get_attrs in ssd backed cache backend.
Pritha Srivastava [Tue, 31 Oct 2023 06:41:30 +0000 (12:11 +0530)]
rgw/cache: fixing update_attrs and get_attrs in ssd backed cache backend.

update_attrs did not check whether an xattr already existed before
modifying it - added check for that. corrected converting bufferlist
to string.

get_attrs did not check whether attribute size is zero or negative -
added check for that.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/d4n: copying over read op params of the filter driver to
Pritha Srivastava [Fri, 13 Oct 2023 06:34:52 +0000 (12:04 +0530)]
rgw/d4n: copying over read op params of the filter driver to
the read op params of 'next' in prepare() method, to be used
while storing objects to the backend store.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: adding d4n config options for ssd backed
Pritha Srivastava [Wed, 27 Sep 2023 10:34:47 +0000 (16:04 +0530)]
rgw/cache: adding d4n config options for ssd backed
cache driver.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: saving dpp in set_client_cb() instead
Pritha Srivastava [Wed, 27 Sep 2023 06:09:41 +0000 (11:39 +0530)]
rgw/cache: saving dpp in set_client_cb() instead
of flush_write_part() so that the correct value
is available everywhere in handle_data().

Saving dpp value in flush_last_part() caused a crash in
rgw in handle_data().

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: This commit fixes two things in ssd backed cache backend:
Pritha Srivastava [Mon, 28 Aug 2023 11:43:38 +0000 (17:13 +0530)]
rgw/cache: This commit fixes two things in ssd backed cache backend:
1. fixing free space calcuation when data is deleted.

2. implementation of append_data().

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: moving LRU related data structures and interfaces to the Policy Manager.
Pritha Srivastava [Mon, 28 Aug 2023 11:42:18 +0000 (17:12 +0530)]
rgw/cache: moving LRU related data structures and interfaces to the Policy Manager.
Replacing the related calls in d4n filter driver with that of the cachepolicy in the
Policy Manager and adding cache block to the block directory
once the block is written to the cache backend and the
policy manager data structure.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: this commit adds `optional_yield` to ssd and redis driver
Samarah [Thu, 3 Aug 2023 20:59:55 +0000 (16:59 -0400)]
rgw/cache: this commit adds `optional_yield` to ssd and redis driver
and squashes the following commits.

RGW: Add `optional_yield` parameter to Cache Driver methods
RGW: Add `optional_yield` parameter to SSD Driver methods
rgw/cache: fix compilation errors related to optional_yield.

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
Co-authored-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: this commit removes indexing from ssd and redis
Pritha Srivastava [Thu, 24 Aug 2023 04:27:07 +0000 (09:57 +0530)]
rgw/cache: this commit removes indexing from ssd and redis
driver and squashes the following commits.

rgw/cache: modifications to the ssd cache driver after removing
indexing from the cache backend.
rgw/cache: modifications to redis cache driver, after
removing indexing from cache backend.
rgw/cache: modifications to policy driver code, commenting out
some pieces which refer to cache apis. Those apis have been
removed as part of modification to remove indexing from the
cache backend. These need to appropriately re-instated, once
we finalize which module will contain the data structures
related to cache policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: Modified APIs after removing indexing from
Pritha Srivastava [Thu, 24 Aug 2023 04:24:27 +0000 (09:54 +0530)]
rgw/cache: Modified APIs after removing indexing from
cache driver implementation.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: prototype implementation of an LRU Cache which includes
Pritha Srivastava [Thu, 24 Aug 2023 04:16:44 +0000 (09:46 +0530)]
rgw/cache: prototype implementation of an LRU Cache which includes
indexing and a list.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: this commit squashes commits related to removing
Pritha Srivastava [Mon, 14 Aug 2023 06:39:50 +0000 (12:09 +0530)]
rgw/cache: this commit squashes commits related to removing
cache_aio_read()/cache_aio_write() from cache api, ssd driver
and redis driver.

rgw/cache: removing cache_aio_read()/cache_aio_write()
interfaces from the Cache API. And moving cache_read_op
from rgw_aio.cc to ssd cache driver.
rgw/cache: changes related to cache_aio_read()/cache_aio_write()
in redis cache driver.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/d4n: completing the read flow for a local read cache,
Pritha Srivastava [Tue, 11 Jul 2023 15:32:20 +0000 (21:02 +0530)]
rgw/d4n: completing the read flow for a local read cache,
with ssd cache driver as the backend.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: fixing the segmentation fault by passing in 'yield'
Pritha Srivastava [Wed, 9 Aug 2023 09:47:29 +0000 (15:17 +0530)]
rgw/cache: fixing the segmentation fault by passing in 'yield'
by value to async_completion.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: implementation of put_async method in ssd driver
Pritha Srivastava [Tue, 11 Jul 2023 06:01:04 +0000 (11:31 +0530)]
rgw/cache: implementation of put_async method in ssd driver
and redis driver and squashes the following commits.

rgw/cache: implementation of async put. the call
does not take into account throttling for now.
rgw/cache: dummy implementation of put_async in redis
driver to fix compilation error.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agoRGW: Update read workflow so attributes get written
Samarah Uriarte [Wed, 5 Jul 2023 19:43:12 +0000 (15:43 -0400)]
RGW: Update read workflow so attributes get written

Signed-off-by: Samarah Uriarte <samarah.uriarte@ibm.com>
15 months agoRGW: Work on oid in cache and CacheBlock block ID
Samarah Uriarte [Thu, 29 Jun 2023 15:09:23 +0000 (11:09 -0400)]
RGW: Work on oid in cache and CacheBlock block ID

Signed-off-by: Samarah Uriarte <samarah.uriarte@ibm.com>
15 months agorgw/cache: This commit adds implementation of Partition in
Samarah Uriarte [Thu, 29 Jun 2023 13:51:02 +0000 (09:51 -0400)]
rgw/cache: This commit adds implementation of Partition in
ssd driver and squashes the following related commits.

RGW: Minor RedisDriver cleanup
RGW: Add partitioning
RGW: Cleanup RedisDriver

Signed-off-by: Samarah Uriarte <samarah.uriarte@ibm.com>
15 months agorgw/cache: This commit squashes commits related to boost::redis library.
Samarah Uriarte [Mon, 26 Jun 2023 19:56:01 +0000 (15:56 -0400)]
rgw/cache: This commit squashes commits related to boost::redis library.

RGW: Add boost_redis library
RGW: Add boost_redis to CMake
RGW: Add aedis methods to RedisDriver
RGW: Minor RedisDriver cleanup
build: Remove build/boost_redis
build: remove src/redis
build: Set src/boost_redis submodule to release version
RGW: Remove broken boost_redis
RGW: Add boost_redis
rgw: Get rgw_redis_driver to build
rgw/redis: Update `boost::redis` library
rgw/cache: fixing compilation error, by removing aedis header
file and src file includes from redis driver

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
Co-authored-by: Adam Emerson <aemerson@redhat.com>
Co-authored-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: fixing free space calcuation when data is deleted
Pritha Srivastava [Mon, 28 Aug 2023 11:43:38 +0000 (17:13 +0530)]
rgw/cache: fixing free space calcuation when data is deleted
from the cache backend.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agorgw/cache: This commit squashes the following commits
Pritha Srivastava [Mon, 3 Jul 2023 05:45:03 +0000 (11:15 +0530)]
rgw/cache: This commit squashes the following commits
related to xattrs in ssd driver.

rgw/cache: code to handle xattrs, which are stored as
file xattrs.
rgw/cache: correcting ssd driver get_attrs.
rgw/cache: fixing set_attrs in ssd cache driver.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agoRGW: Add D4N chunking mechanism
Samarah Uriarte [Fri, 23 Jun 2023 18:41:43 +0000 (14:41 -0400)]
RGW: Add D4N chunking mechanism

Signed-off-by: Samarah Uriarte <samarah.uriarte@ibm.com>
15 months agoRGW: move CacheDriver instance to D4N Filter
Samarah [Mon, 19 Jun 2023 18:00:16 +0000 (14:00 -0400)]
RGW: move CacheDriver instance to D4N Filter

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agoRGW: Fix rgw crash issue
Samarah [Mon, 19 Jun 2023 15:58:36 +0000 (11:58 -0400)]
RGW: Fix rgw crash issue

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agoRGW: Fix find_client method; rgw crash issue
Samarah [Mon, 19 Jun 2023 13:44:01 +0000 (09:44 -0400)]
RGW: Fix find_client method; rgw crash issue

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agorgw/d4n: This commit squashes the following commits related to
Samarah [Thu, 25 May 2023 18:58:26 +0000 (14:58 -0400)]
rgw/d4n: This commit squashes the following commits related to
d4n policy and filter driver.

RGW: Add cache driver base class - add d4n filter driver.
RGW: Update cache and policy files; add RedisDriver into D4N filter
RGW: Add D4N classes and unit testing; update cpp_redis submodule
QA: Add D4N teuthology suite
RGW: Improve D4N readability and structure
RGW: Add base D4N policy and test
RGW: Add GWF policy to D4N
RGW: Add global weight property
RGW: Added D4N namespace
RGW: Update policy driver interface
RGW: Update unit tests
RGW: Add Address struct to all files
RGW: Update D4N names and structure
RGW: Update structure and fix errors
RGW: Add more features to policy driver
RGW: Access local weight in policy code
RGW: Work on D4N workflows
RGW: Add object directory class and remove copy_value method
RGW: Establish object directory in read workflow
RGW: Update cache and policy files; add RedisDriver into D4N filter
RGW: Switch out D4N cache methods with Redis driver methods
RGW: Update RedisDriver to match new CacheDriver structure; define set_attrs method
RGW: Fix D4N read workflow crashes
RGW: Update D4N files to match CacheDriver changes
RGW: fix key_exists method for RedisDriver and clean up rgw_sal_d4n.cc
RGW: Use correct get_block method
RGW: Make CachePolicy a virtual class
RGW: Initialize localWeight if not found and develop find_victim method
RGW: Debugging network failure
RGW: Rebase and debugging network failure
RGW: Update RedisDriver::list_entries and usage in D4N policy driver
RGW: Fix network failure issue; add entries and entry methods
RGW: Update D4N test files to match rebase
RGW: Update D4N policy and RedisDriver with entries
RGW: make localWeight an attribute
rgw/cache: commit to fix compilation issues.

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agorgw/cache: this commit squashes the following commits for ssd backed cache driver.
Pritha Srivastava [Mon, 5 Jun 2023 08:04:32 +0000 (13:34 +0530)]
rgw/cache: this commit squashes the following commits for ssd backed cache driver.

initial implementation of ssd backed cache driver.This contains implementations
for methods related to Entry and Partition.

adding methods for put, get and delete data.
They are all synchronous.

added CacheAioRequest to handle async read and write
requests and added a corresponding read op in aio.h and aio.cc.

added get_async method implementation and
initial implementation to handle async reads using asio.

changing the signature of list_entries to match that with
base class and making it public.

adding implementation for asynchronous get.

modifying namespace from rgw::cal to rgw::cache.
Modifying the SSDDriver constructor for Partition related
modifications.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agoRGW: add entries and entry methods
Samarah [Tue, 13 Jun 2023 16:33:18 +0000 (12:33 -0400)]
RGW: add entries and entry methods

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agorgw/cache: This commit squashes the following commits related to list_entries cache...
Samarah [Thu, 8 Jun 2023 17:10:58 +0000 (13:10 -0400)]
rgw/cache: This commit squashes the following commits related to list_entries cache api.

RGW: Add list_entries method to CacheDriver and to Redis driver.
RGW: Change CacheDriver::list_entries output type
RGW: Update RedisDriver::list_entries and usage in D4N policy driver

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agorgw/cache: This commit squashes the following commits for redis driver.
Samarah [Fri, 22 Dec 2023 11:10:13 +0000 (16:40 +0530)]
rgw/cache: This commit squashes the following commits for redis driver.

RGW: fix key_exists method for RedisDriver and clean up rgw_sal_d4n.cc
RGW: Implement RedisDriver::get_free_space
rgw/cache: modifying namespace from rgw::cal to rgw::cache.
RGW: Update D4N files to match CacheDriver changes
RGW: Fix D4N read workflow crashes
RGW: Update RedisDriver to match new CacheDriver structure; define set_attrs method
RGW: Switch out D4N cache methods with Redis driver methods
RGW: Update Cache Driver structure
RGW: Update cache files.
RGW: create redis cache driver files

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agorgw/cache: This commit squashes modifications to base class.
Pritha Srivastava [Fri, 2 Jun 2023 10:16:23 +0000 (15:46 +0530)]
rgw/cache: This commit squashes modifications to base class.

1. Rename namespace from rgw::cal to rgw::cache
2. Remove Partition from the CacheDriver constructor.
3. Removing rgw_cache_driver.cc since all methods
are pure virtual in rgw_cache_driver.h.
4. Moving structures Partition and Entry out of
class CacheDriver. And modifying the signatures of methods
get_current_partition_info() and get_free_space().
5. modifying the cache driver base class.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
15 months agoRGW: Add cache driver base class
Samarah [Thu, 25 May 2023 18:58:26 +0000 (14:58 -0400)]
RGW: Add cache driver base class

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
15 months agoMerge pull request #55976 from cbodley/wip-qa-quincy-upgrade-distro
Casey Bodley [Tue, 2 Apr 2024 13:14:32 +0000 (14:14 +0100)]
Merge pull request #55976 from cbodley/wip-qa-quincy-upgrade-distro

qa/upgrade/quincy: bump ubuntu from 20->22

Reviewed-by: Adam King <adking@redhat.com>
15 months agoMerge pull request #56279 from ceph/long_processing_timeout
Casey Bodley [Tue, 2 Apr 2024 12:38:33 +0000 (13:38 +0100)]
Merge pull request #56279 from ceph/long_processing_timeout

rgw/s3select: s3select response handler refactor

Reviewed-by: Casey Bodley <cbodley@redhat.com>
15 months agoMerge pull request #56614 from zdover23/wip-doc-2024-03-30-mgr-zabbix
Zac Dover [Tue, 2 Apr 2024 11:21:15 +0000 (21:21 +1000)]
Merge pull request #56614 from zdover23/wip-doc-2024-03-30-mgr-zabbix

doc/mgr: update zabbix information

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
15 months agoMerge pull request #53679 from rhcs-dashboard/alerts-v2-api
Aashish Sharma [Tue, 2 Apr 2024 04:22:01 +0000 (09:52 +0530)]
Merge pull request #53679 from rhcs-dashboard/alerts-v2-api

mgr/dashboard: start using alertmanager v2

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
15 months agoMerge pull request #56596 from tchaikov/wip-on-exit-fix-leaks
Kefu Chai [Tue, 2 Apr 2024 00:43:42 +0000 (08:43 +0800)]
Merge pull request #56596 from tchaikov/wip-on-exit-fix-leaks

test/on_exit: use static variables for on_exit hooks

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
15 months agoMerge pull request #55979 from athanatos/sjust/wip-64546-max-creating-pgs
Samuel Just [Mon, 1 Apr 2024 20:18:15 +0000 (13:18 -0700)]
Merge pull request #55979 from athanatos/sjust/wip-64546-max-creating-pgs

mon: always create pgs in the epoch in which the pool was created

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
15 months agoMerge pull request #56533 from phlogistonjohn/jjm-cpatch-before
Adam King [Mon, 1 Apr 2024 15:32:36 +0000 (11:32 -0400)]
Merge pull request #56533 from phlogistonjohn/jjm-cpatch-before

cpatch: add --run-before command line option

Reviewed-by: Adam King <adking@redhat.com>
15 months agoMerge pull request #56136 from NitzanMordhai/wip-nitzan-init-hash-in-chunk-ref-by...
Yuri Weinstein [Mon, 1 Apr 2024 15:32:12 +0000 (08:32 -0700)]
Merge pull request #56136 from NitzanMordhai/wip-nitzan-init-hash-in-chunk-ref-by-hash

cls/cas/cls_cas_internal: Initialize 'hash' value before decoding

Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>
15 months agoMerge pull request #54991 from ifed01/wip-ifed-some-recover-stuff
Yuri Weinstein [Mon, 1 Apr 2024 15:29:57 +0000 (08:29 -0700)]
Merge pull request #54991 from ifed01/wip-ifed-some-recover-stuff

tools/ceph-objectstore-tool: tricks to tolerate disk errors for "pg export" command.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
15 months agodoc/mgr: update zabbix information 56614/head
Zac Dover [Fri, 29 Mar 2024 11:06:28 +0000 (21:06 +1000)]
doc/mgr: update zabbix information

Include information from John Jasen about how to install Zabbix.

Signed-off-by: Zac Dover <zac.dover@proton.me>
15 months agoMerge pull request #56512 from rhcs-dashboard/remove-token-authentication-multi-cluster
Aashish Sharma [Mon, 1 Apr 2024 14:00:56 +0000 (19:30 +0530)]
Merge pull request #56512 from rhcs-dashboard/remove-token-authentication-multi-cluster

mgr/dashboard: Remove token authentication multi cluster

Reviewed-by: Nizamudeen A <nia@redhat.com>
15 months agomgr/dashboard: remove the option for authentication with token in 56512/head
Aashish Sharma [Wed, 27 Mar 2024 09:45:59 +0000 (15:15 +0530)]
mgr/dashboard: remove the option for authentication with token in
multi-cluster form

Signed-off-by: Aashish Sharma <aasharma@redhat.com>
15 months agoMerge pull request #55860 from ceph/add-multi-cluster-token-ttl
Aashish Sharma [Mon, 1 Apr 2024 07:50:37 +0000 (13:20 +0530)]
Merge pull request #55860 from ceph/add-multi-cluster-token-ttl

mgr/dashboard: Add multi cluster token ttl

Reviewed-by: Nizamudeen A <nia@redhat.com>
15 months agoMerge pull request #56600 from zdover23/wip-doc-2024-04-01-dev-essentials-leads-list
Zac Dover [Mon, 1 Apr 2024 01:36:42 +0000 (11:36 +1000)]
Merge pull request #56600 from zdover23/wip-doc-2024-04-01-dev-essentials-leads-list

doc/dev: update leads list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
15 months agodoc/dev: update leads list 56600/head
Zac Dover [Mon, 1 Apr 2024 01:17:39 +0000 (11:17 +1000)]
doc/dev: update leads list

Remove Joao Luis from the list of component leads.

Signed-off-by: Zac Dover <zac.dover@proton.me>
15 months agorefactor of s3select response handler, adding continuation-response to parquet flow... 56279/head
Gal Salomon [Tue, 19 Mar 2024 11:45:58 +0000 (13:45 +0200)]
refactor of s3select response handler, adding continuation-response to parquet flow, refacor of error response

bug fix (init_success_response).
s3select submodule

fix for json-error-flow

Signed-off-by: Gal Salomon <gal.salomon@gmail.com>
15 months agotest/on_exit: use static variables for on_exit hooks 56596/head
Kefu Chai [Sun, 31 Mar 2024 01:14:21 +0000 (09:14 +0800)]
test/on_exit: use static variables for on_exit hooks

before this change, we allocate memory chunks using malloc(), but
we never free them. and LeakSanitizer points this out

```
Direct leak of 4 byte(s) in 1 object(s) allocated from:
    #0 0x5588bfe532de in malloc (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_on_exit+0xa52de) (BuildId: 7c7a92bf5719592938c5307214bcd9b080bd847f)
    #1 0x5588bfe911d7 in func_scope() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/on_exit.cc:33:22
    #2 0x5588bfe90804 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/on_exit.cc:64:3
    #3 0x7f23081c1d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

Direct leak of 4 byte(s) in 1 object(s) allocated from:
    #0 0x5588bfe532de in malloc (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_on_exit+0xa52de) (BuildId: 7c7a92bf5719592938c5307214bcd9b080bd847f)
    #1 0x5588bfe91160 in func_scope() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/on_exit.cc:29:22
    #2 0x5588bfe90804 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/on_exit.cc:64:3
    #3 0x7f23081c1d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
```

in this change, instead of allocating the variables using `malloc()`,
we keep them in static variables, so that they can be accessed by
`OnExitManager` even if it is a static variable.
with this change, the memory leak reports for this source file go away.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
15 months agoMerge pull request #56591 from tchaikov/wip-unittest_rocksdb_option-fix-leak
Kefu Chai [Sun, 31 Mar 2024 08:32:09 +0000 (16:32 +0800)]
Merge pull request #56591 from tchaikov/wip-unittest_rocksdb_option-fix-leak

tests: unittest_rocksdb_option: do not leak RocksDBStore

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
15 months agoMerge pull request #56592 from nh2/crush-location-doc-fixes
Zac Dover [Sun, 31 Mar 2024 00:08:50 +0000 (10:08 +1000)]
Merge pull request #56592 from nh2/crush-location-doc-fixes

doc/rados/operations: Improve crush_location docs

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>
15 months agodoc/rados/operations: Improve crush_location docs 56592/head
Niklas Hambüchen [Sat, 30 Mar 2024 16:42:48 +0000 (17:42 +0100)]
doc/rados/operations: Improve crush_location docs

* Fix incorrect syntax
* Use underscores for config options, like other ceph docs did
* Fix incorrect statement that crush_location_hook adds fiels; it replaces
* Explain `root=default host=HOSTNAME` is not set if `crush_location` is given
* Remove duplication across sections
* Point out that `root=default` is important

Signed-off-by: Niklas Hambüchen <mail@nh2.me>
15 months agoMerge pull request #56572 from ceph/wip-yuriw-release-process-main
Yuri Weinstein [Sat, 30 Mar 2024 14:57:37 +0000 (07:57 -0700)]
Merge pull request #56572 from ceph/wip-yuriw-release-process-main

doc: added jammy to the supported distro list for quincy

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
15 months agotests: unittest_rocksdb_option: do not leak RocksDBStore 56591/head
Kefu Chai [Sat, 30 Mar 2024 13:21:25 +0000 (21:21 +0800)]
tests: unittest_rocksdb_option: do not leak RocksDBStore

before this change, we allocate an instance of `RocksDBStore` with
`new`, but we never free it. and LeanSanitizer points this out:

```
Direct leak of 952 byte(s) in 1 object(s) allocated from:
    #0 0x55f31440bc2d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_rocksdb_option+0xaebc2d) (BuildId: 81b849dbc41cbc6b05d5e603d9ba8a002dab2d24)
    #1 0x55f3144132fd in RocksDBOption_simple_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/TestRocksdbOptionParse.cc:17:22
    #2 0x55f3144ecf26 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
    #3 0x55f3144a4312 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
    #4 0x55f314453ccc in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
    #5 0x55f314455d02 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
    #6 0x55f31445733b in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
    #7 0x55f3144747c8 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
    #8 0x55f3144f5576 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
    #9 0x55f3144ab1a2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
    #10 0x55f314473b52 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
    #11 0x55f31440f690 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
    #12 0x55f31440e4c3 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/unit.cc:45:10
    #13 0x7f0d32551d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
```

in this change, we manage the life cycle of `RocksDBStore` using
a smart pointer. this should address the leak.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
15 months agoMerge pull request #56587 from tchaikov/wip-bluefs-use-after-free
Kefu Chai [Sat, 30 Mar 2024 11:47:19 +0000 (19:47 +0800)]
Merge pull request #56587 from tchaikov/wip-bluefs-use-after-free

test/objectstore/test_bluefs: fix heap-use-after-free

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
15 months agoMerge pull request #56581 from zdover23/wip-doc-2024-03-29-dev-essentials-leads-list
Zac Dover [Sat, 30 Mar 2024 11:05:10 +0000 (21:05 +1000)]
Merge pull request #56581 from zdover23/wip-doc-2024-03-29-dev-essentials-leads-list

doc/dev: update leads list

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
15 months agoMerge pull request #55436 from tchaikov/mgr-python-3.12
Kefu Chai [Sat, 30 Mar 2024 05:40:49 +0000 (13:40 +0800)]
Merge pull request #55436 from tchaikov/mgr-python-3.12

mgr: use un-deprecated APIs to initialize Python interpretor

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
15 months agotest/objectstore/test_bluefs: fix heap-use-after-free 56587/head
Kefu Chai [Sat, 30 Mar 2024 04:43:56 +0000 (12:43 +0800)]
test/objectstore/test_bluefs: fix heap-use-after-free

in BlueFS.test_shared_alloc, we keep the return value of
`fs.get_perf_counters()`, and deference it after umounting the fs,
but the `PerfCounters*` pointer returned from `fs.get_perf_counters()`
is destroyed in `BlueFS::_shutdown_logger()` which is in turn called
by `BlueFS::umount()`. so ASan points this out:
```
==1662613==ERROR: AddressSanitizer: heap-use-after-free on address 0x6110000b2d80 at pc 0x7f0eefc30644 bp 0x7ffcdbab6430 sp 0x7ffcdbab6428
READ of size 8 at 0x6110000b2d80 thread T0
    #0 0x7f0eefc30643 in ceph::common::PerfCounters::get(int) const /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/perf_counters.cc:246:8
    #1 0x557595ddfc15 in BlueFS_test_shared_alloc_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1182:3
    #2 0x557595eeef66 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
    #3 0x557595ea8b22 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
    #4 0x557595e5974c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
    #5 0x557595e5b782 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
    #6 0x557595e5cdbb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
    #7 0x557595e7a248 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
    #8 0x557595ef7816 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
    #9 0x557595eaf5c2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
    #10 0x557595e795d2 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
    #11 0x557595e05370 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
    #12 0x557595dfc1f5 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1603:10
    #13 0x7f0eed083d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #14 0x7f0eed083e3f in __libc_start_main csu/../csu/libc-start.c:392:3
    #15 0x557595cd46a4 in _start (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs+0x2856a4) (BuildId: 5439261504ca3d7549fe9bcda1d17ef6d4d9b644)

0x6110000b2d80 is located 0 bytes inside of 208-byte region [0x6110000b2d80,0x6110000b2e50)
freed by thread T0 here:
    #0 0x557595d92b1d in operator delete(void*) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs+0x343b1d) (BuildId: 5439261504ca3d7549fe9bcda1d17ef6d4d9b644)
    #1 0x557595f31c43 in BlueFS::_shutdown_logger() /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueFS.cc:462:3
    #2 0x557595f54ab5 in BlueFS::umount(bool) /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueFS.cc:1076:3
    #3 0x557595ddfbd7 in BlueFS_test_shared_alloc_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1180:6
    #4 0x557595eeef66 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
    #5 0x557595ea8b22 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
    #6 0x557595e5974c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
    #7 0x557595e5b782 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
    #8 0x557595e5cdbb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
    #9 0x557595e7a248 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
    #10 0x557595ef7816 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
    #11 0x557595eaf5c2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
    #12 0x557595e795d2 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
    #13 0x557595e05370 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
    #14 0x557595dfc1f5 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1603:10
    #15 0x7f0eed083d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

previously allocated by thread T0 here:
    #0 0x557595d922bd in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs+0x3432bd) (BuildId: 5439261504ca3d7549fe9bcda1d17ef6d4d9b644)
    #1 0x7f0eefc33180 in ceph::common::PerfCountersBuilder::PerfCountersBuilder(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/perf_counters.cc:537:21
    #2 0x557595f30ac9 in BlueFS::_init_logger() /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueFS.cc:221:23
    #3 0x557595f42bc6 in BlueFS::mount() /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueFS.cc:977:3
    #4 0x557595ddd339 in BlueFS_test_shared_alloc_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1139:3
    #5 0x557595eeef66 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
    #6 0x557595ea8b22 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
    #7 0x557595e5974c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
    #8 0x557595e5b782 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
    #9 0x557595e5cdbb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
    #10 0x557595e7a248 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
    #11 0x557595ef7816 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
    #12 0x557595eaf5c2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
    #13 0x557595e795d2 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
    #14 0x557595e05370 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
    #15 0x557595dfc1f5 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1603:10
    #16 0x7f0eed083d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
```

in this change, instead of keeping `logger` across the `umount()` and
`mount()` calls, we get another instance of `logger`, query it for
the perf counter that we are interested, and compare the value
to see if it is unchanged.

this should address the ASan warning above.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
15 months agoMerge pull request #56544 from baum/fix_f68248944815acbf5fb08ef66f36da6d5baef824
Kefu Chai [Sat, 30 Mar 2024 01:30:02 +0000 (09:30 +0800)]
Merge pull request #56544 from baum/fix_f68248944815acbf5fb08ef66f36da6d5baef824

pybind: support RHEL8, whose default python is python 3.6.8

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
15 months agoMerge pull request #56584 from phlogistonjohn/jjm-rpmbuild-fix
Dan Mick [Fri, 29 Mar 2024 18:14:19 +0000 (11:14 -0700)]
Merge pull request #56584 from phlogistonjohn/jjm-rpmbuild-fix

Merged without waiting for tests.  Obvious fix.

15 months agoceph.spec.in: remove command-with-macro line 56584/head
John Mulligan [Fri, 29 Mar 2024 18:04:33 +0000 (14:04 -0400)]
ceph.spec.in: remove command-with-macro line

A comment clearly left as a breadcrumb for a node-proxy manpage is
causing (intermittent) build failures. Remove the line and hope
the manpage is added if/when appropriate.

Fixes: 0dd73643649ddc2366e60de4fe6c078b6e112091
Signed-off-by: John Mulligan <jmulligan@redhat.com>
15 months agoqa/upgrade/quincy: bump ubuntu from 20->22 55976/head
Casey Bodley [Tue, 5 Mar 2024 19:42:24 +0000 (14:42 -0500)]
qa/upgrade/quincy: bump ubuntu from 20->22

Signed-off-by: Casey Bodley <cbodley@redhat.com>
15 months agoMerge pull request #56540 from tchaikov/wip-fix-test-leaks-ceph-context
Kefu Chai [Fri, 29 Mar 2024 16:10:20 +0000 (00:10 +0800)]
Merge pull request #56540 from tchaikov/wip-fix-test-leaks-ceph-context

test: : do not increase ref when creating intrusive_ptr<CephContext>

Reviewed-by: Casey Bodley <cbodley@redhat.com>
15 months agopybind: use syntax supported by python3.6 56544/head
Alexander Indenbaum [Thu, 28 Mar 2024 07:20:11 +0000 (07:20 +0000)]
pybind: use syntax supported by python3.6

Fixes: f68248944815acbf5fb08ef66f36da6d5baef824
It seems the shaman CentOS8 builders, which serve as the base for container images, aren't current with the ":=" syntax. Amend to support RHEL8, whose default python is python 3.6.8.

Error example from https://shaman.ceph.com/builds/ceph/ceph-nvmeof-mon/27188b78bdbaa3c33de518b8514906de629e774a/default/388913/

  File "/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2505-g27188b78/rpm/el8/BUILD/ceph-19.0.0-2505-g27188b78/src/pybind/cephfs/setup.py", line 120
    if ldflags := os.environ.get('LDFLAGS'):
                ^
SyntaxError: invalid syntax

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>
15 months agodoc: added jammy to the supported distro list for quincy 56572/head
Yuri Weinstein [Thu, 28 Mar 2024 22:57:34 +0000 (15:57 -0700)]
doc: added jammy to the supported distro list for quincy

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
15 months agodoc/dev: update leads list 56581/head
Zac Dover [Fri, 29 Mar 2024 11:14:21 +0000 (21:14 +1000)]
doc/dev: update leads list

Update the list of component leads in
doc/dev/developer_guide/essentials.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
15 months agomgr/dashboard: allow the user to add ttl for the connected cluster's 55860/head
Aashish Sharma [Thu, 29 Feb 2024 16:16:23 +0000 (21:46 +0530)]
mgr/dashboard: allow the user to add ttl for the connected cluster's
token expiration

Fixes: https://tracker.ceph.com/issues/65055
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
15 months agoMerge pull request #56190 from rhcs-dashboard/disable-multi-cluster-non-hub
Aashish Sharma [Fri, 29 Mar 2024 05:39:06 +0000 (11:09 +0530)]
Merge pull request #56190 from rhcs-dashboard/disable-multi-cluster-non-hub

mgr/dashboard: disable multi-cluster feature for non-hub clusters

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
15 months agoMerge pull request #54633 from joscollin/wip-F61334-use-snapdiff-api
Venky Shankar [Fri, 29 Mar 2024 05:36:35 +0000 (11:06 +0530)]
Merge pull request #54633 from joscollin/wip-F61334-use-snapdiff-api

cephfs_mirror: use snapdiff api for incremental syncing

Reviewed-by: Venky Shankar <vshankar@redhat.com>
15 months agoMerge pull request #54301 from lxbsz/wip-sparse-read-test
Xiubo Li [Fri, 29 Mar 2024 04:26:26 +0000 (12:26 +0800)]
Merge pull request #54301 from lxbsz/wip-sparse-read-test

ceph_test_rados_api_io_pp: add multiple extents test for sparse-read

15 months agoMerge pull request #56570 from batrick/release-checklist-docs-2
Ilya Dryomov [Thu, 28 Mar 2024 23:06:07 +0000 (00:06 +0100)]
Merge pull request #56570 from batrick/release-checklist-docs-2

doc: reconcile release checklist

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
15 months agodoc: reconcile release checklist 56570/head
Patrick Donnelly [Thu, 28 Mar 2024 22:35:40 +0000 (18:35 -0400)]
doc: reconcile release checklist

Updating the releases documentation existed already but that section should be
moved to after dev freeze when the branch is forked.

Fixes: 7b5017913a9ecf27ca7281367a5472a9b9b043ec
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
15 months agoMerge pull request #56568 from batrick/release-checklist-docs
Ilya Dryomov [Thu, 28 Mar 2024 22:11:42 +0000 (23:11 +0100)]
Merge pull request #56568 from batrick/release-checklist-docs

doc: update release checklist

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
15 months agodoc: update release checklist 56568/head
Patrick Donnelly [Thu, 28 Mar 2024 21:06:41 +0000 (17:06 -0400)]
doc: update release checklist

To include note that release notes must be removed after forking the release
branch.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
15 months agoMerge PR #55337 into main
Patrick Donnelly [Thu, 28 Mar 2024 19:10:15 +0000 (15:10 -0400)]
Merge PR #55337 into main

* refs/pull/55337/head:
mds: explain clients retaining Fb during xlock

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
15 months agoMerge PR #56536 into main
Patrick Donnelly [Thu, 28 Mar 2024 18:40:39 +0000 (14:40 -0400)]
Merge PR #56536 into main

* refs/pull/56536/head:
mds: do not dispatch aborted internal requests
qa: add test for killing quiesce_inode with outstanding remote authpin requests
mds: add lockup debugging command

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
15 months agoMerge pull request #56534 from athanatos/sjust/wip-64975-crimson-scrub-snapmapper-2
Samuel Just [Thu, 28 Mar 2024 16:57:45 +0000 (09:57 -0700)]
Merge pull request #56534 from athanatos/sjust/wip-64975-crimson-scrub-snapmapper-2

crimson: introduce INTERNAL_PG_LOCAL_NS, use for snapmapper

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
15 months agoMerge pull request #56516 from afreen23/wip-ceph-logo
Nizamudeen A [Thu, 28 Mar 2024 15:23:09 +0000 (20:53 +0530)]
Merge pull request #56516 from afreen23/wip-ceph-logo

mgr/dashboard: make ceph logo redirect to dashboard

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
15 months agoMerge pull request #56543 from jrchyang/main
Igor Fedotov [Thu, 28 Mar 2024 09:13:54 +0000 (12:13 +0300)]
Merge pull request #56543 from jrchyang/main

os/bluestore: remove unnecessary std::hex in dout

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
15 months agoMerge pull request #56504 from rhcs-dashboard/replace-active_mds-active_nfs
Nizamudeen A [Thu, 28 Mar 2024 08:20:30 +0000 (13:50 +0530)]
Merge pull request #56504 from rhcs-dashboard/replace-active_mds-active_nfs

mgr/dashboard: s/active_mds/active_nfs in fs attach form

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
15 months agoMerge pull request #56468 from rhcs-dashboard/clone-name-exists-valid-fix
Nizamudeen A [Thu, 28 Mar 2024 08:19:59 +0000 (13:49 +0530)]
Merge pull request #56468 from rhcs-dashboard/clone-name-exists-valid-fix

mgr/dashboard: fix clone unique validator for name validation

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
15 months agoMerge pull request #56470 from rhcs-dashboard/fix-full-name-validation-rgw
Nizamudeen A [Thu, 28 Mar 2024 08:19:50 +0000 (13:49 +0530)]
Merge pull request #56470 from rhcs-dashboard/fix-full-name-validation-rgw

mgr/dashboard: fix rgw display name validation

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
15 months agoos/bluestore: remove unnecessary std::hex in dout 56543/head
jrchyang [Thu, 28 Mar 2024 07:16:58 +0000 (07:16 +0000)]
os/bluestore: remove unnecessary std::hex in dout

Signed-off-by: Jrchyang Yu <yuzhiqiang_yewu@cmss.chinamobile.com>
15 months agotest: : do not increase ref when creating intrusive_ptr<CephContext> 56540/head
Kefu Chai [Thu, 28 Mar 2024 02:01:46 +0000 (10:01 +0800)]
test: : do not increase ref when creating intrusive_ptr<CephContext>

before this change, we increment the refcount when constructing
`cct` instrusive_ptr, but nobody owns this smart pointer. also,
`CephContext` 's constructor set its refcount to 1. so, when the
test finishes, the refcount is 1, and this leads to a leakage of
the `CephContext` instance. and LeakSanitizer points this out:
```
Indirect leak of 10880000 byte(s) in 1 object(s) allocated from:
    #0 0x558d341d837d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_ipaddr+0x19b37d) (BuildId: 1b7e7e5abfc2b58ce2334712e4c00b2441c25870)
    #1 0x7fd74c957559 in __gnu_cxx::new_allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27
    #2 0x7fd74c956933 in std::allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32
    #3 0x7fd74c956933 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::allocate(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2396:39
    #4 0x7fd74c956690 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::initialize_buffer(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2494:18
    #5 0x7fd74c950562 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::circular_buffer(unsigned long, std::allocator<ceph::logging::ConcreteEntry> const&) /opt/ceph/include/boost/circ
ular_buffer/base.hpp:1039:9
    #6 0x7fd74c9435b4 in ceph::logging::Log::Log(ceph::logging::SubsystemMap const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/log/Log.cc:53:5
    #7 0x7fd74bc1891d in ceph::common::CephContext::CephContext(unsigned int, ceph::common::CephContext::create_options const&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:729:16
    #8 0x7fd74bc178eb in ceph::common::CephContext::CephContext(unsigned int, code_environment_t, int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:697:5
    #9 0x558d341f97e9 in pick_address_filtering_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/test_ipaddr.cc:774:47
    #10 0x558d3430c4f6 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
    #11 0x558d342c3fc2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
    #12 0x558d342749dc in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
    #13 0x558d34276a12 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
    #14 0x558d3427804b in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
    #15 0x558d342954d8 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
    #16 0x558d34314d26 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
    #17 0x558d342ca932 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
    #18 0x558d34294862 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
    #19 0x558d34218d80 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
    #20 0x558d34218d11 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googlemock/src/gmock_main.cc:70:10
    #21 0x7fd749331d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
```

so, in this change, we do not increase the refcount when creating cct.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>