Casey Bodley [Wed, 22 Nov 2023 19:42:07 +0000 (14:42 -0500)]
rgw/quota: remove base class AsyncRefreshHandler
instead of using a virtual `allocate_refresh_handler()` to create an
object to call the virtual `AsyncRefreshHandler::init_fetch()` on, just
provide one virtual `init_refresh()` to start the operation. this avoids
the need for `AsyncRefreshHandler` altogether
Casey Bodley [Wed, 22 Nov 2023 19:29:23 +0000 (14:29 -0500)]
rgw/quota: use intrusive_ptr for RefCountedWaitObject
instead of relying on async_refresh_response/fail to manage ref counts
manually, just stash an intrusive_ptr in AsyncRefreshHandler for the
duration of the async operation
Casey Bodley [Wed, 22 Nov 2023 16:20:26 +0000 (11:20 -0500)]
rgw/sal: unify ReadStatsCB for async user/bucket stats
`rgw::sal::Bucket::read_stats_async()` had returned stats as
`map<RGWObjCategory, RGWStorageStats>`, but the only caller in
rgw_quota.cc just summed up the categories into a single instance
of `RGWStorageStats`. moving this summation up into rgw_rados'
`RGWGetBucketStatsContext` allows us to unify these sal callbacks
around `RGWStorageStats` alone
the `RGWGetUserStats_CB`/`RGWGetBucketStats_CB` callbacks had inherited
from `RefCountedObject` and required manual reference counting. switch
to `boost::intrusive_ptr` for scope-based shared ownership
Lucian Petrut [Fri, 18 Aug 2023 12:25:52 +0000 (12:25 +0000)]
common: Windows Unicode CLI support
Windows CLI arguments use either ANSI (main()) or UTF-16 (wmain()).
Meanwhile, Ceph libraries expect UTF-8 and raise exceptions when
trying to use Unicode CLI arguments or log Unicode output:
rbd.exe create test_unicode_șțăâ --size=32M
terminate called after throwing an instance of 'std::runtime_error'
what(): invalid utf8
We'll use a Windows application manifest, setting the "activeCodePage"
property [1][2]. This enables the Windows UCRT UTF-8 mode so that
functions that receive char* arguments will expect UTF-8 instead of ANSI,
including main(). One exception is CreateProcess, which will need the
UTF-16 form (CreateProcessW).
Despite the locale being set to utf-8, we'll have to explicitly set
the console output to utf-8 using SetConsoleOutputCP(CP_UTF8).
In order to use the UTF-8 locale, we'll have to switch the mingw-llvm
runtime from msvcrt to ucrt.
This also fixes ceph-dokan crashes that currently occur when non-ANSI
paths are logged.
Casey Bodley [Tue, 21 Nov 2023 18:44:40 +0000 (13:44 -0500)]
crush: remove unused variables
[161/715] Building CXX object src/crush/CMakeFiles/crush_objs.dir/CrushTester.cc.o
ceph/src/crush/CrushTester.cc:478:7: warning: variable 'num_devices_active' set but not used [-Wunused-but-set-variable]
int num_devices_active = 0;
^
1 warning generated.
[165/715] Building CXX object src/crush/CMakeFiles/crush_objs.dir/CrushWrapper.cc.o
ceph/src/crush/CrushWrapper.cc:1579:9: warning: variable 'local_changed' set but not used [-Wunused-but-set-variable]
int local_changed = 0;
^
```
// SnapTrimEvent is a background operation,
// it's lifetime is not guarnteed since the caller
// returned future is being ignored. We should capture
// a self reference thourhgout the entire execution
// progress (not only on finally() continuations).
// See: PG::on_active_actmap()
```
kernel callstack:
#0 0x55e310e0ace7 in seastar::shared_mutex::unlock() (/usr/bin/ceph-osd+0x1edd0ce7)
#1 0x55e313325d9c in auto seastar::futurize_invoke<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::ExitBarrier<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::BlockingEvent::Trigger<crimson::osd::SnapTrimEvent> >::exit()::{lambda()#1}&>(crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::ExitBarrier<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::BlockingEvent::Trigger<crimson::osd::SnapTrimEvent> >::exit()::{lambda()#1}&) (/usr/bin/ceph-osd+0x212ebd9c)
#2 0x55e3133260ef in _ZN7seastar20noncopyable_functionIFNS_6futureIvEEvEE17direct_vtable_forIZNS2_4thenIZN7crimson23OrderedConcurrentPhaseTINS7_3osd13SnapTrimEvent9WaitSubopEE11ExitBarrierINSC_13BlockingEvent7TriggerISA_EEE4exitEvEUlvE_S2_EET0_OT_EUlDpOT_E_E4callEPKS4_ (/usr/bin/ceph-osd+0x212ec0ef)
0x61500013365c is located 92 bytes inside of 472-byte region [0x615000133600,0x6150001337d8)
freed by thread T2 here:
#0 0x7fb345ab73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf)
#1 0x55e313474863 in crimson::osd::SnapTrimEvent::~SnapTrimEvent() (/usr/bin/ceph-osd+0x2143a863)
previously allocated by thread T2 here:
#0 0x7fb345ab6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367)
#1 0x55e31183ac18 in auto crimson::OperationRegistryI::create_operation<crimson::osd::SnapTrimEvent, crimson::osd::PG*, SnapMapper&, snapid_t const&, bool const&>(crimson::osd::PG*&&, SnapMapper&, snapid_t const&, bool const&) (/usr/bin/ceph-osd+0x1f800c18)
SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-osd+0x1edd0ce7) in seastar::shared_mutex::unlock()
```
Zac Dover [Fri, 17 Nov 2023 09:24:14 +0000 (19:24 +1000)]
doc/start: explain "OSD"
Explain the initialism "OSD" and link to its definition in the glossary.
This PR is raised in response to an anonymous documentation bug that reads
"Paragraph 2 uses the acronym OSD without any explanation.
This makes it very difficult to understand this part of
the documentation as there is no indication of what this
acronym is until much further into the documentation. Replace
first occurence of OSD with Object Storage Daemon (OSD) or
link it to the glossary."
-- https://pad.ceph.com/p/Report_Documentation_Bugs
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
The interface to the load_bucket call changed when tenant and bucket
name are specified and user is no longer required. This updates the
flight call to load_bucket to reflect those changes.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
#0 0x5653c613c071 in seastar::shared_mutex::unlock() (/usr/bin/ceph-osd+0x1ed27071)
#1 0x5653c8670acf in auto seastar::futurize_invoke<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::ExitBarrier<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::BlockingEvent::Trigger<crimson::osd::SnapTrimEvent> >::exit()::{lambda()#1}&>(crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::ExitBarrier<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::BlockingEvent::Trigger<crimson::osd::SnapTrimEvent> >::exit()::{lambda()#1}&) (/usr/bin/ceph-osd+0x2125bacf)
#2 0x5653c8670e22 in _ZN7seastar20noncopyable_functionIFNS_6futureIvEEvEE17direct_vtable_forIZNS2_4thenIZN7crimson23OrderedConcurrentPhaseTINS7_3osd13SnapTrimEvent9WaitSubopEE11ExitBarrierINSC_13BlockingEvent7TriggerISA_EEE4exitEvEUlvE_S2_EET0_OT_EUlDpOT_E_E4callEPKS4_ (/usr/bin/ceph-osd+0x2125be22)
freed by thread T1 here:
#0 0x7f10628b73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf)
#1 0x5653c8794bff in crimson::osd::SnapTrimEvent::~SnapTrimEvent() (/usr/bin/ceph-osd+0x2137fbff)
previously allocated by thread T1 here:
#0 0x7f10628b6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367)
SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-osd+0x1ed27071) in seastar::shared_mutex::unlock()
```
John Mulligan [Tue, 14 Nov 2023 00:15:25 +0000 (19:15 -0500)]
cephadm: workaround issues running cephadm with relative path
Implement a workaround for the jinja2 package loader not correctly
finding a template inside the cephadmlib package when run as
a zipapp. See docstring in the shim class for more details.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
'ceph-volume raw list' is broken for a specific use case (rook).
rook copies devices from /dev/ to /mnt for specific/internal needs.
when ceph-volume raw list is passed a device from /mnt then
ceph-volume ignores it and return an empty dict.
That prevent rook from creating OSDs properly.
Ronen Friedman [Thu, 9 Nov 2023 15:21:42 +0000 (09:21 -0600)]
tests/standalone: fix scrub-related tests following command changes
Using
ceph tell $pgid [deep]-scrub
to initiate an 'operator initiated' scrub, and
ceph tell $pgid schedule[-deep]-scrub
for causing a 'periodic scrub' to be scheduled.
mgr/cephadm: Expose nvmeof gateway configuration parameters through specifications
- min_controller_id, max_controller_id: Enable the specification of minimum and maximum controller IDs utilized by the SPDK.
Having distinct controller IDs is vital for configuring multipath setups.
- enable_spdk_discovery_controller: Manage whether the SPDK or ceph-nvmeof discovery service is employed. The default value is set to False.
Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>
Zac Dover [Tue, 14 Nov 2023 13:40:42 +0000 (23:40 +1000)]
doc/glossary: add "Quorum" to glossary
Add the term "Quorum" to the glossary and link to the part of
architecture.rst concerning Monitors. The sticky header at the top of
the docs.ceph.com website gets in the way of the location linked to in
this commit, but fatigue and disgust prevent me from spending time today
trial-and-erroring my way through the hostile and ill-documented
wilderness of scroll-margin so that the link goes where it should.
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
John Mulligan [Sun, 5 Nov 2023 17:57:45 +0000 (12:57 -0500)]
cephadm: eliminate the _update_pids_limit function
Eliminate the _update_pids_limit function by moving the logic into the
various daemon type classes `customize_container_args` functions allow
each class to decide if it needs to run with unlimited-pids or not.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 5 Nov 2023 16:44:48 +0000 (11:44 -0500)]
cephadm: use class methods when setting up ceph based daemon containers
Now that the ceph and ceph exporter classes have all the necessary
methods, rework the get_container function to use those class methods
in a common manner.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 21 Oct 2023 00:18:36 +0000 (20:18 -0400)]
cephadm: add container envs methods to ceph and ceph exporter classes
These, currently unused, methods will be used in a subsequent change to
move logic for setting up containers out of the get_containers function
into the daemon type classes in a common way.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 5 Nov 2023 16:12:54 +0000 (11:12 -0500)]
cephadm: add methods to ceph daemon type class
Add unused methods to the ceph class in order to prepare for moving
the logic for setting up the various ceph deaemon types to be
handled by the class.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 5 Nov 2023 16:09:27 +0000 (11:09 -0500)]
cephadm: add more ContainerDeamonForm methods to the monitoring class
Add methods customize_{container_args,process_args} & default_entrypoint
to the monitoring daemon type class. Use those methods in the
get_container function.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 5 Nov 2023 16:00:54 +0000 (11:00 -0500)]
cephadm: add more ContainerDeamonForm methods to the tracing class
Add methods customize_{container_envs,process_args} & default_entrypoint
to the tracing daemon type class. Use those methods in the get_container
function.
This change tries to adapt to, but not fix, the rather strange
set_configuration static method that side-effects the class's(!)
property.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 5 Nov 2023 13:09:10 +0000 (08:09 -0500)]
cephadm: add more ContainerDeamonForm methods to the nfs class
Add methods customize_{container_envs,process_args} & default_entrypoint
to the nfs daemon type class. Use those methods in the get_container
function.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 5 Nov 2023 12:18:55 +0000 (07:18 -0500)]
cephadm: add more ContainerDeamonForm methods to the custom container class
Add methods customize_{process_args,container_envs,container_args} and
default_entrypoint to the custom container daemon type class. Use those
methods in the get_container function.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Tue, 7 Nov 2023 22:57:43 +0000 (17:57 -0500)]
cephadm: update additional container daemon form base class methods
Fix the function signature for the previously added stub
customize_container_args.
Add methods for customize_process_args and customize_container_envs.
The "process args" are args passed to the daemon process / entrypoint.
Container args are passed to the container engine.
Add a default_entrypoint method to return the default entrypoint for
a particular daemon-in-container.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
and it placed the node-exporter daemons on vm-00
and vm-02 but not vm-01. Obviously there are more
advanced scenarios that justify this than listing
two hosts, but using "|" as an OR like that is an
example of something you can't do with the fnmatch
version of the host pattern
Ramana Raja [Thu, 2 Nov 2023 21:44:10 +0000 (17:44 -0400)]
pybind/mgr: remove __del__() of mgr modules
It's strongly recommended for objects that have references to
external resources (e.g., files) to explicitly release them.
Python doesn't guarantee garbage collection of objects and hence
doesn't guarantee freeing of external resources that occur on
garbage collection.
The __del__() methods in the python mgr modules may not even be
called since garbage collection of objects is not guaranteed in python.
And some of the __del__() methods try to cleanup that seem redundant.
- In volumes/module.py, vc.shutdown() is called in Module.shutdown().
No need to call it again in Module.__del__()
- In telegraf/basesocket.py, BaseSocker.close() is called in
BaseSocket.__exit__(). No need to call it again in
BaseSocket.__del__().
- In mgr_module.py, MgrModuleLoggingMixin._unconfigure_logging() is
called in MgrModule.__init__() and MgrStandbyModule.__init__(). No
need to call it in MgrModule.__del__() and
MgrStandbyModule.__del__().|
- In dashboard/services/cephfs.py, the libcephfs mount is not
shutdown explicitly by the mgr module. However, the cython libcephfs
bindings has a LibCephFS.__dealloc__() finalizer method that calls
LibCephFS.shutdown(). This should unmount and cleanup the ceph mount
handle.
Remove the __del__() of the python mgr modules.
Fixes: https://tracker.ceph.com/issues/63421 Signed-off-by: Ramana Raja <rraja@redhat.com>
projects / ceph.git / log