Kefu Chai [Thu, 19 Aug 2021 09:24:32 +0000 (17:24 +0800)]
common/options: validate see-also
y2c.py is like a compiler which translates .yaml to .cc and .h files,
it does not have access to all .yaml files. to validate the dangling
see-also issue, we need to do this with a "linker".
in this change, validate-options.py is introduced to check if any of
option name included by the see-also property is valid.
J. Eric Ivancich [Wed, 28 Jul 2021 18:07:09 +0000 (14:07 -0400)]
rgw-multisite: metadata conflict not computed correctly
The former logic with a conditional based on `++i == 0` would never
execute. So this uses a boolean to differentiate the first from other
iterations and tries to clarify the code through commenting and an
explicit declaration. Additionally a warning is eliminated by
initializing a variable.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
add `event_loop` and `tkey` object to with_cephadm_module, and create MockEventLoopThread in fixtures.py to test async functions of ssh.py.
rewrite test_offline to be compatible with asyncssh
Fixes: https://tracker.ceph.com/issues/44676 Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>
mgr/cephadm: use _remote_connection (ssh.py), _execute_command, _check_execute_command in _run_cephadm
remove _get_connection from module.py and _remote_connection in serve.py, replacing with _remote_connection in ssh.py.
also, replace remoto.process.check with _execute_command and _check_execute_command in ssh.py
Fixes: https://tracker.ceph.com/issues/44676 Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>
mgr/cephadm: remove remotes.py, replace old _write_remote_file in serve.py with write_remote_file in ssh.py
remove remotes.py because it is specific to execnet/remoto.
_write_remote_file in ssh.py now fulfills the function of write_file in remotes.py and the old _write_remote_file in serve.py
Fixes: https://tracker.ceph.com/issues/44676 Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>
mgr/cephadm: create thread to start event loop for ssh.py, and return results of the async functions with get_result
The EventLoopThread class starts a thread and an event loop which runs forever. Coroutines are scheduled on the event loop by the `get_result` method which uses `run_coroutine_threadsafe` to return a concurrent.futures.Future, and ultimately the result with .result()
Fixes: https://tracker.ceph.com/issues/44676 Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>
mgr/cephadm: create async function _write_remote_file to write files on remote host
_write_remote_file uses _check_execute_command in ssh.py which calls _execute_command which uses shlex quote. Thus, any commands with an int will need to be transformed into a str because shlex quote does not take int objects
Fixes: https://tracker.ceph.com/issues/44676 Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>
mgr/cephadm: execute commands run over ssh via asyncssh
_execute_command will run commands over ssh using the asyncssh `run` method: https://asyncssh.readthedocs.io/en/latest/api.html#asyncssh.SSHClientConnection.run
_check_execute_command will check the output of _execute_command and raise OrchestratorError if command fails on the remote host.
All commands run over ssh are prepended with sudo in `_execute_command` and shell-escaped with shlex quote.
If the cached ssh connection is closed or broken, the connection object will be removed from the cache, added to the `offline_hosts`, and an OrchestratorError will be raised. On the next call, the connection object will attempt to be recreated.
Exceptions involving asyncssh methods should be handled otherwise errors like TypeError: __init__() missing 1 required positional argument: 'reason' could occur due to the asyncssh error interacting with `raise_if_exception`
Fixes: https://tracker.ceph.com/issues/44676 Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>
mgr/cephadm: create and cache asyncssh connection objects, and handle asyncssh connection errors
Create asyncssh connection object in async `_remote_connection` function and cache in `self.cons`
Create a handler for asyncssh log redirection and output ssh log if a connection error occurs
Disable asyncssh logger from propagating because the asyncssh info messages are verbose
Fixes: https://tracker.ceph.com/issues/44676 Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>
Kefu Chai [Fri, 20 Aug 2021 14:50:40 +0000 (22:50 +0800)]
cmake: exclude "grafonnet-lib" target from "all"
so we don't build this target when running "make", and hence avoid
accessing the internet in a building envronment where the internest
access is not allowed.
As part of removing RGWObjManifest from the Zipper API, we need to remove
WriteOp. Fortunately, with the multipart upload changes, it's no longer
needed outside the RadosStore.
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
Deepika Upadhyay [Thu, 19 Aug 2021 09:00:33 +0000 (14:30 +0530)]
run-make-check: fix do_cmake not consuming run-make-check opts
run-make-check.sh uses run-make.sh to `prepare`(install dependencies)
and `configure` cmake options, without quotes these options containing
special characters(hypens mostly) are skipped, hence we see not all
options supplied at cmake configure step.
Resolves(focused to solve issues in jenkins build env):
- missing cmake options:
cmake_opts+=" -DCMAKE_CXX_COMPILER=$cxx_compiler -DCMAKE_C_COMPILER=$c_compiler"
cmake_opts+=" -DCMAKE_CXX_FLAGS_DEBUG=\-Werror"
- Ninja not being used as cmake generator
Remove the remaining rgw_raw_obj from the API. This consists of:
- move get/delete raw_obj from Store to RadosStore; it's only needed by
RadosStore code
- Add a layout formatter to the API
- remove head_obj and get_manifest() from ReadOp
- Remove GCChain (no longer needed)
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
Ronen Friedman [Tue, 17 Aug 2021 05:11:15 +0000 (05:11 +0000)]
common: create a faster & cleaner alternative to argv_to_vec()
New function signature follows current C++ guidance regarding
returning values from functions. It is also faster
- 1.3X for a sample test of 10 arguments.
Joseph Sawaya [Wed, 11 Aug 2021 15:01:25 +0000 (11:01 -0400)]
mgr/rook: fix convert_size method in DefaultFetcher
This commit fixes the convert_size method by getting it
to use the re python module to split the digits and letters
to support all units a PV could be expressed in.
Joseph Sawaya [Mon, 19 Jul 2021 13:37:58 +0000 (09:37 -0400)]
mgr/rook: OSD creation using Creator classes
This commit implements the apply_drivegroups method in the
RookOrchestrator class and creates the DefaultCreator and
LSOCreator classes that handle creating OSDs. The add_osds
method in RookCluster will use each creator based on what
storage class the user provided in the ceph config.
Joseph Sawaya [Mon, 19 Jul 2021 13:26:03 +0000 (09:26 -0400)]
mgr/rook: add supplemental information to device object in LSOFetcher
This commit adds the PV name, node name, vendor and model to the Device object
created in the LSOFetcher device method, this information is useful for adding
OSDs.
myoungwon oh [Tue, 17 Aug 2021 12:51:29 +0000 (21:51 +0900)]
test: allowing >= the real refcount to avoid false alarm
Current dedup allow to contain multiple same sources using
multiset, which results in inconsistent situation as follow
(during set_chunk, but not confined in set_chunk).
1. User issues set_chunk
2. OSD receives the set_chunk, and sends increment message
to an object in the low tier (INPROGRESS).
3. OSD map is changed (841 → 843)
3.5. on_change() is called
4. the set_chunk op is reenqueued by requeue_op()
5. OSD handles the duplicated set_chunk, but it is not able to
know the set_chunk is duplicated because it does not log on the disk yet.
6. OSD issues increment message again to the object
in the low tier. (increment operation is executed twice)
To fix this, this commit allows >= the real refcount in test cases
projects / ceph.git / log