osd/scrub: add "queue my request" flag to replica reservation messages

Up-to-date primaries will set this flag when sending a reservation
request. The replica OSD, if too busy to handle the request immediately, will queue
it until such time that the number of concurrent reservations is below the
configured limit. The queued requests are honored in FIFO order.

Old primaries will not set this flag, and will receive the expected
grant or deny reply immediately.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: add synchronous request to AsyncReserver API

To be used when handling replica reservation requests from "old"
primaries, that expect an immediate grant/deny reply.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55281 from Matan-B/wip-matanb-crimson-cyanstore-rmcoll

crimson/os/cyanstore: support OP_RMCOLL

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #55333 from zdover23/wip-doc-2024-01-27-radosgw-index-verb-disagreement

doc/radosgw: fix verb disagreement - index.html

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: fix verb disagreement - index.html

Fix a tricky verb disagreement and rewrite a few sentences for what I
hope is greater clarity.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54423 from guits/dmcrypt-optim

ceph-volume: use 'no workqueue' options with dmcrypt

ceph-volume: fix partitions support in disk.get_devices()

The following:
```
is_part = get_file_contents(os.path.join(_sys_dev_block_path, item, 'partition')) == "1"
```
assumes any `/sys/dev/block/x:y/partition` contains '1' which is wrong.
This file actually contains the corresponding partition number.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #55323 from ceph/dependabot-github_actions-gregsdennis-dependencies-action-1.3.2

.github: Bump gregsdennis/dependencies-action from 1.2.3 to 1.3.2

Merge pull request #55331 from ceph/revert-55096-sjust/for-review/wip-crush-msr

Revert "crush: add multistep retry rules"

ceph-volume: use 'no workqueue' options with dmcrypt

CloudFlare engineers made some testing and realized that using
workqueues with encryption on flash devices has a bad effect.

See [1] for details.

With this patch it will make ceph-volume call crypsetup with
`--perf-no_read_workqueue` and `--perf-no_write_workqueue` options
when the device is not a rotational.

[1] https://blog.cloudflare.com/speeding-up-linux-disk-encryption/

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
Co-Authored-by: Stefan Kooman <stefan@kooman.org>
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Revert "crush: add multistep retry rules"

This PR was merged by accident before it was ready.
Let's revert for now and open a new PR.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #55096 from athanatos/sjust/for-review/wip-crush-msr

crush: add multistep retry rules

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #55303 from cbodley/wip-63130-debug

cmake/arrow: don't treat warnings as errors

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #55234 from ajarr/wip-64063

rbd-nbd: use netlink interface by default

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55121 from zhscn/fix-ambiguous-error

common: fix ambiguous error when using gcc 13

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #54896 from cyx1231st/wip-crimson-save-conn-foreign-copy

crimson/osd: drop a foreign-copy to shard-0 for every pg operation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52496 from adamemerson/wip-rgw-surface-neorados

rgw: Surface neorados

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

.github: Bump gregsdennis/dependencies-action from 1.2.3 to 1.3.2

Bumps [gregsdennis/dependencies-action](https://github.com/gregsdennis/dependencies-action) from 1.2.3 to 1.3.2.
- [Release notes](https://github.com/gregsdennis/dependencies-action/releases)
- [Commits](https://github.com/gregsdennis/dependencies-action/compare/80b5ffec566913b1494d5a8577ab0d60e476271d...f98d55eee1f66e7aaea4a60e71892736ae2548c7)

---
updated-dependencies:
- dependency-name: gregsdennis/dependencies-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #55308 from ljflores/wip-dependabot

Merge pull request #55307 from zdover23/wip-doc-2024-01-25-radosgw-admin-usage

doc/radosgw: edit "Usage" admin.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit "Usage" admin.rst

Edit "Usage" in doc/radosgw/admin.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55315 from cbodley/wip-moncommand-dencoder

mon: zero-initialize MonCommand::flags

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

rbd-nbd: map using netlink interface by default

Mapping rbd images to nbd devices using ioctl interface is not
robust. It was discovered that the device size or the md5 checksum
of the nbd device was incorrect immediately after mapping using
ioctl method. When using the nbd netlink interface to map RBD images
the issue was not encountered. Switch to using nbd netlink interface
for mapping.

Fixes: https://tracker.ceph.com/issues/64063
Signed-off-by: Ramana Raja <rraja@redhat.com>

crimson/os/cyanstore: support OP_RMCOLL

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

mon: zero-initialize MonCommand::flags

causing failures in check-generated.sh:

2 MonCommand
/tmp/typ-qmnDZ1ahR /tmp/typ-lIgJTZiUe differ: byte 100, line 6
**** MonCommand test 1 dump_json check failed ****
   ceph-dencoder type MonCommand select_test 1 dump_json > /tmp/typ-qmnDZ1ahR
   ceph-dencoder type MonCommand select_test 1 encode decode dump_json > /tmp/typ-lIgJTZiUe
6c6
<     "flags": 94661933599904
---
>     "flags": 94342467308704

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55287 from ajarr/wip-64139

rbd-nbd: fix resize of images mapped using netlink

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55270 from afreen23/fix-cap-inconsistency-multisite

mgr/dashboard: Fix inconsistency in capitalisation of "Multi-site"

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: rosinL <NOT@FOUND>

Merge pull request #55182 from rkachach/fix_issue_64029

mgr/rook: adding some basic rook e2e testing

Merge pull request #55266 from athanatos/sjust/wip-63996

crimson: retain map references in OSDSingletonState::store_maps

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/shard_services: retain map references in OSDSingletonState::store_maps

Introduced: 3f11cd94
Fixes: https://tracker.ceph.com/issues/63996
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/shard_service.cc: convert to newer logging machinery

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd.cc: migrate logging to new style

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #55288 from athanatos/sjust/wip-64140

Revert "crimson/os/alienstore/alien_log: _flush concurrently"

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

rgw: Thread site config into RADOS driver

Make this available so multisite sync doesn't have to rely on the zone
service.

Signed-off-by: Adam Emerson <aemerson@redhat.com>

rgw: Asio frontend shares `io_context` with the rest of RGW

Pull the `io_context` and threads out of `AsioFrontend`, pass in a
reference to `io_context_pool` so it can be shut down at `AsioFrontend::join`

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #54987 from batrick/i63822

pybind/mgr/devicehealth: skip legacy objects that cannot be loaded

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge pull request #54491 from jianwei1216/fix_osd_pg_stat_report_interval_max_cmain

fix: resolve inconsistent judgment of osd_pg_stat_report_interval_max

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #53250 from YiteGu/add-perfcount-for-allocator

os/bluestore: add perfcount for bluestore/bluefs allocator

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #52530 from amathuria/wip-amat-fix-59531

osd: Add memstore to unsupported objstores for QoS

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: SAL drivers take `boost::asio::io_context`

`RadosDriver` needs it, and since SAL generally uses `optional_yield`,
other stores are going to need it to implement that sensibly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph/async: `io_context_pool` constructor/start takes init function

Add overloads of the constructor and `start` that take an init
function, for setting up the thread before dropping into
`io_context::run`.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/async: `io_context_pool` constructor takes int64_t

Just because this is what Ceph's config uses and it saves a narrowing
conversion. If we want to set a max value on the thread count, we
should do it in config.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

include/inline_memory.h: Add missing includes to fix build

Signed-off-by: Adam Emerson <aemerson@redhat.com>

.github: fix length of dependabot commit message prefix

Followup to commit 265f727481c729fbe78dce7410def7de6fe869d1.

Fixes this error mesage under GitHub Insights (https://github.com/ceph/ceph/network/updates):

```
The property '#/updates/1/commit-message/prefix' was not of a maximum string length of 15
```

Signed-off-by: Laura Flores <lflores@ibm.com>

rbd-nbd: log errors during netlink_resize() using derr

When using rbd CLI to map the images to NBD devices via netlink,
any errors that arose during image resizing in netlink_resize()
were not logged. Switching the error logging from using cerr to
derr helps log the errors from netlink_resize().

Signed-off-by: Ramana Raja <rraja@redhat.com>

rbd_nbd: fix resize of images mapped using netlink

Include device identifier or cookie in the message sent to the kernel
to resize images mapped to NBD devices using netlink. Otherwise,
netlink_resize() fails and the size of the device isn't updated.

Fixes: https://tracker.ceph.com/issues/64139
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #49462 from rzarzynski/wip-bug-53789

osdc: fix the ENOCONN normalization in Objecter::_linger_reconnect()

Merge pull request #55219 from samarahu/rgw_asio_frontend_asserts

rgw/asio: Add asserts to rgw_asio_frontend.cc

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mgr/rook: increase minikube mem to 6GB to avoid stability issues

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #55192 from dparmar18/fix_docstrings_ceph_test_case

qa: typo fixes in ceph_test_case docstrings

Merge pull request #53320 from jzhu116-bloomberg/wip-62710

rgw/multisite: maintain endpoints connectable status and retry the requests to them when appropriate

Reviewed-by: Mark Kogan <mkogan@ibm.com>

Merge pull request #54941 from samsungceph/vstart_network_v2

vstart: Pick only CIDR-formatted routes when cephadm enabled

Reviewed-by: Adam King <adking@redhat.com>

cmake/arrow: don't treat warnings as errors

with arrow's default BUILD_WARNING_LEVEL, -Werror is added to cflags and
debug builds fail on a warning about _FORTIFY_SOURCE

Fixes: https://tracker.ceph.com/issues/63130
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53668 from mdw-at-linuxbox/wip-master-update-kmip-1

Update libkmip to pull in some portability changes.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/rook: adding some basic rook e2e testing
Fixes: https://tracker.ceph.com/issues/64029
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Revert "crimson/os/alienstore/alien_log: _flush concurrently"

While submitting the log line asyncronously is reasonable,
with this implementation the EntryVector &q parameter does
not necessarily outlive the submission continuation.

This reverts commit 511af83e2747361350b60ce0ce88e67a726d9343.

Fixes: https://tracker.ceph.com/issues/64140
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #55183 from galsalomon66/s3select_fixes_QE_bugs

rgw/s3select: bug fixes per QE recent defects

Update libkmip submodule to pull in some portability changes.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #55217 from ronen-fr/wip-rf-old-reserv

osd/scrub: check reservation replies for relevance

Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #55067 from yaarith/telemetry-pool-flags

mgr/telemetry: add pool flags

Merge pull request #55240 from rosinL/wip-fix-64032

install-deps: Force remove ceph-libboost* packages

Merge pull request #55278 from Himura2la/patch-2

Docs: Specify correct fs type for mkfs on volume creation

Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

bug fixes per QE recent defects
update for the engine_version message
s3select submodule

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

Merge pull request #55277 from yuvalif/wip-yuval-63578

rgw/lua: fix compilation issue when lua packages are disabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: specify correct fs type for mkfs

The default value is ext2, which is not supported (anymore?).

Signed-off-by: Vladislav Glagolev <vladislav.glagolev@devexpress.com>

rgw/lua: fix compilation issue when lua packages are disabled

Fixes: https://tracker.ceph.com/issues/63578#change-253102
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/dashboard: Fix inconsistency in capitalisation of "Multi-site"

fixes https://tracker.ceph.com/issues/64125

Across the dashboard, two instances are present: Multi-site and
Multi-Site.
Making it consistent all over by using Multi-site.

Signed-off-by: Afreen <afreen23.git@gmail.com>

osd/scrub: check reservation replies for relevance

Compare a token (nonce) carried in the reservation reply with the remembered
token of the reservation request.  If they don't match, the reply is
stale and should be ignored (and logged).

Fixes: https://tracker.ceph.com/issues/64052
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55107 from ronen-fr/wip-rf-rm-penaltyq

osd/scrub: remove the 'penalty queue' from the scrubber

Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #55269 from zdover23/wip-doc-2024-01-23-radosgw-admin-read-write-global-rate-limit-config

doc/radosgw: edit "read/write global rate limit" admin.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit "read/write global rate limit" admin.rst

Edit "Reading/Writing Global Rate Limit Configuration" in
doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55223 from athanatos/sjust/wip-64055

crimson: clear obc_registry on interval change

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rgw/multisite: add multisite test cases with some rgw instances down

Signed-off-by: Juan Zhu <jzhu4@dev-10-34-20-139.pw1.bcc.bloomberg.com>

Merge pull request #55070 from pdvian/wip-fix-progressevent

mon: initialize ProgressEvent::add_to_ceph_s

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #53154 from ifed01/wip-ifed-no-death-tests

test/store_test: get rid off assert_death.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #49415 from ljflores/wip-update-telemetry-upgrade

qa/workunits: update telemetry quincy workunits with `basic_pool_options_bluestore` collection

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

osd/scrub: update job's NB on failure

When a scrub job fails, update its NB to the current time plus a
fixed delay.  This prevents the job from being scheduled again
immediately.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: fix set_last_deep_scrub_stamp()

The call should update last_scrub_stamp, too, without
requiring an extra call to on_scrub_schedule_input_change()

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: introduce a 'not before' attribute for scrub jobs

The NB enables the OSD to delay the next attempt to schedule a specific
scrub job.  This is useful for jobs that have failed for whatever
reason, especially if the primary has failed to acquire the replicas.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: remove the 'penalized jobs' queue

The 'penalized jobs' queue was used to track scrub jobs that had failed
to acquire their replicas, and to prevent those jobs from being retried
too quickly.  This functionality will be replaced by a
simple 'not before' delay (see the next commits).

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson/osd/osd_operations/scrub_events: fix to use get_peering_request_pipeline()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: drop a foreign-copy to shard-0 for every pg operation

By using ConnectionRef before pg submission, and after that, change to
use ConnectionXcoreRef.

The intent is to drop the foreign copy of the connection to shard 0 at
pg submission time. This should remove two pairs of crosscore
communications in shard 0 for each I/O, one for connection-ref foreign
copy, another for connection-ref destruction.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #55262 from Matan-B/wip-matanb-crimson-bluestore-submit

crimson/os/alienstore/alien_log: _flush concurrently

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations/client_request: don't use and move pgref at the same time

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations/client_request: cleanup shard_services

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/alienstore/alien_log: _flush concurrently

In continuation to c15e56e386251403a876454f6a4aa186284565e1

Authored-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55190 from zdover23/wip-doc-2024-01-16-radosgw-admin-enable-disable-bucket-rate-limit

doc/radosgw: edit "Enable/Disable Bucket Rate Limit"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53288 from rzarzynski/wip-crimson-dont-shadow-store-in-ecbackend

crimson: drop store from ECBackend to not shadow PGBackend::store

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54813 from amathuria/wip-crimson-amat-fix-config-set-cmd

src/crimson: Add support for the OSD to receive config changes

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55127 from idryomov/wip-63341

librbd: improve rbd_diff_iterate2() performance in fast-diff mode

Reviewed-by: Mykola Golub <mgolub@suse.com>

PendingReleaseNotes: add rbd_diff_iterate2 note

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: try to preserve object map for diff-iterate in fast-diff mode

As an optimization, try to ensure that the object map for the end
version is preloaded through the acquisition of exclusive lock and
as a consequence remains around until exclusive lock is released.
If it's not around, DiffRequest would (re)load it on each call.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/object_map: potentially use in-memory object map in DiffRequest

If the object map for the end version is around (already loaded in
memory, either due to the end version being a snapshot or due to
exclusive lock being held), use it to run diff-iterate against the
beginning of time.  Since it's the only object map needed in that
case, such calls would be satisfied locally.

Fixes: https://tracker.ceph.com/issues/63341
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/object_map: decouple object map processing in DiffRequest

In preparation for potentially using in-memory object map, decouple
object map processing from loading object maps and place the logic in
prepare_for_object_map() and process_object_map().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

common/bit_vector: fix iterator vs reference constness confusion

T (ConstIterator or Iterator) is confused with const T here:
IteratorImpl dereference operator is wrongly overloaded on const
and returns Reference instead of ConstReference for ConstIterator.
This then fails inside bufferlist bowels because Reference is
incompatible with bufferlist::const_iterator.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/object_map: make object map in handle_load_object_map() local

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/object_map: don't resize object map in handle_load_object_map()

Currently it's done in two cases:

- if the loaded object map is larger than expected based on byte size,
  it's truncated to expected number of objects
- in case of deep-copy, if the loaded object map is smaller than diff
  state, it's expanded to get "track the largest of all versions in the
  set" semantics

Both of these cases can be easily dealt with without modifying the
object map.  Being able to process a const object map is needed for
working on in-memory object map which is external to DiffRequest.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

common/bit_vector: fix IteratorImpl post-increment operator

It's totally broken: instead of returning the current position and
moving to the next position, it returns the next position and doesn't
move anywhere.  Luckily it hasn't been used until now.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: drop DiffIterate::diff_object_map() declaration

This is a leftover from commit 2b3a46801d39 ("librbd: switch
diff-iterate API to use new object-map diff helper").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: propagate diff-iterate range to parent in fast-diff mode

When getting parent diff, pass the overlap-reduced image extent instead
of the entire 0..overlap range to avoid a similar quadratic slowdown on
cloned images.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>