]> git-server-git.apps.pok.os.sepia.ceph.com Git - ceph-ci.git/commit
mon: add NVMe-oF gateway monitor and HA
authorLeonid Chernin <lechernin@gmail.com>
Tue, 17 Oct 2023 13:25:07 +0000 (13:25 +0000)
committerAlexander Indenbaum <aindenba@redhat.com>
Wed, 19 Nov 2025 18:41:51 +0000 (20:41 +0200)
commitddfe00ee36d0baf113fd042632bf025ba437a7c5
tree3760716876e0583fde5e66bbfb0ae1a8b57d41ab
parent83c6ad27a40495bb2b7b74e8d5aa67d82f856666
mon: add NVMe-oF gateway monitor and HA

- gateway submodule

Fixes: https://tracker.ceph.com/issues/64777
This PR adds high availability support for the nvmeof Ceph service. High availability means that even in the case that a certain GW is down, there will be another available path for the initiator to be able to continue the IO through another GW. High availability is achieved by running nvmeof service consisting of at least 2 nvmeof GWs in the Ceph cluster. Every GW will be seen by the host (initiator) as a separate path to the nvme namespaces (volumes).

The implementation consists of the following main modules:

- NVMeofGWMon - a PaxosService. It is a monitor that tracks the status of the nvmeof running services, and take actions in case that services fail, and in case services restored.
- NVMeofGwMonitorClient – It is an agent that is running as a part of each nvmeof GW. It is sending beacons to the monitor to signal that the GW is alive. As a part of the beacon, the client also sends information about the service. This information is used by the monitor to take decisions and perform some operations.
- MNVMeofGwBeacon – It is a structure used by the client and the monitor to send/recv the beacons.
- MNVMeofGwMap – The map is tracking the nvmeof GWs status. It also defines what should be the new role of every GW. So in the events of GWs go down or GWs restored, the map will reflect the new role of each GW resulted by these events. The map is distributed to the NVMeofGwMonitorClient on each GW, and it knows to update the GW with the required changes.

It is also adding 3 new mon commands:
- nvme-gw create
- nvme-gw delete
- nvme-gw show

The commands are used by the ceph adm to update the monitor that a new GW is deployed. The monitor will update the map accordingly and will start tracking this GW until it is deleted.

Signed-off-by: Leonid Chernin <lechernin@gmail.com>
Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>
(cherry picked from commit 5843c6b04bacf9a7c981fca5d874ab3400f855db)
37 files changed:
.gitmodules
PendingReleaseNotes
ceph.spec.in
src/CMakeLists.txt
src/ceph_nvmeof_monitor_client.cc [new file with mode: 0644]
src/common/options/global.yaml.in
src/common/options/mon.yaml.in
src/messages/MNVMeofGwBeacon.h [new file with mode: 0644]
src/messages/MNVMeofGwMap.h [new file with mode: 0644]
src/mon/CMakeLists.txt
src/mon/MonCommands.h
src/mon/Monitor.cc
src/mon/Monitor.h
src/mon/NVMeofGwMap.cc [new file with mode: 0755]
src/mon/NVMeofGwMap.h [new file with mode: 0755]
src/mon/NVMeofGwMon.cc [new file with mode: 0644]
src/mon/NVMeofGwMon.h [new file with mode: 0644]
src/mon/NVMeofGwSerialize.h [new file with mode: 0755]
src/mon/NVMeofGwTypes.h [new file with mode: 0755]
src/mon/mon_types.h
src/msg/Message.cc
src/msg/Message.h
src/nvmeof/NVMeofGwClient.cc [new file with mode: 0644]
src/nvmeof/NVMeofGwClient.h [new file with mode: 0644]
src/nvmeof/NVMeofGwMonitorClient.cc [new file with mode: 0644]
src/nvmeof/NVMeofGwMonitorClient.h [new file with mode: 0644]
src/nvmeof/NVMeofGwMonitorGroupClient.cc [new file with mode: 0644]
src/nvmeof/NVMeofGwMonitorGroupClient.h [new file with mode: 0644]
src/nvmeof/gateway [new submodule]
src/pybind/mgr/cephadm/services/nvmeof.py
src/pybind/mgr/cephadm/templates/services/nvmeof/ceph-nvmeof.conf.j2
src/python-common/ceph/deployment/service_spec.py
src/test/CMakeLists.txt
src/test/test_nvmeof_mon_encoding.cc [new file with mode: 0644]
src/tools/ceph-dencoder/CMakeLists.txt
src/tools/ceph-dencoder/nvmeof_types.cc [new file with mode: 0644]
src/tools/ceph-dencoder/nvmeof_types.h [new file with mode: 0644]