--- /dev/null
+==============================
+ Block Devices and Nomad
+==============================
+
+Like Kubernetes, Nomad can use Ceph Block Device thanks to `ceph-csi`_,
+which allow to dinamacially provision RBD images or import existing one.
+
+Every nomad version can use `ceph-csi`_, however we'll here describe the
+latest version available at writing time, Nomad v1.1.2 .
+
+To use Ceph Block Devices with Nomad, you must install
+and configure ``ceph-csi`` within your Nomad environment. The following
+diagram depicts the Nomad/Ceph technology stack.
+
+.. ditaa::
+ +-------------------------+-------------------------+
+ | Container | ceph--csi |
+ | | node |
+ | ^ | ^ |
+ | | | | |
+ +----------+--------------+-------------------------+
+ | | | |
+ | v | |
+ | Nomad | |
+ | | |
+ +---------------------------------------------------+
+ | ceph--csi |
+ | controller |
+ +--------+------------------------------------------+
+ | |
+ | configures maps |
+ +---------------+ +----------------+
+ | |
+ v v
+ +------------------------+ +------------------------+
+ | | | rbd--nbd |
+ | Kernel Modules | +------------------------+
+ | | | librbd |
+ +------------------------+-+------------------------+
+ | RADOS Protocol |
+ +------------------------+-+------------------------+
+ | OSDs | | Monitors |
+ +------------------------+ +------------------------+
+
+.. note::
+ Nomad has many task drivers, but we'll only use a Docker container in this example.
+
+.. important::
+ ``ceph-csi`` uses the RBD kernel modules by default which may not support all
+ Ceph `CRUSH tunables`_ or `RBD image features`_.
+
+Create a Pool
+=============
+
+By default, Ceph block devices use the ``rbd`` pool. Create a pool for
+Nopmad persistent storage. Ensure your Ceph cluster is running, then create
+the pool. ::
+
+ $ ceph osd pool create nomad
+
+See `Create a Pool`_ for details on specifying the number of placement groups
+for your pools, and `Placement Groups`_ for details on the number of placement
+groups you should set for your pools.
+
+A newly created pool must be initialized prior to use. Use the ``rbd`` tool
+to initialize the pool::
+
+ $ rbd pool init nomad
+
+Configure ceph-csi
+==================
+
+Setup Ceph Client Authentication
+--------------------------------
+
+Create a new user for nomad and `ceph-csi`. Execute the following and
+record the generated key::
+
+ $ ceph auth get-or-create client.nomad mon 'profile rbd' osd 'profile rbd pool=nomad' mgr 'profile rbd pool=nomad'
+ [client.nomad]
+ key = AQAlh9Rgg2vrDxAARy25T7KHabs6iskSHpAEAQ==
+
+
+Configure Nomad
+---------------
+
+By default Nomad doesn't allow containers to use privileged mode.
+Edit the nomad configuration file by adding this configuration block to `/etc/nomad.d/nomad.hcl`::
+
+ plugin "docker" {
+ config {
+ allow_privileged = true
+ }
+ }
+
+
+Nomad must have `rbd` module loaded, check if it's the case.::
+
+ $ lsmod |grep rbd
+ rbd 94208 2
+ libceph 364544 1 rbd
+
+If it's not the case, load it.::
+
+ $ modprobe rbd
+
+And restart Nomad.
+
+
+
+Create ceph-csi controller and plugin nodes
+===========================================
+
+The `ceph-csi`_ plugin requieres two components:
+
+- **Controller plugin**: Communicates with the provider's API.
+- **Node plugin**: execute tasks on the client.
+
+.. note::
+ We'll set the ceph-csi's version in those files see `ceph-csi release`_ for other versions.
+
+Configure controller plugin
+---------------------------
+
+The controller plugin requires Cpeh monitor addresses of for the Ceph cluster.
+Collect both the Ceph cluster unique `fsid` and the monitor addresses::
+
+ $ ceph mon dump
+ <...>
+ fsid b9127830-b0cc-4e34-aa47-9d1a2e9949a8
+ <...>
+ 0: [v2:192.168.1.1:3300/0,v1:192.168.1.1:6789/0] mon.a
+ 1: [v2:192.168.1.2:3300/0,v1:192.168.1.2:6789/0] mon.b
+ 2: [v2:192.168.1.3:3300/0,v1:192.168.1.3:6789/0] mon.c
+
+Generate a `ceph-csi-plugin-controller.nomad` file similar to the example below, substituting
+the `fsid` for "clusterID", and the monitor addresses for "monitors"::
+
+
+ job "ceph-csi-plugin-controller" {
+ datacenters = ["dc1"]
+ group "controller" {
+ network {
+ port "metrics" {}
+ }
+ task "ceph-controller" {
+ template {
+ data = <<EOF
+ [{
+ "clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
+ "monitors": [
+ "192.168.1.1",
+ "192.168.1.2",
+ "192.168.1.3"
+ ]
+ }]
+ EOF
+ destination = "local/config.json"
+ change_mode = "restart"
+ }
+ driver = "docker"
+ config {
+ image = "quay.io/cephcsi/cephcsi:v3.3.1"
+ volumes = [
+ "./local/config.json:/etc/ceph-csi-config/config.json"
+ ]
+ mounts = [
+ {
+ type = "tmpfs"
+ target = "/tmp/csi/keys"
+ readonly = false
+ tmpfs_options = {
+ size = 1000000 # size in bytes
+ }
+ }
+ ]
+ args = [
+ "--type=rbd",
+ "--controllerserver=true",
+ "--drivername=rbd.csi.ceph.com",
+ "--endpoint=unix://csi/csi.sock",
+ "--nodeid=${node.unique.name}",
+ "--instanceid=${node.unique.name}-controller",
+ "--pidlimit=-1",
+ "--logtostderr=true",
+ "--v=5",
+ "--metricsport=$${NOMAD_PORT_metrics}"
+ ]
+ }
+ resources {
+ cpu = 500
+ memory = 256
+ }
+ service {
+ name = "ceph-csi-controller"
+ port = "metrics"
+ tags = [ "prometheus" ]
+ }
+ csi_plugin {
+ id = "ceph-csi"
+ type = "controller"
+ mount_dir = "/csi"
+ }
+ }
+ }
+ }
+
+Configure plugin node
+---------------------
+Generate a `ceph-csi-plugin-node.nomad` file similar to the example below, substituting
+the `fsid` for "clusterID", and the monitor addresses for "monitors"::
+
+
+ job "ceph-csi-plugin-nodes" {
+ datacenters = ["dc1"]
+ type = "system"
+ group "nodes" {
+ network {
+ port "metrics" {}
+ }
+
+ task "ceph-node" {
+ driver = "docker"
+ template {
+ data = <<EOF
+ [{
+ "clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
+ "monitors": [
+ "192.168.1.1",
+ "192.168.1.2",
+ "192.168.1.3"
+ ]
+ }]
+ EOF
+ destination = "local/config.json"
+ change_mode = "restart"
+ }
+ config {
+ image = "quay.io/cephcsi/cephcsi:v3.3.1"
+ volumes = [
+ "./local/config.json:/etc/ceph-csi-config/config.json"
+ ]
+ mounts = [
+ {
+ type = "tmpfs"
+ target = "/tmp/csi/keys"
+ readonly = false
+ tmpfs_options = {
+ size = 1000000 # size in bytes
+ }
+ }
+ ]
+ args = [
+ "--type=rbd",
+ "--drivername=rbd.csi.ceph.com",
+ "--nodeserver=true",
+ "--endpoint=unix://csi/csi.sock",
+ "--nodeid=${node.unique.name}",
+ "--instanceid=${node.unique.name}-nodes",
+ "--pidlimit=-1",
+ "--logtostderr=true",
+ "--v=5",
+ "--metricsport=$${NOMAD_PORT_metrics}"
+ ]
+ privileged = true
+ }
+ resources {
+ cpu = 500
+ memory = 256
+ }
+ service {
+ name = "ceph-csi-nodes"
+ port = "metrics"
+ tags = [ "prometheus" ]
+ }
+ csi_plugin {
+ id = "ceph-csi"
+ type = "node"
+ mount_dir = "/csi"
+ }
+ }
+ }
+ }
+
+Start plugin controller and node
+--------------------------------
+Run::
+
+ nomad job run ceph-csi-plugin-controller.nomad
+ nomad job run ceph-csi-plugin-nodes.nomad
+
+`ceph-csi`_ image will be downloaded, after few minutes check plugin status::
+
+ $ nomad plugin status ceph-csi
+ ID = ceph-csi
+ Provider = rbd.csi.ceph.com
+ Version = 3.3.1
+ Controllers Healthy = 1
+ Controllers Expected = 1
+ Nodes Healthy = 1
+ Nodes Expected = 1
+
+ Allocations
+ ID Node ID Task Group Version Desired Status Created Modified
+ 23b4db0c a61ef171 nodes 4 run running 3h26m ago 3h25m ago
+ fee74115 a61ef171 controller 6 run running 3h26m ago 3h25m ago
+
+Using Ceph Block Devices
+========================
+
+Create rbd image
+----------------
+
+`ceph-csi` requires the cephx credentials for communicating with the Ceph
+cluster. Generate a `ceph-volume.hcl` file similar to the example below,
+using the newly created nomad user id and cephx key::
+
+ id = "ceph-mysql"
+ name = "ceph-mysql"
+ type = "csi"
+ plugin_id = "ceph-csi"
+ capacity_max = "200G"
+ capacity_min = "100G"
+
+ capability {
+ access_mode = "single-node-writer"
+ attachment_mode = "file-system"
+ }
+
+ secrets {
+ userID = "admin"
+ userKey = "AQAlh9Rgg2vrDxAARy25T7KHabs6iskSHpAEAQ=="
+ }
+
+ parameters {
+ clusterID = "b9127830-b0cc-4e34-aa47-9d1a2e9949a8"
+ pool = "nomad"
+ imageFeatures = "layering"
+ }
+
+Once generated, create the volume::
+
+ $ nomad volume create ceph-volume.hcl
+
+Use rbd image with a container
+------------------------------
+
+As example we'll modify Hashicorp learn `nomad sateful`_ example
+
+Generate a mysql.nomad file similar to the example below.::
+
+ job "mysql-server" {
+ datacenters = ["dc1"]
+ type = "service"
+ group "mysql-server" {
+ count = 1
+ volume "ceph-mysql" {
+ type = "csi"
+ attachment_mode = "file-system"
+ access_mode = "single-node-writer"
+ read_only = false
+ source = "ceph-mysql"
+ }
+ network {
+ port "db" {
+ static = 3306
+ }
+ }
+ restart {
+ attempts = 10
+ interval = "5m"
+ delay = "25s"
+ mode = "delay"
+ }
+ task "mysql-server" {
+ driver = "docker"
+ volume_mount {
+ volume = "ceph-mysql"
+ destination = "/srv"
+ read_only = false
+ }
+ env {
+ MYSQL_ROOT_PASSWORD = "password"
+ }
+ config {
+ image = "hashicorp/mysql-portworx-demo:latest"
+ args = ["--datadir", "/srv/mysql"]
+ ports = ["db"]
+ }
+ resources {
+ cpu = 500
+ memory = 1024
+ }
+ service {
+ name = "mysql-server"
+ port = "db"
+ check {
+ type = "tcp"
+ interval = "10s"
+ timeout = "2s"
+ }
+ }
+ }
+ }
+ }
+
+Start the job::
+
+ $ nomad job run mysql.nomad
+
+Check job's status::
+
+ nomad job status mysql-server
+ ...
+ Status = running
+ ...
+ Allocations
+ ID Node ID Task Group Version Desired Status Created Modified
+ 38070da7 9ad01c63 mysql-server 0 run running 6s ago 3s ago
+
+To check data are actually persistant, you can modify database, purge the job then create it using the same file.
+It will reuse the same RBD image.
+
+.. _ceph-csi: https://github.com/ceph/ceph-csi/
+.. _csi: https://www.nomadproject.io/docs/internals/plugins/csi
+.. _Create a Pool: ../../rados/operations/pools#createpool
+.. _Placement Groups: ../../rados/operations/placement-groups
+.. _CRUSH tunables: ../../rados/operations/crush-map/#tunables
+.. _RBD image features: ../rbd-config-ref/#image-features
+.. _nomad sateful: https://learn.hashicorp.com/tutorials/nomad/stateful-workloads-csi-volumes?in=nomad/stateful-workloads#create-the-job-file
+.. _ceph-csi release: https://github.com/ceph/ceph-csi#ceph-csi-container-images-and-release-compatibility
\ No newline at end of file
projects / ceph.git / commitdiff