From: Etienne Menguy <etienne.menguy@croit.io>
Date: Sat, 3 Jul 2021 18:58:07 +0000 (+0200)
Subject: Initial commit to add rbd and nomad integration documentation
X-Git-Tag: v17.1.0~1122^2~4
X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=849e2bb78202b997b58eeb91dc8e5aa0fa3a2018;p=ceph.git

Initial commit to add rbd and nomad integration documentation

Signed-off-by: Etienne Menguy <etienne.menguy@croit.io>
---

diff --git a/.organizationmap b/.organizationmap
index 0169f117d032..590f38f5de4f 100644
--- a/.organizationmap
+++ b/.organizationmap
@@ -141,6 +141,7 @@ Corvisa LLC <contact@corvisa.com> Walter Huf <hufman@gmail.com>
 Credit Mutuel Arkea <contact@arkea.com> Eric Mourgaya <eric.mourgaya@arkea.com>
 Croit <contact@croit.io> Fabian Bonk <fabian.bonk@croit.io>
 Croit <contact@croit.io> Paul Emmerich <paul.emmerich@croit.io>
+Croit <contact@croit.io> Etienne Menguy <etienne.menguy@croit.io>
 CypressXt Networking <cypressxt.net> Clément Hampaï <clement.hampai@cypressxt.net>
 Datatom <contact@datatom.com> Wang Yong <wang.yong@datatom.com>
 Day By Day <contact@daybyday.nl> Gerben Meijer <gerben@daybyday.nl>
@@ -486,7 +487,6 @@ OpenSUSE <contact@opensuse.org> Richard Brown <RBrownCCB@opensuse.org>
 Opower <contact@opower.com> Derrick Schneider <derrick.schneider@opower.com>
 Oracle <contact@oracle.com> Nikita Gerasimov <nikita.gerasimov@oracle.com>
 OVH <ceph@ovh.net> Bartłomiej Święcki <bartlomiej.swiecki@corp.ovh.com>
-OVH <ceph@ovh.net> Etienne Menguy <etienne.menguy@corp.ovh.com>
 OVH <ceph@ovh.net> Paweł Sadowski <pawel.sadowski@corp.ovh.com>
 OVH <ceph@ovh.net> Piotr Dałek <piotr.dalek@corp.ovh.com>
 Pacific Northwest National Laboratory <contact@pnl.gov> Brown, David M JR <david.brown@pnl.gov>
diff --git a/doc/rbd/rbd-nomad.rst b/doc/rbd/rbd-nomad.rst
new file mode 100644
index 000000000000..a5cf4e245cf3
--- /dev/null
+++ b/doc/rbd/rbd-nomad.rst
@@ -0,0 +1,431 @@
+==============================
+ Block Devices and Nomad
+==============================
+
+Like Kubernetes, Nomad can use Ceph Block Device thanks to `ceph-csi`_, 
+which allow to dinamacially provision RBD images or import existing one.
+
+Every nomad version can use `ceph-csi`_, however we'll here describe the
+latest version available at writing time, Nomad v1.1.2 .
+
+To use Ceph Block Devices with Nomad, you must install
+and configure ``ceph-csi`` within your Nomad environment. The following
+diagram depicts the Nomad/Ceph technology stack.
+
+.. ditaa::
+            +-------------------------+-------------------------+
+            |      Container          |          ceph--csi      |
+            |                         |            node         |
+            |          ^              |                 ^       |
+            |          |              |                 |       |
+            +----------+--------------+-------------------------+
+            |          |                                |       |
+            |          v                                |       |
+            |                       Nomad               |       |
+            |                                           |       |
+            +---------------------------------------------------+
+            |                       ceph--csi                   |
+            |                       controller                  |
+            +--------+------------------------------------------+
+                     |                                  |
+                     | configures       maps            |
+                     +---------------+ +----------------+
+                                     | | 
+                                     v v
+            +------------------------+ +------------------------+
+            |                        | |        rbd--nbd        |
+            |     Kernel Modules     | +------------------------+
+            |                        | |         librbd         |
+            +------------------------+-+------------------------+
+            |                   RADOS Protocol                  |
+            +------------------------+-+------------------------+
+            |          OSDs          | |        Monitors        |
+            +------------------------+ +------------------------+
+
+.. note::
+    Nomad has many task drivers, but we'll only use a Docker container in this example.
+
+.. important::
+   ``ceph-csi`` uses the RBD kernel modules by default which may not support all
+   Ceph `CRUSH tunables`_ or `RBD image features`_.
+
+Create a Pool
+=============
+
+By default, Ceph block devices use the ``rbd`` pool. Create a pool for
+Nopmad persistent storage. Ensure your Ceph cluster is running, then create
+the pool. ::
+
+        $ ceph osd pool create nomad
+
+See `Create a Pool`_ for details on specifying the number of placement groups
+for your pools, and `Placement Groups`_ for details on the number of placement
+groups you should set for your pools.
+
+A newly created pool must be initialized prior to use. Use the ``rbd`` tool
+to initialize the pool::
+
+        $ rbd pool init nomad
+
+Configure ceph-csi
+==================
+
+Setup Ceph Client Authentication
+--------------------------------
+
+Create a new user for nomad and `ceph-csi`. Execute the following and
+record the generated key::
+
+    $ ceph auth get-or-create client.nomad mon 'profile rbd' osd 'profile rbd pool=nomad' mgr 'profile rbd pool=nomad'
+    [client.nomad]
+        key = AQAlh9Rgg2vrDxAARy25T7KHabs6iskSHpAEAQ==
+
+
+Configure Nomad  
+---------------
+
+By default Nomad doesn't allow containers to use privileged mode.
+Edit the nomad configuration file by adding this configuration block to `/etc/nomad.d/nomad.hcl`::
+
+    plugin "docker" {
+        config {
+        allow_privileged = true
+        }
+    }
+
+
+Nomad must have `rbd` module loaded, check if it's the case.::
+
+        $ lsmod |grep rbd
+        rbd                    94208  2
+        libceph               364544  1 rbd
+
+If it's not the case, load it.::
+
+        $ modprobe rbd
+
+And restart Nomad.
+
+
+
+Create ceph-csi controller and plugin nodes
+===========================================
+
+The `ceph-csi`_ plugin requieres two components:
+
+- **Controller plugin**: Communicates with the provider's API.
+- **Node plugin**: execute tasks on the client.
+
+.. note::
+    We'll set the ceph-csi's version in those files see `ceph-csi release`_ for other versions.
+
+Configure controller plugin
+---------------------------
+
+The controller plugin requires Cpeh monitor addresses of for the Ceph cluster.
+Collect both the Ceph cluster unique `fsid` and the monitor addresses::
+
+        $ ceph mon dump
+        <...>
+        fsid b9127830-b0cc-4e34-aa47-9d1a2e9949a8
+        <...>
+        0: [v2:192.168.1.1:3300/0,v1:192.168.1.1:6789/0] mon.a
+        1: [v2:192.168.1.2:3300/0,v1:192.168.1.2:6789/0] mon.b
+        2: [v2:192.168.1.3:3300/0,v1:192.168.1.3:6789/0] mon.c
+
+Generate a `ceph-csi-plugin-controller.nomad` file similar to the example below, substituting
+the `fsid` for "clusterID", and the monitor addresses for "monitors"::
+
+
+        job "ceph-csi-plugin-controller" {
+          datacenters = ["dc1"]
+        group "controller" {
+            network {
+              port "metrics" {}
+            }
+            task "ceph-controller" {
+        template {
+                data        = <<EOF
+        [{
+            "clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
+            "monitors": [
+                "192.168.1.1",
+          "192.168.1.2",
+          "192.168.1.3"
+            ]
+        }]
+        EOF
+                destination = "local/config.json"
+                change_mode = "restart"
+              }
+              driver = "docker"
+              config {
+                image = "quay.io/cephcsi/cephcsi:v3.3.1"
+                volumes = [
+                  "./local/config.json:/etc/ceph-csi-config/config.json"
+                ]
+                mounts = [
+                  {
+                    type     = "tmpfs"
+                    target   = "/tmp/csi/keys"
+                    readonly = false
+                    tmpfs_options = {
+                      size = 1000000 # size in bytes
+                    }
+                  }
+                ]
+                args = [
+                  "--type=rbd",
+                  "--controllerserver=true",
+                  "--drivername=rbd.csi.ceph.com",
+                  "--endpoint=unix://csi/csi.sock",
+                  "--nodeid=${node.unique.name}",
+            "--instanceid=${node.unique.name}-controller",
+                  "--pidlimit=-1",
+            "--logtostderr=true",
+                  "--v=5",
+                  "--metricsport=$${NOMAD_PORT_metrics}"
+                ]
+              }
+           resources {
+                cpu    = 500
+                memory = 256
+              }
+              service {
+                name = "ceph-csi-controller"
+                port = "metrics"
+                tags = [ "prometheus" ]
+              }
+        csi_plugin {
+                id        = "ceph-csi"
+                type      = "controller"
+                mount_dir = "/csi"
+              }
+            }
+          }
+        }
+
+Configure plugin node
+---------------------
+Generate a `ceph-csi-plugin-node.nomad` file similar to the example below, substituting
+the `fsid` for "clusterID", and the monitor addresses for "monitors"::
+
+
+        job "ceph-csi-plugin-nodes" {
+          datacenters = ["dc1"]
+          type        = "system"
+          group "nodes" {
+            network {
+              port "metrics" {}
+            }
+        
+            task "ceph-node" {
+              driver = "docker"
+              template {
+                data        = <<EOF
+        [{
+            "clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
+            "monitors": [
+                "192.168.1.1",
+          "192.168.1.2",
+          "192.168.1.3"
+            ]
+        }]
+        EOF
+                destination = "local/config.json"
+                change_mode = "restart"
+              }
+              config {
+                image = "quay.io/cephcsi/cephcsi:v3.3.1"
+                volumes = [
+                  "./local/config.json:/etc/ceph-csi-config/config.json"
+                ]
+                mounts = [
+                  {
+                    type     = "tmpfs"
+                    target   = "/tmp/csi/keys"
+                    readonly = false
+                    tmpfs_options = {
+                      size = 1000000 # size in bytes
+                    }
+                  }
+                ]
+                args = [
+                  "--type=rbd",
+                  "--drivername=rbd.csi.ceph.com",
+                  "--nodeserver=true",
+                  "--endpoint=unix://csi/csi.sock",
+                  "--nodeid=${node.unique.name}",
+                  "--instanceid=${node.unique.name}-nodes",
+                  "--pidlimit=-1",
+            "--logtostderr=true",
+                  "--v=5",
+                  "--metricsport=$${NOMAD_PORT_metrics}"
+                ]
+                privileged = true
+              }
+           resources {
+                cpu    = 500
+                memory = 256
+              }
+              service {
+                name = "ceph-csi-nodes"
+                port = "metrics"
+                tags = [ "prometheus" ]
+              }
+        csi_plugin {
+                id        = "ceph-csi"
+                type      = "node"
+                mount_dir = "/csi"
+              }
+            }
+          }
+        }
+
+Start plugin controller and node
+--------------------------------
+Run::
+
+        nomad job run ceph-csi-plugin-controller.nomad
+        nomad job run ceph-csi-plugin-nodes.nomad
+
+`ceph-csi`_ image will be downloaded, after few minutes check plugin status::
+
+        $ nomad plugin status ceph-csi
+        ID                   = ceph-csi
+        Provider             = rbd.csi.ceph.com
+        Version              = 3.3.1
+        Controllers Healthy  = 1
+        Controllers Expected = 1
+        Nodes Healthy        = 1
+        Nodes Expected       = 1
+
+        Allocations
+        ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
+        23b4db0c  a61ef171  nodes       4        run      running  3h26m ago  3h25m ago
+        fee74115  a61ef171  controller  6        run      running  3h26m ago  3h25m ago
+
+Using Ceph Block Devices
+========================
+
+Create rbd image
+----------------
+
+`ceph-csi` requires the cephx credentials for communicating with the Ceph
+cluster. Generate a `ceph-volume.hcl` file similar to the example below,
+using the newly created nomad user id and cephx key::
+
+        id = "ceph-mysql"
+        name = "ceph-mysql"
+        type = "csi"
+        plugin_id = "ceph-csi"
+        capacity_max = "200G"
+        capacity_min = "100G"
+
+        capability {
+          access_mode     = "single-node-writer"
+          attachment_mode = "file-system"
+        }
+
+        secrets {
+          userID  = "admin"
+          userKey = "AQAlh9Rgg2vrDxAARy25T7KHabs6iskSHpAEAQ=="
+        }
+
+        parameters {
+          clusterID = "b9127830-b0cc-4e34-aa47-9d1a2e9949a8"
+          pool = "nomad"
+          imageFeatures = "layering"
+        }
+
+Once generated, create the volume::
+
+        $ nomad volume create ceph-volume.hcl
+
+Use rbd image with a container
+------------------------------
+
+As example we'll modify Hashicorp learn `nomad sateful`_ example 
+
+Generate a mysql.nomad file similar to the example below.::
+
+        job "mysql-server" {
+          datacenters = ["dc1"]
+          type        = "service"
+          group "mysql-server" {
+            count = 1
+            volume "ceph-mysql" {
+              type      = "csi"
+                attachment_mode = "file-system"
+                access_mode     = "single-node-writer"
+              read_only = false
+              source    = "ceph-mysql"
+            }
+            network {
+              port "db" {
+                static = 3306
+              }
+            }
+            restart {
+              attempts = 10
+              interval = "5m"
+              delay    = "25s"
+              mode     = "delay"
+            }
+            task "mysql-server" {
+              driver = "docker"
+              volume_mount {
+                volume      = "ceph-mysql"
+                destination = "/srv"
+                read_only   = false
+              }
+              env {
+                MYSQL_ROOT_PASSWORD = "password"
+              }
+              config {
+                image = "hashicorp/mysql-portworx-demo:latest"
+                args  = ["--datadir", "/srv/mysql"]
+                ports = ["db"]
+              }
+              resources {
+                cpu    = 500
+                memory = 1024
+              }
+              service {
+                name = "mysql-server"
+                port = "db"
+                check {
+                  type     = "tcp"
+                  interval = "10s"
+                  timeout  = "2s"
+                }
+              }
+            }
+          }
+        }
+
+Start the job::
+
+        $ nomad job run mysql.nomad
+
+Check job's status::
+
+        nomad job status mysql-server
+        ...
+        Status        = running
+        ...
+        Allocations
+        ID        Node ID   Task Group    Version  Desired  Status   Created  Modified
+        38070da7  9ad01c63  mysql-server  0        run      running  6s ago   3s ago
+
+To check data are actually persistant, you can modify database, purge the job then create it using the same file.
+It will reuse the same RBD image.
+
+.. _ceph-csi: https://github.com/ceph/ceph-csi/
+.. _csi: https://www.nomadproject.io/docs/internals/plugins/csi
+.. _Create a Pool: ../../rados/operations/pools#createpool
+.. _Placement Groups: ../../rados/operations/placement-groups
+.. _CRUSH tunables: ../../rados/operations/crush-map/#tunables
+.. _RBD image features: ../rbd-config-ref/#image-features
+.. _nomad sateful: https://learn.hashicorp.com/tutorials/nomad/stateful-workloads-csi-volumes?in=nomad/stateful-workloads#create-the-job-file
+.. _ceph-csi release: https://github.com/ceph/ceph-csi#ceph-csi-container-images-and-release-compatibility
\ No newline at end of file