From 25324164dc92c14037a22cb45bbde42acce2fc9c Mon Sep 17 00:00:00 2001 From: Alexandre Marangone Date: Tue, 24 Oct 2017 13:51:13 -0700 Subject: [PATCH] doc: Kube + Helm installation Signed-off-by: Alexandre Marangone --- doc/index.rst | 1 + doc/start/kube-helm.rst | 344 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 345 insertions(+) create mode 100644 doc/start/kube-helm.rst diff --git a/doc/index.rst b/doc/index.rst index 703f6bb8b194f..2b02e2af7ab28 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -94,6 +94,7 @@ about Ceph, see our `Architecture`_ section. start/intro start/index install/index + start/kube-helm rados/index cephfs/index rbd/index diff --git a/doc/start/kube-helm.rst b/doc/start/kube-helm.rst new file mode 100644 index 0000000000000..829dc82716fc4 --- /dev/null +++ b/doc/start/kube-helm.rst @@ -0,0 +1,344 @@ +================================ +Installation (Kubernetes + Helm) +================================ + +The ceph-helm_ project enables you to deploy Ceph in a Kubernetes environement. +This documentation assumes a Kubernetes environement is available. + +Current limitations +=================== + + - The public and cluster networks must be the same + - If the storage class user id is not admin, you will have to manually create the user + in your Ceph cluster and create its secret in Kubernetes + - ceph-mgr can only run with 1 replica + +Install and start helm +====================== + +Helm can be installed by following these instructions_. + +Once installed, run:: + + $ helm init + $ helm serve & + $ helm repo add local http://localhost:8879/charts + +Add ceph-helm to Helm local repos +================================== +:: + + $ git clone https://github.com/ceph/ceph-helm + $ cd ceph-helm/ceph + $ make + +Configure your Ceph cluster +=========================== + +Create a ``~/ceph-overrides.yaml`` that will contain your Ceph configuration:: + + $ cat ~/ceph-overrides.yaml + network: + public: 172.21.0.0/20 + cluster: 172.21.0.0/20 + + osd_devices: + - name: dev-sdd + device: /dev/sdd + zap: "1" + - name: dev-sde + device: /dev/sde + zap: "1" + + storageclass: + name: ceph-rbd + pool: rbd + user_id: k8s + +.. note:: If journal is not set it will be colocated with device + +.. note:: The ``ceph-helm/ceph/ceph/values.yaml`` file contains the full + list of option that can be set + +Configure RBAC permissions +========================== + +Kubernetes >=v1.6 makes RBAC the default admission controller. ceph-helm provides RBAC roles and permissions for each component:: + + $ kubectl create -f ~/ceph-helm/ceph/rbac.yaml + +The ``rbac.yaml`` file assumes that the Ceph cluster will be deployed in the ``ceph`` namespace. + +Label kubelets +============== + +The following labels need to be set to deploy a Ceph cluster: + - ceph-mon=enabled + - ceph-mgr=enabled + - ceph-osd=enabled + - ceph-osd-device-=enabled + +The ``ceph-osd-device-`` label is created based on the osd_devices name value defined in our ``ceph-overrides.yaml``. +From our example above we will have the two following label: ``ceph-osd-device-dev-sdb`` and ``ceph-osd-device-dev-sdc``. + +For each Ceph Monitor:: + + $ kubectl label node ceph-mon=enabled ceph-mgr=enabled + +For each OSD node:: + + $ kubectl label node ceph-osd=enabled ceph-osd-device-dev-sdb=enabled ceph-osd-device-dev-sdc=enabled + +Ceph Deployment +=============== + +Run the helm install command to deploy Ceph:: + + $ helm install --name=ceph local/ceph --namespace=ceph -f ~/ceph-overrides.yaml + NAME: ceph + LAST DEPLOYED: Wed Oct 18 22:25:06 2017 + NAMESPACE: ceph + STATUS: DEPLOYED + + RESOURCES: + ==> v1/Secret + NAME TYPE DATA AGE + ceph-keystone-user-rgw Opaque 7 1s + + ==> v1/ConfigMap + NAME DATA AGE + ceph-bin-clients 2 1s + ceph-bin 24 1s + ceph-etc 1 1s + ceph-templates 5 1s + + ==> v1/Service + NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE + ceph-mon None 6789/TCP 1s + ceph-rgw 10.101.219.239 8088/TCP 1s + + ==> v1beta1/DaemonSet + NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE + ceph-mon 3 3 0 3 0 ceph-mon=enabled 1s + ceph-osd-dev-sde 3 3 0 3 0 ceph-osd-device-dev-sde=enabled,ceph-osd=enabled 1s + ceph-osd-dev-sdd 3 3 0 3 0 ceph-osd-device-dev-sdd=enabled,ceph-osd=enabled 1s + + ==> v1beta1/Deployment + NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE + ceph-mds 1 1 1 0 1s + ceph-mgr 1 1 1 0 1s + ceph-mon-check 1 1 1 0 1s + ceph-rbd-provisioner 2 2 2 0 1s + ceph-rgw 1 1 1 0 1s + + ==> v1/Job + NAME DESIRED SUCCESSFUL AGE + ceph-mgr-keyring-generator 1 0 1s + ceph-mds-keyring-generator 1 0 1s + ceph-osd-keyring-generator 1 0 1s + ceph-rgw-keyring-generator 1 0 1s + ceph-mon-keyring-generator 1 0 1s + ceph-namespace-client-key-generator 1 0 1s + ceph-storage-keys-generator 1 0 1s + + ==> v1/StorageClass + NAME TYPE + ceph-rbd ceph.com/rbd + +The output from helm install shows us the different types of ressources that will be deployed. + +A StorageClass named ``ceph-rbd`` of type ``ceph.com/rbd`` will be created with ``ceph-rbd-provisioner`` Pods. These +will allow a RBD to be automatically provisioned upon creation of a PVC. RBDs will also be formatted when mapped for the first +time. All RBDs will use the ext4 filesystem. ``ceph.com/rbd`` does not support the ``fsType`` option. +By default, RBDs will use image format 2 and layering. You can overwrite the following storageclass' defaults in your values file:: + + storageclass: + name: ceph-rbd + pool: rbd + user_id: k8s + user_secret_name: pvc-ceph-client-key + image_format: "2" + image_features: layering + +Check that all Pods are running with the command below. This might take a few minutes:: + + $ kubectl -n ceph get pods + NAME READY STATUS RESTARTS AGE + ceph-mds-3804776627-976z9 0/1 Pending 0 1m + ceph-mgr-3367933990-b368c 1/1 Running 0 1m + ceph-mon-check-1818208419-0vkb7 1/1 Running 0 1m + ceph-mon-cppdk 3/3 Running 0 1m + ceph-mon-t4stn 3/3 Running 0 1m + ceph-mon-vqzl0 3/3 Running 0 1m + ceph-osd-dev-sdd-6dphp 1/1 Running 0 1m + ceph-osd-dev-sdd-6w7ng 1/1 Running 0 1m + ceph-osd-dev-sdd-l80vv 1/1 Running 0 1m + ceph-osd-dev-sde-6dq6w 1/1 Running 0 1m + ceph-osd-dev-sde-kqt0r 1/1 Running 0 1m + ceph-osd-dev-sde-lp2pf 1/1 Running 0 1m + ceph-rbd-provisioner-2099367036-4prvt 1/1 Running 0 1m + ceph-rbd-provisioner-2099367036-h9kw7 1/1 Running 0 1m + ceph-rgw-3375847861-4wr74 0/1 Pending 0 1m + +.. note:: The MDS and RGW Pods are pending since we did not label any nodes with + ``ceph-rgw=enabled`` or ``ceph-mds=enabled`` + +Once all Pods are running, check the status of the Ceph cluster from one Mon:: + + $ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- ceph -s + cluster: + id: e8f9da03-c2d2-4ad3-b807-2a13d0775504 + health: HEALTH_OK + + services: + mon: 3 daemons, quorum mira115,mira110,mira109 + mgr: mira109(active) + osd: 6 osds: 6 up, 6 in + + data: + pools: 0 pools, 0 pgs + objects: 0 objects, 0 bytes + usage: 644 MB used, 5555 GB / 5556 GB avail + pgs: + +Configure a Pod to use a PersistentVolume from Ceph +=================================================== + +Create a keyring for the k8s user defined in the ``~/ceph-overwrite.yaml`` and convert +it to base64:: + + $ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- bash + # ceph auth get-or-create-key client.k8s mon 'allow r' osd 'allow rwx pool=rbd' | base64 + QVFCLzdPaFoxeUxCRVJBQUVEVGdHcE9YU3BYMVBSdURHUEU0T0E9PQo= + # exit + +Edit the user secret present in the ``ceph`` namespace:: + + $ kubectl -n ceph edit secrets/pvc-ceph-client-key + +Add the base64 value to the key value with your own and save:: + + apiVersion: v1 + data: + key: QVFCLzdPaFoxeUxCRVJBQUVEVGdHcE9YU3BYMVBSdURHUEU0T0E9PQo= + kind: Secret + metadata: + creationTimestamp: 2017-10-19T17:34:04Z + name: pvc-ceph-client-key + namespace: ceph + resourceVersion: "8665522" + selfLink: /api/v1/namespaces/ceph/secrets/pvc-ceph-client-key + uid: b4085944-b4f3-11e7-add7-002590347682 + type: kubernetes.io/rbd + +We are going to create a Pod that consumes a RBD in the default namespace. +Copy the user secret from the ``ceph`` namespace to ``default``:: + + $ kubectl -n ceph get secrets/pvc-ceph-client-key -o json | jq '.metadata.namespace = "default"' | kubectl create -f - + secret "pvc-ceph-client-key" created + $ kubectl get secrets + NAME TYPE DATA AGE + default-token-r43wl kubernetes.io/service-account-token 3 61d + pvc-ceph-client-key kubernetes.io/rbd 1 20s + +Create and initialize the RBD pool:: + + $ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- ceph osd pool create rbd 256 + pool 'rbd' created + $ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- rbd pool init rbd + +.. important:: Kubernetes uses the RBD kernel module to map RBDs to hosts. Luminous requires + CRUSH_TUNABLES 5 (Jewel). The minimal kernel version for these tunables is 4.5. + If your kernel does not support these tunables, run ``ceph osd crush tunables hammer`` + + +.. important:: Since RBDs are mapped on the host system. Hosts need to be able to resolve + the ceph-mon.ceph.svc.cluster.local name managed by the kube-dns service. To get the + IP address of the kube-dns service, run ``kubectl -n kube-system get svc/kube-dns`` + +Create a PVC:: + + $ cat pvc-rbd.yaml + kind: PersistentVolumeClaim + apiVersion: v1 + metadata: + name: ceph-pvc + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 20Gi + storageClassName: ceph-rbd + + $ kubectl create -f pvc-rbd.yaml + persistentvolumeclaim "ceph-pvc" created + $ kubectl get pvc + NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE + ceph-pvc Bound pvc-1c2ada50-b456-11e7-add7-002590347682 20Gi RWO ceph-rbd 3s + +You can check that the RBD has been created on your cluster:: + + $ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- rbd ls + kubernetes-dynamic-pvc-1c2e9442-b456-11e7-9bd2-2a4159ce3915 + $ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- rbd info kubernetes-dynamic-pvc-1c2e9442-b456-11e7-9bd2-2a4159ce3915 + rbd image 'kubernetes-dynamic-pvc-1c2e9442-b456-11e7-9bd2-2a4159ce3915': + size 20480 MB in 5120 objects + order 22 (4096 kB objects) + block_name_prefix: rbd_data.10762ae8944a + format: 2 + features: layering + flags: + create_timestamp: Wed Oct 18 22:45:59 2017 + +Create a Pod that will use the PVC:: + + $ cat pod-with-rbd.yaml + kind: Pod + apiVersion: v1 + metadata: + name: mypod + spec: + containers: + - name: busybox + image: busybox + command: + - sleep + - "3600" + volumeMounts: + - mountPath: "/mnt/rbd" + name: vol1 + volumes: + - name: vol1 + persistentVolumeClaim: + claimName: ceph-pvc + + $ kubectl create -f pod-with-rbd.yaml + pod "mypod" created + +Check the Pod:: + + $ kubectl get pods + NAME READY STATUS RESTARTS AGE + mypod 1/1 Running 0 17s + $ kubectl exec mypod -- mount | grep rbd + /dev/rbd0 on /mnt/rbd type ext4 (rw,relatime,stripe=1024,data=ordered) + +Logging +======= + +OSDs and Monitor logs can be accessed via the ``kubectl logs [-f]`` command. Monitors have multiple stream of logging, +each stream is accessible from a container running in the ceph-mon Pod. + +There are 3 containers running in the ceph-mon Pod: + - ceph-mon, equivalent of ceph-mon.hostname.log on baremetal + - cluster-audit-log-tailer, equivalent of ceph.audit.log on baremetal + - cluster-log-tailer, equivalent of ceph.log on baremetal or ``ceph -w`` + +Each container is accessible via the ``--container`` or ``-c`` option. +For instance, to access the cluster-tail-log, one can run:: + + $ kubectl -n ceph logs ceph-mon-cppdk -c cluster-log-tailer + +.. _ceph-helm: https://github.com/ceph/ceph-helm/ +.. _instructions: https://github.com/kubernetes/helm/blob/master/docs/install.md -- 2.39.5