In this post I will show you how can you use CEPH RBD with CSI driver for persistent storagi on Kubernetes.

Parst of the Kubernetes series

Parst of the Kubernetes series

• Part1a: Install K8S with ansible
• Part1b: Install K8S with kubeadm
• Part1c: Install K8S with containerd and kubeadm
• Part1d: Install K8S with kubeadm in HA mode
• Part2: Intall metal-lb with K8S
• Part2: Intall metal-lb with BGP
• Part3: Install Nginx ingress to K8S
• Part4: Install cert-manager to K8S
• Part5a: Use local persisten volume with K8S
• Part5b: Use ceph persisten volume with K8S
• Part5c: Use ceph CSI persisten volume with K8S
• Part5d: Use Project Longhorn as persisten volume with K8S
• Part5e: Use OpenEBS as persisten volume with K8S
• Part5f: vSphere persistent storage for K8S
• Part6a: Install k8s with IPVS mode
• Part6b: Install k8s with IPVS mode
• Part7: Use Helm with K8S
• Part8: Tillerless helm2 install
• Part9: Kubernetes Dashboard SSO
• Part10: Kuberos for K8S
• Part11: Gangway for K8S
• Part12: Velero Backup for K8S

The Container Storage Interface (CSI) is a standard for exposing arbitrary block and file storage storage systems to Kubernetes. Using CSI third-party storage providers can write and deploy plugins exposing storage systems in Kubernetes. Bbefore we begin lets ensure that we have the following requirements:

• Kubernetes cluster v1.14+
• allow-privileged flag enabled for both kubelet and API server
• Running Ceph cluster

git clone https://github.com/ceph/ceph-csi.git
cd ceph-csi/deploy/rbd/kubernetes/v1.14+/

kubectl create -f csi-nodeplugin-rbac.yaml
kubectl create -f csi-provisioner-rbac.yaml

nano csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "k8s-ceph",
        "monitors": [
          "192.168.1.31:6789",
          "192.168.1.32:6789",
          "192.168.1.33:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config


kubectl create -fcsi-config-map.yaml

kubectl create -f csi-rbdplugin-provisioner.yaml
kubectl create -f csi-rbdplugin.yaml

ceph auth get-key client.admin|base64
QVFDTDliVmNEb21I32SHoPxXNGhmRkczTFNtcXM0ZW5VaXlTZEE977==

nano csi-rbd-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: default
data:
  userID: admin
  userKey: QVFDTDliVmNEb21I32SHoPxXNGhmRkczTFNtcXM0ZW5VaXlTZEE977==

nano rbd-csi-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd
provisioner: rbd.csi.ceph.com
parameters:
   monitors: 192.168.1.31:6790,192.168.1.32:6790,192.168.1.33:6790
   clusterID: k8s-ceph
   pool: rbd
   imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: default
   csi.storage.k8s.io/node-publish-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-publish-secret-namespace: default
   adminid: admin
   csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
mountOptions:
   - discard

kubectl create -f csi-rbd-secret.yaml
kubectl create -f rbd-csi-sc.yaml

kubectl get storageclass
NAME      PROVISIONER        AGE
csi-rbd   rbd.csi.ceph.com   15s

nano raw-block-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd

kubectl create -f raw-block-pvc.yaml

kubectl get pvc
NAME            STATUS    VOLUME                                  
raw-block-pvc   Bound     pvc-fd66b4d6-757d-22e9-8f9e-4f86e2356a59