一佳互联
虚拟化容器,大数据,DBA,中间件,监控。

Kubernets对接kadalu

22 11月
作者:admin|分类:容器虚拟化

介绍

Kadalu 是一个在容器生态系统（如 kubernetes、openshift、RKE 等）中提供持久存储的项目，基于csi实现。github.com/kadalu/kadal

部署

可以通过两种方式来进行部署，即二进制部署和yaml部署。下面分别进行介绍。

二进制部署

部署在Kubernetes上：

#curl -LO https://github.com/kadalu/kadalu/releases/latest/download/kubectl-kadalu
#chmod +x ./kubectl-kadalu
#sudo mv ./kubectl-kadalu /usr/local/bin/kubectl-kadalu

部署在Openshift上：

curl -LO https://github.com/kadalu/kadalu/releases/latest/download/kubectl-kadalu
chmod +x ./kubectl-kadalu
sudo mv ./kubectl-kadalu /usr/local/bin/oc-kadalu

安装Operator：

#kubectl-kadalu version
#kubectl-kadalu install

yaml 部署

kubectl apply -f https://github.com/kadalu/kadalu/releases/latest/download/kadalu-operator.yaml

检查部署结果

kubectl get pods -n kadalu

对接external gluster cluster

  • 部署storageclass(external-config.yaml)，执行： kubectl create -f external-config.yaml
---
apiVersion: kadalu-operator.storage/v1alpha1
kind: KadaluStorage
metadata:
  name: ext-config
spec:
  type: External
  kadalu_format: native
  details:
    gluster_host: 10.9.9.149 #对应gluster集群host
    gluster_volname: vol1 #对应gluster volume
    gluster_options: log-level=DEBUG


执行：kubect get sc，查看部署的storageclass

  • 部署pvc (gluster_pvc.yaml)，执行： kubectl create -f gluster_pvc.yaml
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pv-ext-kadalu-1
spec:
  storageClassName: kadalu.ext-config
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  • 部署pod(pod.yaml)
apiVersion: v1
kind: Pod
metadata:
  name: web-server-1
spec:
  containers:
   - name: web-server
     image: nginx
     volumeMounts:
       - mountPath: /var/lib/www/html
         name: mypvc
  volumes:
   - name: mypvc
     persistentVolumeClaim:
       claimName: pv-ext-kadalu-1
       readOnly: false

External gluster设置quota(必须enable gluster volume的quota)

  • 拷贝gluster cluster节点上的私钥到k8s master节点的任意目录
scp root@<ip>:/root/.ssh/id.ras /tmp/id.ras
  • 创建secret
kubectl create secret generic glusterquota-ssh-secret --from-literal=glusterquota-ssh-username=<username> --from-file=ssh-privatekey=<ssh_privatekey_path> -n kadalu
  • 重启operator，使生效

对接kadalu storages

  • 部署internal gluster cluster
kubectl kadalu storage-add storage-pool --type=Replica1 --device k8s-1.xxx:/dev/vdb  # k8s-1.xxx为主机名
kubect get pods -n kadalu
kubectl get kadalustorage
  • 检查storage-pool是否能通
kubectl -n kadalu exec -it sts/kadalu-csi-provisioner -c kadalu-logging -- sh -c 'ping -c 5 server-storage-pool-0-0.storage-pool; nc -zv server-storage-pool-0-0.storage-pool 24007'
  • 创建pvc
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pv3
spec:
  storageClassName: kadalu.storage-pool
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

kadalu VS heketi

kadalu实现原理

待补充

参考文档

  1. 设置quota：github.com/kadalu/kadal
  2. 对接external gluster storage：kadalu.io/docs/k8s-stor
  3. 部署kadalu storage：github.com/kadalu/kadal
  4. 为什么kadalu用python实现，而不用golang：github.com/kadalu/kadal
  5. kadalu对接internal gluster问题总结：
    1. github.com/kadalu/kadal
    2. cloud.tencent.com/devel
  6. thoughtexpo.com/explori kadalu使用

部署yaml

---
kind: Namespace
apiVersion: v1
metadata:
  name: kadalu
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kadalu-operator
  namespace: kadalu
---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: kadalu-csi-nodeplugin
  namespace: kadalu
---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: kadalu-csi-provisioner
  namespace: kadalu
---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: kadalu-server-sa
  namespace: kadalu


---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: kadalustorages.kadalu-operator.storage
  namespace: kadalu
spec:
  group: kadalu-operator.storage
  names:
    kind: KadaluStorage
    listKind: KadaluStorageList
    plural: kadalustorages
    singular: kadalustorage
  scope: Namespaced
  versions:
    - name: v1alpha1
      storage: true
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                disperse:
                  type: object
                  properties:
                    data:
                      type: integer
                    redundancy:
                      type: integer
                type:
                  type: string
                pvReclaimPolicy:
                  type: string
                  default: delete
                volume_id:
                  type: string
                kadalu_format:
                  type: string
                  default: native
                storage:
                  type: array
                  items:
                    type: object
                    properties:
                      node:
                        type: string
                      device:
                        type: string
                      path:
                        type: string
                      pvc:
                        type: string
                      decommissioned:
                        type: string
                tiebreaker:
                  type: object
                  properties:
                    deployment:
                      type: string
                    node:
                      type: string
                    path:
                      type: string
                    port:
                      type: integer
                details:
                  type: object
                  properties:
                    gluster_host:
                      type: string
                    gluster_hosts:
                      type: array
                      items:
                        type: string
                    gluster_volname:
                      type: string
                    gluster_options:
                      type: string

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: kadalu
  name: pod-exec
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["create", "get", "list", "watch", "update", "delete", "patch"]
- apiGroups: [""]
  resources: ["pods", "pods/exec"]
  verbs: ["create", "get", "list", "watch", "update", "delete", "patch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kadalu-operator
  namespace: kadalu
rules:
  - apiGroups: [""]
    resources:
      - pods
      - services
      - endpoints
      - persistentvolumes
      - persistentvolumeclaims
      - persistentvolumeclaims/status
      - customresourcedefinitions
      - events
      - configmaps
      - secrets
      - serviceaccounts
      - clusterroles
      - clusterrolebindings
      - roles
      - rolebindings
    verbs: ["create", "get", "list", "watch", "update", "delete", "patch"]
  - apiGroups:
      - storage.k8s.io
    resources:
      - storageclasses
      - volumeattachments
      - volumeattachments/status
    verbs: ["create", "get", "list", "watch", "update", "delete", "patch"]
  - apiGroups:
      - rbac.authorization.k8s.io
    resources:
      - roles
      - rolebindings
      - clusterroles
      - clusterrolebindings
    verbs: ["create", "get", "list", "watch", "update", "delete", "patch"]
  - apiGroups: ["apps"]
    resources:
      - deployments
      - daemonsets
      - replicasets
      - statefulsets
    verbs: ["create", "get", "list", "watch", "update", "delete", "patch"]
  - apiGroups: ["monitoring.coreos.com"]
    resources:
      - servicemonitors
    verbs: ["get", "create"]
  - apiGroups: ["kadalu-operator.storage"]
    resources:
      - kadalustorages
    verbs: ["create", "get", "list", "watch", "update", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["csi.storage.k8s.io"]
    resources: ["csinodeinfos"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims/status"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch", "delete", "get"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["create", "get", "list", "watch", "update", "delete"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["create", "list", "watch", "delete"]
  - apiGroups: ["csi.storage.k8s.io"]
    resources: ["csidrivers"]
    verbs: ["get", "create", "delete", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csidrivers"]
    verbs: ["get", "create", "delete", "patch"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "watch", "list", "delete", "update", "create"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "get", "list", "watch", "update", "delete", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kadalu-operator
subjects:
  - kind: ServiceAccount
    name: kadalu-operator
    namespace: kadalu
  - kind: ServiceAccount
    name: pod-exec
    namespace: kadalu
roleRef:
  kind: ClusterRole
  name: kadalu-operator
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: operator
  namespace: kadalu
  labels:
    app.kubernetes.io/part-of: kadalu
    app.kubernetes.io/component: operator
    app.kubernetes.io/name: operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: kadalu
  template:
    metadata:
      labels:
        name: kadalu
    spec:
      serviceAccountName: kadalu-operator
      containers:
        - name: kadalu-operator
          securityContext:
            capabilities: {}
            privileged: true
          image: docker.io/kadalu/kadalu-operator:0.8.4
          imagePullPolicy: IfNotPresent
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "kadalu-operator"
            - name: DOCKER_USER
              value: "kadalu"
            - name: KADALU_VERSION
              value: "0.8.4"
            - name: KADALU_NAMESPACE
              value: "kadalu"
            - name: KUBELET_DIR
              value: "/var/lib/kubelet"
            - name: K8S_DIST
              value: "kubernetes"
            - name: VERBOSE
              value: "no"
            - name: DEPLOY_SOCAT
              value: "no"

更多文章推荐

历史上的今天
11月
22
浏览906 评论0
返回
目录 返回
首页
GlusterFS分布式存储集群部署记录-相关补充 GlusterFS分布式存储数据的恢复机制(AFR)的说明
↑