一佳互联
虚拟化容器,大数据,DBA,中间件,监控。

基于Kubernetes构建企业Jenkins CI/CD平台

29 12月
作者:admin|分类:容器虚拟化

基于Kubernetes构建企业Jenkins CI/CD平台

基于Kubernetes构建企业Jenkins CI/CD平台

1.蓝绿发布

项目逻辑上分为AB组，在项目升级时，首先把A组从负 载均衡中摘除，进行新版本的部署。

B组仍然继续提供 服务。A组升级完成上线，B组从负载均衡中摘除。

 

特点：

  • 策略简单
  • 升级/回滚速度快
  • 用户无感知，平滑过渡

 

缺点：

  • 需要两倍以上服务器资源
  • 短时间内浪费一定资源成本

image.png

 

2.灰度发布

灰度发布：只升级部分服务，即让一部分用户继续用 老版本，一部分用户开始用新版本，如果用户对新版 本没有什么意见，那么逐步扩大范围，把所有用户都 迁移到新版本上面来。

特点：

  • 保证整体系统稳定性
  • 用户无感知，平滑过渡

 

缺点：

  • 自动化要求高

image.png

 

k8s中的落地方式
image.png

3.滚动发布

滚动发布：

每次只升级一个或多个服务，升级完成 后加入生产环境，不断执行这个过程，直到集群中 的全部旧版升级新版本。 特点：

  • 用户无感知，平滑过渡

缺点：

  • 部署周期长
  • 发布策略较复杂
  • 不易回滚

image.png

 

4.发布流程

image.png

image.png

 

5在Kubernetes中部署Jenkins

image.png

部署文档：https://github.com/jenkinsci/kubernetes-plugin/tree/fc40c869edfd9e3904a9a56b0f80c5a25e988fa1/src/main/kubernetes

 

mkdir k8s-ci/jenkins -p && cd k8s-cli/jenkins

rabc.yml

--- # 创建名为jenkins的ServiceAccount apiVersion: v1 kind: ServiceAccount metadata:  name: jenkins  --- # 创建名为jenkins的Role，授予允许管理API组的资源Pod kind: Role apiVersion: rbac.authorization.k8s.io/v1beta1 metadata:  name: jenkins rules: - apiGroups: [""]  resources: ["pods"]  verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""]  resources: ["pods/exec"]  verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""]  resources: ["pods/log"]  verbs: ["get","list","watch"] - apiGroups: [""]  resources: ["secrets"]  verbs: ["get"]  --- # 将名为jenkins的Role绑定到名为jenkins的ServiceAccount apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata:  name: jenkins roleRef:  apiGroup: rbac.authorization.k8s.io  kind: Role  name: jenkins subjects: - kind: ServiceAccount  name: jenkins

statefulset.yml

apiVersion: apps/v1 kind: StatefulSet metadata:  name: jenkins  labels:  name: jenkins spec:  serviceName: jenkins  replicas: 1  updateStrategy:  type: RollingUpdate  selector:  matchLabels:  name: jenkins  template:  metadata:  name: jenkins  labels:  name: jenkins  spec:  terminationGracePeriodSeconds: 10  serviceAccountName: jenkins  containers:  - name: jenkins  image: jenkins/jenkins:lts-alpine  imagePullPolicy: Always  ports:  - containerPort: 8080  - containerPort: 50000  resources:  limits:  cpu: 1  memory: 1Gi  requests:  cpu: 0.5  memory: 500Mi  env:  - name: LIMITS_MEMORY  valueFrom:  resourceFieldRef:  resource: limits.memory  divisor: 1Mi  - name: JAVA_OPTS  value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MAR GIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 volumeMounts:  - name: jenkins-home  mountPath: /var/jenkins_home  livenessProbe:  httpGet:  path: /login  port: 8080  initialDelaySeconds: 60  timeoutSeconds: 5  failureThreshold: 12  readinessProbe:  httpGet:  path: /login  port: 8080  initialDelaySeconds: 60  timeoutSeconds: 5  failureThreshold: 12  securityContext:  fsGroup: 1000  volumeClaimTemplates:  - metadata:  name: jenkins-home  spec:  storageClassName: "managed-nfs-storage"  accessModes: [ "ReadWriteOnce" ]  resources:  requests:  storage: 1Gi

service.yml

apiVersion: v1 kind: Service metadata:  name: jenkins spec:  selector:  name: jenkins  type: NodePort  ports:  -  name: http  port: 80  targetPort: 8080  protocol: TCP  nodePort: 30006  -  name: agent  port: 50000  protocol: TCP

ingress.yml

apiVersion: extensions/v1beta1 kind: Ingress metadata:  name: jenkins  annotations:  nginx.ingress.kubernetes.io/ssl-redirect: "true"  kubernetes.io/tls-acme: "true"  # 如果上传插件超出默认会报"413 Request Entity Too Large", 增加 client_max_body_size  nginx.ingress.kubernetes.io/proxy-body-size: 50m  nginx.ingress.kubernetes.io/proxy-request-buffering: "off"  # nginx-ingress controller版本小于 0.9.0.beta-18 的配置  ingress.kubernetes.io/ssl-redirect: "true"  ingress.kubernetes.io/proxy-body-size: 50m  ingress.kubernetes.io/proxy-request-buffering: "off" spec:  rules:  - host: jenkins.example.com  http:  paths:  - path: /  backend:  serviceName: jenkins  servicePort: 80

批量完成创建

image.png

这样创建之后 就会通过 sts 控制器在node上 拉起一个jenkins 容器，可以通过node ip:svc port 访问jenkins

插件默认先不安装--》创建个admin 用户---》更换默认插件源 ---》重启jenkins 容器----》安装 git  pipline kubernetes插件

 

 更改配置 使用api 重启jenkins   页面访问 http://192.168.31.65:30006/restart

jenkins master/slave架构

这种架构主要解决 单jenkins执行效率低，资源不足等问题，jenkins master 调度任务到 slave上 并发执行任务。

提升任务执行的效率

image.png

image.png

 

传统添加jenkins 节点是在jenkins页面 jenins nodes 添加

image.png

那么在 k8s 中如何动态创建jenkins slave 节点呢？

jienkins中的Kubernetes插件：Jenkins在Kubernetes集群中运行动态代理 插件介绍：https://github.com/jenkinsci/kubernetes-plugin

 

 

一个 slave节点具体的条件：


1.git

2.镜像构建，推送仓库（docker in docker）

3.slave.jar （jenkins slave）  从jenkins 中获取   http://192.168.31.65:30006/jnlpJars/slave.jar

4.maven  jdk （java 项目）

 

mkdir k8s-ci/jenkins-slave && cd k8s-ci/jenkins-slave

 

jenkins  slave  Dockerfile

FROM centos:7 LABEL maintainer lizhenliang  RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \  yum clean all && \  rm -rf /var/cache/yum/* && \  mkdir -p /usr/share/jenkins  COPY slave.jar /usr/share/jenkins/slave.jar COPY jenkins-slave /usr/bin/jenkins-slave COPY settings.xml /etc/maven/settings.xml RUN chmod +x /usr/bin/jenkins-slave  ENTRYPOINT ["jenkins-slave"]

slave.jarsettings.xml

jenkins-slave       用来设置 这个镜像的  entrypoint  启动jenkins slave

#!/usr/bin/env sh  if [ $# -eq 1 ]; then   # if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image  exec "$@"  else   # if -tunnel is not provided try env vars  case "$@" in  *"-tunnel "*) ;;  *)  if [ ! -z "$JENKINS_TUNNEL" ]; then  TUNNEL="-tunnel $JENKINS_TUNNEL"  fi ;;  esac   # if -workDir is not provided try env vars  if [ ! -z "$JENKINS_AGENT_WORKDIR" ]; then  case "$@" in  *"-workDir"*) echo "Warning: Work directory is defined twice in command-line arguments and the environment variable" ;;  *)  WORKDIR="-workDir $JENKINS_AGENT_WORKDIR" ;;  esac  fi   if [ -n "$JENKINS_URL" ]; then  URL="-url $JENKINS_URL"  fi   if [ -n "$JENKINS_NAME" ]; then  JENKINS_AGENT_NAME="$JENKINS_NAME"  fi   if [ -z "$JNLP_PROTOCOL_OPTS" ]; then  echo "Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior"  JNLP_PROTOCOL_OPTS="-Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true"  fi   # If both required options are defined, do not pass the parameters  OPT_JENKINS_SECRET=""  if [ -n "$JENKINS_SECRET" ]; then  case "$@" in  *"${JENKINS_SECRET}"*) echo "Warning: SECRET is defined twice in command-line arguments and the environment variable" ;;  *)  OPT_JENKINS_SECRET="${JENKINS_SECRET}" ;;  esac  fi   OPT_JENKINS_AGENT_NAME=""  if [ -n "$JENKINS_AGENT_NAME" ]; then  case "$@" in  *"${JENKINS_AGENT_NAME}"*) echo "Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable" ;;  *)  OPT_JENKINS_AGENT_NAME="${JENKINS_AGENT_NAME}" ;;  esac  fi   #TODO: Handle the case when the command-line and Environment variable contain different values.  #It is fine it blows up for now since it should lead to an error anyway.   exec java $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@" fi

 

构建 jenkins slave 镜像 并推送至 私有仓库

 

 docker build -t 192.168.31.70/library/jenkins-slave-jdk:1.8 .  docker push 192.168.31.70/library/jenkins-slave-jdk:1.8

 

测试 job

image.png

配置插件

manage jenkins --》configur system ---》拉到最底下 add cloud 选择kubernetes

image.png

image.png

测试的pipeline 脚本

从私有仓库拉取 jenkins-slave镜像，并起一个pod，完成 打印123

实际过程是，jenkins 会提交一个 yaml文件 给k8s 编排生成一个pod

// 公共 def registry = "192.168.31.70"  podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [  containerTemplate(  name: 'jnlp',  image: "${registry}/library/jenkins-slave-jdk:1.8"  ),  ],  volumes: [  hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),  hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker')  ], ) {  node("jenkins-slave"){  // 第一步  stage('拉取代码'){  echo "123"  }  } }

点击构建 最终会创建出一个 jenkins-slave pod提供 构建功能

image.png

image.png

 

 

6.k8s完整发布流程

添加凭据

添加harbor 凭据

image.png

添加git凭据

image.png

 

安装插件

 

Kubernetes Continuous Deploy插件：

用于将资源配置部署到Kubernetes。

插件介绍：https://plugins.jenkins.io/kubernetes-cd

支持以下资源类型:

  • Deployment
  • Replica Set
  • Daemon Set
  • StatefulSet
  • Pod • Job
  • Service
  • Ingress
  • Secret

生成config文件

 

mkdir admin-cert && cd admin-cert

admin-csr.json

 

{  "CN": "admin",  "hosts": [],  "key": {  "algo": "rsa",  "size": 2048  },  "names": [  {  "C": "CN",  "ST": "BeiJing",  "L": "BeiJing",  "O": "system:masters",  "OU": "System"  }  ] }

ca-config.json

{  "signing": {  "default": {  "expiry": "87600h"  },  "profiles": {  "kubernetes": {  "expiry": "87600h",  "usages": [  "signing",  "key encipherment",  "server auth",  "client auth"  ]  }  }  } }

 

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

 

cp admin*pem /opt/kubernetes/ssl/ export KUBE_APISERVER="https://192.168.31.63:6443" kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=${KUBE_APISE RVER}  kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=${KUBE_APISE RVER}  kubectl config set-credentials admin --client-certificate=/opt/kubernetes/ssl/admin.pem --embed-certs=true --client-key=/opt/kube rnetes/ssl/admin-key.pem  kubectl config set-context kubernetes --cluster=kubernetes --user=admin kubectl config use-context kubernetes

 

image.png

image.png

image.png

既然定义了 部署时使用的yaml  ，那么可以将yaml文件存放在git 仓库中。


image.png

将以下deploy.yaml复制到 页面git中,这次拉群 代码时也会将这个yam 拉取下来。

deploy.yaml

--- apiVersion: apps/v1 kind: Deployment metadata:  name: web spec:  replicas: 3  selector:  matchLabels:  app: java-demo  template:  metadata:  labels:  app: java-demo  spec:  imagePullSecrets:  - name: $SECRET_NAME  containers:  - name: tomcat  image: $IMAGE_NAME  ports:  - containerPort: 8080  name: web  livenessProbe:  httpGet:  path: /  port: 8080  initialDelaySeconds: 60  timeoutSeconds: 5  failureThreshold: 12  readinessProbe:  httpGet:  path: /  port: 8080  initialDelaySeconds: 60  timeoutSeconds: 5  failureThreshold: 12   --- apiVersion: v1 kind: Service metadata:  name: web spec:  type: NodePort  selector:  app: java-demo  ports:  - protocol: TCP  port: 80  targetPort: 8080  --- apiVersion: extensions/v1beta1 kind: Ingress metadata:  name: web spec:  rules:  - host: java.example.com  http:  paths:  - path: /  backend:  serviceName: web  servicePort: 80

 

创建一个登录registry的secret

 

kubectl create secret docker-registry docker-regsitry-auth --docker-username=admin --docker-password=Harbor12345 --docker-server=192.16 8.31.70

 

 

完整发布应用的 pipeline 脚本

Jenkinsfile

// 公共 def registry = "192.168.31.70" // 项目 def project = "welcome" def app_name = "demo" def image_name = "${registry}/${project}/${app_name}:${Branch}-${BUILD_NUMBER}" def git_address = "http://192.168.31.61:9999/root/java-demo.git" // 认证 def secret_name = "docker-regsitry-auth" def docker_registry_auth = "6170ebd8-3b94-409b-ad69-2a4753701041" def git_auth = "3ca1f434-a566-442b-b598-188ac3d07ae2" def k8s_auth = "58cc1edc-4a58-4541-9e74-cbc29c776b9e"  podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [  containerTemplate(  name: 'jnlp',  image: "${registry}/library/jenkins-slave-jdk:1.8"  ),  ],  volumes: [  hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),  hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker')  ], ) {  node("jenkins-slave"){  // 第一步  stage('拉取代码'){  checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"] ]]) }  // 第二步  stage('代码编译'){  sh "mvn clean package -Dmaven.test.skip=true"  }  // 第三步  stage('构建镜像'){  withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')])  {  sh """  echo '  FROM lizhenliang/tomcat  RUN rm -rf /usr/local/tomcat/webapps/*  ADD target/*.war /usr/local/tomcat/webapps/ROOT.war  ' > Dockerfile  docker build -t ${image_name} .  docker login -u ${username} -p '${password}' ${registry}  docker push ${image_name}  """  }  }  // 第四步  stage('部署到K8S平台'){  sh """  sed -i 's#\$IMAGE_NAME#${image_name}#' deploy.yaml  sed -i 's#\$SECRET_NAME#${secret_name}#' deploy.yaml  """  kubernetesDeploy configs: 'deploy.yaml', kubeconfigId: "${k8s_auth}"  }  } }

执行构建等待构建完成

image.png

7.验证部署

验证 svc  和 pod 的部署

image.png

image.png

 

验证ingress

image.png

在本地hosts文件 添加对应的域名记录

image.png

image.png

 

8.总结：

使用Jenkins的插件

  • Git
  • Kubernetes
  • Pipeline
  • Kubernetes Continuous Deploy

 

CI/CD环境特点

  • Slave弹性伸缩
  • 基于镜像隔离构建环境
  • 流水线发布，易维护

 

Jenkins参数化构建

  • 可帮助你完成更复杂环境CI/CD
posted @ 2020-04-29 11:09  benjamin杨  阅读( 2423)  评论( 1)  编辑  收藏  举报
刷新评论刷新页面返回顶部

更多文章推荐

历史上的今天
12月
29
浏览466 评论0
返回
目录 返回
首页
prometheus监控k8s 使用ELK收集k8s集群日志
↑