K8S-Devops实操
- Kubernetes
- 2024-12-27
- 20892热度
- 0评论
K8S-Devops实操
Jenkins 凭证 Credentials
Harbor 的账号密码、GitLab 的私钥、Kubernetes 的证书均使用 Jenkins 的 Credentials 管理
配置 Kubernetes 证书
找到集群中的 KUBECONFIG，一般是 kubectl 节点的~/.kube/config 文件，或者是 KUBECONFIG 环境变量所指向的文件（/etc/kubernetes/admin.conf）
接着证书文件放置于 Jenkins 的 Credentials 中即可，登录Jenkins界面，依次单击左侧导航栏的Dashboard→Manage Jenkins→Credentials按钮，随后出现Credentials和Stores scoped to jenkins两个导航栏，单击（global）按钮的下拉箭头，单击Add credentials添加凭证
在弹出的 New credentials界面，修改 Kind 为 Secret file，单击浏览按钮上传集群的认证文件，ID设置为jxcai_kubernetes，Description字段按需填写即可，最后单击创建按钮
配置 Harbor 证书
创建完成后，单击右上方的 Add Credentials 按钮，继续添加 Harbor仓库凭证，修改 Kind 为 Username with password ，填写 Harbor的账号和密码，ID 设置为 Harbor_Account，Description字段按需填写即可，最后单击创建按钮
配置 Gitlab 证书
继续单击右上方的 Add Credentials 按钮，继续添加 Gitlab 仓库凭证，修改 Kind 为 SSH Username with private key ，ID 设置为 Gitlab_Account，Description字段按需填写即可，填写登录的用户名（非强制），勾选 Private Key下面的 Enter directly 选项，单击Add 按钮添加Jenkins服务器的私钥，最后单击创建按钮
创建完成后，界面如下
配置Agent
通常情况下，Jenkins Slave 会通过 Jenkins Master 节点的 50000 端口与之通信，所以需要开启 Agent 的 50000 端口，在配置完成后，该配置默认已经放行 50000 端口
登录 Jenkins UI界面，依次单击导航栏 Dashboard→Manage Jenkins→Security按钮，下拉至 Agents配置项，检查 Fixed配置项是否为 50000
Jenkins 配置 Kubernetes 多集群
登录 Jenkins UI界面，依次单击导航栏 Dashboard→Manage Jenkins→Clouds→New cloud按钮，添加 Cloud name名称为jxcai_kubernetes，并勾选 Type为 Kubernetes，最后单击 Create 按钮
随后需要添加凭据，选择在上一步创建的 Kubernetes集群凭证，选择完成后单击连接测试按钮，测试成功后，界面会提示集群的版本，最后单击 Save按钮，添加完 Kubernetes 后，在 Jenkinsfile 的 Agent 中，就可以选择该集群作为创建 Slave 的集群。如果想要添加多个集群，重复上述的步骤即可。首先添加 Kubernetes 凭证，然后添加 Cloud即可。
自动化构建 Java 应用
将Java项目导入到Gitlab内，打开 Gitlab UI界面，选择在环境搭建时创建的组，进入到Project 内，依次单击New Project →Import project按钮，但是此时会提示
No import options available. Contact an administrator to enable options for importing your project.（没有导入选项，请联系管理员开启选项）这是因为在新版本的 Gitlab 安装后，默认不打开 Import选项，此时找到 Settings选项。单击 General 按钮（注意：要在admin用户下），找到 Import and export settings选项，勾选要导入的源即可，随后单击 Save changes 按钮即可生效
切换回之前创建的组，或者直接找到 Projects 选项，依次单击右上方的 New project→Import project按钮，即可看到导入选项
单击Gitee按钮，会提示需要Token才能进行访问
此时需要登录Gitee仓库，单击右上方的个人头像，点击设置按钮，在右侧导航栏单击私人令牌选项，生成新令牌，勾选令牌拥有的权限和描述，单击提交即可。将生成的令牌保存好，并记录下来
切换回 Gitlab UI界面内，填写生成的令牌密码，单击 Authenticate 按钮即可，随后即可看到在个人仓库内的 Project，选择对应的 Porject，单击 Import按钮即可导入
导入后，界面如下
定义 Jenkinsfile
在 Project 的源代码中添加 Jenkinsfile。首先点击代码首页的 "+"号，然后点击 New file
在弹出的窗口内，填写文件名称（Jenkinsfile），并填写如下内容
注意：此处先对Jenkinsfile文件进行解析，完整配置放在解析之后（解析的配置文件并非完全契合实验环境，在解析后面的配置文件才是完整的）
首先是顶层的 Agent，定义的是 Kubernetes 的 Pod 作为 Jenkins 的 Slave。注意：jnlp非常容器报错，比如镜像版本太旧，或者是JDK版本有问题，都会报错
java.io.IOException: Remote call on JNLP4-connect connection from 192.168.132.172/192.168.132.172:62529 failedSlaveVersion has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 52.0agent {
# 定义使用 Kubernetes 作为 agent
kubernetes {
# 选择的云为之前配置的名字
cloud 'jxcai_kubernetes'
slaveConnectTimeout 1200
# 将 workspace 改成 hostPath，因为该 Slave 会固定节点创建，如果有存储可用，可以改成 PVC 的模式
workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)
yaml '''
apiVersion: v1
kind: Pod
spec:
containers:
# jnlp 容器，和 Jenkins 主节点通信
- args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)']
image: 'registry.cn-beijing.aliyuncs.com/dotbalo/jnlp-agent-docker:latest'
name: jnlp
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
# build 容器，包含执行构建的命令，比如 Java 的需要 mvn 构建，就可以用一个 maven 的镜像
- command: ["cat"]
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/maven:3.5.3"
imagePullPolicy: "IfNotPresent"
name: "build"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
# Pod 单独创建了一个缓存的 volume，将其挂载到了 maven 插件的缓存目录，默认是 /root/.m2
- mountPath: "/root/.m2/"
name: "cachedir"
readOnly: false
# 发版容器，因为最终是发版至 Kubernetes 的，所以需要有一个 kubectl 命令
- command: ["cat"]
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
imagePullPolicy: IfNotPresent
name: "kubectl"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
# 用于生成镜像的容器，需要包含 docker 命令
- command: ["cat"]
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"
imagePullPolicy: "IfNotPresent"
name: "docker"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
# 由于容器没有启动 docker 服务，所以将宿主机的 docker 经常挂载至容器即可
- mountPath: "/var/run/docker.sock"
name: "dockersock"
readOnly: false
restartPolicy: "Never"
# 固定节点部署
nodeSelector:
build: "true"
securityContext: {}
volumes:
# Docker socket volume
- hostPath:
path: "/var/run/docker.sock"
name: "dockersock"
- hostPath:
path: "/usr/share/zoneinfo/Asia/Shanghai"
name: "localtime"
# 缓存目录
- name: "cachedir"
hostPath:
path: "/opt/m2"
'''
}
}Jenkinsfile 环境变量和 parameters 的配置，请读者注意，TAG和COMMIT_ID变量留空的原因是为了动态赋值，在 Pipeline 执行的过程中，局部变量会根据当前的构建版本和分支版本动态生成 TAG 变量，此时局部变量会覆盖掉全局变量
# 定义一些全局的环境变量
environment {
COMMIT_ID = ""
HARBOR_ADDRESS = "CHANGE_HERE_FOR_YOUR_HARBOR_URL" # Harbor 地址
REGISTRY_DIR = "kubernetes" # Harbor 的项目目录
IMAGE_NAME = "spring-boot-project" # 镜像的名称
NAMESPACE = "kubernetes" # 该应用在 Kubernetes 中的命名空间
TAG = "" # 镜像的 Tag，在此用 BUILD_TAG+COMMIT_ID 组成
}
parameters {
# 之前讲过一些 choice、input 类型的参数，本次使用的是 GitParameter 插件
# 该字段会在 Jenkins 页面生成一个选择分支的选项
gitParameter(
branch: '',
branchFilter: 'origin/(.*)',
defaultValue: '',
description: 'Branch for build and deploy',
name: 'BRANCH',
quickFilterEnabled: false,
selectedValue: 'NONE',
sortMode: 'NONE',
tagFilter: '*',
type: 'PT_BRANCH'
)
}gitParameter 是 Jenkins Git Parameter 插件中的一个参数类型，它用于在 Jenkins Pipeline 中动态选择 Git 分支或标签。具体来说，这些参数定义了在 Jenkins 构建时用户可以选择的 Git 分支（BRANCH）或标签。
| 参数 | 含义 |
|---|---|
| branch | 默认情况下，这个字段为空。它指定了从 Git 仓库中提取的分支或标签。如果没有指定，gitParameter 会列出所有的分支 |
| branchFilter | 正则表达式，用于过滤仓库中的分支。'origin/(.*)' 表示仅显示远程仓库 origin 下的所有分支（不包括本地分支）。例如，它可以用来匹配所有的远程分支名称，如 origin/master，origin/develop 等 |
| defaultValue | 默认值字段，当用户没有选择任何分支时，Jenkins 将自动选择此默认值 |
| description | 在 Jenkins 的 UI 中，description 会显示为提示文本，帮助用户理解该参数的功能。例如，'Branch for build and deploy' 意味着这是用于选择构建和部署的 Git 分支 |
| name | 参数的名称。在这里是 'BRANCH'，它将在 Jenkins 中作为一个变量来引用。例如，用户在 UI 中选择的分支会存储在名为 BRANCH 的环境变量中，后续在 Pipeline 的其他地方可以通过 ${BRANCH} 来使用该值 |
| quickFilterEnabled | 是否启用快速过滤功能。如果设置为 true，用户可以在选择分支时通过输入部分分支名称来进行过滤。通常情况下，设置为 false 以避免干扰选择过程 |
| selectedValue | 定义了在没有指定分支时选中的默认值。'NONE' 意味着没有默认选择的分支。这通常用于明确要求用户手动选择分支，而不是自动选择一个默认值 |
| sortMode | 定义分支显示的排序方式。'NONE' 表示没有排序，分支会按其在 Git 仓库中的顺序显示 |
| tagFilter | 过滤标签的正则表达式。'*' 表示匹配所有标签 |
| type | 指定了选择的参数类型，'PT_BRANCH' 表示该参数是一个 Git 分支选择器（PT_BRANCH 是指选择分支）。除了 PT_BRANCH，还有其他类型的参数，例如 PT_TAG 用于选择标签，PT_COMMIT 用于选择提交等 |
拉取代码的 stage，这个 stage 是一个并行的 stage，因为考虑了该流水线是手动触 发还是触发，注意 credentialsId 字段的值需要跟创建的Gitlab私钥名称相同，否则构建时会提示 stderr:Permission denied, please try again. 或者是仓库不存在/拉取失败，请读者读者，在 stage('Pulling Code by Jenkins') 过程中，变量是正常被赋值了，可以被打印出来
stage('Pulling Code') {
parallel {
stage('Pulling Code by Jenkins') {
when {
expression {
# 如果 env.gitlabBranch 为空，说明是手动触发流水线，执行该 stage
env.gitlabBranch == null
}
}
steps {
# 使用 git 插件拉取代码，BRANCH 变量来自于之前定义的 parameters 配置
git(changelog: true, poll: true, url: 'git@xxxxxx:root/spring-boot-project.git', branch: "${BRANCH}", credentialsId: 'gitlab-key')
script {
# 获取最近一次提交的 Commit ID
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
# 生成 TAG，通常用于 Docker 镜像的标签
TAG = BUILD_TAG + '-' + COMMIT_ID
println "Current is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
}
}
}
stage('Pulling Code by trigger') {
when {
expression {
# 如果 env.gitlabBranch 不为空，说明是通过 webhook 触发流水线，执行该 stage
env.gitlabBranch != null
}
}
steps {
# 此时拉取代码时使用的是 webhook 触发时传入的分支信息
git(url: 'git@xxxxxxxxxxx:root/spring-boot-project.git', branch: env.gitlabBranch, changelog: true, poll: true, credentialsId: 'gitlab-key')
script {
# 获取最近一次提交的 Commit ID
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
# 生成 TAG，通常用于 Docker 镜像的标签
TAG = BUILD_TAG + '-' + COMMIT_ID
println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
}
}
}
}
}代码拉下来后，就可以执行构建命令，由于本次实验是 Java 示例，所以需要使用 mvn 命令进行构建（非常慢）
stage('Building') {
steps {
# 使用 Pod 模板里面的 build 容器进行构建
container(name: 'build') {
sh """
# 编译命令， 需要根据自己项目的实际情况进行修改，可能会不一致
mvn clean install -DskipTests
# 构建完成后，一般情况下会在 target 目录下生成 jar 包
ls target/*
"""
}
}
}生成编译产物后，需要根据该产物生成对应的镜像，此时可以使用 Pod 模板的 docker 容器，注意：sh """ """ 包围的代码片段才能正常打印 TAG变量出来，如果修改成了 sh ''' ''' 包围代码片段，打印的 TAG 变量为空
stage('Docker build for creating image') {
# 首先取出来 HARBOR 的账号密码
environment {
HARBOR_USER = credentials('HARBOR_ACCOUNT')
}
steps {
# 指定使用 docker 容器
container(name: 'docker') {
sh """
# 执行 build 命令，Dockerfile 会在下一小节创建，也是放在代码仓库，和 Jenkinsfile 同级
docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
# 登录 Harbor，HARBOR_USER_USR 和 HARBOR_USER_PSW 由上述 environment 生成
docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
# 将镜像推送至镜像仓库
docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
"""
}
}
}最后一步就是将该镜像发版至 Kubernetes 集群中，此时使用的是包含 kubectl 命令的容器
stage('Deploying to K8s') {
# 获取连接 Kubernetes 集群证书
environment {
MY_KUBECONFIG = credentials('study-k8s-kubeconfig')
}
steps {
# 指定使用 kubectl 容器
container(name: 'kubectl') {
sh """
# 直接 set 更改 Deployment 的镜像即可
/usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image \
deploy -l app=${IMAGE_NAME} \
${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE
"""
}
}
}完整配置如下
pipeline {
environment {
COMMIT_ID = ""
HARBOR_ADDRESS = "192.168.132.168"
REGISTRY_DIR = "kubernetes"
IMAGE_NAME = "spring-boot-project"
NAMESPACE = "kubernetes"
TAG = ""
}
parameters {
gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH')
}
agent {
kubernetes {
cloud 'jxcai_kubernetes'
slaveConnectTimeout 1200
workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)
yaml '''
apiVersion: v1
kind: Pod
spec:
containers:
- args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)']
image: 'registry.cn-beijing.aliyuncs.com/dotbalo/jnlp-agent-docker:latest'
name: jnlp
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- command: ["cat"]
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/maven:3.5.3"
imagePullPolicy: IfNotPresent
name: "build"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
- mountPath: "/root/.m2/"
name: "cachedir"
readOnly: false
- command: ["cat"]
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
imagePullPolicy: IfNotPresent
name: "kubectl"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- command: ["cat"]
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"
imagePullPolicy: IfNotPresent
name: "docker"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "dockersock"
readOnly: false
restartPolicy: "Never"
nodeSelector:
build: "true"
securityContext: {}
volumes:
- hostPath:
path: "/var/run/docker.sock"
name: "dockersock"
- hostPath:
path: "/usr/share/zoneinfo/Asia/Shanghai"
name: "localtime"
- name: "cachedir"
hostPath:
path: "/opt/m2"
'''
}
}
stages {
stage('Pull Code') {
parallel {
stage('Pulling Code by Jenkins') {
when {
expression {
env.gitlabBranch == null
}
}
steps {
git changelog: true, poll: true, url: 'git@192.168.132.167:caijx/spring-boot-project.git', branch: "${BRANCH}", credentialsId: 'Gitlab_Account'
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
echo "Current branch is ${BRANCH}，Commit ID is ${COMMIT_ID}，Image TAG is ${TAG}"
}
}
}
stage('Pulling Code by trigger') {
when {
expression {
env.gitlabBranch != null
}
}
steps {
git url: 'git@192.168.132.167:caijx/spring-boot-project.git', branch: env.gitlabBranch, changelog: true, poll: true, credentialsId: 'Gitlab_Account'
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
echo "Current branch is ${env.gitlabBranch}，Commit ID is ${COMMIT_ID}，Image TAG is ${TAG}"
}
}
}
}
}
stage('Maven Building') {
steps {
container(name: 'build') {
sh '''
curl -I repo.maven.apache.org || exit 1
mvn clean install -DskipTests
ls target/*
'''
}
}
}
stage('Docker build for creating image') {
environment {
HARBOR_USER = credentials('Harbor_Account')
}
steps {
container(name: 'docker') {
sh """
echo "Login Harbor：${HARBOR_USER_USR} ${HARBOR_USER_PSW}"
echo "${COMMIT_ID} ${TAG}"
docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
"""
}
}
}
stage('Deploying to K8S') {
environment {
MY_KUBECONFIG = credentials('jxcai_kubernetes')
}
steps {
container(name: 'kubectl') {
sh """
/usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE
"""
}
}
}
}
}定义 Dockerfile
在执行流水线过程时，需要将代码的编译产物做成镜像。Dockerfile 主要写的是如何生成公司业务的镜像。而本次示例是 Java 项目，只需要把 Jar 包放在有 Jre 环境的镜像中，然后启动该 Jar 包即可
# 基础镜像可以按需修改，可以更改为公司自有镜像
FROM registry.cn-beijing.aliyuncs.com/dotbalo/jre:8u211-data
# jar 包名称改成实际的名称，本示例为 spring-cloud-eureka-0.0.1-SNAPSHOT.jar
COPY target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar ./
# 启动 Jar 包
CMD java -jar spring-cloud-eureka-0.0.1-SNAPSHOT.jar
定义 Kubernetes 资源
在 GitLab 创建的 Group 为 kubernetes，可以将其认为是一个项目，同一个项目可以部署至 Kubernetes 集群中同一个 Namespace 中，本示例为 kubernetes 命名空间。由于使用的是私有仓库，因此也需要先配置拉取私有仓库镜像的密钥（注意：由于此处还未进行资源的发版，所以先使用nginx的进行部署，对应监听的端口为Java项目的端口，进行按需修改，Ingress服务配置的原因是对外发布）
[root@master-01 ~]# kubectl create namespace kubernetes
namespace/kubernetes created[root@master-01 ~]# kubectl create secret docker-registry harborkey --docker-server=192.169.132.168 --docker-username=admin --docker-password=Harbor12345 --docker-email=jxcai@coremail.cn -n kubernetes
secret/harborkey created配置Java应用的 Deployment模板
[root@master-01 ~]# cat spring-boot-project.yaml
---
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: spring-boot-project
name: spring-boot-project
namespace: kubernetes
spec:
ports:
- name: web
port: 8761
protocol: TCP
targetPort: 8761
selector:
app: spring-boot-project
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
creationTimestamp: null
name: spring-boot-project
namespace: kubernetes
spec:
ingressClassName: nginx
rules:
- host: spring-boot-project.test.com
http:
paths:
- backend:
service:
name: spring-boot-project
port:
number: 8761
path: /
pathType: ImplementationSpecific
status:
loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: spring-boot-project
name: spring-boot-project
namespace: kubernetes
spec:
replicas: 1
selector:
matchLabels:
app: spring-boot-project
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: spring-boot-project
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- spring-boot-project
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- env:
- name: TZ
value: Asia/Shanghai
- name: LANG
value: C.UTF-8
image: registry.cn-guangzhou.aliyuncs.com/caijxlinux/nginx:v1.15.1
imagePullPolicy: IfNotPresent
lifecycle: {}
livenessProbe:
failureThreshold: 2
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 8761
timeoutSeconds: 2
name: spring-boot-project
ports:
- containerPort: 8761
name: web
protocol: TCP
readinessProbe:
failureThreshold: 2
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 8761
timeoutSeconds: 2
resources:
limits:
cpu: 994m
memory: 1170Mi
requests:
cpu: 10m
memory: 55Mi
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: harborkey
restartPolicy: Always
securityContext: {}
serviceAccountName: default创建资源
[root@master-01 ~]# kubectl apply -f spring-boot-project.yaml
Warning: resource services/spring-boot-project is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
service/spring-boot-project configured
Warning: resource ingresses/spring-boot-project is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
ingress.networking.k8s.io/spring-boot-project configured
deployment.apps/spring-boot-project created创建 Jenkins 任务
打开 Jenkins UI 界面，在 Dashboard 界面单击 New Item按钮，在弹出的 New Item 界面填写 item name 为 spring-boot-project ，并选择 item type 为 Pipeline，随后单击 OK 按钮进行创建
对创建的 Item 进行配置，在弹出的 Configure 界面，单击左侧导航栏的 Pipeline按钮，将 Pipeline Definition 选项修改为 Pipeline script from SCM ，将 SCM 类型修改为 Git，在 Repository URl 复选框填写 Gitlab项目地址，Credentials 复选框选择上一章节配置的 Gitlab 密钥，随后单击 Save按钮保存配置（注意：在没有选择 Credentials之前，会提示无法连接仓库，只有该提示消失，才能证明仓库连接成功）
创建后，会自动跳转到 Project 界面，此时单击左侧导航栏中的 Build Now 按钮进行构建（注意：由于 Jenkins 参数由 Jenkinsfile 生成，所以第一次执行流水线会失败）
刷新界面，可以观察到左侧导航栏的 Build Now 按钮变成了 Build wiith Parameters，单击该按钮后，选择 master 分支，单击 Build 按钮，再次进行构建
在界面的左下方，可以看到当前构建的进度，单击对应构建次数的右侧下拉箭头，可以选择 Console Output 按钮查看构建输出的日志，也可以选择打开 Bule Ocean 查看构建过程对应执行的命令和输出的日志信息（注意：在构建过程当中可能会多次报错，请根据日志输出检查对应的环境和命令）
Blue Ocean界面如下图所示，白色背景为执行的命令，黑色背景为对应输出的日志
查看Console Output日志，构建日志输出如下，对每部分进行拆分和解释
第一部分：构建Pod，为 Agent 指定 Pod
Started by user Cai Jun Xian
Obtained Jenkinsfile from git git@192.168.132.167:caijx/spring-boot-project.git
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
[PodInfo] default/spring-boot-project-2-k3kr5-9tp0m-tjt47
Container [build] waiting [ContainerCreating] No message
Container [docker] waiting [ContainerCreating] No message
Container [jnlp] waiting [ContainerCreating] No message
Container [kubectl] waiting [ContainerCreating] No message
Pod [Pending][ContainersNotReady] containers with unready status: [jnlp build kubectl docker]
Created Pod: jxcai_kubernetes default/spring-boot-project-2-k3kr5-9tp0m-tjt47
[PodInfo] default/spring-boot-project-2-k3kr5-9tp0m-tjt47
Container [build] waiting [ContainerCreating] No message
Container [docker] waiting [ContainerCreating] No message
Container [jnlp] waiting [ContainerCreating] No message
Container [kubectl] waiting [ContainerCreating] No message
Pod [Pending][ContainersNotReady] containers with unready status: [jnlp build kubectl docker]
Agent spring-boot-project-2-k3kr5-9tp0m-tjt47 is provisioned from template spring-boot-project_2-k3kr5-9tp0m
---
apiVersion: "v1"
kind: "Pod"
metadata:
annotations:
kubernetes.jenkins.io/last-refresh: "1735253242339"
buildUrl: "http://192.168.132.168:8088/job/spring-boot-project/2/"
runUrl: "job/spring-boot-project/2/"
labels:
jenkins: "slave"
jenkins/label-digest: "21ba5b673bfb97933570841c1660c7d36046d355"
jenkins/label: "spring-boot-project_2-k3kr5"
kubernetes.jenkins.io/controller: "http___192_168_132_168_8088x"
name: "spring-boot-project-2-k3kr5-9tp0m-tjt47"
...省略部分输出...第二部分：拉取对应的仓库文件
Running on spring-boot-project-2-k3kr5-9tp0m-tjt47 in /home/jenkins/agent/workspace/spring-boot-project
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
using credential Gitlab_Account
Fetching changes from the remote Git repository
> git rev-parse --resolve-git-dir /home/jenkins/agent/workspace/spring-boot-project/.git # timeout=10
> git config remote.origin.url git@192.168.132.167:caijx/spring-boot-project.git # timeout=10
Fetching upstream changes from git@192.168.132.167:caijx/spring-boot-project.git
> git --version # timeout=10
> git --version # 'git version 2.40.1'
using GIT_SSH to set credentials Gitlab私钥
Checking out Revision 8ecc6d19facd50df3db2a1371d7cb526e2636385 (refs/remotes/origin/master)
> git fetch --tags --force --progress -- git@192.168.132.167:caijx/spring-boot-project.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git config core.sparsecheckout # timeout=10
> git checkout -f 8ecc6d19facd50df3db2a1371d7cb526e2636385 # timeout=10
Commit message: "Update Jenkinsfile"
> git rev-list --no-walk 8ecc6d19facd50df3db2a1371d7cb526e2636385 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Pull Code)
[Pipeline] parallel
[Pipeline] { (Branch: Pulling Code by Jenkins)
[Pipeline] { (Branch: Pulling Code by trigger)
[Pipeline] stage
[Pipeline] { (Pulling Code by Jenkins)
[Pipeline] stage
[Pipeline] { (Pulling Code by trigger)
Stage "Pulling Code by trigger" skipped due to when conditional
[Pipeline] getContext
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] git
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
using credential Gitlab_Account
Fetching changes from the remote Git repository
Checking out Revision 8ecc6d19facd50df3db2a1371d7cb526e2636385 (refs/remotes/origin/master)
Commit message: "Update Jenkinsfile"
[Pipeline] script
[Pipeline] {
[Pipeline] sh
> git rev-parse --resolve-git-dir /home/jenkins/agent/workspace/spring-boot-project/.git # timeout=10
> git config remote.origin.url git@192.168.132.167:caijx/spring-boot-project.git # timeout=10
Fetching upstream changes from git@192.168.132.167:caijx/spring-boot-project.git
> git --version # timeout=10
> git --version # 'git version 2.40.1'
using GIT_SSH to set credentials Gitlab私钥
> git fetch --tags --force --progress -- git@192.168.132.167:caijx/spring-boot-project.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git config core.sparsecheckout # timeout=10
> git checkout -f 8ecc6d19facd50df3db2a1371d7cb526e2636385 # timeout=10
> git branch -a -v --no-abbrev # timeout=10
> git branch -D master # timeout=10
> git checkout -b master 8ecc6d19facd50df3db2a1371d7cb526e2636385 # timeout=10
+ git log -n 1 '--pretty=format:%h'
[Pipeline] echo
Current branch is master，Commit ID is 8ecc6d1，Image TAG is jenkins-spring-boot-project-2-8ecc6d1
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Maven Building)
[Pipeline] container
[Pipeline] {
[Pipeline] sh第三部分： BUILD SUCCESS 说明 mvn 编译已经通过
+ curl -I repo.maven.apache.org
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 139 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
HTTP/1.1 501 HTTPS Required
Connection: close
Content-Length: 139
Server: Varnish
Content-Type: text/plain
Accept-Ranges: bytes
Date: Thu, 26 Dec 2024 22:47:47 GMT
Via: 1.1 varnish
X-Served-By: cache-nrt-rjtf7700100-NRT
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1735253268.719388,VS0,VE0
+ mvn clean install -DskipTests
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< com.testjava:spring-cloud-eureka >------------------
[INFO] Building spring-cloud-eureka 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ spring-cloud-eureka ---
[INFO] Deleting /home/jenkins/agent/workspace/spring-boot-project/target
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ spring-cloud-eureka ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ spring-cloud-eureka ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/jenkins/agent/workspace/spring-boot-project/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ spring-cloud-eureka ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/spring-boot-project/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ spring-cloud-eureka ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ spring-cloud-eureka ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ spring-cloud-eureka ---
[INFO] Building jar: /home/jenkins/agent/workspace/spring-boot-project/target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.1.9.RELEASE:repackage (repackage) @ spring-cloud-eureka ---
[INFO] Replacing main artifact with repackaged archive
[INFO]
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ spring-cloud-eureka ---
[INFO] Installing /home/jenkins/agent/workspace/spring-boot-project/target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar to /root/.m2/repository/com/testjava/spring-cloud-eureka/0.0.1-SNAPSHOT/spring-cloud-eureka-0.0.1-SNAPSHOT.jar
[INFO] Installing /home/jenkins/agent/workspace/spring-boot-project/pom.xml to /root/.m2/repository/com/testjava/spring-cloud-eureka/0.0.1-SNAPSHOT/spring-cloud-eureka-0.0.1-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 22.636 s
[INFO] Finished at: 2024-12-26T22:48:15Z
[INFO] ------------------------------------------------------------------------
+ ls target/classes target/generated-sources target/maven-archiver target/maven-status target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar.original
target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar
target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar.original
target/classes:
application.yml
com
logback.xml
target/generated-sources:
annotations
target/maven-archiver:
pom.properties
target/maven-status:
maven-compiler-plugin第四部分：登录 Harbor 仓库构建镜像并推送
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Docker build for creating image)
[Pipeline] withCredentials
Masking supported pattern matches of $HARBOR_USER or $HARBOR_USER_PSW
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
Affected argument(s) used the following variable(s): [HARBOR_USER_PSW]
See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ echo 'Login Harbor：admin ****'
Login Harbor：admin ****
+ echo '8ecc6d1 jenkins-spring-boot-project-2-8ecc6d1'
8ecc6d1 jenkins-spring-boot-project-2-8ecc6d1
+ docker build -t 192.168.132.168/kubernetes/spring-boot-project:jenkins-spring-boot-project-2-8ecc6d1 .
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.1s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 214B done
#2 DONE 0.1s
#3 [internal] load metadata for registry.cn-beijing.aliyuncs.com/dotbalo/jr...
#3 DONE 0.7s
#4 [1/2] FROM registry.cn-beijing.aliyuncs.com/dotbalo/jre:8u211-data@sha25...
#4 DONE 0.0s
#5 [internal] load build context
#5 transferring context: 47.19MB 1.1s done
#5 DONE 1.2s
#4 [1/2] FROM registry.cn-beijing.aliyuncs.com/dotbalo/jre:8u211-data@sha25...
#4 CACHED
#6 [2/2] COPY target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar ./
#6 DONE 0.4s
#7 exporting to image
#7 exporting layers
#7 exporting layers 0.4s done
#7 writing image sha256:5e6e08651ec42bd877a54a764b00433d2a5e5ba7ed148e42ad20629807abbf6b done
#7 naming to 192.168.132.168/kubernetes/spring-boot-project:jenkins-spring-boot-project-2-8ecc6d1 done
#7 DONE 0.4s
+ docker login -u admin -p **** 192.168.132.168
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
+ docker push 192.168.132.168/kubernetes/spring-boot-project:jenkins-spring-boot-project-2-8ecc6d1
The push refers to repository [192.168.132.168/kubernetes/spring-boot-project]
e031dc8bc8f7: Preparing
eff47b948517: Preparing
f25edfe5114d: Preparing
46f3c01a700a: Preparing
37cd0af63488: Preparing
ed211f7d10e3: Preparing
f1b5933fe4b5: Preparing
ed211f7d10e3: Waiting
f1b5933fe4b5: Waiting
46f3c01a700a: Layer already exists
37cd0af63488: Layer already exists
f25edfe5114d: Layer already exists
eff47b948517: Layer already exists
f1b5933fe4b5: Layer already exists
ed211f7d10e3: Layer already exists
e031dc8bc8f7: Pushed
jenkins-spring-boot-project-2-8ecc6d1: digest: sha256:da6aa6c5db938fd2adbea77d09cad26785382306efebc68ca75730a3df6fa4c9 size: 1785第五部分：发版至 Kubernetes
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Deploying to K8S)
[Pipeline] withCredentials
Masking supported pattern matches of $MY_KUBECONFIG
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
Affected argument(s) used the following variable(s): [MY_KUBECONFIG]
See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ /usr/local/bin/kubectl --kubeconfig **** set image deploy -l 'app=spring-boot-project' 'spring-boot-project=192.168.132.168/kubernetes/spring-boot-project:jenkins-spring-boot-project-2-8ecc6d1' -n kubernetes
deployment.apps/spring-boot-project image updated
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESSFinished：SUCCESS 说明该流水线正常结束。接下来可以查看 Deployment 的镜像
[root@master-01 ~]# kubectl get pods -n kubernetes
NAME READY STATUS RESTARTS AGE
spring-boot-project-6d857c4b97-h4zvq 1/1 Running 0 34m
You have new mail in /var/spool/mail/root
[root@master-01 ~]# kubectl get pods -n kubernetes -oyaml | grep image
image: 192.168.132.168/kubernetes/spring-boot-project:jenkins-spring-boot-project-2-8ecc6d1
imagePullPolicy: IfNotPresent
imagePullSecrets:
image: 192.168.132.168/kubernetes/spring-boot-project:jenkins-spring-boot-project-2-8ecc6d1
imageID: 192.168.132.168/kubernetes/spring-boot-project@sha256:da6aa6c5db938fd2adbea77d09cad26785382306efebc68ca75730a3df6fa4c9查看对应的 Service 和 Ingress
[root@master-01 ~]# kubectl get service -n kubernetes
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
spring-boot-project ClusterIP 10.96.170.103 <none> 8761/TCP 2d8h[root@master-01 ~]# kubectl get ingress -n kubernetes
NAME CLASS HOSTS ADDRESS PORTS AGE
spring-boot-project nginx spring-boot-project.test.com 192.168.132.172 80 2d8h在物理机上修改 hosts文件，添加对应的解析，即可访问创建的资源，至此整个Jenkins流程结束
自动化构建 Vue/H5 前端应用
本节介绍自动化构建 Vue/H5 应用，其构建方式和自动化构建 Java 基本相同，重点是更改Deployment、Jenkinsfile 和 Dockerfile 即可。 前端应用测试项目地址：https://gitee.com/caijunxianlinux/vue-project，可以参考 Java 小节的方式，导入前端项目到 GitLab 中，当然也可以使用公司自己的项目。
定义Jenkinsfile
pipeline {
environment {
COMMIT_ID = ""
HARBOR_ADDRESS = "192.168.132.168"
REGISTRY_DIR = "kubernetes"
IMAGE_NAME = "vue-project"
NAMESPACE = "kubernetes"
TAG = ""
}
parameters {
gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH')
}
agent {
kubernetes {
cloud 'jxcai_kubernetes'
slaveConnectTimeout 1200
workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)
yaml '''
apiVersion: v1
kind: Pod
spec:
containers:
- args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)']
image: 'registry.cn-beijing.aliyuncs.com/dotbalo/jnlp-agent-docker:latest'
name: jnlp
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- command: ["cat"]
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/node:lts"
imagePullPolicy: IfNotPresent
name: "build"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
- mountPath: "/root/.m2/"
name: "cachedir"
readOnly: false
- command: ["cat"]
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
imagePullPolicy: IfNotPresent
name: "kubectl"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- command: ["cat"]
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"
imagePullPolicy: IfNotPresent
name: "docker"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "dockersock"
readOnly: false
restartPolicy: "Never"
nodeSelector:
build: "true"
securityContext: {}
volumes:
- hostPath:
path: "/var/run/docker.sock"
name: "dockersock"
- hostPath:
path: "/usr/share/zoneinfo/Asia/Shanghai"
name: "localtime"
- name: "cachedir"
hostPath:
path: "/opt/m2"
'''
}
}
stages {
stage('Pull Code') {
parallel {
stage('Pulling Code by Jenkins') {
when {
expression {
env.gitlabBranch == null
}
}
steps {
git changelog: true, poll: true, url: 'git@192.168.132.167:caijx/vue-project.git', branch: "${BRANCH}", credentialsId: 'Gitlab_Account'
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
echo "Current branch is ${BRANCH}，Commit ID is ${COMMIT_ID}，Image TAG is ${TAG}"
}
}
}
stage('Pulling Code by trigger') {
when {
expression {
env.gitlabBranch != null
}
}
steps {
git url: 'git@192.168.132.167:caijx/vue-project.git', branch: env.gitlabBranch, changelog: true, poll: true, credentialsId: 'gitlab-key'
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
echo "Current branch is ${env.gitlabBranch}，Commit ID is ${COMMIT_ID}，Image TAG is ${TAG}"
}
}
}
}
}
stage('Building') {
steps {
container(name: 'build') {
sh '''
npm install --registry=https://registry.npmjs.org
npm run build
'''
}
}
}
stage('Docker build for creating image') {
environment {
HARBOR_USER = credentials('Harbor_Account')
}
steps {
container(name: 'docker') {
sh """
echo "Login Harbor：${HARBOR_USER_USR} ${HARBOR_USER_PSW}"
echo "${COMMIT_ID} ${TAG}"
docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
"""
}
}
}
stage('Deploying to K8S') {
environment {
MY_KUBECONFIG = credentials('jxcai_kubernetes')
}
steps {
container(name: 'kubectl') {
sh """
/usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE
"""
}
}
}
}
}定义Dockerfile
FROM registry.cn-guangzhou.aliyuncs.com/caijxlinux/nginx:v1.15.1
COPY dist/* /usr/share/nginx/html/定义Kubernetes资源
[root@master-01 ~]# cat vue-project.yaml
---
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: vue-project
name: vue-project
namespace: kubernetes
spec:
ports:
- name: web
port: 80
protocol: TCP
targetPort: 80
selector:
app: vue-project
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
creationTimestamp: null
name: vue-project
namespace: kubernetes
spec:
ingressClassName: nginx
rules:
- host: vue-project.test.com
http:
paths:
- backend:
service:
name: vue-project
port:
number: 80
path: /
pathType: ImplementationSpecific
status:
loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: vue-project
name: vue-project
namespace: kubernetes
spec:
replicas: 1
selector:
matchLabels:
app: vue-project
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: vue-project
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- vue-project
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- env:
- name: TZ
value: Asia/Shanghai
- name: LANG
value: C.UTF-8
image: registry.cn-guangzhou.aliyuncs.com/caijxlinux/nginx:v1.15.1
imagePullPolicy: IfNotPresent
lifecycle: {}
livenessProbe:
failureThreshold: 2
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 80
timeoutSeconds: 2
name: vue-project
ports:
- containerPort: 80
name: web
protocol: TCP
readinessProbe:
failureThreshold: 2
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 80
timeoutSeconds: 2
resources:
limits:
cpu: 994m
memory: 1170Mi
requests:
cpu: 10m
memory: 55Mi
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: harborkey
restartPolicy: Always
securityContext: {}
serviceAccountName: default创建资源
[root@master-01 ~]# kubectl create -f vue-project.yaml
service/vue-project created
ingress.networking.k8s.io/vue-project created
deployment.apps/vue-project created创建 Jenkins 任务
此步骤与上小节内容一致，此处不再赘述，构建完毕后，在集群验证容器和镜像，物理机添加映射，访问域名进行验证
[root@master-01 ~]# kubectl get pods -n kubernetes -l app=vue-project
NAME READY STATUS RESTARTS AGE
vue-project-564c8b9997-4jk4z 1/1 Running 0 31m[root@master-01 ~]# kubectl get pods -n kubernetes -l app=vue-project -oyaml | grep image
image: 192.168.132.168/kubernetes/vue-project:jenkins-vue-project-4-c5e764c
imagePullPolicy: IfNotPresent
imagePullSecrets:
image: 192.168.132.168/kubernetes/vue-project:jenkins-vue-project-4-c5e764c
imageID: 192.168.132.168/kubernetes/vue-project@sha256:57769fcba17a9e7339e3ca1b31da7f764a401690db2eecc4e02e0d3bfcc02472
自动化构建 Golang 项目
定义Jenkinsfile
pipeline {
environment {
COMMIT_ID = ""
HARBOR_ADDRESS = "192.168.132.168"
REGISTRY_DIR = "kubernetes"
IMAGE_NAME = "go-project"
NAMESPACE = "kubernetes"
TAG = ""
}
parameters {
gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH')
}
agent {
kubernetes {
cloud 'jxcai_kubernetes'
slaveConnectTimeout 1200
workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)
yaml '''
apiVersion: v1
kind: Pod
spec:
containers:
- args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)']
image: 'registry.cn-beijing.aliyuncs.com/dotbalo/jnlp-agent-docker:latest'
name: jnlp
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- command: ["cat"]
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/golang:1.15"
imagePullPolicy: IfNotPresent
name: "build"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
- mountPath: "/go/pkg/"
name: "cachedir"
readOnly: false
- command: ["cat"]
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
imagePullPolicy: IfNotPresent
name: "kubectl"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- command: ["cat"]
env:
- name: "LANGUAGE"
value: "en_US:en"
- name: "LC_ALL"
value: "en_US.UTF-8"
- name: "LANG"
value: "en_US.UTF-8"
image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"
imagePullPolicy: IfNotPresent
name: "docker"
tty: true
volumeMounts:
- mountPath: "/etc/localtime"
name: "localtime"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "dockersock"
readOnly: false
restartPolicy: "Never"
nodeSelector:
build: "true"
securityContext: {}
volumes:
- hostPath:
path: "/var/run/docker.sock"
name: "dockersock"
- hostPath:
path: "/usr/share/zoneinfo/Asia/Shanghai"
name: "localtime"
- name: "cachedir"
hostPath:
path: "/opt/gopkg"
'''
}
}
stages {
stage('Pull Code') {
parallel {
stage('Pulling Code by Jenkins') {
when {
expression {
env.gitlabBranch == null
}
}
steps {
git changelog: true, poll: true, url: 'git@192.168.132.167:caijx/go-project.git', branch: "${BRANCH}", credentialsId: 'Gitlab_Account'
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
echo "Current branch is ${BRANCH}，Commit ID is ${COMMIT_ID}，Image TAG is ${TAG}"
}
}
}
stage('Pulling Code by trigger') {
when {
expression {
env.gitlabBranch != null
}
}
steps {
git url: 'git@192.168.132.167:caijx/go-project.git', branch: env.gitlabBranch, changelog: true, poll: true, credentialsId: 'gitlab-key'
script {
COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
TAG = BUILD_TAG + '-' + COMMIT_ID
echo "Current branch is ${env.gitlabBranch}，Commit ID is ${COMMIT_ID}，Image TAG is ${TAG}"
}
}
}
}
}
stage('Building') {
steps {
container(name: 'build') {
sh '''
export GO111MODULE=on
go env -w GOPROXY=https://goproxy.cn,direct
go build
'''
}
}
}
stage('Docker build for creating image') {
environment {
HARBOR_USER = credentials('Harbor_Account')
}
steps {
container(name: 'docker') {
sh """
echo "Login Harbor：${HARBOR_USER_USR} ${HARBOR_USER_PSW}"
echo "${COMMIT_ID} ${TAG}"
docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
"""
}
}
}
stage('Deploying to K8S') {
environment {
MY_KUBECONFIG = credentials('jxcai_kubernetes')
}
steps {
container(name: 'kubectl') {
sh """
/usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE
"""
}
}
}
}
}定义Dockerfile
FROM registry.cn-beijing.aliyuncs.com/dotbalo/alpine-glibc:alpine-3.9
COPY conf/ ./conf
COPY ./go-project ./
ENTRYPOINT [ "./go-project"]定义Kubernetes资源
[root@master-01 ~]# cat go-project.yaml
---
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: go-project
name: go-project
namespace: kubernetes
spec:
ports:
- name: web
port: 8080
protocol: TCP
targetPort: 8080
selector:
app: go-project
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
creationTimestamp: null
name: go-project
namespace: kubernetes
spec:
ingressClassName: nginx
rules:
- host: go-project.test.com
http:
paths:
- backend:
service:
name: go-project
port:
number: 8080
path: /
pathType: ImplementationSpecific
status:
loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: go-project
name: go-project
namespace: kubernetes
spec:
replicas: 1
selector:
matchLabels:
app: go-project
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: go-project
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- go-project
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- env:
- name: TZ
value: Asia/Shanghai
- name: LANG
value: C.UTF-8
image: registry.cn-guangzhou.aliyuncs.com/caijxlinux/nginx:v1.15.1
imagePullPolicy: IfNotPresent
lifecycle: {}
livenessProbe:
failureThreshold: 2
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 8080
timeoutSeconds: 2
name: go-project
ports:
- containerPort: 8080
name: web
protocol: TCP
readinessProbe:
failureThreshold: 2
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 8080
timeoutSeconds: 2
resources:
limits:
cpu: 994m
memory: 1170Mi
requests:
cpu: 10m
memory: 55Mi
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: harborkey
restartPolicy: Always
securityContext: {}
serviceAccountName: default创建 Jenkins 任务
此步骤与上小节内容一致，此处不再赘述，构建完毕后，在集群验证容器和镜像，物理机添加映射，访问域名进行验证
[root@master-01 ~]# kubectl get pods -n kubernetes -l app=go-project
NAME READY STATUS RESTARTS AGE
go-project-75f978d77d-26zs8 1/1 Running 0 2m32s[root@master-01 ~]# kubectl get pods -n kubernetes -l app=go-project -oyaml | grep image
image: 192.168.132.168/kubernetes/go-project:jenkins-go-project-3-da0e4a2
imagePullPolicy: IfNotPresent
imagePullSecrets:
image: 192.168.132.168/kubernetes/go-project:jenkins-go-project-2-da0e4a2
imageID: 192.168.132.168/kubernetes/go-project@sha256:e3cad33ed08b20076aad34a038ca16e9361105243e7cdbd4ac950d108d7fa453
自动触发构建
之前的构建都是采用手动选择分支进行构建的，实际使用时，项目可能有很多，如果都是手动触发可能比较消耗人力。所以推荐可以按需配置自动触发，即提交代码后自动触发 Jenkins 进行构建任务。
本次用 Java 项目进行演示。首先找到Jenkins Java 项目的 Job，单击左侧导航栏的 Configure 按钮，找到 Build Triggers选项，勾选 Build when a change is pushed to GitLab. GitLab webhook URL: http://192.168.132.168:8088/project/spring-boot-project 选项，在启用 triggers 选项选择 Push Events选项，随后单击 Advanced 选项
下拉选项条，Allowed branches 选项选择允许所有分支（实验使用，生产环境可以使用正则表达式进行限制），找到 Secret token 选项，单击 Generate 按钮生成 token，并记录下来，用以与 Gitlab 进行认证
切换到 Gitlab UI界面，切换到管理员界面下，依次单击左侧导航栏的 Settings→Network→Outbound requests按钮，勾选 Allow requests to the local network from webhooks and integrations 选项（允许从 webhook 和集成向本地网络发出请求，如果不勾选此选项，由于Jenkins为内网地址，在配置Webhook时会报错 invaild url），随后单击 Save changes 按钮保存配置
切换回Gitlab Java 项目内，依次单击左侧导航栏的 Settings→Webhooks→Add new webhook按钮，在 Webhook 界面的 URL 选项填写在 Jenkins 记录下来的GitLab webhook URL，在 Secret token 选项填写在 Jenkins 生成的secret token，勾选 Trigger 选项 Push events 和 All branches 子选项（生产环境下可以勾选其他的触发事件），随后单击 Save changes 选项保存配置
随后，自动返回 Webhooks 界面，单击 Test 按钮，选项 Push events，开始测试
回到 Jenkins 界面可以观察到流水线已经被自动触发了
打开 Bule Ocean 界面，可以观察到 Pipeline 走了 trigger分支，证明自动触发构建任务成功
一次构建多次部署
在 UAT 和生产环境，一般不需要再次构建，而是选择其它环境产生的镜像进行发版，本节实验演示如何进行不构建进行发版
在 Jenkins UI 界面创建新 Item，名称为 go-project-uat，类型为 Pipeline，单击 OK按钮，在 General 选项卡勾选 This project is parameterized ，选择 Image Tag Parameter
Name选项卡填写IMAGE_TAG，Image Name 填写为 kubernetes/go-project（对应Harbor仓库内的镜像名称），Default Tag 填写为 blank（按需填写），Description 填写为 选择对应的镜像版本
填写完成后，单击 Advanced 按钮，填写 Harbor仓库的地址和凭证，单击 Save 按钮保存配置
注意：Registry URL 填写的协议是 HTTPS，并使用域名而不是 IP地址，这是因为Jenkins会查询 CA证书，由于Harbor是自签的证书，并且证书签署的是域名而不是IP，所以在填写时要注意对应的名称和协议
随后可以在主界面单击 Build with Parameters 按钮进行验证，但是依然报错
HTTP status: Certificate for <192.168.132.168> doesn't match any of the subject alternative names: [] #这个报错是证书和认证的名称不一致PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target #这个报错是因为Jenkins没有找到对应的根证书解决方法：
1、打开谷歌浏览器，访问Harbor，单击网站不安全标识→证书详情→详细信息→导出(x)按钮，导出时选择保存类型为 DER 编码二进制，单一证书（*.der）
2、将导出的证书上传到 Jenkins所在的物理机内，通过docker cp命令导入 Jenkins 容器
[root@harbor ~]# docker cp harbor.der f7885d84206e:/opt
Successfully copied 3.07kB to f7885d84206e:/opt3、登录 Jenkins 容器导入证书（注意：需要对容器提权，否则导入时会提示权限不足，拒绝访问，如果需要输入密码，请输入默认密码 changeit）
[root@harbor ~]# docker exec -u root -it --privileged f7885d84206e bash[root@f7885d84206e /]# keytool --import --alias harbor --keystore /opt/java/openjdk/lib/security/cacerts -file /opt/harbor.der
Warning: use -cacerts option to access cacerts keystore
Owner: CN=harbor, O=Default Company Ltd, L=GZ, ST=GZ, C=CN
Issuer: O=Default Company Ltd, L=GZ, ST=GZ, C=CN
Serial number: b23cb907a8fac8b3
Valid from: Tue Apr 30 16:02:31 GMT 2024 until: Fri Apr 28 16:02:31 GMT 2034
Certificate fingerprints:
SHA1: EF:F6:A6:F2:9B:94:54:6F:E9:9D:AD:92:93:70:2B:6E:2D:DD:55:C7
SHA256: E2:91:1B:34:8D:0C:E9:27:3F:37:EF:3A:3E:F3:EE:46:37:8F:7C:A8:0D:D1:33:D3:B2:64:71:B9:1D:30:D8:9C
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 3072-bit RSA key
Version: 1
Trust this certificate? [no]: yes
Certificate was added to keystore4、导入完成后，刷新 Jenkins 界面，可以观察到 Harbor 内的镜像已经被正确显示了
验证成功后，添加 Pipeline脚本
pipeline {
agent {
kubernetes {
cloud 'jxcai_kubernetes'
slaveConnectTimeout 1200
yaml '''
apiVersion: v1
kind: Pod
spec:
containers:
- args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)']
image: 'registry.cn-beijing.aliyuncs.com/dotbalo/jnlp-agent-docker:latest'
name: jnlp
imagePullPolicy: IfNotPresent
- command: ['cat']
env:
- name: 'LANGUAGE'
value: 'en_US:en'
- name: 'LC_ALL'
value: 'en_US.UTF-8'
- name: 'LANG'
value: 'en_US.UTF-8'
image: 'registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17'
imagePullPolicy: 'IfNotPresent'
name: 'kubectl'
tty: true
restartPolicy: 'Never'
'''
}
}
stages {
stage('Deploy') {
environment {
MY_KUBECONFIG = credentials('jxcai_kubernetes')
}
steps {
container(name: 'kubectl') {
sh """
echo ${IMAGE_TAG}
kubectl --kubeconfig=${MY_KUBECONFIG} set image deployment \
-l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${IMAGE_TAG} -n ${NAMESPACE}
kubectl --kubeconfig=${MY_KUBECONFIG} get po -l app=${IMAGE_NAME} -n ${NAMESPACE} -w
"""
}
}
}
}
environment {
HARBOR_ADDRESS = "192.168.132.168"
NAMESPACE = "kubernetes"
IMAGE_NAME = "go-project"
TAG = ""
}
}执行构建，通过日志的输出可以观察到，跳过了构建的流程，使用选定的镜像进行构建，节省了发版的时间，同时也可以利用此功能实现镜像的回滚（读者可以选择不同的镜像版本进行验证）
Running on go-project-uat-2-shnjn-ld59z-r1x4s in /home/jenkins/agent/workspace/go-project-uat
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Deploy)
[Pipeline] withCredentials
Masking supported pattern matches of $MY_KUBECONFIG
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
Affected argument(s) used the following variable(s): [MY_KUBECONFIG]
See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ echo kubernetes/go-project:jenkins-go-project-3-da0e4a2
kubernetes/go-project:jenkins-go-project-3-da0e4a2
+ kubectl '--kubeconfig=****' set image deployment -l 'app=go-project' 'go-project=192.168.132.168/kubernetes/go-project:jenkins-go-project-3-da0e4a2' -n kubernetes
+ kubectl '--kubeconfig=****' get po -l 'app=go-project' -n kubernetes -w
NAME READY STATUS RESTARTS AGE
go-project-75f978d77d-26zs8 1/1 Running 0 5h4m