二进制安装高可用Kubernetes1.30.1
- Kubernetes
- 2024-05-28
- 628热度
- 0评论
二进制安装高可用Kubernetes1.30.1
高可用Kubernetes集群规划
//注意：宿主机网段、K8S Service网段、Pod网段不能重复
| 主机名 | IP地址 | 角色 | 版本 |
|---|---|---|---|
| master-01 | 192.168.132.169 | master节点-1 | Centos7.9 |
| master-02 | 192.168.132.170 | master节点-2 | Centos7.9 |
| master-03 | 192.168.132.171 | master节点-3 | Centos7.9 |
| master-lb | 192.168.132.236 | VIP | KeepAlived+HAProxy |
| node-01 | 192.168.132.172 | node节点-1 | Centos7.9 |
| node-02 | 192.168.132.173 | node节点-2 | Centos7.9 |
软件和其他网段规划
| 软件 | 版本 | 网段 |
|---|---|---|
| Containerd | 20.10.x | |
| Pod | 172.16.0.0/16 | |
| Service | 10.96.0.0/16 |
安装过程（5个结点都需要操作的，仅以master-01结点操作，读者需要自行在其余主机内执行相同的命令）
初始化操作
所有节点修改主机名（其他节点按照规划表进行修改）
[root@localhost ~]# hostnamectl set-hostname master-01
[root@localhost ~]# bash
[root@master-01 ~]#master-01节点修改/etc/hosts文件
[root@master-01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.132.169 master-01
192.168.132.170 master-02
192.168.132.171 master-03
# 如果不是高可用集群，该IP为Master01的IP
192.168.132.236 master-lb
192.168.132.172 node-01
192.168.132.173 node-02master-01执行ping命令，检查与其他4台主机的连通性，由于192.168.132.236是VIP地址，所以当前地址不可达
[root@master-01 ~]# for i in {master-01,master-02,master-03,master-lb,node-01,node-02}; do ping $i -c 1; done
PING master-01 (192.168.132.169) 56(84) bytes of data.
64 bytes from master-01 (192.168.132.169): icmp_seq=1 ttl=64 time=0.040 ms
--- master-01 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.040/0.040/0.040/0.000 ms
PING master-02 (192.168.132.170) 56(84) bytes of data.
64 bytes from master-02 (192.168.132.170): icmp_seq=1 ttl=64 time=0.651 ms
--- master-02 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.651/0.651/0.651/0.000 ms
PING master-03 (192.168.132.171) 56(84) bytes of data.
64 bytes from master-03 (192.168.132.171): icmp_seq=1 ttl=64 time=0.460 ms
--- master-03 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.460/0.460/0.460/0.000 ms
PING master-lb (192.168.132.236) 56(84) bytes of data.
From master-01 (192.168.132.169) icmp_seq=1 Destination Host Unreachable
--- master-lb ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
PING node-01 (192.168.132.172) 56(84) bytes of data.
64 bytes from node-01 (192.168.132.172): icmp_seq=1 ttl=64 time=0.453 ms
--- node-01 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.453/0.453/0.453/0.000 ms
PING node-02 (192.168.132.173) 56(84) bytes of data.
64 bytes from node-02 (192.168.132.173): icmp_seq=1 ttl=64 time=0.842 ms
--- node-02 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.842/0.842/0.842/0.000 ms
master-01节点配置对其他主机的免密登录
[root@master-01 ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:y8j+HKRG7GQCAZWmdHpqM5rKK4FKbvTotHzyuR5+eXY root@master-01
The key's randomart image is:
+---[RSA 2048]----+
|.oo. |
| .oo |
|.o+ |
|.. o . |
|. o . = S |
|oB B = . |
|B+=. B + |
|O=ooo= + E |
|****o +.+ |
+----[SHA256]-----+
[root@master-01 ~]# for i in {master-01,master-02,master-03,node-01,node-02}; do ssh-copy-id $i; done
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'master-01 (192.168.132.169)' can't be established.
ECDSA key fingerprint is SHA256:3jDHkD+/2lQF89uZBEBLWtHSQLcv34cdY/oRhyo507I.
ECDSA key fingerprint is MD5:ca:ec:2f:b0:c6:65:d9:50:11:f6:b1:38:84:64:f1:c4.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@master-01's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'master-01'"
and check to make sure that only the key(s) you wanted were added.
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'master-02 (192.168.132.170)' can't be established.
ECDSA key fingerprint is SHA256:3jDHkD+/2lQF89uZBEBLWtHSQLcv34cdY/oRhyo507I.
ECDSA key fingerprint is MD5:ca:ec:2f:b0:c6:65:d9:50:11:f6:b1:38:84:64:f1:c4.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@master-02's password:
Number of key(s) added: 1
...省略部分输出...master-01节点通过scp命令将/etc/hosts文件分发到其他4台主机上
[root@master-01 ~]# for i in {master-01,master-02,master-03,node-01,node-02}; do scp /etc/hosts root@$i:/etc/hosts; done
hosts 100% 364 506.1KB/s 00:00
hosts 100% 364 150.4KB/s 00:00
hosts 100% 364 355.4KB/s 00:00
hosts 100% 364 397.2KB/s 00:00
hosts 所有节点配置仓库源
[root@master-01 ~]# curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
[root@master-01 ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
[root@master-01 ~]# yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@master-01 ~]# sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo所有节点安装初始化工具
[root@master-01 ~]# yum install wget jq psmisc vim net-tools telnet git bash-completion -y
[root@master-01 ~]# bash所有节点修改IP地址为静态获取，修改网卡配置文件，删除UUID选项，对应的设备接口、网关地址、
DNS地址需要读者根据实际环境进行修改
[root@master-01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.132.169
NETMASK=255.255.255.0
#PREFIX=24
GATEWAY=192.168.132.2
DNS1=192.168.132.2所有节点关闭NetworkManager服务，重启所有节点的网络，查看IP地址是否成功获取，查看主机是否能访问互联网，如果检查配置无误，发现无法获取IP地址或者无法访问互联网，检查IP地址是否被占用，网关和域名解析是否正确
[root@master-01 ~]# systemctl stop NetworkManager
[root@master-01 ~]# systemctl disable NetworkMange
[root@master-01 ~]# ip address show | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 192.168.132.169/24 brd 192.168.132.255 scope global ens33
[root@master-01 ~]# ping blog.caijxlinux.work
PING blog.caijxlinux.work (8.138.107.10) 56(84) bytes of data.
64 bytes from blog.caijxlinux.work (8.138.107.10): icmp_seq=1 ttl=128 time=45.7 ms
--- blog.caijxlinux.work ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 45.753/45.753/45.753/0.000 ms所有节点关闭防火墙、Selinux和Swap
[root@master-01 ~]# systemctl stop firewalld.service
[root@master-01 ~]# systemctl disable firewalld.service
[root@master-01 ~]# setenforce 0
[root@master-01 ~]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
[root@master-01 ~]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
[root@master-01 ~]# swapoff -a && sysctl -w vm.swappiness=0
vm.swappiness = 0
[root@master-01 ~]# sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab所有节点安装时间同步服务，修改系统时区，使用阿里云的时间服务器进行时间同步，并写入定时任务
[root@master-01 ~]# rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
[root@master-01 ~]# yum install ntpdate -y
[root@master-01 ~]# ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
[root@master-01 ~]# echo 'Asia/Shanghai' >/etc/timezone
[root@master-01 ~]# ntpdate time2.aliyun.com
19 May 00:51:05 ntpdate[3180]: adjust time server 203.107.6.88 offset 0.008514 sec
[root@master-01 ~]# crontab -e
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com
[root@master-01 ~]# date
Sun May 19 00:55:22 CST 2024调整系统的文件描述符限制
[root@master-01 ~]# ulimit -SHn 65535
[root@master-01 ~]# vim /etc/security/limits.conf
#ftp hard nproc 0
#@student - maxlogins 4
# End of file
#在配置文件的末尾加入
* soft nofile 65536
* hard nofile 131072
* soft nproc 65535
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited所有节点软件升级
[root@master-01 ~]# yum update -y --exclude=kernel*master-01节点下载K8S所需源码文件（读者需要注意下载的目录）
[root@master-01 ~]# git clone https://gitee.com/dukuan/k8s-ha-install.git
Cloning into 'k8s-ha-install'...
remote: Enumerating objects: 907, done.
remote: Counting objects: 100% (25/25), done.
remote: Compressing objects: 100% (21/21), done.
...省略部分输出...master-01节点下载4.19内核版本，读者可以通过链接进行下载，也可以联系编者提供压缩包
[root@master-01 ~]# wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
[root@master-01 ~]# wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
[root@master-01 ~]# ls
anaconda-ks.cfg kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpmmaster-01节点分发内核文件到其他4个节点
[root@master-01 ~]# for i in {master-02,master-03,node-01,node-02} ; do scp kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm root@$i:/root/ ; done
kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm 100% 46MB 61.0MB/s 00:00
kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm 100% 12MB 55.0MB/s 00:00
...省略部分输出...所有节点安装新内核
[root@master-01 ~]# yum localinstall -y kernel-ml*所有节点更改内核启动顺序
[root@master-01 ~]# grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg
[root@master-01 ~]# grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"所有节点检查内核版本是否为4.19
[root@master-01 ~]# grubby --default-kernel
/boot/vmlinuz-4.19.12-1.el7.elrepo.x86_64所有节点安装ipvsadm、ipset等管理工具，并添加ipvs模块
[root@master-01 ~]# yum -y install ipvsadm ipset sysstat conntrack libseccomp
[root@master-01 ~]# modprobe -- ip_vs
[root@master-01 ~]# modprobe -- ip_vs_rr
[root@master-01 ~]# modprobe -- ip_vs_wrr
[root@master-01 ~]# modprobe -- ip_vs_sh
[root@master-01 ~]# modprobe -- nf_conntrack
[root@master-01 ~]# cat /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip所有节点重新加载内核模块
[root@master-01 ~]# systemctl enable --now systemd-modules-load.service所有节点配置K8S内核所需参数，并立即重新加载内核参数
[root@master-01 ~]# cat <<EOF > /etc/sysctl.d/k8s.conf
> net.ipv4.ip_forward = 1
> net.bridge.bridge-nf-call-iptables = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> fs.may_detach_mounts = 1
> net.ipv4.conf.all.route_localnet = 1
> vm.overcommit_memory=1
> vm.panic_on_oom=0
> fs.inotify.max_user_watches=89100
> fs.file-max=52706963
> fs.nr_open=52706963
> net.netfilter.nf_conntrack_max=2310720
> net.ipv4.tcp_keepalive_time = 600
> net.ipv4.tcp_keepalive_probes = 3
> net.ipv4.tcp_keepalive_intvl =15
> net.ipv4.tcp_max_tw_buckets = 36000
> net.ipv4.tcp_tw_reuse = 1
> net.ipv4.tcp_max_orphans = 327680
> net.ipv4.tcp_orphan_retries = 3
> net.ipv4.tcp_syncookies = 1
> net.ipv4.tcp_max_syn_backlog = 16384
> net.ipv4.ip_conntrack_max = 65536
> net.ipv4.tcp_max_syn_backlog = 16384
> net.ipv4.tcp_timestamps = 0
> net.core.somaxconn = 16384
> EOF
[root@master-01 ~]# sysctl --system
* Applying /usr/lib/sysctl.d/00-system.conf ...
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...所有节点配置完成后，重启节点，确保内核参数正确（虚拟机环境建议添加快照）
[root@master-01 ~]# reboot
[root@master-01 ~]# lsmod | grep --color=auto -e ip_vs -e nf_conntrack
ip_vs_ftp 16384 0
nf_nat 32768 1 ip_vs_ftp
ip_vs_sed 16384 0
ip_vs_nq 16384 0
ip_vs_fo 16384 0
...省略部分输出...
[root@master-01 ~]# uname -a
Linux master-01 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@master-01 ~]# getenforce
DisabledCotainerd安装（Runtime）
所有节点添加阿里云docker源并更新缓存（有时候更新的方法可能会改变，具体参照阿里云官方文档，https://developer.aliyun.com/mirror/docker-ce/?spm=a2c6h.25603864.0.0.5bef61d5dsLmOy）
[root@master-01 ~]# yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@master-01 ~]# sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo
[root@master-01 ~]# yum makecache fast所有节点安装Containerd
[root@master-01 ~]# yum -y install docker-ce-23.* docker-ce-cli-23.* containerd.io所有节点配置Containerd所需模块，并重新加载模块
[root@master-01 ~]# cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
> overlay
> br_netfilter
> EOF
overlay
br_netfilter
[root@master-01 ~]# modprobe -- overlay
[root@master-01 ~]# modprobe -- br_netfilter
[root@master-01 ~]# cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
> net.bridge.bridge-nf-call-iptables = 1
> net.ipv4.ip_forward = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> EOF
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
[root@master-01 ~]# sysctl --system
* Applying /usr/lib/sysctl.d/00-system.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
* Applying /usr/lib/sysctl.d/50-default.conf ...
...省略部分输出...所有节点生成Containerd默认配置文件，修改配置文件/etc/containerd/config.toml的SystemdCgroup字段（令Cgroup方式为Systemd）
[root@master-01 ~]# containerd config default | tee /etc/containerd/config.toml
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
...省略部分输出...
[root@master-01 ~]# vim /etc/containerd/config.toml
SystemdCgroup = true
[root@master-01 ~]# cat /etc/containerd/config.toml | grep SystemdCgroup -A 5 -B 5
...省略部分输出...
IoUid = 0
NoNewKeyring = false
NoPivotRoot = false
Root = ""
ShimCgroup = ""
SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
...省略部分输出...所有节点修改配置文件/etc/containerd/config.toml的sandbox_image字段，修改pause镜像拉取地址
[root@master-01 ~]# cat /etc/containerd/config.toml | grep -A 2 -B 2 sandbox_image
...省略部分输出...
netns_mounts_under_state_dir = false
restrict_oom_score_adj = false
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"
selinux_category_range = 1024
stats_collect_period = 10
...省略部分输出...所有节点启动Containerd服务，配置服务开机自启并查看服务状态（部分节点会提示关于CNI的错误，无需理会，网络插件安装完成后，会解决错误）
[root@master-01 ~]# systemctl daemon-reload
[root@master-01 ~]# systemctl enable --now containerd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/containerd.service to /usr/lib/systemd/system/containerd.service.
[root@master-01 ~]# systemctl status containerd.service
● containerd.service - containerd container runtime
Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2024-05-19 20:49:16 CST; 5s ago
Docs: https://containerd.io
Process: 12753 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 12757 (containerd)
Tasks: 9
Memory: 16.1M
CGroup: /system.slice/containerd.service
└─12757 /usr/bin/containerd
May 19 20:49:16 master-01 containerd[12757]: time="2024-05-19T20:49:16.958294487+08:00" level=info msg=serv...trpc
May 19 20:49:16 master-01 containerd[12757]: time="2024-05-19T20:49:16.958855971+08:00" level=info msg=serv...sock
May 19 20:49:16 master-01 containerd[12757]: time="2024-05-19T20:49:16.966456119+08:00" level=info msg="Sta...ent"
May 19 20:49:16 master-01 containerd[12757]: time="2024-05-19T20:49:16.966664799+08:00" level=info msg="Sta...ate"
May 19 20:49:16 master-01 containerd[12757]: time="2024-05-19T20:49:16.966778189+08:00" level=info msg="Sta...tor"
May 19 20:49:16 master-01 containerd[12757]: time="2024-05-19T20:49:16.966835873+08:00" level=info msg="Sta...cer"
May 19 20:49:16 master-01 containerd[12757]: time="2024-05-19T20:49:16.966859864+08:00" level=info msg="Sta...ult"
May 19 20:49:16 master-01 containerd[12757]: time="2024-05-19T20:49:16.966879131+08:00" level=info msg="Sta...ver"
May 19 20:49:16 master-01 containerd[12757]: time="2024-05-19T20:49:16.967603120+08:00" level=info msg="con...25s"
May 19 20:49:16 master-01 systemd[1]: Started containerd container runtime.
Hint: Some lines were ellipsized, use -l to show in full.所有节点配置crictl客户端连接的运行时位置
[root@master-01 ~]# cat > /etc/crictl.yaml <<EOF
> runtime-endpoint: unix:///run/containerd/containerd.sock
> image-endpoint: unix:///run/containerd/containerd.sock
> timeout: 10
> debug: false
> EOFK8S与ETCD安装
查看当前需要安装K8S的版本，网址如下
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md
master01节点下载K8S压缩包（网络或许会很慢，建议读者可以直接访问网页使用浏览器进行下载）
注：读者需要自行下载自己需要的版本
[root@master-01 ~]# wget https://dl.k8s.io/v1.30.1/kubernetes-server-linux-amd64.tar.gzmaster01节点下载Etcd压缩包
[root@master-01 ~]# wget https://github.com/etcd-io/etcd/releases/download/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz
[root@master-01 ~]# ls | egrep "kube|etcd"
etcd-v3.5.13-linux-amd64.tar.gz
kubernetes-server-linux-amd64.tar.gzmaster01节点解压下载完成的K8S压缩包
[root@master-01 ~]# tar -xf kubernetes-server-linux-amd64.tar.gz --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
[root@master-01 ~]# cd /usr/local/bin/
[root@master-01 bin]# ls
kube-apiserver kube-controller-manager kubectl kubelet kube-proxy kube-schedulermaster01节点解压下载完成的Etcd压缩包
[root@master-01 ~]# tar -zxvf etcd-v3.5.13-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.5.13-linux-amd64/etcd{,ctl}
etcd-v3.5.13-linux-amd64/etcdctl
etcd-v3.5.13-linux-amd64/etcd
[root@master-01 ~]# ls /usr/local/bin/
etcd etcdctl kube-apiserver kube-controller-manager kubectl kubelet kube-proxy kube-schedulermaster01节点查看当前软件版本
[root@master-01 ~]# kubelet --version
Kubernetes v1.30.1
[root@master-01 ~]# etcd --version
etcd Version: 3.5.13
Git SHA: c9063a0dc
Go Version: go1.21.8
Go OS/Arch: linux/amd64master01节点将当前安装的组件对应发送到相应结点
[root@master-01 ~]# MasterNodes='master-02 master-03'
[root@master-01 ~]# WorkNodes='node-01 node-02'
[root@master-01 ~]# for NODE in $MasterNodes; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done
master-02
kubelet 100% 95MB 62.4MB/s 00:01
kubectl 100% 49MB 59.3MB/s 00:00
kube-apiserver 100% 108MB 54.4MB/s 00:01
kube-controller-manager 100% 102MB 57.1MB/s 00:01
kube-scheduler 100% 56MB 55.6MB/s 00:01
kube-proxy 100% 55MB 54.9MB/s 00:01
etcd 100% 22MB 56.4MB/s 00:00
etcdctl 100% 17MB 54.4MB/s 00:00
master-03
kubelet 100% 95MB 58.1MB/s 00:01
kubectl 100% 49MB 59.0MB/s 00:00
kube-apiserver 100% 108MB 54.1MB/s 00:01
kube-controller-manager 100% 102MB 54.4MB/s 00:01
kube-scheduler 100% 56MB 35.7MB/s 00:01
kube-proxy 100% 55MB 46.6MB/s 00:01
etcd 100% 22MB 47.1MB/s 00:00
etcdctl
[root@master-01 ~]# for NODE in $WorkNodes; do scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done
kubelet 100% 95MB 57.0MB/s 00:01
kube-proxy 100% 55MB 54.9MB/s 00:01
kubelet 100% 95MB 60.8MB/s 00:01
kube-proxy master01节点切换分支
[root@master-01 ~]# cd /root/k8s-ha-install && git checkout manual-installation-v1.30.x
Branch manual-installation-v1.30.x set up to track remote branch manual-installation-v1.30.x from origin.
Switched to a new branch 'manual-installation-v1.30.x'证书生成（谨慎操作）
master01节点下载证书生成工具，并授予执行权限（下载缓慢可以直接访问网址进行下载）
[root@master-01 ~]# wget "https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssl_linux-amd64" -O /usr/local/bin/cfssl
[root@master-01 ~]# wget "https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson
[root@master-01 ~]# chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson所有master节点创建etcd证书目录
[root@master-01 ~]# mkdir -p /etc/etcd/ssl所有节点创建kubernetes证书目录
[root@master-01 ~]# mkdir -p /etc/kubernetes/pkimaster01节点生成etcd CA证书和CA证书的密钥
[root@master-01 ~]# cd /root/k8s-ha-install/pki/
[root@master-01 pki]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca
2024/05/26 22:26:35 [INFO] generating a new CA key and certificate from CSR
2024/05/26 22:26:35 [INFO] generate received request
2024/05/26 22:26:35 [INFO] received CSR
2024/05/26 22:26:35 [INFO] generating key: rsa-2048
2024/05/26 22:26:35 [INFO] encoded CSR
2024/05/26 22:26:35 [INFO] signed certificate with serial number 138811720879307370144961486812110114884014831127
[root@master-01 pki]# ls /etc/etcd/ssl
etcd-ca.csr etcd-ca-key.pem etcd-ca.pem| 文件 | 解释 |
|---|---|
| etcd-ca.csr | 证书签名请求（Certificate Signing Request） |
| etcd-ca-key.pem | CA 私钥 |
| etcd-ca.pem | CA 证书 |
master01节点使用CA证书对Etcd证书进行签发
[root@master-01 pki]# cfssl gencert \
> -ca=/etc/etcd/ssl/etcd-ca.pem \
> -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
> -config=ca-config.json \
> -hostname=127.0.0.1,master-01,master-02,master-03,192.168.132.169,192.168.132.170,192.168.132.171 \
> -profile=kubernetes \
> etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd
2024/05/26 22:34:05 [INFO] generate received request
2024/05/26 22:34:05 [INFO] received CSR
2024/05/26 22:34:05 [INFO] generating key: rsa-2048
2024/05/26 22:34:05 [INFO] encoded CSR
2024/05/26 22:34:05 [INFO] signed certificate with serial number 59745135163596986953649756537535579125140135568
[root@master-01 pki]# ls /etc/etcd/ssl/
etcd-ca.csr etcd-ca-key.pem etcd-ca.pem etcd.csr etcd-key.pem etcd.pemmaster01节点将证书分发到其他master节点
[root@master-01 pki]# MasterNodes='master-02 master-03'
[root@master-01 pki]# for NODE in $MasterNodes; do
> ssh $NODE "mkdir -p /etc/etcd/ssl"
> for FILE in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem; do
> scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}
> done
> done
etcd-ca-key.pem 100% 1675 1.3MB/s 00:00
etcd-ca.pem 100% 1367 543.0KB/s 00:00
etcd-key.pem 100% 1675 1.7MB/s 00:00
etcd.pem 100% 1497 487.5KB/s 00:00
etcd-ca-key.pem 100% 1675 500.2KB/s 00:00
etcd-ca.pem 100% 1367 497.0KB/s 00:00
etcd-key.pem 100% 1675 845.1KB/s 00:00
etcd.pem master01节点生成kubernetesCA证书和CA证书的密钥
[root@master-01 pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/camaster01节点使用CA证书对kubernetes apiserver的证书进行签发（10.96.0.0/16是K8S service的网段，如果说需要更改K8S service网段，那就需要更改命令中的10.96.0.1参数，如果不是高可用集群，192.168.132.236需修改为master01节点的IP地址，通过指定多个主机名和 IP 地址，确保了证书可以在多种网络环境下正确验证）
[root@master-01 pki]# cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json -hostname=10.96.0.1,192.168.132.236,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,192.168.132.169,192.168.132.170,192.168.132.171 -profile=kubernetes apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver
2024/05/26 23:01:23 [INFO] generate received request
2024/05/26 23:01:23 [INFO] received CSR
2024/05/26 23:01:23 [INFO] generating key: rsa-2048
2024/05/26 23:01:23 [INFO] encoded CSR
2024/05/26 23:01:23 [INFO] signed certificate with serial number 322672452083631196538519798225527739153614469419master01节点生成apiserver聚合证书（警告提示缺乏hosts字段，不适用于网页，可以忽略提示）
[root@master-01 pki]# cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca
2024/05/26 23:07:05 [INFO] generating a new CA key and certificate from CSR
2024/05/26 23:07:05 [INFO] generate received request
2024/05/26 23:07:05 [INFO] received CSR
2024/05/26 23:07:05 [INFO] generating key: rsa-2048
2024/05/26 23:07:05 [INFO] encoded CSR
2024/05/26 23:07:05 [INFO] signed certificate with serial number 356915599476623614142101401642379897667782943133
You have new mail in /var/spool/mail/root
[root@master-01 pki]# cfssl gencert -ca=/etc/kubernetes/pki/front-proxy-ca.pem -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem -config=ca-config.json -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client
2024/05/26 23:07:26 [INFO] generate received request
2024/05/26 23:07:26 [INFO] received CSR
2024/05/26 23:07:26 [INFO] generating key: rsa-2048
2024/05/26 23:07:26 [INFO] encoded CSR
2024/05/26 23:07:26 [INFO] signed certificate with serial number 193948965250604211203560268473855762486375548809
2024/05/26 23:07:26 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").master01节点生成controller-manage证书
[root@master-01 pki]# cfssl gencert \
> -ca=/etc/kubernetes/pki/ca.pem \
> -ca-key=/etc/kubernetes/pki/ca-key.pem \
> -config=ca-config.json \
> -profile=kubernetes \
> manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager
2024/05/26 23:10:27 [INFO] generate received request
2024/05/26 23:10:27 [INFO] received CSR
2024/05/26 23:10:27 [INFO] generating key: rsa-2048
2024/05/26 23:10:28 [INFO] encoded CSR
2024/05/26 23:10:28 [INFO] signed certificate with serial number 177468267181248985462034971264524373695621703403
2024/05/26 23:10:28 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").master01节点生成一份关于controller-manage的kubeconfig文件，文件内包含了集群的名称、用户上下文关联到指定的集群和用户，可使用指定客户端证书和密钥的用户等选项（注意，如果不是高可用集群，192.168.132.236:8443改为master01的地址，8443改为apiserver的端口，默认是6443）
[root@master-01 pki]# kubectl config set-cluster kubernetes \
> --certificate-authority=/etc/kubernetes/pki/ca.pem \
> --embed-certs=true \
> --server=https://192.168.132.236:8443 \
> --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
Cluster "kubernetes" set.
[root@master-01 pki]# kubectl config set-context system:kube-controller-manager@kubernetes \
> --cluster=kubernetes \
> --user=system:kube-controller-manager \
> --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
Context "system:kube-controller-manager@kubernetes" created.
[root@master-01 pki]# kubectl config set-credentials system:kube-controller-manager \
--client-key=/etc/kubernetes/pki/controller-manager-key.pem \
> --client-certificate=/etc/kubernetes/pki/controller-manager.pem \
> --client-key=/etc/kubernetes/pki/controller-manager-key.pem \
> --embed-certs=true \
> --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
User "system:kube-controller-manager" set.
[root@master-01 pki]# kubectl config use-context system:kube-controller-manager@kubernetes \
> --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
Switched to context "system:kube-controller-manager@kubernetes".master01节点生成scheduler证书
[root@master-01 pki]# cfssl gencert \
> -ca=/etc/kubernetes/pki/ca.pem \
> -ca-key=/etc/kubernetes/pki/ca-key.pem \
> -config=ca-config.json \
> -profile=kubernetes \
> scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler
2024/05/26 23:24:41 [INFO] generate received request
2024/05/26 23:24:41 [INFO] received CSR
2024/05/26 23:24:41 [INFO] generating key: rsa-2048
2024/05/26 23:24:42 [INFO] encoded CSR
2024/05/26 23:24:42 [INFO] signed certificate with serial number 452803404300607432338568233091353835661262675031
2024/05/26 23:24:42 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").master01节点生成一份关于scheduler的kubeconfig文件
[root@master-01 pki]# kubectl config set-cluster kubernetes \
> --certificate-authority=/etc/kubernetes/pki/ca.pem \
> --embed-certs=true \
> --server=https://192.168.132.236:8443 \
> --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Cluster "kubernetes" set.
[root@master-01 pki]# kubectl config set-context system:kube-scheduler@kubernetes \
> --cluster=kubernetes \
> --user=system:kube-scheduler \
> --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Context "system:kube-scheduler@kubernetes" created.
[root@master-01 pki]# kubectl config set-credentials system:kube-scheduler \
> --client-certificate=/etc/kubernetes/pki/scheduler.pem \
> --client-key=/etc/kubernetes/pki/scheduler-key.pem \
> --embed-certs=true \
> --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
User "system:kube-scheduler" set.
[root@master-01 pki]# kubectl config use-context system:kube-scheduler@kubernetes \
> --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Switched to context "system:kube-scheduler@kubernetes".master01节点生成kubernetes-admin证书
[root@master-01 pki]# cfssl gencert \
> -ca=/etc/kubernetes/pki/ca.pem \
> -ca-key=/etc/kubernetes/pki/ca-key.pem \
> -config=ca-config.json \
> -profile=kubernetes \
> admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin
2024/05/27 21:37:05 [INFO] generate received request
2024/05/27 21:37:05 [INFO] received CSR
2024/05/27 21:37:05 [INFO] generating key: rsa-2048
2024/05/27 21:37:06 [INFO] encoded CSR
2024/05/27 21:37:06 [INFO] signed certificate with serial number 695925473107716579746577398805633207026353020531
2024/05/27 21:37:06 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").master01节点生成一份关于kubernetes-admin的kubeconfig文件
[root@master-01 pki]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.132.236:8443 --kubeconfig=/etc/kubernetes/admin.kubeconfig
Cluster "kubernetes" set.
[root@master-01 pki]# kubectl config set-context kubernetes-admin@kubernetes --cluster=kubernetes --user=kubernetes-admin --kubeconfig=/etc/kubernetes/admin.kubeconfig
Context "kubernetes-admin@kubernetes" created.
[root@master-01 pki]# kubectl config set-credentials kubernetes-admin --client-certificate=/etc/kubernetes/pki/admin.pem --client-key=/etc/kubernetes/pki/admin-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/admin.kubeconfig
User "kubernetes-admin" set.
[root@master-01 pki]# kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=/etc/kubernetes/admin.kubeconfig
Switched to context "kubernetes-admin@kubernetes".master01节点创建ServiceAccount（SA）的私钥
[root@master-01 pki]# openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
Generating RSA private key, 2048 bit long modulus
.......................+++
..+++
e is 65537 (0x10001)master01节点根据刚才创建的私钥生成公钥，用于配置K8S集群
[root@master-01 pki]# openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub
writing RSA keymaster01节点查看证书文件并检查证书数量
[root@master-01 pki]# ls /etc/kubernetes/pki/
admin.csr apiserver.csr ca.csr controller-manager.csr front-proxy-ca.csr front-proxy-client.csr sa.key scheduler-key.pem
admin-key.pem apiserver-key.pem ca-key.pem controller-manager-key.pem front-proxy-ca-key.pem front-proxy-client-key.pem sa.pub scheduler.pem
admin.pem apiserver.pem ca.pem controller-manager.pem front-proxy-ca.pem front-proxy-client.pem scheduler.csr
[root@master-01 pki]# ls /etc/kubernetes/pki/ | wc -l
23master01节点将证书分发到其他的master节点
[root@master-01 pki]# for NODE in master-02 master-03; do
> for FILE in $(ls /etc/kubernetes/pki); do
> scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE};
> done;
> for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do
> scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE};
> done;
> done
admin.csr 100% 1025 731.5KB/s 00:00
admin-key.pem 100% 1679 564.1KB/s 00:00
admin.pem 100% 1444 554.0KB/s 00:00
apiserver.csr 100% 1029 317.7KB/s 00:00
apiserver-key.pem 100% 1679 607.0KB/s 00:00
...省略部分输出...高可用配置
所有master节点安装KeepAlived和HAproxy工具
[root@master-01 ~]# yum install keepalived haproxy -y所有master节点配置HAproxy（所有master节点配置一样）
[root@master-01 ~]# vim /etc/haproxy/haproxy.cfg
global
maxconn 2000
ulimit-n 16384
log 127.0.0.1 local0 err
stats timeout 30s
defaults
log global
mode http
option httplog
timeout connect 5000
timeout client 50000
timeout server 50000
timeout http-request 15s
timeout http-keep-alive 15s
frontend k8s-master
bind 0.0.0.0:8443
bind 127.0.0.1:8443
mode tcp
option tcplog
tcp-request inspect-delay 5s
default_backend k8s-master
backend k8s-master
mode tcp
option tcplog
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server master-01 192.168.132.169:6443 check
server master-02 192.168.132.170:6443 check
server master-03 192.168.132.171:6443 check
所有master节点配置KeepAlived（但节点配置不一样，读者注意区分，不同点：网卡名称、节点IP地址、承担角色）
master01节点
[root@master-01 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state MASTER
interface ens33
mcast_src_ip 192.168.132.169
virtual_router_id 51
priority 101
nopreempt
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.132.236
}
track_script {
chk_apiserver
} }master02节点
[root@master-02 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
mcast_src_ip 192.168.132.170
virtual_router_id 51
priority 100
nopreempt
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.132.236
}
track_script {
chk_apiserver
} }master03节点
[root@master-03 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
mcast_src_ip 192.168.132.171
virtual_router_id 51
priority 100
nopreempt
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.132.236
}
track_script {
chk_apiserver
} }所有master节点编写健康检查脚本，并赋予执行权限
[root@master-01 ~]# vim /etc/keepalived/check_apiserver.sh
#!/bin/bash
err=0
for k in $(seq 1 3)
do
check_code=$(pgrep haproxy)
if [[ $check_code == "" ]]; then
err=$(expr $err + 1)
sleep 1
continue
else
err=0
break
fi
done
if [[ $err != "0" ]]; then
echo "systemctl stop keepalived"
/usr/bin/systemctl stop keepalived
exit 1
else
exit 0
fi
[root@master-01 ~]# chmod +x /etc/keepalived/check_apiserver.sh所有master节点启动KeepAlived和HAproxy服务
[root@master-01 ~]# systemctl daemon-reload
[root@master-01 ~]# systemctl enable --now haproxy
Created symlink from /etc/systemd/system/multi-user.target.wants/haproxy.service to /usr/lib/systemd/system/haproxy.service.
[root@master-01 ~]# systemctl enable --now keepalived
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.所有节点ping命令测试与VIP的连通性
[root@master-01 ~]# ping 192.168.132.236 -c 2
PING 192.168.132.236 (192.168.132.236) 56(84) bytes of data.
64 bytes from 192.168.132.236: icmp_seq=1 ttl=64 time=0.060 ms
64 bytes from 192.168.132.236: icmp_seq=2 ttl=64 time=0.052 ms
--- 192.168.132.236 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1012ms
rtt min/avg/max/mdev = 0.052/0.056/0.060/0.004 ms
[root@master-02 ~]# ping 192.168.132.236 -c 2
PING 192.168.132.236 (192.168.132.236) 56(84) bytes of data.
64 bytes from 192.168.132.236: icmp_seq=1 ttl=64 time=1.10 ms
64 bytes from 192.168.132.236: icmp_seq=2 ttl=64 time=0.498 ms
--- 192.168.132.236 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.498/0.802/1.106/0.304 ms
[root@node-01 ~]# ping 192.168.132.236 -c 2
PING 192.168.132.236 (192.168.132.236) 56(84) bytes of data.
64 bytes from 192.168.132.236: icmp_seq=1 ttl=64 time=0.603 ms
64 bytes from 192.168.132.236: icmp_seq=2 ttl=64 time=0.404 ms
--- 192.168.132.236 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.404/0.503/0.603/0.102 ms
...省略部分节点输出...所有节点telnet命令测试与VIP的连通性
[root@master-01 ~]# telnet 192.168.132.236 8443
Trying 192.168.132.236...
Connected to 192.168.132.236.
Escape character is '^]'.
Connection closed by foreign host.K8S组件配置
etcd配置（各master节点大致相同，读者需要注意主机名和IP地址）
master01节点
[root@master-01 ~]# vim /etc/etcd/etcd.config.yml
name: 'master-01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.132.169:2380'
listen-client-urls: 'https://192.168.132.169:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.132.169:2380'
advertise-client-urls: 'https://192.168.132.169:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'master-01=https://192.168.132.169:2380,master-02=https://192.168.132.170:2380,master-03=https://192.168.132.171:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
peer-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
peer-client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: falsemaster02节点
[root@master-02 ~]# vim /etc/etcd/etcd.config.yml
name: 'master-02'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.132.170:2380'
listen-client-urls: 'https://192.168.132.170:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.132.170:2380'
advertise-client-urls: 'https://192.168.132.170:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'master-01=https://192.168.132.169:2380,master-02=https://192.168.132.170:2380,master-03=https://192.168.132.171:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
peer-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
peer-client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: falsemaster03节点
[root@master-03 ~]# vim /etc/etcd/etcd.config.yml
name: 'master-03'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.132.171:2380'
listen-client-urls: 'https://192.168.132.171:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.132.171:2380'
advertise-client-urls: 'https://192.168.132.171:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'master-01=https://192.168.132.169:2380,master-02=https://192.168.132.170:2380,master-03=https://192.168.132.171:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
peer-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
peer-client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false所有master节点创建etcd service并启动
[root@master-01 ~]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target
[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
Alias=etcd3.service所有master节点创建etcd的证书目录并重新加载配置文件，设置服务开机自启
[root@master-01 ~]# mkdir /etc/kubernetes/pki/etcd
[root@master-01 ~]# ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/
[root@master-01 ~]# systemctl daemon-reload
[root@master-01 ~]# systemctl enable --now etcd
Created symlink from /etc/systemd/system/etcd3.service to /usr/lib/systemd/system/etcd.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
[root@master-01 ~]# ls /etc/kubernetes/pki/etcd/
etcd-ca.csr etcd-ca-key.pem etcd-ca.pem etcd.csr etcd-key.pem etcd.pem任意master节点查看etcd状态，指定etcdctl使用 v3 API 进行操作
[root@master-01 ~]# export ETCDCTL_API=3
[root@master-01 ~]# etcdctl --endpoints="192.168.132.171:2379,192.168.132.170:2379,192.168.132.169:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem endpoint status --write-out=table
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.132.171:2379 | 29f5c3be0672c3e5 | 3.5.13 | 25 kB | true | false | 2 | 8 | 8 | |
| 192.168.132.170:2379 | 65d0eb3943d30458 | 3.5.13 | 20 kB | false | false | 2 | 8 | 8 | |
| 192.168.132.169:2379 | 98e7ad14ffa67577 | 3.5.13 | 20 kB | false | false | 2 | 8 | 8 | |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+所有master节点创建kube-apiserver service，注意，K8S Service网段为10.96.0.0/16，该网段不能和宿主机的网段、Pod网段的重复，请按需修改
master01节点
[root@master-01 ~]# vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-apiserver \
--v=2 \
--allow-privileged=true \
--bind-address=0.0.0.0 \
--secure-port=6443 \
--advertise-address=192.168.132.169 \
--service-cluster-ip-range=10.96.0.0/16 \
--service-node-port-range=30000-32767 \
--etcd-servers=https://192.168.132.169:2379,https://192.168.132.170:2379,https://192.168.132.171:2379 \
--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
--etcd-certfile=/etc/etcd/ssl/etcd.pem \
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
--client-ca-file=/etc/kubernetes/pki/ca.pem \
--tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
--service-account-key-file=/etc/kubernetes/pki/sa.pub \
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
--authorization-mode=Node,RBAC \
--enable-bootstrap-token-auth=true \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
--requestheader-allowed-names=aggregator \
--requestheader-group-headers=X-Remote-Group \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-username-headers=X-Remote-User
# --token-auth-file=/etc/kubernetes/token.csv
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
[Install]
WantedBy=multi-user.targetmaster02节点
[root@master-02 ~]# vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-apiserver \
--v=2 \
--allow-privileged=true \
--bind-address=0.0.0.0 \
--secure-port=6443 \
--advertise-address=192.168.132.170 \
--service-cluster-ip-range=10.96.0.0/16 \
--service-node-port-range=30000-32767 \
--etcd-servers=https://192.168.132.169:2379,https://192.168.132.170:2379,https://192.168.132.171:2379 \
--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
--etcd-certfile=/etc/etcd/ssl/etcd.pem \
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
--client-ca-file=/etc/kubernetes/pki/ca.pem \
--tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
--service-account-key-file=/etc/kubernetes/pki/sa.pub \
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
--authorization-mode=Node,RBAC \
--enable-bootstrap-token-auth=true \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
--requestheader-allowed-names=aggregator \
--requestheader-group-headers=X-Remote-Group \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-username-headers=X-Remote-User
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
[Install]
WantedBy=multi-user.targetmaster03节点
[root@master-03 ~]# vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-apiserver \
--v=2 \
--allow-privileged=true \
--bind-address=0.0.0.0 \
--secure-port=6443 \
--advertise-address=192.168.132.171 \
--service-cluster-ip-range=10.96.0.0/16 \
--service-node-port-range=30000-32767 \
--etcd-servers=https://192.168.132.169:2379,https://192.168.132.170:2379,https://192.168.132.171:2379 \
--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
--etcd-certfile=/etc/etcd/ssl/etcd.pem \
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
--client-ca-file=/etc/kubernetes/pki/ca.pem \
--tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
--service-account-key-file=/etc/kubernetes/pki/sa.pub \
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
--authorization-mode=Node,RBAC \
--enable-bootstrap-token-auth=true \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
--requestheader-allowed-names=aggregator \
--requestheader-group-headers=X-Remote-Group \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-username-headers=X-Remote-User
# --token-auth-file=/etc/kubernetes/token.csv
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target所有master节点开启kube-apiserver服务
[root@master-01 ~]# systemctl daemon-reload && systemctl enable --now kube-apiserver
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.所有master节点查看kube-apiserver服务状态，此时出现RBAC报错可忽略
[root@master-03 ~]# systemctl status kube-apiserver -l
● kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2024-05-27 22:54:19 CST; 1min 58s ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 2343 (kube-apiserver)
Tasks: 8
Memory: 281.1M
CGroup: /system.slice/kube-apiserver.service
└─2343 /usr/local/bin/kube-apiserver --v=2 --allow-privileged=true --bind-address=0.0.0.0 --secure-port=6443 --advertise-address=192.168.132.171 --service-cluster-ip-range=10.96.0.0/16 --service-node-port-range=30000-32767 --etcd-servers=https://192.168.132.169:2379,https://192.168.132.170:2379,https://192.168.132.171:2379 --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem --etcd-certfile=/etc/etcd/ssl/etcd.pem --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem --client-ca-file=/etc/kubernetes/pki/ca.pem --tls-cert-file=/etc/kubernetes/pki/apiserver.pem --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-account-issuer=https://kubernetes.default.svc.cluster.local --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota --authorization-mode=Node,RBAC --enable-bootstrap-token-auth=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem --requestheader-allowed-names=aggregator --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-username-headers=X-Remote-User
May 27 22:54:26 master-03 kube-apiserver[2343]: [-]poststarthook/rbac/bootstrap-roles failed: not finished
May 27 22:54:26 master-03 kube-apiserver[2343]: I0527 22:54:26.836026 2343 healthz.go:255] poststarthook/rbac/bootstrap-roles check failed: readyz
May 27 22:54:26 master-03 kube-apiserver[2343]: [-]poststarthook/rbac/bootstrap-roles failed: not finished
May 27 22:54:26 master-03 kube-apiserver[2343]: I0527 22:54:26.935095 2343 healthz.go:255] poststarthook/rbac/bootstrap-roles check failed: readyz
May 27 22:54:26 master-03 kube-apiserver[2343]: [-]poststarthook/rbac/bootstrap-roles failed: not finished
May 27 22:54:27 master-03 kube-apiserver[2343]: I0527 22:54:27.034962 2343 healthz.go:255] poststarthook/rbac/bootstrap-roles check failed: readyz
May 27 22:54:27 master-03 kube-apiserver[2343]: [-]poststarthook/rbac/bootstrap-roles failed: not finished
May 27 22:54:27 master-03 kube-apiserver[2343]: W0527 22:54:27.156796 2343 lease.go:265] Resetting endpoints for master service "kubernetes" to [192.168.132.169 192.168.132.170 192.168.132.171]
May 27 22:54:27 master-03 kube-apiserver[2343]: I0527 22:54:27.159680 2343 controller.go:615] quota admission added evaluator for: endpoints
May 27 22:54:27 master-03 kube-apiserver[2343]: I0527 22:54:27.172352 2343 controller.go:615] quota admission added evaluator for: endpointslices.discovery.k8s.ioController Manager配置
所有master节点配置kube-controller-manager service（所有master节点配置一样）
[root@master-01 ~]# vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
--v=2 \
--root-ca-file=/etc/kubernetes/pki/ca.pem \
--cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
--service-account-private-key-file=/etc/kubernetes/pki/sa.key \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
--authentication-kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
--authorization-kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
--leader-elect=true \
--use-service-account-credentials=true \
--node-monitor-grace-period=40s \
--node-monitor-period=5s \
--controllers=*,bootstrapsigner,tokencleaner \
--allocate-node-cidrs=true \
--cluster-cidr=172.16.0.0/16 \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
--node-cidr-mask-size=24
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target所有master节点启动kube-controller-manager服务
[root@master-01 ~]# systemctl daemon-reload && systemctl enable --now kube-controller-manager
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.所有master节点查看kube-controller-manager服务启动状态
[root@master-01 ~]# systemctl status kube-controller-manager
● kube-controller-manager.service - Kubernetes Controller Manager
Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2024-05-27 23:06:27 CST; 59s ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 5363 (kube-controller)
Tasks: 6
Memory: 141.5M
CGroup: /system.slice/kube-controller-manager.service
└─5363 /usr/local/bin/kube-controller-manager --v=2 --root-ca-file=/etc/kubernetes/pki/ca.pem --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem --cluster-signing-key-file=/etc/kubernetes/...
May 27 23:06:43 master-01 kube-controller-manager[5363]: I0527 23:06:43.312942 5363 reflector.go:359] Caches populated for *v1.FlowSchema from k8s.io/client-go/informers/factory.go:160
May 27 23:06:43 master-01 kube-controller-manager[5363]: I0527 23:06:43.360962 5363 reflector.go:359] Caches populated for *v1.ValidatingAdmissionPolicyBinding from k8s.io/client-go/inform...ctory.go:160
May 27 23:06:43 master-01 kube-controller-manager[5363]: I0527 23:06:43.411667 5363 reflector.go:359] Caches populated for *v1.ClusterRoleBinding from k8s.io/client-go/informers/factory.go:160
May 27 23:06:43 master-01 kube-controller-manager[5363]: I0527 23:06:43.458915 5363 reflector.go:359] Caches populated for *v1.IngressClass from k8s.io/client-go/informers/factory.go:160
May 27 23:06:43 master-01 kube-controller-manager[5363]: I0527 23:06:43.509881 5363 reflector.go:359] Caches populated for *v1.RuntimeClass from k8s.io/client-go/informers/factory.go:160
May 27 23:06:43 master-01 kube-controller-manager[5363]: I0527 23:06:43.564492 5363 reflector.go:359] Caches populated for *v1.PriorityLevelConfiguration from k8s.io/client-go/informers/factory.go:160
May 27 23:06:43 master-01 kube-controller-manager[5363]: I0527 23:06:43.583770 5363 shared_informer.go:320] Caches are synced for garbage collector
May 27 23:06:43 master-01 kube-controller-manager[5363]: I0527 23:06:43.583820 5363 garbagecollector.go:281] "synced garbage collector" logger="garbage-collector-controller"
May 27 23:06:43 master-01 kube-controller-manager[5363]: I0527 23:06:43.590601 5363 shared_informer.go:320] Caches are synced for garbage collector
May 27 23:06:43 master-01 kube-controller-manager[5363]: I0527 23:06:43.590637 5363 garbagecollector.go:157] "All resource monitors have synced. Proceeding to collect garbage" logger="garb...-controller"
Hint: Some lines were ellipsized, use -l to show in full.Scheduler
所有master节点配置kube-scheduler service（所有master节点配置一样）
[root@master-03 ~]# vim /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-scheduler \
--v=2 \
--leader-elect=true \
--authentication-kubeconfig=/etc/kubernetes/scheduler.kubeconfig \
--authorization-kubeconfig=/etc/kubernetes/scheduler.kubeconfig \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target所有master节点启动kube-controller-manager服务，并查看服务启动状态
[root@master-01 ~]# systemctl daemon-reload && systemctl enable --now kube-scheduler
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/ systemd/system/kube-scheduler.service.
[root@master-01 ~]# systemctl status kube-scheduler
● kube-scheduler.service - Kubernetes Scheduler
Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2024-05-27 23:10:12 CST; 32s ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 5606 (kube-scheduler)
Tasks: 8
Memory: 72.0M
CGroup: /system.slice/kube-scheduler.service
└─5606 /usr/local/bin/kube-scheduler --v=2 --leader-elect=true --authentication-kubeconfig=/etc/kubernetes/scheduler.kubeconfig --authorization-kubeconfig=/etc/kubernetes/scheduler.kubeconfig ...
May 27 23:10:14 master-01 kube-scheduler[5606]: I0527 23:10:14.965113 5606 leaderelection.go:250] attempting to acquire leader lease kube-system/kube-scheduler...
May 27 23:10:14 master-01 kube-scheduler[5606]: I0527 23:10:14.982456 5606 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::reques...lient-ca-file
May 27 23:10:14 master-01 kube-scheduler[5606]: I0527 23:10:14.982576 5606 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
May 27 23:10:14 master-01 kube-scheduler[5606]: I0527 23:10:14.983362 5606 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-...
May 27 23:10:14 master-01 kube-scheduler[5606]: I0527 23:10:14.983846 5606 tlsconfig.go:200] "Loaded serving cert" certName="Generated self signed cert" certDetail="\"localhost@1716822614\" [serving] ...
May 27 23:10:14 master-01 kube-scheduler[5606]: I0527 23:10:14.984502 5606 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1...
May 27 23:10:14 master-01 kube-scheduler[5606]: I0527 23:10:14.984807 5606 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-...
May 27 23:10:14 master-01 kube-scheduler[5606]: I0527 23:10:14.984851 5606 tlsconfig.go:178] "Loaded client CA" index=1 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-...
May 27 23:10:14 master-01 kube-scheduler[5606]: I0527 23:10:14.985145 5606 tlsconfig.go:200] "Loaded serving cert" certName="Generated self signed cert" certDetail="\"localhost@1716822614\" [serving] ...
May 27 23:10:14 master-01 kube-scheduler[5606]: I0527 23:10:14.985381 5606 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1...
Hint: Some lines were ellipsized, use -l to show in full.TLS Bootstrapping
master01节点创建bootstrap
[root@master-01 ~]# cd /root/k8s-ha-install/bootstrap/
[root@master-01 bootstrap]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.132.236:8443 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
Cluster "kubernetes" set.
[root@master-01 bootstrap]# kubectl config set-credentials tls-bootstrap-token-user --token=c8ad9c.2e4d610cf3e7426e --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
User "tls-bootstrap-token-user" set.
[root@master-01 bootstrap]# kubectl config set-context tls-bootstrap-token-user@kubernetes --cluster=kubernetes --user=tls-bootstrap-token-user --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
Context "tls-bootstrap-token-user@kubernetes" created.
[root@master-01 bootstrap]# kubectl config use-context tls-bootstrap-token-user@kubernetes --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
Switched to context "tls-bootstrap-token-user@kubernetes".master01节点复制admin.kubeconfig文件
[root@master-01 bootstrap]# mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/configmaster01节点查询集群状态，如果出现错误，需要检查集群组件暴露的问题
[root@master-01 bootstrap]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy okmaster01节点创建资源
[root@master-01 bootstrap]# kubectl create -f bootstrap.secret.yaml
secret/bootstrap-token-c8ad9c created
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-certificate-rotation created
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver createdNode节点配置
master01节点复制证书到其余节点
[root@master-01 bootstrap]# cd /etc/kubernetes/
[root@master-01 kubernetes]# for NODE in master-02 master-03 node-01 node-02; do
> ssh $NODE mkdir -p /etc/kubernetes/pki
> for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do
> scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}
> done
> done
ca.pem 100% 1411 1.5MB/s 00:00
ca-key.pem 100% 1679 671.1KB/s 00:00
front-proxy-ca.pem 100% 1143 1.3MB/s 00:00
bootstrap-kubelet.kubeconfig 100% 2302 1.7MB/s 00:00
ca.pem 100% 1411 1.1MB/s 00:00
ca-key.pem 100% 1679 709.7KB/s 00:00
...省略部分输出...所有节点创建kubelet相关目录
[root@master-01 ~]# mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/所有节点配置kubelet service
[root@master-01 ~]# vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/local/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target所有节点配置kubelet service的子文件
[root@master-01 ~]# vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--container-runtime-endpoint=unix:///run/containerd/containerd.sock"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS所有节点创建kubelet的配置文件，如果更改了k8s的service网段，需要更改kubelet-conf.yml 的clusterDNS:配置，改成K8S Service网段的第十个地址，比如10.96.0.10
[root@master-01 ~]# vim /etc/kubernetes/kubelet-conf.yml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s所有节点启动kubelet服务
[root@master-01 ~]# systemctl daemon-reload && systemctl enable --now kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.所有节点查看当前消息，没有提示资源没找到或权限不足加入失败类的报错即可继续下一步
[root@master-01 ~]# tail /var/log/messages
May 28 00:01:13 master-01 kube-controller-manager: I0528 00:01:13.868424 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="master-01" timeStamp="2024-05-28 00:01:13.868405355 +0800 CST m=+3286.183445191"
May 28 00:01:13 master-01 kube-controller-manager: I0528 00:01:13.868486 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="master-03" timeStamp="2024-05-28 00:01:13.868480346 +0800 CST m=+3286.183520180"
May 28 00:01:13 master-01 kube-controller-manager: I0528 00:01:13.868518 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="node-02" timeStamp="2024-05-28 00:01:13.868514164 +0800 CST m=+3286.183553997"
May 28 00:01:13 master-01 kube-controller-manager: I0528 00:01:13.868540 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="master-02" timeStamp="2024-05-28 00:01:13.868536795 +0800 CST m=+3286.183576630"
May 28 00:01:13 master-01 kube-controller-manager: I0528 00:01:13.868561 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="node-01" timeStamp="2024-05-28 00:01:13.868557008 +0800 CST m=+3286.183596841"
May 28 00:01:18 master-01 kube-controller-manager: I0528 00:01:18.869259 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="master-03" timeStamp="2024-05-28 00:01:18.869225845 +0800 CST m=+3291.184265670"
May 28 00:01:18 master-01 kube-controller-manager: I0528 00:01:18.869347 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="node-02" timeStamp="2024-05-28 00:01:18.869338396 +0800 CST m=+3291.184378220"
May 28 00:01:18 master-01 kube-controller-manager: I0528 00:01:18.869376 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="master-02" timeStamp="2024-05-28 00:01:18.869372142 +0800 CST m=+3291.184411967"
May 28 00:01:18 master-01 kube-controller-manager: I0528 00:01:18.869397 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="node-01" timeStamp="2024-05-28 00:01:18.86939412 +0800 CST m=+3291.184433945"
May 28 00:01:18 master-01 kube-controller-manager: I0528 00:01:18.869466 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="master-01" timeStamp="2024-05-28 00:01:18.869456073 +0800 CST m=+3291.184495897"master01节点查看集群状态
[root@master-01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master-01 NotReady <none> 3m42s v1.30.1
master-02 NotReady <none> 3m44s v1.30.1
master-03 NotReady <none> 3m50s v1.30.1
node-01 NotReady <none> 3m44s v1.30.1
node-02 NotReady <none> 3m46s v1.30.1Kube-proxy
master01节点生成kube-proxy证书
[root@master-01 ~]# cd /root/k8s-ha-install/pki/
[root@master-01 pki]# cfssl gencert \
> -ca=/etc/kubernetes/pki/ca.pem \
> -ca-key=/etc/kubernetes/pki/ca-key.pem \
> -config=ca-config.json \
> -profile=kubernetes \
> kube-proxy-csr.json | cfssljson -bare /etc/kubernetes/pki/kube-proxy
2024/05/28 00:06:35 [INFO] generate received request
2024/05/28 00:06:35 [INFO] received CSR
2024/05/28 00:06:35 [INFO] generating key: rsa-2048
2024/05/28 00:06:36 [INFO] encoded CSR
2024/05/28 00:06:36 [INFO] signed certificate with serial number 456439669351705529547005512216275440161338961523
2024/05/28 00:06:36 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").master01节点生成一份关于kube-proxy的kubeconfig文件
[root@master-01 pki]# kubectl config set-cluster kubernetes \
> --certificate-authority=/etc/kubernetes/pki/ca.pem \
> --embed-certs=true \
> --server=https://192.168.132.236:8443 \
> --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
Cluster "kubernetes" set.
[root@master-01 pki]# kubectl config set-context system:kube-proxy@kubernetes \
> --cluster=kubernetes \
> --user=system:kube-proxy \
> --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
Context "system:kube-proxy@kubernetes" created.
[root@master-01 pki]# kubectl config set-credentials system:kube-proxy \
> --client-certificate=/etc/kubernetes/pki/kube-proxy.pem \
> --client-key=/etc/kubernetes/pki/kube-proxy-key.pem \
> --embed-certs=true \
> --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
User "system:kube-proxy" set.
[root@master-01 pki]# kubectl config use-context system:kube-proxy@kubernetes \
> --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
Switched to context "system:kube-proxy@kubernetes".
master01节点将kube-proxy.kubeconfig文件分发到其他节点
[root@master-01 pki]# for NODE in master-02 master-03 node-01 node-02; do
> scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig
> done
kube-proxy.kubeconfig 100% 6479 4.9MB/s 00:00
kube-proxy.kubeconfig 100% 6479 1.5MB/s 00:00
kube-proxy.kubeconfig 100% 6479 4.1MB/s 00:00
kube-proxy.kubeconfig 100% 6479 2.6MB/s 00:00所有节点添加kube-proxy service
[root@master-01 ~]# vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-proxy \
--config=/etc/kubernetes/kube-proxy.yaml \
--v=2
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target所有节点添加kube-proxy配置文件
[root@master-01 ~]# vim /etc/kubernetes/kube-proxy.yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
acceptContentTypes: ""
burst: 10
contentType: application/vnd.kubernetes.protobuf
kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
qps: 5
clusterCIDR: 172.16.0.0/16
configSyncPeriod: 15m0s
conntrack:
max: null
maxPerCore: 32768
min: 131072
tcpCloseWaitTimeout: 1h0m0s
tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
masqueradeAll: false
masqueradeBit: 14
minSyncPeriod: 0s
syncPeriod: 30s
ipvs:
masqueradeAll: true
minSyncPeriod: 5s
scheduler: "rr"
syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms所有节点启动kube-proxy并设置服务开机自启
[root@master-01 ~]# systemctl daemon-reload && systemctl enable --now kube-proxy
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.所有节点再次查看日志信息，如果无其他资源类的错误，安装完网络插件后，报错会消失，读者无需理会，如果出现权限和资源类的错误，读者需要检查前面的证书配置是否有问题
[root@master-01 ~]# tail /var/log/messages
May 28 00:13:34 master-01 kube-controller-manager: I0528 00:13:34.033092 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="master-01" timeStamp="2024-05-28 00:13:34.033087502 +0800 CST m=+4026.348127337"
May 28 00:13:34 master-01 kube-controller-manager: I0528 00:13:34.033020 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="master-02" timeStamp="2024-05-28 00:13:34.033006293 +0800 CST m=+4026.348046115"
May 28 00:13:34 master-01 kube-controller-manager: I0528 00:13:34.033102 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="master-03" timeStamp="2024-05-28 00:13:34.033098225 +0800 CST m=+4026.348138055"
May 28 00:13:36 master-01 kubelet: E0528 00:13:36.436605 8732 kubelet.go:2900] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
May 28 00:13:39 master-01 kube-controller-manager: I0528 00:13:39.033806 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="master-03" timeStamp="2024-05-28 00:13:39.033790281 +0800 CST m=+4031.348830107"
May 28 00:13:39 master-01 kube-controller-manager: I0528 00:13:39.033880 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="node-02" timeStamp="2024-05-28 00:13:39.033873662 +0800 CST m=+4031.348913490"
May 28 00:13:39 master-01 kube-controller-manager: I0528 00:13:39.033906 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="master-02" timeStamp="2024-05-28 00:13:39.033903062 +0800 CST m=+4031.348942889"
May 28 00:13:39 master-01 kube-controller-manager: I0528 00:13:39.033924 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="node-01" timeStamp="2024-05-28 00:13:39.033921655 +0800 CST m=+4031.348961483"
May 28 00:13:39 master-01 kube-controller-manager: I0528 00:13:39.033953 5363 node_lifecycle_controller.go:787] "Node is NotReady. Adding it to the Taint queue" logger="node-lifecycle-controller" node="master-01" timeStamp="2024-05-28 00:13:39.033950485 +0800 CST m=+4031.348990311"
May 28 00:13:41 master-01 kubelet: E0528 00:13:41.437477 8732 kubelet.go:2900] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"Calico
master01节点切换目录，并修改文件内calico网段（网段需要根据实际的Pod网段进行修改）
[root@master-01 ~]# cd /root/k8s-ha-install/calico/
[root@master-01 calico]# sed -i "s#POD_CIDR#172.16.0.0/16#g" calico.yaml
[root@master-01 calico]# grep "IPV4POOL_CIDR" calico.yaml -A 1
- name: CALICO_IPV4POOL_CIDR
value: "172.16.0.0/16"master01节点创建calico资源
[root@master-01 calico]# kubectl apply -f calico.yaml
poddisruptionbudget.policy/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
serviceaccount/calico-cni-plugin created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpfilters.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrole.rbac.authorization.k8s.io/calico-cni-plugin created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-cni-plugin created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers createdmaster01节点查看容器运行状态（小心网络问题，附带错误日志）
containerd
May 28 21:22:35 master-01 containerd[1101]: time="2024-05-28T21:22:35.005398224+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:calico-node-vfp28,Uid:9fefa741-080d-4c70-b...om/google_cont
May 28 21:22:48 master-01 containerd[1101]: time="2024-05-28T21:22:48.993941322+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:calico-node-vfp28,Uid:9fefa741-080d-4c70-bb...m,Attempt:0,}"
May 28 21:22:49 master-01 containerd[1101]: time="2024-05-28T21:22:49.004771859+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://registry.cn-hangzhou.aliyuncs.com/v2/g...
May 28 21:22:49 master-01 containerd[1101]: time="2024-05-28T21:22:49.005685498+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:calico-node-vfp28,Uid:9fefa741-080d-4c70-b...om/google_cont
kube-controller
May 28 21:05:22 master-01 kube-controller-manager[1097]: E0528 21:05:22.098239 1097 reflector.go:150] runtime/asm_amd64.s:1695: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https:...
kube-scheduler
May 28 21:17:44 master-01 kube-scheduler[1085]: I0528 21:17:44.215197 1085 schedule_one.go:1049] "Unable to schedule pod; no fit; waiting" pod="kube-system/calico-kube-controllers-744d658cd-6mhns" err="0/5 nodes are available: 5 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling."
describe pods
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m37s default-scheduler 0/5 nodes are available: 5 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.[root@master-01 calico]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-744d658cd-6mhns 0/1 Terminating 0 9m42s
calico-kube-controllers-744d658cd-kt6p2 0/1 ContainerCreating 0 21s
calico-node-5fkq7 0/1 Init:2/3 0 22s
calico-node-5spl2 0/1 Init:2/3 0 21s
calico-node-9549x 0/1 Init:2/3 0 21s
calico-node-dxpkh 0/1 Init:2/3 0 21s
calico-node-rjqtv 0/1 Init:2/3 0 21s
[root@master-01 calico]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-744d658cd-kt6p2 0/1 ContainerCreating 0 114s
calico-node-5fkq7 1/1 Running 0 115s
calico-node-5spl2 1/1 Running 0 114s
calico-node-9549x 1/1 Running 0 114s
calico-node-dxpkh 1/1 Running 0 114s
calico-node-rjqtv 1/1 Running 0 114s
[root@master-01 calico]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-744d658cd-kt6p2 1/1 Running 0 118s
calico-node-5fkq7 1/1 Running 0 119s
calico-node-5spl2 1/1 Running 0 118s
calico-node-9549x 1/1 Running 0 118s
calico-node-dxpkh 1/1 Running 0 118s
calico-node-rjqtv 1/1 Running 0 118smaster01节点查看容器内日志命令示例
1. Kubectl logs -f POD_NAME -n kube-system
2. Kubectl logs -f POD_NAME -c upgrade-ipam -n kube-system CoreDNS
master01节点切换目录，修改CoreDNS配置文件
[root@master-01 ~]# cd /root/k8s-ha-install/CoreDNS/
[root@master-01 CoreDNS]# COREDNS_SERVICE_IP=`kubectl get svc | grep kubernetes | awk '{print $3}'`0
[root@master-01 CoreDNS]# echo $COREDNS_SERVICE_IP
10.96.0.10
[root@master-01 CoreDNS]# sed -i "s#KUBEDNS_SERVICE_IP#${COREDNS_SERVICE_IP}#g" coredns.yaml
[root@master-01 CoreDNS]# cat coredns.yaml | grep 10.96.0.10 -A 2 -B 2
k8s-app: kube-dns
app.kubernetes.io/name: coredns
clusterIP: 10.96.0.10
ports:
- name: dnsmaster01节点创建CoreDNS资源
[root@master-01 CoreDNS]# kubectl create -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns createdmaster01节点查看CoreDNS容器运行状态
[root@master-01 CoreDNS]# kubectl get pods -n kube-system | grep coredns
coredns-5664d5ddc6-2xxbm 1/1 Running 0 38sMetrics Server
master01节点安装Metrics Server
[root@master-01 CoreDNS]# cd /root/k8s-ha-install/metrics-server/
[root@master-01 metrics-server]# ls
comp.yaml
[root@master-01 metrics-server]# kubectl create -f comp.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io createdmaster01节点查看Metrics Server容器启动状态
[root@master-01 metrics-server]# kubectl get pods -n kube-system | grep metrics
metrics-server-959599786-dtcfk 1/1 Running 0 94s剩余的集群可用性验证请读者参考kubeadm章节完成测试，此处不再赘述。