Centos7.9 arm架构搭建 k8s v1.24.0 (超新)

由于现在使用的是m1电脑,导致虚拟机安装都是 arm架构的, 正好今年2022.5.3 k8s发布了最新的 1.24 版本的k8s,有一些改动最重要的是 删除 Dockershim, 正好在arm架构下尝试部署一下最新的 1.24版本 , 里面的坑还是比较多的.

虚拟机准备

如果你也在使用m1的电脑, 可以参考我上一篇的 m1 安装 centos7.9 arm 虚拟机,我这里准备了3台虚拟机信息如下:

ip	主机名	内存	硬盘	系统(arm)
172.16.225.100	k8s-master	8G	100G	centos7.9
172.16.225.101	k8s-node1	8G	100G	centos7.9
172.16.225.102	K8s-node2	8G	100G	centos7.9

1. 3台节点前置准备

一下配置都要在 3台节点上进行一些前置准备

1.1 arm centos7 yum 镜像源配置

注意要下载 arm 源

https://developer.aliyun.com/mirror/centos?spm=a2c6h.13651102.0.0.3e221b114IVHmo

wget http://mirrors.aliyun.com/repo/Centos-altarch-7.repo -O /etc/yum.repos.d/CentOS-Base.repo

yum clean all

yum makecache #重新生成缓存 

#可以尝试yum 安装一个 比如 lrzsz 看看是否报错

yum配置好后可以尝试yum 安装一个比如 lrzsz 看看是否报错,

如果报GPG签名错误,把 CentOS-Base.repo 中的gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-7-aarch64 改成这个官方的, 我这里确实报错了,不知道阿里后续会不会修复

1.2 /etc/hosts配置

1.3 hostname设置

hostnamectl set-hostname k8s-master #master节点上执行
hostnamectl set-hostname k8s-node1  #node1 节点上执行
hostnamectl set-hostname k8s-node2 #node2 节点上执行

1.4 防火墙关闭

systemctl stop firewalld #停止
systemctl disable firewalld #开启不启动

1.5 swap分区关闭

swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab

1.6 iptables 检查桥接流量

iptables

所有节点都需要执行此操作
加载 br_netfilter 模块以启用内核 ipv4 转发，1.20+强依赖该模块
ip_conntrack 模块来记录iptables 网络包的状态，并把每条记录保存到table 里

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# 设置所需的 sysctl 参数，参数在重新启动后保持不变  添加网桥过滤及内核转发配置文件
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# 应用 sysctl 参数而不重新启动
sudo sysctl --system

1.7 selinux 关闭

setenforce 0 #临时关闭
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config #永久关闭

1.8 时间同步

简单点直接如下

crontab -e" #后添加下面内容 保存

0 */1 * * * /usr/sbin/ntpdate time1.aliyun.com

1.9 设置时区

#设置上海时区，东八区
timedatectl set-timezone Asia/Shanghai*

1.10 ipset ipvsadm 配置

yum -y install ipset ipvsadm #安装 ipset ipvsadm
 

cat > /etc/sysconfig/modules/ipvs.module <<EOF
modprobe -- ip_vs
modprobe -- ip_vs_sh
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- nf_conntrack
EOF

1.11 安装必要工具 yum-utils device-mapper-persistent-data lvm2

yum install -y yum-utils device-mapper-persistent-data lvm2

2. 3台节点安装 docker 并且配置

2.1 安装docker

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

yum install docker-ce

systemctl enable --now docker
systemctl start docker

2.2 配置 crgoup 和 mirrors


sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://9o4i1clo.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF


systemctl enable docker && systemctl restart docker #重启

3. 3台节点安装 cri-dockerd (替代 Dockershim 垫片)

3.1 下载安装 cri-dockerd

 tar -xf cri-dockerd-0.2.5.arm64.tgz  # 注意 arm架构需要下载 arm架构的
 cp cri-dockerd/cri-dockerd /usr/bin/
 chmod +x /usr/bin/cri-dockerd

3.2 配置 cri-dockerd

# 配置启动文件

cat <<"EOF" > /usr/lib/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket

[Service]
Type=notify

ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7

ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

StartLimitBurst=3

StartLimitInterval=60s

LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target

EOF





# 生成socket 文件

cat <<"EOF" > /usr/lib/systemd/system/cri-docker.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service

[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

EOF

3.3 启动cri-docker

systemctl daemon-reload
systemctl start cri-docker
systemctl enable cri-docker
systemctl status cri-docker

4. 配置kubernetes.repo 阿里镜像源

4.1 3台节点配置kubernetes.repo

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-aarch64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum makecache 失败

修改kubernetes镜像源，使用国内的阿里云，验证的GPG key验证问题，验证不了，所以报错。

解决:Github上找到解决方式：编辑 /etc/yum.repos.d/kubernetes.repo 文件修改一下参数：repo_gpgcheck=0然后重试yum update，就解决了。这边是禁用了GPG key的验证机制。

再次执行就成功了

4.2 3台节点安装 kubeadm kubectl kubelet 并启用

# 下载安装 1.24.0版本
yum -y install kubeadm-1.24.0 kubectl-.1.24.0 kubelet-1.24.0 

#启用 kubelet
systemctl enable kubelet

或者你可以直接去下载并且指定 nogpgcheck

Tips:

# 查看所有的可用版本
yum list kubeadm kubelet kubectl --showduplicates | sort -r

5. master节点配置kubeadm.yaml 文件

5.1 主节点执行初始化 kubeadm.yaml 文件

可以通过初始化配置文件然后修改编辑配置文件进行 kubeadm init ,比较清晰

#初始化 kubeadm.yaml配置文件
kubeadm config print init-defaults > kubeadm.yaml

5.2 修改 kubeadm.yaml 配置文件

修改advertiseAddress 指定master节点ip
修改criSocket 指定 cri-dockerd.sock #指定上面安装的cri-dockerd
修改imageRepository registry.aliyuncs.com/google_containers #配置阿里云镜像仓库
修改 podSubnet: 10.224.0.0/16 # pod子网 #和网络插件要保持一致
name: k8s-master #主节点名称

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 172.16.225.100  #指定主master的 ip
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/cri-dockerd.sock  #cri-dockerd
  imagePullPolicy: IfNotPresent
  name: k8s-master  #主节点
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers #配置 阿里云镜像仓库
kind: ClusterConfiguration
kubernetesVersion: 1.24.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.224.0.0/16 # pod子网  #和 网络插件要保持一致
scheduler: {}


---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

5.3 拉取镜像

提前拉取镜像方便待会 kubeadm init 更快一点

kubeadm config images list --config kubeadm.yaml #查看 需要下载的镜像

kubeadm config images pull --config kubeadm.yaml # 根据指定的配置文件 下载所需要的镜像

6. kubeadm init 初始化 k8s集群

当前面都配置好后终于可以开始执行初始化集群操作啦!!! 恭喜

6.1 kubeadm init

kubeadm init --config=kubeadm.yaml

[root@localhost ~]# kubeadm init --config=kubeadm.yaml
[init] Using Kubernetes version: v1.24.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.225.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [172.16.225.100 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [172.16.225.100 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 4.502711 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.225.100:6443 --token abcdef.0123456789abcdef 
  --discovery-token-ca-cert-hash sha256:cccfbed3011b4a1f107e6c355bafb7929df7a43a14babc0c5496030d3c225df3

6.2 node 节点添加进集群

`如果需要重置 需要在对应的node节点上执行 ,我这里是 因为hostname忘了设置导致`
#先删除 
kubectl delete node localhost.localdomain

#reset 的时候 一定要指定--cri-socket  这是新版本的要求
kubeadm reset --cri-socket unix:///var/run/cri-dockerd.sock

reset 的时候一定要指定–cri-socket 这是新版本的要求

7. calico 集群网络准备 (每个节点都要执行)

我这里选择了 calico

7.1 calico安装

#下载 calico.yaml
wget https://docs.projectcalico.org/manifests/calico.yaml --no-check-certificate

#编辑 calico 修改 CALICO_IPV4POOL_CIDR 为上面 podSubnet: 10.224.0.0/16 # pod子网  #和 网络插件要保持一致
vim  calico.yaml 
................
  name: CALICO_IPV4POOL_CIDR
  value: "10.244.0.0/16"
................

[root@kubesphere ~]# kubectl apply -f calico.yaml 
监视kube-system命名空间中pod运行情况
[root@k8s-master01 ~]# watch kubectl get pods -n kube-system

7.2 验证集群可用性

#查看所有的节点

[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
k8s-master01   Ready    control-plane   28m   v1.24.1
k8s-node1      Ready    <none>          26m   v1.24.1
k8s-node2      Ready    <none>          26m   v1.24.1
查看集群健康情况
[root@k8s-master01 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0

搞个 nginx 验证成功

8. node 节点执行 kubectl 命令报错

kubectl 命令报错：
[root@node01 ~]# kubectl get nodes
The connection to the server localhost:8080 was refused - did you specify the right host or port?

`报错原因：该节点未对kubernetes进行权限配置

解决办法：
`1、拷贝master01节点的/etc/kubernetes/admin.conf配置文件到/etc/kubernetes/目录下
[root@node01 ~]# scp /etc/kubernetes/admin.conf master01:/etc/kubernetes/
`2、执行如下命令：
To start using your cluster, you need to run the following as a regular user:

[root@node01 ~]# mkdir -p $HOME/.kube
[root@node01 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@node01 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

3、配置成功
[root@node01 ~]# kubectl get nodes
NAME       STATUS   ROLES                  AGE     VERSION
master01   Ready    control-plane,master   5h47m   v1.20.6
node01     Ready    <none>                 5h36m   v1.20.6

总结

至此已经在我的m1的虚拟机中 arm架构下把最新版v1.24.0 k8s 搭建成功,

介绍了k8s 新版本中Dockershim 的替代品 cri-dockerd 的安装(替代 Dockershim 垫片 )

有一些坑需要注意:

yum配置好后可以尝试yum 安装一个比如 lrzsz 看看是否报错, 如果报GPG签名错误,把 CentOS-Base.repo 中的gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-7-aarch64 改成这个官方的, 我这里确实报错了,不知道阿里后续会不会修复
修改kubernetes镜像源，使用国内的阿里云，验证的GPG key验证问题，验证不了，所以报错 ,

编辑 /etc/yum.repos.d/kubernetes.repo 文件修改一下参数：repo_gpgcheck=0然后重试yum update，就解决了。这边是禁用了GPG key的验证机制。
如果需要重置需要在对应的node节点上执行 ,我这里是因为hostname忘了设置导致先删除 kubectl delete node localhost.localdomain

reset 的时候一定要指定--cri-socket 这是新版本的要求kubeadm reset –cri-socket unix:///var/run/cri-dockerd.sock

Mac M1 上面虚拟机搭建 k8s v 1.19.3 集群命令参考

基于Kubernetes v1.24.0的集群搭建（二）

ARM(aarch64)CentOS阿里源更新报签名校验错误

node 节点执行 kubectl 命令报错

centos 阿里云的arm 配置的GPG签名错误需要改成 https://www.centos.org/keys/RPM-GPG-KEY-CentOS-7-aarch64

# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the 
# remarked out baseurl= line instead.
#
#
 
[base]
name=CentOS-$releasever - Base
baseurl=https://mirrors.aliyun.com/centos-altarch/$releasever/os/$basearch/
gpgcheck=1
gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-7-aarch64
 
#released updates 
[updates]
name=CentOS-$releasever - Updates
baseurl=https://mirrors.aliyun.com/centos-altarch/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-7-aarch64
 
#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
baseurl=https://mirrors.aliyun.com/centos-altarch/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-7-aarch64
enabled=1
 
#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
baseurl=https://mirrors.aliyun.com/centos-altarch/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-7-aarch64