关于Kubernetes中kube-apiserver使用token、kubeconfig认证的一些笔记

写在前面

学习K8s涉及，整理笔记记忆
博文偏实战，内容涉及：

token方式的API Server认证Demo
Kubeconfig文件方式的API Server认证Demo
Kubeconfig文件的创建Demo

只有能做到“尽人事而听天命”，一个人才能永远保持心情的平衡。—– 《季羡林谈人生》

API Server认证管理

Kubernetes集群中所有资源的访问和变更都是通过Kubernetes API Server的REST API来实现的,所以集群安全的关键点就在于如何鉴权和授权

一个简单的Demo,在master节点上,我们通过root用户可以直接通kubectl来请求API Service从而获取集群信息,但是我们通过其他用户登录就没有这个权限,这就涉及到k8s的一个认证问题.

root用户可以正常访问

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$kubectl  get pods
NAME                                                  READY   STATUS        RESTARTS       AGE
liruilong-grafana-5955564c75-zpbjq                    3/3     Terminating   0              8h
liruilong-kube-prometheus-operator-5cb699b469-fbkw5   1/1     Terminating   0              8h
liruilong-prometheus-node-exporter-vm7s9              1/1     Terminating   2 (109m ago)   8h
prometheus-liruilong-kube-prometheus-prometheus-0     2/2     Terminating   0              8h
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$

切换tom用户来访问，没有权限，报错找不到集群ＡＰＩ的位置，那么为什么会这样呢？

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$su tom
[tom@vms81 k8s-helm-create]$ kubectl  get pods
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[tom@vms81 k8s-helm-create]$ exit
exit

为了演示认证，我们需要在集群外的机器上安装一个客户端工具kubectl，用于和集群的入口api-Service交互

┌──[root@liruilongs.github.io]-[~]
└─$ yum install -y kubectl-1.22.2-0 --disableexcludes=kubernetes

可以通过kubectl cluster-info来查看集群的相关信息

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$kubectl cluster-info
Kubernetes control plane is running at https://192.168.26.81:6443
CoreDNS is running at https://192.168.26.81:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://192.168.26.81:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$

Kubernetes集群提供了3种级别的客户端身份认证方式

HTTP Token认证：通过一个Token来识别合法用户。
HTTPS 证书认证：基于CA根证书签名的双向数字证书认证方式
HTTP Base认证：通过用户名+密码的方式认证，这个只有1.19之前的版本适用，之后的版本不在支持

下面就Token和SSL和小伙伴分享下，Bash因为在高版本的K8s中不在支持，所以我们这里不聊。关于上面的普通用户访问集群的问题，我们也会解答

HTTP Token认证

HTTP Token的认证是用一个很长的特殊编码方式的并且难以被模仿的字符串Token来表明客户身份的一种方式。

每个Token对应一个用户名,存储在APIServer能访问的一个文件中。当客户端发起API调用请求时,需要在HTTP Header里放入Token,这样一来, API Server就能识别合法用户和非法用户了。

当 API 服务器的命令行设置了--token-auth-file=SOMEFILE选项时，会从文件中读取持有者令牌。目前，令牌会长期有效，并且在不重启 API 服务器的情况下无法更改令牌列表。下面我们一个通过Demo来演示通过静态Token的用户认证,

通过openssl生成一个令牌

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$openssl rand -hex 10
4bf636c8214b7ff0a0fb

令牌文件是一个 CSV 文件，包含至少 3 个列：令牌、用户名和用户的 UID。其余列被视为可选的组名。这里需要注意的是，令牌文件要放到/etc/kubernetes/pki目录下才可以，可能默认读取令牌的位置即是这个位置

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$echo "4bf636c8214b7ff0a0fb,admin2,3" > /etc/kubernetes/pki/liruilong.csv
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$cat /etc/kubernetes/pki/liruilong.csv
4bf636c8214b7ff0a0fb,admin2,3

通过Sed添加kube-apiserver服务启动参数，- --token-auth-file=/etc/kubernetes/pki/liruilong.csv

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$sed  '17a     - --token-auth-file=/etc/kubernetes/pki/liruilong.csv' /etc/kubernetes/manifests/kube-apiserver.yaml | grep -A 5  command
  - command:
    - kube-apiserver
    - --advertise-address=192.168.26.81
    - --allow-privileged=true
    - --token-auth-file=/etc/kubernetes/liruilong.csv
    - --authorization-mode=Node,RBAC
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$sed -i  '17a     - --token-auth-file=/etc/kubernetes/pki/liruilong.csv' /etc/kubernetes/manifests/kube-apiserver.yaml

检查修改的启动参数

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$cat -n /etc/kubernetes/manifests/kube-apiserver.yaml | grep -A 5  command
    14    - command:
    15      - kube-apiserver
    16      - --advertise-address=192.168.26.81
    17      - --allow-privileged=true
    18      - --token-auth-file=/etc/kubernetes/pki/liruilong.csv
    19      - --authorization-mode=Node,RBAC
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$

重启kubelet服务

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$systemctl restart kubelet
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]
└─$

确认集群能够正常访问

┌──[root@vms81.liruilongs.github.io]-[/etc/kubernetes/pki]
└─$kubectl get nodes
NAME                         STATUS     ROLES                  AGE   VERSION
vms81.liruilongs.github.io   Ready      control-plane,master   34d   v1.22.2
vms82.liruilongs.github.io   Ready      <none>                 34d   v1.22.2
vms83.liruilongs.github.io   NotReady   <none>                 34d   v1.22.2
┌──[root@vms81.liruilongs.github.io]-[/etc/kubernetes/pki]
└─$

在集群外的客户机访问集群信息，这里提示我们admin2用户没有访问的权限，说明已经认证成功了，只是没有权限

┌──[root@liruilongs.github.io]-[~]
└─$ kubectl -s="https://192.168.26.81:6443" --insecure-skip-tls-verify=true  --token="4bf636c8214b7ff0a0fb" get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "admin2" cannot list resource "pods" in API group "" in the namespace "kube-system"
┌──[root@liruilongs.github.io]-[~]
└─$

这里我们修改一些token的字符串，Token和集群的Token文件不对应，会提示我们没有获得授权，即认证失败

┌──[root@liruilongs.github.io]-[~]
└─$ kubectl -s="https://192.168.26.81:6443" --insecure-skip-tls-verify=true  --token="4bf636c8214b7ff0a0f" get pods -n kube-system
error: You must be logged in to the server (Unauthorized)

kubeconfig文件认证

在回到我们之前的那个问题，为什么使用root用户可以访问集群信息，但是通过tom用户去不能够访问集群信息，这里就涉及到一个kubeconfig 文件认证的问题

在通过kubeadm创建集群的时候，不知道小伙伴没还记不记得下面这个文件admin.conf，这个文件就是kubeadm帮我们生成的kubeconfig文件

┌──[root@vms81.liruilongs.github.io]-[~/.kube]
└─$ll /etc/kubernetes/admin.conf
-rw------- 1 root root 5676 12月 13 02:13 /etc/kubernetes/admin.conf
┌──[root@vms81.liruilongs.github.io]-[~/.kube]
└─$

我们把这个文件拷贝到tom用户的目录下，修改权限

┌──[root@vms81.liruilongs.github.io]-[~/.kube]
└─$cp /etc/kubernetes/admin.conf   ~tom/
┌──[root@vms81.liruilongs.github.io]-[~/.kube]
└─$chown tom:tom ~tom/admin.conf

这个时候发现通过 --kubeconfig=admin.conf 指定这个文件，就可以访问集群信息

[tom@vms81 home]$ cd tom/
[tom@vms81 ~]$ ls
admin.conf
[tom@vms81 ~]$ kubectl get pods
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[tom@vms81 ~]$ kubectl get pods -A --kubeconfig=admin.conf
NAMESPACE                  NAME                                                  READY   STATUS             RESTARTS          AGE
ingress-nginx              ingress-nginx-controller-744d4fc6b7-t9n4l             1/1     Running            6 (8h ago)        44h
kube-system                calico-kube-controllers-78d6f96c7b-85rv9              1/1     Running            193               31d
kube-system                calico-node-6nfqv                                     1/1     Running            254               34d
kube-system                calico-node-fv458                                     0/1     Running            50                34d
kube-system                calico-node-h5lsq                                     1/1     Running            94 (7h10m ago)    34d
kube-system       ..........................

那个，kubeconfig文件是个什么东西，官方文档中这样描述：

使用 kubeconfig 文件来组织有关集群、用户、命名空间和身份认证机制的信息。kubectl 命令行工具使用 kubeconfig 文件来查找选择集群所需的信息，并与集群的 API 服务器进行通信。

换句话讲，通过kubeconfig与集群的 API 服务器进行通信，类似上面的Token的作用，我们要说的HTTPS证书认证就是放到这里

默认情况下，kubectl 在 $HOME/.kube 目录下查找名为 config 的文件。

┌──[root@vms81.liruilongs.github.io]-[~]
└─$ls ~/.kube/config
/root/.kube/config
┌──[root@vms81.liruilongs.github.io]-[~]
└─$ll ~/.kube/config
-rw------- 1 root root 5663 1月  16 02:33 /root/.kube/config

将kubeconfig文件复制到 $HOME/.kube 目录下改名为 config 发现tom用户依旧可以访问

[tom@vms81 ~]$ ls
admin.conf
[tom@vms81 ~]$ cp admin.conf  .kube/config
[tom@vms81 ~]$ kubectl get pods -n kube-system
NAME                                                 READY   STATUS    RESTARTS          AGE
calico-kube-controllers-78d6f96c7b-85rv9             1/1     Running   193               31d
calico-node-6nfqv                                    1/1     Running   254               34d
calico-node-fv458                                    0/1     Running   50                34d
calico-node-h5lsq                                    1/1     Running   94 (7h13m ago)    34d
。。。。。。。

也可以通过设置 KUBECONFIG 环境变量或者设置 --kubeconfig参数来指定其他kubeconfig文件。

[tom@vms81 ~]$ export KUBECONFIG=admin.conf
[tom@vms81 ~]$ kubectl get pods -n kube-system
NAME                                                 READY   STATUS    RESTARTS          AGE
calico-kube-controllers-78d6f96c7b-85rv9             1/1     Running   193               31d
calico-node-6nfqv                                    1/1     Running   254               34d
calico-node-fv458                                    0/1     Running   50                34d
calico-node-h5lsq                                    1/1     Running   94 (7h11m ago)    34d
..............

当我们什么都不设置时，tom用户获取不到kubeconfig文件，没有认证信息，无法访问

[tom@vms81 ~]$ unset KUBECONFIG
[tom@vms81 ~]$ kubectl get pods -n kube-system
The connection to the server localhost:8080 was refused - did you specify the right host or port?

查看kubeconfig文件的配置信息

┌──[root@vms81.liruilongs.github.io]-[~/.kube]
└─$kubectl config view

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://192.168.26.81:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: liruilong-rbac-create
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED
┌──[root@vms81.liruilongs.github.io]-[~/.kube]
└─$

所以我们要想访问集群信息，只需要把这个kubeconfig 文件拷贝到客户机上就OK了

创建 kubeconfig 文件

一个kubeconfig 文件包括一下几部分：

集群信息：

集群CA证书
集群地址

上下文信息

所有上下文信息
当前上下文

用户信息

用户CA证书
用户私钥

要创建 kubeconfig 文件的话，我们需要一个私钥，以及集群 CA 授权颁发的证书。同理我们不能直接用私钥生成公钥，而必须是用私钥生成证书请求文件(申请书)，然后根据证书请求文件向 CA(权威机构)申请证书(身份证)，CA 审核通过之后会颁发证书。

环境准备

┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$kubectl create ns liruilong-rbac-create
namespace/liruilong-rbac-create created
┌──[root@vms81.liruilongs.github.io]-[~/ansible]
└─$mkdir k8s-rbac-create;cd k8s-rbac-create
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$kubectl config set-context $(kubectl config current-context) --namespace=liruilong-rbac-create
Context "kubernetes-admin@kubernetes" modified.
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$

申请证书

生成一个 2048 位的私钥 iruilong.key 文件

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$openssl genrsa -out liruilong.key 2048
Generating RSA private key, 2048 bit long modulus
....................+++
...........................................................................................................+++
e is 65537 (0x10001)

查看私钥文件

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$cat liruilong.key
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAt9OBnwaA3VdFfjdiurJPtcaiXOGPc1AWFmrlgocq4vT5WZgq
..............................
..................................
LHd0n1yCKpwbYMGghF4iGmEGIIdsCVZP+EV6lduPKjqEm9kjuLROKzRZHFoGyASO
Krb3VR4CKHvnZAPVctv7Pu+4JgMliJHl8GVYhqM5UykbLRMdNHSNIQ==
-----END RSA PRIVATE KEY-----
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$

利用刚生成的私有 liruilong.key 生成证书请求文件 liruilong.key：这里CN的值 liruilong，就是后面我们授权的用户。

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$openssl req -new -key liruilong.key -out liruilong.csr -subj "/CN=liruilong/O=cka2020"
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$ls
liruilong.csr  liruilong.key

对证书请求文件进行 base64 编码

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$cat liruilong.csr | base64 |tr -d "n"
LS0tLS1CRUdJTiBDRVJUSUZJ...............

编写申请证书请求文件的 yaml 文件:cat csr.yaml

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: liruilong
spec:
  signerName: kubernetes.io/kube-apiserver-client
  request: LS0tLS1CRUdJTiBDRVJUSUZJ...............
  usages:
  - client auth

这里 request 里的是 base64 编码之后的证书请求文件。申请证书

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$kubectl apply -f csr.yaml
certificatesigningrequest.certificates.k8s.io/liruilong created

查看已经发出证书申请请求：

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$kubectl  get csr
NAME        AGE   SIGNERNAME                            REQUESTOR          REQUESTEDDURATION   CONDITION
liruilong   15s   kubernetes.io/kube-apiserver-client   kubernetes-admin   <none>              Pending

批准证书：

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$kubectl certificate approve liruilong
certificatesigningrequest.certificates.k8s.io/liruilong approved

查看审批通过的证书：

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$kubectl get csr/liruilong -o yaml

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"certificates.k8s.io/v1","kind":"CertificateSigningRequest","metadata":{"annotations":{},"name":"liruilong"},"spec":{"request":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2F6Q0NBVk1DQVFBd0pqRVNNQkFHQTFVRUF3d0piR2x5ZFdsc2IyNW5NUkF3RGdZRFZRUUtEQWRqYTJFeQpNREl3TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF0OU9CbndhQTNWZEZmamRpCnVySlB0Y2FpWE9HUGMxQVdGbXJsZ29jcTR2VDVXWmdxd1g5T0RvSnpDREJZZVFJQ3h0Wm5uUk9XY1B2dVB6K1IKb1Eybk83K3FnNUNjZzlWZmVOWFRwUDB0VXZsQ21ZVVg2dkRDdlgxUDR3VnNFdXNydlZBdkF4NmdqZTZzNW94VgphZTIwcXFBRXpTUXJhczhPeldsZ1Frd0xjNU5MZ2k3bWlpNHNzaVpQRXU1ZFZIRWs5dHdCeUZTV0dsanJETkhvCnN4UkFFNXlrWjBnODBWSzN1U1JNNmFHSEJ0QmVpbysxa2d0U0xDMlVScy9QWUwwRGNSQm9zUUx0c3JublFSMTkKSE5NWTkweUhYN3Jta3ZqcHdOdkRZWjNIWUVvbGJQZThWZjhBTFpsbDVBTnJ5SUJqbXNrY01QM2lRMzdxWGZUNwptSzhKeHdJREFRQUJvQUF3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUFwa09aUUNTTGxGYWJTVk9zNmtzL1ZyCmd3b3FTdjFOM1YzUC84SmNzR1pFUTc4TkpJb2R0REExS3EwN25DWjJWUktselZDN1kyMCszZUVzTXZNWnFMc1MKbUtaS0w2SFE3N2RHa1liUjhzKzRMaFo4YXR6cXVMSnlqZUZKODQ2N1ZrUXF5T1R6by9wZ3E4YWJJY01XNzlKMgoxWEkybi92RWlIMEgvWU9DaWExVHRqTnpSWGtlL2hPQTZ4Y29CcVRpdWtkUHBqZDJSaWFTRUNUS1h4ZGNOS0xLCmZVbFhkb2s5UkVkQ2V3bU9ISUdvVG9qUGRWdWlPdkYzZkFqUXZNNDJ3UjJDdklHMWs1YUQzdWVlbzcwd0pnUlQKYzhZNnUwY2padEI5ZW5xUStmRFFqdUUyZElrMDJLbm5HQVppK0wxUnRnSnA2Tm1udEg5WUc3RlBLSXYrakFZPQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K","signerName":"kubernetes.io/kube-apiserver-client","usages":["client auth"]}}
  creationTimestamp: "2022-01-16T15:25:24Z"
  name: liruilong
  resourceVersion: "1185668"
  uid: 51837659-7214-4dec-bcd4-b7a9129ee2bb
spec:
  groups:
  - system:masters
  - system:authenticated
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2F6Q0NBVk1DQVFBd0pqRVNNQkFHQTFVRUF3d0piR2x5ZFdsc2IyNW5NUkF3RGdZRFZRUUtEQWRqYTJFeQpNREl3TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF0OU9CbndhQTNWZEZmamRpCnVySlB0Y2FpWE9HUGMxQVdGbXJsZ29jcTR2VDVXWmdxd1g5T0RvSnpDREJZZVFJQ3h0Wm5uUk9XY1B2dVB6K1IKb1Eybk83K3FnNUNjZzlWZmVOWFRwUDB0VXZsQ21ZVVg2dkRDdlgxUDR3VnNFdXNydlZBdkF4NmdqZTZzNW94VgphZTIwcXFBRXpTUXJhczhPeldsZ1Frd0xjNU5MZ2k3bWlpNHNzaVpQRXU1ZFZIRWs5dHdCeUZTV0dsanJETkhvCnN4UkFFNXlrWjBnODBWSzN1U1JNNmFHSEJ0QmVpbysxa2d0U0xDMlVScy9QWUwwRGNSQm9zUUx0c3JublFSMTkKSE5NWTkweUhYN3Jta3ZqcHdOdkRZWjNIWUVvbGJQZThWZjhBTFpsbDVBTnJ5SUJqbXNrY01QM2lRMzdxWGZUNwptSzhKeHdJREFRQUJvQUF3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUFwa09aUUNTTGxGYWJTVk9zNmtzL1ZyCmd3b3FTdjFOM1YzUC84SmNzR1pFUTc4TkpJb2R0REExS3EwN25DWjJWUktselZDN1kyMCszZUVzTXZNWnFMc1MKbUtaS0w2SFE3N2RHa1liUjhzKzRMaFo4YXR6cXVMSnlqZUZKODQ2N1ZrUXF5T1R6by9wZ3E4YWJJY01XNzlKMgoxWEkybi92RWlIMEgvWU9DaWExVHRqTnpSWGtlL2hPQTZ4Y29CcVRpdWtkUHBqZDJSaWFTRUNUS1h4ZGNOS0xLCmZVbFhkb2s5UkVkQ2V3bU9ISUdvVG9qUGRWdWlPdkYzZkFqUXZNNDJ3UjJDdklHMWs1YUQzdWVlbzcwd0pnUlQKYzhZNnUwY2padEI5ZW5xUStmRFFqdUUyZElrMDJLbm5HQVppK0wxUnRnSnA2Tm1udEg5WUc3RlBLSXYrakFZPQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth
  username: kubernetes-admin
status:
  certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURDekNDQWZPZ0F3SUJBZ0lRUC9aR05rUjdzVy9sdHhkQTNGQjBoekFOQmdrcWhraUc5dzBCQVFzRkFEQVYKTVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1CNFhEVEl5TURFeE5qRTFNakV3TWxvWERUSXpNREV4TmpFMQpNakV3TWxvd0pqRVFNQTRHQTFVRUNoTUhZMnRoTWpBeU1ERVNNQkFHQTFVRUF4TUpiR2x5ZFdsc2IyNW5NSUlCCklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF0OU9CbndhQTNWZEZmamRpdXJKUHRjYWkKWE9HUGMxQVdGbXJsZ29jcTR2VDVXWmdxd1g5T0RvSnpDREJZZVFJQ3h0Wm5uUk9XY1B2dVB6K1JvUTJuTzcrcQpnNUNjZzlWZmVOWFRwUDB0VXZsQ21ZVVg2dkRDdlgxUDR3VnNFdXNydlZBdkF4NmdqZTZzNW94VmFlMjBxcUFFCnpTUXJhczhPeldsZ1Frd0xjNU5MZ2k3bWlpNHNzaVpQRXU1ZFZIRWs5dHdCeUZTV0dsanJETkhvc3hSQUU1eWsKWjBnODBWSzN1U1JNNmFHSEJ0QmVpbysxa2d0U0xDMlVScy9QWUwwRGNSQm9zUUx0c3JublFSMTlITk1ZOTB5SApYN3Jta3ZqcHdOdkRZWjNIWUVvbGJQZThWZjhBTFpsbDVBTnJ5SUJqbXNrY01QM2lRMzdxWGZUN21LOEp4d0lECkFRQUJvMFl3UkRBVEJnTlZIU1VFRERBS0JnZ3JCZ0VGQlFjREFqQU1CZ05WSFJNQkFmOEVBakFBTUI4R0ExVWQKSXdRWU1CYUFGR0RjS1N1dVY1TTV5Wk5CR1AxLzZoN0xZNytlTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFCagpOelREMmZ5bTc3bXQ4dzlacXRZN3NQelhmNHJQTXpWUzVqV3NzenpidlhEUzhXcFNMWklIYkQ3VU9vYlYxcFYzClYzRW02RXlpWUEvbjhMYTFRMnZra0EyUDk1d3JqWlBuemZIeUhWVFpCTUY4YU1MSHVpVHZ5WlVVV0JYMTg1UFAKQ2MxRncwanNmVThJMDBzbUNOeURBZjVMejFjRUVrNWlGYUswMDJRblUyNk5lcDF3U3BMcVZWWVptSW9UVU9DOApCNzNpU3J6Y0wyVmdBejRCaUQxdUVlUkFMM20zRTB2VVpsQjduKzF1MllrNDFCajdGYnpWR2w1dFpYT3hDMVhxCjJVc0hSbmkzY1VYZ203QlloZDU3aTFHclRRRFJpckRwVFV1RDB3ZlFYTjZLdEx1TmVDYUc0alc4ZTl4QkQrTjIKOFE4Z25UZjdPSEI3VWZkUzVnMWQKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  conditions:
  - lastTransitionTime: "2022-01-16T15:26:02Z"
    lastUpdateTime: "2022-01-16T15:26:01Z"
    message: This CSR was approved by kubectl certificate approve.
    reason: KubectlApprove
    status: "True"
    type: Approved

导出证书文件：

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$kubectl get csr liruilong -o jsonpath='{.status.certificate}'| base64 -d > liruilong.crt

给用户授权，这里给 liruilong 一个集群角色 cluster-role(类似于root一样的角色)，这样 liruilong 具有管理员权限

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$kubectl create clusterrolebinding test  --clusterrole=cluster-admin --user=liruilong
clusterrolebinding.rbac.authorization.k8s.io/test created
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$

创建 kubeconfig 文件

拷贝 CA 证书

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$ls
csr.yaml      #(申请证书请求文件yaml)  
liruilong.crt #公钥(证书文件)  
liruilong.csr #(证书请求文件)  
liruilong.key #私钥
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$ls /etc/kubernetes/pki/
apiserver.crt              apiserver.key                 ca.crt  front-proxy-ca.crt      front-proxy-client.key  sa.pub
apiserver-etcd-client.crt  apiserver-kubelet-client.crt  ca.key  front-proxy-ca.key      liruilong.csv
apiserver-etcd-client.key  apiserver-kubelet-client.key  etcd    front-proxy-client.crt  sa.key
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$cp  /etc/kubernetes/pki/ca.crt .

设置集群字段，这里包含集群名字，服务地址和集群证书

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$kubectl config --kubeconfig=kc1 set-cluster cluster1  --server=https://192.168.26.81:6443 --certificate-authority=ca.crt --embed-certs=true
Cluster "cluster1" set.

在上面集群中创建一个上下文context1

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$kubectl config --kubeconfig=kc1 set-context context1 --cluster=cluster1 --namespace=default --user=liruilong
Context "context1" created.

这里–embed-certs=true 的意思是把证书内容写入到此 kubeconfig 文件里。
设置用户字段，包含用户名字，用户证书，用户私钥

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$kubectl config --kubeconfig=kc1 set-credentials liruilong --client-certificate=liruilong.crt --client-key=liruilong.key --embed-certs=true
User "liruilong" set.

查看创建的kubeconfig文件信息

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$cat kc1

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1USXhNakUyTURBME1sb1hEVE14TVRJeE1ERTJNREEwTWxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTkdkCisrWnhFRDJRQlR2Rm5ycDRLNFBrd2lsYXUrNjdXNTVobVdwc09KSHF6ckVoWUREY3l4ZTU2Z1VJVDFCUTFwbU0KcGFrM0V4L0JZRStPeHY4ZmxtellGbzRObDZXQjl4VXovTW5HQi96dHZsTGpaVEVHZy9SVlNIZTJweCs2MUlSMQo2Mkh2OEpJbkNDUFhXN0pmR3VXNDdKTXFUNTUrZUNuR00vMCtGdnI2QUJnT2YwNjBSSFFuaVlzeGtpSVJmcjExClVmcnlPK0RFTGJmWjFWeDhnbi9tcGZEZ044cFgrVk9FNFdHSDVLejMyNDJtWGJnL3A0emd3N2NSalpSWUtnVlUKK2VNeVIyK3pwaTBhWW95L2hLYmg4RGRUZ3FZeERDMzR6NHFoQ3RGQnVia1hmb3Ftc3FGNXpQUm1ZS051RUgzVAo2c1FNSFl4emZXRkZvSGQ2Y0JNQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZHRGNLU3V1VjVNNXlaTkJHUDEvNmg3TFk3K2VNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBRVE0SUJhM0hBTFB4OUVGWnoyZQpoSXZkcmw1U0xlanppMzkraTdheC8xb01SUGZacElwTzZ2dWlVdHExVTQ2V0RscTd4TlFhbVVQSFJSY1RrZHZhCkxkUzM5Y1UrVzk5K3lDdXdqL1ZrdzdZUkpIY0p1WCtxT1NTcGVzb3lrOU16NmZxNytJUU9lcVRTbGpWWDJDS2sKUFZxd3FVUFNNbHFNOURMa0JmNzZXYVlyWUxCc01EdzNRZ3N1VTdMWmg5bE5TYVduSzFoR0JKTnRndjAxdS9MWAo0TnhKY3pFbzBOZGF1OEJSdUlMZHR1dTFDdEFhT21CQ2ZjeTBoZHkzVTdnQXh5blR6YU1zSFFTamIza0JDMkY5CkpWSnJNN1FULytoMStsOFhJQ3ZLVzlNM1FlR0diYm13Z1lLYnMvekswWmc1TE5sLzFJVThaTUpPREhTVVBlckQKU09ZPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://192.168.26.81:6443
  name: cluster1
contexts:
- context:
    cluster: cluster1
    namespace: default
    user: liruilong
  name: context1
current-context: ""
kind: Config
preferences: {}
users:
- name: liruilong
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURDekNDQWZPZ0F3SUJBZ0lRUC9aR05rUjdzVy9sdHhkQTNGQjBoekFOQmdrcWhraUc5dzBCQVFzRkFEQVYKTVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1CNFhEVEl5TURFeE5qRTFNakV3TWxvWERUSXpNREV4TmpFMQpNakV3TWxvd0pqRVFNQTRHQTFVRUNoTUhZMnRoTWpBeU1ERVNNQkFHQTFVRUF4TUpiR2x5ZFdsc2IyNW5NSUlCCklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF0OU9CbndhQTNWZEZmamRpdXJKUHRjYWkKWE9HUGMxQVdGbXJsZ29jcTR2VDVXWmdxd1g5T0RvSnpDREJZZVFJQ3h0Wm5uUk9XY1B2dVB6K1JvUTJuTzcrcQpnNUNjZzlWZmVOWFRwUDB0VXZsQ21ZVVg2dkRDdlgxUDR3VnNFdXNydlZBdkF4NmdqZTZzNW94VmFlMjBxcUFFCnpTUXJhczhPeldsZ1Frd0xjNU5MZ2k3bWlpNHNzaVpQRXU1ZFZIRWs5dHdCeUZTV0dsanJETkhvc3hSQUU1eWsKWjBnODBWSzN1U1JNNmFHSEJ0QmVpbysxa2d0U0xDMlVScy9QWUwwRGNSQm9zUUx0c3JublFSMTlITk1ZOTB5SApYN3Jta3ZqcHdOdkRZWjNIWUVvbGJQZThWZjhBTFpsbDVBTnJ5SUJqbXNrY01QM2lRMzdxWGZUN21LOEp4d0lECkFRQUJvMFl3UkRBVEJnTlZIU1VFRERBS0JnZ3JCZ0VGQlFjREFqQU1CZ05WSFJNQkFmOEVBakFBTUI4R0ExVWQKSXdRWU1CYUFGR0RjS1N1dVY1TTV5Wk5CR1AxLzZoN0xZNytlTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFCagpOelREMmZ5bTc3bXQ4dzlacXRZN3NQelhmNHJQTXpWUzVqV3NzenpidlhEUzhXcFNMWklIYkQ3VU9vYlYxcFYzClYzRW02RXlpWUEvbjhMYTFRMnZra0EyUDk1d3JqWlBuemZIeUhWVFpCTUY4YU1MSHVpVHZ5WlVVV0JYMTg1UFAKQ2MxRncwanNmVThJMDBzbUNOeURBZjVMejFjRUVrNWlGYUswMDJRblUyNk5lcDF3U3BMcVZWWVptSW9UVU9DOApCNzNpU3J6Y0wyVmdBejRCaUQxdUVlUkFMM20zRTB2VVpsQjduKzF1MllrNDFCajdGYnpWR2w1dFpYT3hDMVhxCjJVc0hSbmkzY1VYZ203QlloZDU3aTFHclRRRFJpckRwVFV1RDB3ZlFYTjZLdEx1TmVDYUc0alc4ZTl4QkQrTjIKOFE4Z25UZjdPSEI3VWZkUzVnMWQKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdDlPQm53YUEzVmRGZmpkaXVySlB0Y2FpWE9HUGMxQVdGbXJsZ29jcTR2VDVXWmdxCndYOU9Eb0p6Q0RCWWVRSUN4dFpublJPV2NQdnVQeitSb1Eybk83K3FnNUNjZzlWZmVOWFRwUDB0VXZsQ21ZVVgKNnZEQ3ZYMVA0d1ZzRXVzcnZWQXZBeDZnamU2czVveFZhZTIwcXFBRXpTUXJhczhPeldsZ1Frd0xjNU5MZ2k3bQppaTRzc2laUEV1NWRWSEVrOXR3QnlGU1dHbGpyRE5Ib3N4UkFFNXlrWjBnODBWSzN1U1JNNmFHSEJ0QmVpbysxCmtndFNMQzJVUnMvUFlMMERjUkJvc1FMdHNybm5RUjE5SE5NWTkweUhYN3Jta3ZqcHdOdkRZWjNIWUVvbGJQZTgKVmY4QUxabGw1QU5yeUlCam1za2NNUDNpUTM3cVhmVDdtSzhKeHdJREFRQUJBb0lCQUJsS0I3TVErZmw1WUI0VgpFSWdPcjlpYUV3d2tHOUFKWElDSkJEb0l6bVdhdmhNTlZCUjZwd3BuOTl0UWkxdGFZM2RuVjZuTVlBMzdHck9vCjB5Z004TXpQZVczUUh6Z2p5cGFkRkJqR204Mm1iUHNoekVDT0RyeHkyT0txaEV1MS9yWjBxWU1NVzVvckU2NUQKOEJ3NmozaEp1MTlkY251bk1Lb2hyUlJ4MGNGOGJrVHpRcWk1Y0xGZ0lBUlArNklFcnQrS0FVSGFVQ0wvSUtTYgoyblJnSmhvSWszTUJnQnM3eVl0NFpVNlpkSVFLRU56RzRUd2VjNG5ESXE2TkllU1ZpSEVtbmdjRXVOdTNOaWEvCmxwTWd5ZHJKZWJKYStrc0FiZU4zU21TRGc2MXpVWUpHV2FOZUFPemdmbkZRTVp2Y2FyNEQ1b0xCQUE5Rlpic0EKc0hUZjBBRUNnWUVBNWVmdDJSV25kM2pueWJBVVExa3dRTzlzN1BmK241OXF3ekIwaEdrcC9qcjh2dGRYZkxMNgo0THBoNWFhKzlEM0w5TEpMdk8xa0cyaG5UbFE4Q25uVXpVUUUreldiYlRJblR0eE1hRzB5VjJlZ3NaQkFseERYClk1K2tZZ29EVXIyN2dWbE5QZ29SWVkzRG1ZZHplVmp0NEFHOE9JNWRPUlJ6bFE3VHN0Nk9XUUVDZ1lFQXpMQ3kKM2Q0SkRZRjBpeXFFc2FsNFNsckdEc3hmY2xxeUZaVWpmcUcvdGFHeEFTeE9zL0h3SDNwR1cwQXh3c2tPdVNkUwpWcGxOOC9uZjBMQjdPV0hQL2FjTlJLbHdJeUZrNG9EU0VMOVJ5d0FFVEF3NHdrYitoRzNZdUFHU2YvY1dEWXNtCjNJUUxlMVdFS0JSZDVBS1lkYXdyYlJtclFkSndSaFFkalNzOTJzY0NnWUVBck1WSVpwVHhUc1ViV3VQcHRscjEKK2paekl2bVM3WjI5ZTRXVWFsVWxhNW9raWI0R1R2MnBydXdoMlpVZmR5aGhkemZ0MXNLSE1sbVpHTElRbE1iTgpkcHdoS2k4MDZEQ0NmYTdyOUtYcTZPaEZTR3JoUHlVMjEvVUdjVzZZNUxzVWg3WDJhQ0xrd096cUN4eFJXT1hOCmpVT0FrUGZiY3FPOTRFeE9KdU05RWdFQ2dZRUFwNUVqN0xPL0wzcENBVWVlZDU3bjVkN244dWRtWDhSVnM0dHoKRWxDeUU2dzVybDhxVXUrR0J3N2ZtQVkyZG1LSUZoVlZ0NlVyQnNjUmJkTjhIUjZ3MmRNdTduM1RXajhWU3NQdwp0RnNiUjVkTTdVQzRHbnRxRXRtbUtBVEpmTTYzRkFGTm9Bck5KM3Q3aENBZ09PL1RCY29iaHVZVHAvL3hmNzBwCjhBNXRSYk1DZ1lCMkJTb0ZTbW1sVFN5RjdnZnhmM281dTNXM3lwTENIRFV0cFZkYlAxZm9vemJwZUs3V29IY2YKTEhkMG4xeUNLcHdiWU1HZ2hGNGlHbUVHSUlkc0NWWlArRVY2bGR1UEtqcUVtOWtqdUxST0t6UlpIRm9HeUFTTwpLcmIzVlI0Q0tIdm5aQVBWY3R2N1B1KzRKZ01saUpIbDhHVllocU01VXlrYkxSTWROSFNOSVE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$

修改kubeconfig文件当前的上下文为之前创建的上下文

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$sed  's#current-context: ""#current-context: "context1"#' kc1 | grep current-context
current-context: "context1"
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$sed -i  's#current-context: ""#current-context: "context1"#' kc1
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$cat  kc1 | grep current-context
current-context: "context1"

这样 kubeconfig 文件就创建完毕了，下面开始验证 kubeconfig 文件。

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$kubectl auth can-i list pods --as liruilong #检查是否具有 list 当前命名空间里的 pod 的权限
yes
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$kubectl auth can-i list pods -n kube-system --as liruilong #检查 是否具有 list 命名空间 kube-system 里 pod 的权限
yes
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$

拷贝证书到客户机

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-rbac-create]
└─$scp kc1  root@192.168.26.55:~

客户机指定证书访问测试

┌──[root@liruilongs.github.io]-[~]
└─$ kubectl --kubeconfig=kc1 get pods -n kube-system
NAME                                                 READY   STATUS        RESTARTS        AGE
calico-kube-controllers-78d6f96c7b-85rv9             1/1     Running       194 (14h ago)   33d
calico-node-6nfqv                                    0/1     Running       255 (14h ago)   35d
calico-node-fv458                                    0/1     Running       50              35d
calico-node-h5lsq                                    1/1     Running       94 (38h ago)    35d
。。。。。。。。。。。。
┌──[root@liruilongs.github.io]-[~]
└─$