k8s1.13.0二进制部署-keepalived+双主master节点(三)

标题内容

前言

我们在上节通过flannel把节点的网络打通,实现容器跨主机通信。本节部署master中的几个重要组件 kube-apiserver ,kube-controller-manager ,kube-schedular 以及通过keepalived实现双机master高可用

架构图

部署

  • 创建证书

cat >>kubernetes-csr.json<<EOF

{
“CN”: “kubernetes”,
“hosts”: [
“127.0.0.1”,
192.168.137.104“,
192.168.137.107“,
192.168.137.200“,
10.0.0.1“,
“kubernetes”,
“kubernetes.default”,
“kubernetes.default.svc”,
“kubernetes.default.svc.cluster”,
“kubernetes.default.svc.cluster.local”
],
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“names”: [
{
“C”: “CN”,
“ST”: “BeiJing”,
“L”: “BeiJing”,
“O”: “k8s”,
“OU”: “System”
}
]
}

EOF

红色是apiserver的地址,一个是master-1的ip 一个是master-2的ip   一个是vip

蓝色是集群内部cluster地址

  • 生成kubernetes证书和私钥
cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
   -ca-key=/opt/kubernetes/ssl/ca-key.pem \
   -config=/opt/kubernetes/ssl/ca-config.json \
   -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
  •   拷贝证书到本地证书目录

cp kubernetes*.pem /opt/kubernetes/ssl/

生成的证书只需要这两个:  kubernetes-key.pem   kubernetes.pem

  • 下载软件包

wget https://dl.k8s.io/v1.13.0/kubernetes-server-linux-amd64.tar.gz

tar -xvzf kubernetes-server-linux-amd64.tar.gz -C /usr/local

cd /usr/local/kubernetes/server/bin/

cp -a kube-apiserver kube-scheduler kube-controller-manager kubectl /opt/kubernetes/bin/

  • 创建kube-apiserver使用的客户端token文件

export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ‘ ‘)

cat > /opt/kubernetes/cfg/token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,”system:kubelet-bootstrap”
EOF

创建kube-apisecarver配置文件

vim /opt/kubernetes/cfg/kube-apiserver

KUBE_APISERVER_OPTS=”–logtostderr=false \
–v=4 \
–log-dir=/opt/kubernetes/log \
–etcd-servers=https://192.168.137.104:2379,https://192.168.137.105:2379,https://192.168.168.137.106:2379 \
–bind-address=0.0.0.0 \
–secure-port=6443 \
–advertise-address=192.168.137.104 \
–allow-privileged=true \
–service-cluster-ip-range=10.0.0.0/24 \
–enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
–authorization-mode=RBAC,Node \
–enable-bootstrap-token-auth \
–token-auth-file=/opt/kubernetes/cfg/token.csv \
–service-node-port-range=30000-50000 \
–tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \
–tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
–client-ca-file=/opt/kubernetes/ssl/ca.pem \
–service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
–etcd-cafile=/opt/kubernetes/ssl/ca.pem \
–etcd-certfile=/opt/kubernetes/ssl/etcd.pem \
–etcd-keyfile=/opt/kubernetes/ssl/etcd-key.pem”

注意:

蓝色部分的路径要和之前创建token文件的路径一致

–bind-address  监听–seure-port的IP地址。被关联的接口必须能够被集群其它节点和CLI/web客户端访问。如果为空,则将使用所有接口(0.0.0.0)。(默认值0.0.0.0)

我们master要做高可用,实际就是kube-apiserver高可用。配合keepalived要保证bind-address为0.0.0.0 。否则vip(192.168.137.200)无法绑定服务端口,导致报错如下

参数说明:

–logtostderr 启用日志
–v 日志等级
–etcd-servers etcd集群地址
–bind-address 监听地址
–secure-port https安全端口
–advertise-address 集群通告地址
–allow-privileged 启用授权
–service-cluster-ip-range Service虚拟IP地址段
–enable-admission-plugins 准入控制模块
–authorization-mode 认证授权,启用RBAC授权和节点自管理
–enable-bootstrap-token-auth 启用TLS bootstrap功能,后面会讲到
–token-auth-file token文件
–service-node-port-range Service Node类型默认分配端口范围

  • 创建kube-apiserver服务

vim /usr/lib/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
apiserver

  • 启动服务

systemctl daemon-reload

service kube-apiserver start

chkconfig kube-apiserver on

  • 通过URI访问api接口

[root@master-1 ssl]# curl -L –cacert /opt/kubernetes/ssl/ca.pem https://192.168.137.104:6443/api
{
“kind”: “APIVersions”,
“versions”: [
“v1”
],
“serverAddressByClientCIDRs”: [
{
“clientCIDR”: “0.0.0.0/0”,
“serverAddress”: “192.168.137.104:6443”
}
]
}

部署Controller Manager

  • 创建配置文件

vim /opt/kubernetes/cfg/kube-controller-manager

KUBE_CONTROLLER_MANAGER_OPTS=”–logtostderr=true \
–v=4 \
–log-dir=/opt/kubernetes/log \
–master=127.0.0.1:8080 \
–leader-elect=true \
–address=127.0.0.1 \
–service-cluster-ip-range=10.0.0.0/24 \
–cluster-name=kubernetes \
–cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
–cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
–root-ca-file=/opt/kubernetes/ssl/ca.pem \
–service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
–experimental-cluster-signing-duration=87600h0m0s”

  • 创建服务文件

vim /usr/lib/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

  • 启动服务

systemctl daemon-reload

service kube-controller-manager start

chkconfig   kube-controller-manager on

  • 部署kube-scheduler

vim /opt/kubernetes/cfg/kube-scheduler

KUBE_SCHEDULER_OPTS=”–logtostderr=true \
–v=4 \
–log-dir=/opt/kubernetes/log \
–master=127.0.0.1:8080 \
–leader-elect”

  • 创建服务文件

vim /usr/lib/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

EOF

  • 启动服务

systemctl daemon-reload

service kube-scheduler start

chkconfig kube-scheduler on

  • 分发证书文件以及配置到master-2

直接把kubernetes目录拷贝过去

scp -r -p 9777 /opt/kubernetes/ master-2:/opt/

拷贝服务文件

scp -P 9777 /usr/lib/systemd/system/kube-* master-2:/usr/lib/systemd/system/

master-2修改apiserver 节点ip 为192.168.137.107

[root@master-2 ssl]# curl -L –cacert /opt/kubernetes/ssl/ca.pem https://192.168.137.107:6443/api
{
“kind”: “APIVersions”,
“versions”: [
“v1”
],
“serverAddressByClientCIDRs”: [
{
“clientCIDR”: “0.0.0.0/0”,
“serverAddress”: “192.168.137.107:6443”
}
]
}

可以看到的master-2主机的apiserver能够正常工作

  • 在master-2启动kube-controller-manager  kube-scheduler  并设置开机启动

配置kube-apiserver高可用

  • 前提条件

apiserver 配置文件中的  –bind-address=0.0.0.0

controller-manager和scheduler设置了–leader-elect=true

  • 安装配置keepalived

master-1的配置(角色master)

! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id master #二者相同
}
vrrp_script check_api {
script “/etc/keepalived/check_api.sh”
}
vrrp_instance VI_1 {
state MASTER # 需要不同
interface ens33
virtual_router_id 51 #vrrp 路由Id 实例,每个示例唯一
priority 100 #需要不同
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.137.200
}
track_script {
check_api
}
}


master-2的配置(角色:backup)

! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id master #二者相同
}
vrrp_script check_api {
script “/etc/keepalived/check_api.sh”
}
vrrp_instance VI_1 {
state BACKUP # 需要不同
interface ens33
virtual_router_id 51 #vrrp 路由Id 实例,每个示例唯一
priority 90 #需要不同
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.137.200
}
track_script {
check_api
}
}


check_api.sh脚本如下

#!/bin/bash
date=`date +%F`
count=`ps -ef |grep kube-apiserver |grep -v grep|wc -l`
if [ “$count” -eq 0 ];
then
service keepalived stop
echo “现在时间是$date,keepalived开始故障转移” >>/etc/keepalived/failover.log
fi


  • 启动keepalived服务

service keepalived start 

chkconfig keepalived on

  • 验证效果

查看master-1的keepalived日志

vip – 192.168.137.200 已经绑定在master-1的ens33网卡上

curl  apiserver  的vip 查看当前真实apiserver地址为matser-1的

关闭master-1的apiserver服务再次curl验证

备注:由于apiserver关闭,根据脚本定义。此时我们的keepalived服务也会关闭,然后心跳检测master-1为响应,接着进行故障转移,此时vip已经绑定master-2上了。如果master-1的apiserver恢复了,需要再次启动master-1的keepalived才能让vip重新漂移到master-1。

查看master-2 ip地址

vip已经绑定到master-2


配置kubectl客户端命令工具(双master都需要配置)

进入 cd /root/ssl /

  • 创建admin证书签名请求

vim admin-csr.json

{
“CN”: “admin”,
“hosts”: [],
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“names”: [
{
“C”: “CN”,
“ST”: “BeiJing”,
“L”: “BeiJing”,
“O”: “system:masters”,
“OU”: “System”
}
]
}

  • 生成admin证书和密钥

cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem -ca-key=/opt/kubernetes/ssl/ca-key.pem -config=/opt/kubernetes/ssl/ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

  • 拷贝证书密钥到本机证书目录(双主证书一致)

cp  admin*.pem /opt/kubernetes/ssl/

备注:同时给master-2拷贝过去。用于master-2的客户端

  • 设置集群参数(双主都需要配置)

备注:此处声明了集群apiserver通信以vip 192.168.137.200:6443通信。

kubectl config set-cluster kubernetes –certificate-authority=/opt/kubernetes/ssl/ca.pem –embed-certs=true –server=https://192.168.137.200:6443

  • 设置客户端认证参数(双主都需配置)

kubectl config set-credentials admin –client-certificate=/opt/kubernetes/ssl/admin.pem –embed-certs=true –client-key=/opt/kubernetes/ssl/admin-key.pem

  • 设置上下文参数(双主配置)

kubectl config set-context kubernetes –cluster=kubernetes –user=admin

  • 设置默认上下文

kubectl config use-context kubernetes

最终效果

master-1上查看信息

停掉master-1的apiserver切换到master-2。同样在master-1执行命令。效果一致。

直接在master-2上执行也可以。

 

 

 

点赞

发表评论

电子邮件地址不会被公开。 必填项已用*标注

Loading...