kubernetes集群搭建

2020-10-27 650 0

简介

学习k8s前建议先学习docker容器及k8s基础知识,有助于理解,使用k8s环境有很多方式,可以用minikube快速试用,也可以通过在线环境进行学习https://www.katacoda.com/courses/kubernetes
本文主要介绍模拟生产环境多节点的k8s集群搭建过程

1. 准备3台虚拟机节点

本示例节点都选用centos7操作系统环境

具体安装方式参考文章
http://www.hushowly.com/articles/1683

1.centos7-1(主节点)

分配的IP为:192.168.56.201

2.centos7-2(工作节点)

分配的IP为:192.168.56.202

3.centos7-3(工作节点)

分配的IP为:192.168.56.203

file

2. docker安装

集群搭建好后,需要为每个节点装docker环境
docker安装有好几种方案,此处采用最简单的官方的sh安装方案

https://docs.docker.com/engine/install/centos/

2.1 安装docker
$ curl -fsSL https://get.docker.com -o get-docker.sh
$ sudo sh get-docker.sh
2.2 启动docker服务
su root
sudo systemctl start docker

file

3. 安装三架马车

注:集群各节点都需要安装此工具

  • kubeadm
    Kubeadm用于快速部署Kubernetes集群工具,提供kubeadm init和kubeadm join
  • kubelet
    运行在cluster所有节点上,负责启动POD和容器
  • kubectl
    kubenetes命令行工具,通过kubectl远程或本机方式操作部署和管理应用,查看各种资源,创建,删除和更新组件

说明:以下都是基于centos安装和配置,如果是其它操作系统有此许差异,具体可参考官网

官网安装说明 https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

3.1 安装源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
3.2 Set SELinux in permissive mode
# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
3.3 安装
# 安装
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

#设置为开机自动激活单元并现在立刻启动
sudo systemctl enable --now kubelet

4 初始化集群

4.1 初始化master节点

也可以通过配置文件初始化,可以方便制定更多的参数

kubeadm init 
--apiserver-advertise-address=192.168.56.201 
--image-repository registry.aliyuncs.com/google_containers 
--service-cidr=10.1.0.0/16 
--pod-network-cidr=10.244.0.0/16

参数说明:

  • image-repository 设置镜像源,因为国内网络,必须设置一个可用的镜源
  • apiserver-advertise-address 通过该 ip 地址向集群其他节点公布 api server 的信息,必须能够被其他节点访问
  • pod-network-cidr 指定pod网络地址范围
  • service-cidr 指定service网络地址范围
4.1.1 遇到错误1
[ERROR Swap]: running with swap on is not supported. Please disable swap
  • 解决:需要关闭swap
--永久关闭(建议)
vi /etc/fstab 注掉以下行
/swap.img   none    swap    sw  0   0

--临时关闭
swapoff -a 
4.1.2 遇到错误2
[init] Using Kubernetes version: v1.19.2
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
  • 修改docker配置

    cat > /etc/docker/daemon.json <<EOF
    {
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
    "max-size": "100m"
    },
    "storage-driver": "overlay2"
    }
    EOF
  • 重启docker服务

    systemctl restart docker
4.1.3 遇到错误3
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1

处理办法:

echo "1" >/proc/sys/net/bridge/bridge-nf-call-iptables
4.1.4 遇到错误4
registry.aliyuncs.com/google_containers/coredns/coredns:v1.8.0 无法下载

处理办法:

# 在master想办法通过其它registry pullcoredns镜像
docker pull coredns/coredns:v1.8.0
# 调整tag符合当前image-repository版本信息
docker tag k8s.gcr.io/coredns:v1.8.0 registry.aliyuncs.com/google_containers/coredns/coredns:v1.8.0

4.2 集群master节点初始化成功

[root@centos7-1 ~]# kubeadm init 
> --apiserver-advertise-address=192.168.56.201 
> --image-repository registry.aliyuncs.com/google_containers 
> --service-cidr=10.1.0.0/16 
> --pod-network-cidr=10.244.0.0/16
.........
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.56.201:6443 --token ng780y.14hb11g0iya7djc6 
    --discovery-token-ca-cert-hash sha256:342b40ba65b93f2316492176cc479a5415b6c60250b2723f4c7050aa2ab3ea07 
4.6 根据成功提示配置kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf
$HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
4.7 测试kubectl命令
[root@centos7-1 vagrant]# kubectl get nodes
NAME        STATUS   ROLES    AGE   VERSION
centos7-1   Ready    master   55m   v1.19.3
4.8 记录集群加入join地址

其它工作节点需要通过此命令加入到集群中

注:需要等网络插件成功安装,才能正常加入集群

kubeadm join 192.168.56.201:6443 --token ng780y.14hb11g0iya7djc6 
--discovery-token-ca-cert-hash sha256:342b40ba65b93f2316492176cc479a5415b6c60250b2723f4c7050aa2ab3ea07 

5. 安装网络插件

5.1 查看master节点组件状态
[root@centos7-1 vagrant]# kubectl get pod --all-namespaces
NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d56c8448f-fg7g4            0/1     Pending   0          38m
kube-system   coredns-6d56c8448f-mw84l            0/1     Pending   0          38m
kube-system   etcd-centos7-1                      1/1     Running   0          38m
kube-system   kube-apiserver-centos7-1            1/1     Running   0          38m
kube-system   kube-controller-manager-centos7-1   1/1     Running   0          38m
kube-system   kube-proxy-8xbkk                    1/1     Running   0          38m
kube-system   kube-scheduler-centos7-1            1/1     Running   0          38m

我们发现,因为corednspod没有启动,因为缺少网络pod

5.2 安装网络插件

k8s网络插件很多,此处选择flannel

https://kubernetes.io/zh/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-networking-

  • 5.2.1 部署flannel网络

https://github.com/coreos/flannel#flannel

在线方式:

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

备注:有时网络问题无法下载,请切换网络试下

离线方式:
想办法下载kube-flannel.yml到本地

kubectl apply -f kube-flannel.yml

遇到问题1:

[root@centos7-1 vagrant]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
The connection to the server raw.githubusercontent.com was refused - did you specify the right host or port?

解决: 配置KUBECONFIG环境变量
因为kubectl命令需要使用kubernetes-admin来运行,所以需要通过KUBECONFIG找配置文件

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source ~/.bash_profile
5.3 检查网络插件状态(网络组件启动要一会)

从以下看出,网络插件和corednspod都运行正常了

[root@centos7-1 vagrant]# kubectl get pod --all-namespaces
NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d56c8448f-fg7g4            1/1     Running   0          70m
kube-system   coredns-6d56c8448f-mw84l            1/1     Running   0          70m
kube-system   etcd-centos7-1                      1/1     Running   0          70m
kube-system   kube-apiserver-centos7-1            1/1     Running   0          70m
kube-system   kube-controller-manager-centos7-1   1/1     Running   1          70m
kube-system   kube-flannel-ds-wx592               1/1     Running   0          8m36s
kube-system   kube-proxy-8xbkk                    1/1     Running   0          70m
kube-system   kube-scheduler-centos7-1            1/1     Running   0          70m

6. 加入工作节点到集群

前提:各工作节点要已安装docker和三架马车相关工具,各工作节点和master是网络是互通的

6.1 获取集群加入join命令

两种方式获取

  1. 通过以上4.8章节master节点集群初始成功后记录join地址
    2.在master节点再次执行以下命令获取

    kubeadm token create --print-join-command

6.1 完成工作节点加入集群

在相应的工作节点使用root用户执行以下命令:

kubeadm join 192.168.56.201:6443 --token ng780y.14hb11g0iya7djc6 
    --discovery-token-ca-cert-hash sha256:342b40ba65b93f2316492176cc479a5415b6c60250b2723f4c7050aa2ab3ea07 

问题:有可能失败,参考以上4.2-4.4的错误处理

  • 未启动docker服务
  • 禁用swap

工作节点加入集群成功结果:

[root@centos7-2 ~]# kubeadm join 192.168.56.201:6443 --token ng780y.14hb11g0iya7djc6 
>     --discovery-token-ca-cert-hash sha256:342b40ba65b93f2316492176cc479a5415b6c60250b2723f4c7050aa2ab3ea07 
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

6.2 查看节点集群状态

回到master节点执行以下命令,查看node加入状态

[root@centos7-1 ~]# kubectl get nodes
NAME        STATUS   ROLES    AGE    VERSION
centos7-1   Ready    master   6m2s   v1.19.3
centos7-2   Ready    <none>   102s   v1.19.3

7. 体验k8s上应用部署

开启命令补全功能,方便命令输入

yum install bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)

7.1 创建部署

kubectl create deployment hello-echo --image=registry.aliyuncs.com/google_containers/echoserver:1.10

7.2 发布应用

kubectl expose deployment hello-echo --type=NodePort --port=8080

参数说明:

  • --type = NodePort 指定 Service 的类型

7.2 查看应用的外部仿问地址

[root@centos7-1 vagrant]# kubectl get service
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
hello-echo   NodePort    10.1.184.187   <none>        8080:31038/TCP   15m
kubernetes   ClusterIP   10.1.0.1       <none>        443/TCP          3h21m

从以上输入PORT中看出,hello-echo对应的外部端口为31038

7.3 仿问发布的应用

file

8. 安装dashboard

https://kubernetes.io/zh/docs/tasks/access-application-cluster/web-ui-dashboard/

dashboard 可以通过kubectl proxy和NodePort方式对外开放,因为proxy只能在本机仿问,所以以下介绍NodePort方式

8.1 下载dashboard的yaml
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml

备注:有时网络问题无法下载,请切换网络试下

8.2 修改recommended.yaml服务暴露方式

暴露到端口: 30443

file

8.3. 部署dashboard
kubectl apply -f recommended.yaml

注:需要等待一会的时间,dashboard相关镜像拉取和容器初始化费点时间,可以使用kubectl get pods --all-namespaces查看部署状态

8.4. 仿问dashboard

注意使用https方式仿问
dashboard正常启动了,但是需要身份验证才能登陆

file

8.5 创建帐号得到令牌

注: 请在控制面板所在机器操作

  • 创建account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
---
# Create ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system
  • 执行创建

    kubectl create -f account.yaml
  • 获取admin-user用户的令牌

    kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

    file

  • 登陆dashboard
    file

相关文章

使用docker快速搭建php开发调试环境
linux下快速开启内网穿透(ssh隧道)