Open in app

Sign up

Sign in

Write

Sign up

Sign in

Kubernetes 於 Ubuntu 自架且建立多節點 ( Nodes ) 教學 — 上篇

Seachaos
tree.rocks
Published in
18 min readSep 21, 2020

--

前言

本教學主要在 Ubuntu 20.04 Server 上安裝 Kubernetes (k8s)，並且在多台 Server 上執行 Node 與 Master ( 這裡使用 VM 來模擬多台 Server 情況 )

執行本教學請先至少準備/安裝好兩台 Server ( 本教學使用 VM clone 然後修改 hostname 的方式)，一台我們的主機稱為 Master (主控)，另外一台或多台的 Server 我們稱為 Node ( 節點 )

另外本教學幾乎指令都需要 root 權限，請自行判斷 sudo 與否

本文使用 KVM 來模擬多台 Server 作為 k8s Node

1. Kubelet / Kubeadm 安裝與概念

於欲執行 Kubernetes 的機上，必須要有 kubelet / kubeadm ( Master 與 node 皆要執行此步驟 )
這是 Kubernetes 的主要程式，然後透過 runtime 來實現 pods
這裡的 runtime 可以是 Docker / CRI-O / Containerd

( 詳細參考資料 :
kubernetes components : https://kubernetes.io/docs/concepts/overview/components/
kubernetes runtime : https://kubernetes.io/docs/setup/production-environment/container-runtimes/ )

我們執行官方的安裝範例

a. 先設定 k8s server上網路

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOFsudo sysctl --system

b. 安裝 kubeadm / kubelet

sudo apt-get update && sudo apt-get install -y apt-transport-https curlcurl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOFsudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

c. 驗證

只要輸入 kubeadm 就可以知道是否安裝成功 ( 如下畫面 )

安裝成功

2. 安裝 CRI-O for K8s

CRI-O 就是設計給 Kubernetes 的 Container Runtime Interface，k8s 當然也可以使用 Docker 作為 Container Runtime，但是我們這邊遵循選擇 cri-o 為主
( cri-o github 於此 : https://github.com/cri-o/cri-o )

本步驟全部以 Root 身份執行

a. 先設定 Linux kernel 模組

modprobe overlay
modprobe br_netfilter

b. 設定網路給 CRI-O

cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOFsysctl --system

c. 安裝 CRI-O

請注意這邊的 VERSION 與 OS，如果你選擇非本教學的版本請自行修改
詳細參數可以在官網看到

export VERSION=1.18
export OS=xUbuntu_20.04echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.listecho "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.listcurl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | apt-key add -curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | apt-key add -apt-get update
apt-get install -y cri-o cri-o-runc

d. 啟動 CRI-O

如果無意外應該可以成功

systemctl daemon-reload
systemctl start crio

e. 關閉 swap

K8S 預設不希望我們系統有 swap 存在，所以 :
編輯 /etc/fstab ( 執行 sudo vim /etc/fstab )，將 swap 註解掉( 下行 )

# /swap.img     none    swap    sw      0       0

然後執行 Ubuntu 關閉 Swap 指令

sudo swapoff -a

3. 設定服務自動重啟

( 本步驟全部以 Root 身份執行，後略 )

我們要給 kubelet 預設的設定 ( /etc/default/kubelet )

cat > /etc/default/kubelet <<EOF
KUBELET_EXTRA_ARGS=--feature-gates="AllAlpha=false,RunAsGroup=true" --container-runtime=remote --cgroup-driver=systemd --container-runtime-endpoint='unix:///var/run/crio/crio.sock' --runtime-request-timeout=5m
EOF

還有我們需要這個流程，確保所有的服務在重開機後會自己起來 ( k8s reboot 後依然有效 )

systemctl enable kubelet
systemctl enable crio

4. K8s Master Node 啟動 : kubeadm init

如果你是使用 VM，可以選擇在這步驟完成之前先 snapshot / clone ，或是在另外一台 Server 上執行以上安裝步驟

因為我們要建立 master node 了，其實相當簡單，就是輸入
( 真相不是以下指令這麼簡單，先看完此步驟全文再輸入，或是參考此步驟小總結 )

kubeadm init

然後很有可能會遇到各種挫折… 😜

狀況1 — 機器上 Docker 同時存在，導致 kubeadm 分不清楚是要用 docker 還是 cri-0

錯誤訊息如下 :

Found multiple CRI sockets, please use --cri-socket to select one: /var/run/dockershim.sock, /var/run/crio/crio.sockTo see the stack trace of this error execute with --v=5 or higher

解決辦法: 指定我們使用 crio 來執行

kubeadm init --cri-socket=/var/run/crio/crio.sock

狀況2 — 機器 swap 沒關

預設情況下 k8s 機器不能有 swap 存在
錯誤訊息如下:

W0920 04:02:50.683554    7537 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io][init] Using Kubernetes version: v1.19.2[preflight] Running pre-flight checkserror execution phase preflight: [preflight] Some fatal errors occurred:[ERROR Swap]: running with swap on is not supported. Please disable swap[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`To see the stack trace of this error execute with --v=5 or higher

解決辦法: 關閉 swap
編輯 /etc/fstab ( 執行 sudo vim /etc/fstab )，將 swap 註解掉( 下行 )

# /swap.img     none    swap    sw      0       0

然後執行 Ubuntu 關閉 Swap 指令

sudo swapoff -a

然後再一次嘗試啟動

kubeadm init --cri-socket=/var/run/crio/crio.sock

狀況3 — Timeout

其實這是因為網路設定不正確導致 kubeadm 與 kubelet 無法驗證

錯誤訊息主要類似 Unfortunately, an error has occurred: timed out waiting for the condition :

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the conditionThis error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.Here is one example how you may list all Kubernetes containers running in cri-o/containerd using crictl:- 'crictl --runtime-endpoint /var/run/crio/crio.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint /var/run/crio/crio.sock logs CONTAINERID'error execution phase wait-control-plane: couldn't initialize a Kubernetes clusterTo see the stack trace of this error execute with --v=5 or higher

解決辦法:
先看一下 crio-bridge 的網路設定
( 執行 cat /etc/cni/net.d/100-crio-bridge.conf )

應該會類似如下

{
  "cniVersion": "0.3.1",
  "name": "crio",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": true,
  "hairpinMode": true,
  "ipam": {
    "type": "host-local",
    "routes": [
      { "dst": "0.0.0.0/0" },
      { "dst": "1100:200::1/24" }
    ],
    "ranges": [
      [{ "subnet": "10.85.0.0/16" }],
      [{ "subnet": "1100:200::/24" }]
    ]
  }
}

注意上面粗斜體的部分，這邊是 crio 的網路位置，一般來說大家應該都是 10.85.0.0/16 ，如果不一樣的話請自行修改以下指令

用以下指令啟動

export CIDR=10.85.0.0/16
kubeadm init \
    --pod-network-cidr=$CIDR \
    --cri-socket=/var/run/crio/crio.sock

狀況 4 — 已經存在

錯誤訊息類似 : ERROR FileAvailable — etc-kubernetes-manifests-kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml already exists

W0920 04:18:45.722186   14058 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io][init] Using Kubernetes version: v1.19.2[preflight] Running pre-flight checkserror execution phase preflight: [preflight] Some fatal errors occurred:[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists[ERROR Port-10250]: Port 10250 is in use[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`To see the stack trace of this error execute with --v=5 or higher

解決辦法 :
輸入指令清除

kubeadm reset --cri-socket=/var/run/crio/crio.sock

小總結

要避開以上狀況，請先執行狀況 2 ( 關閉 swap ) ，然後狀況 3 設定網路
理論上就可以執行以下指令

export CIDR=10.85.0.0/16
kubeadm init \
    --pod-network-cidr=$CIDR \
    --cri-socket=/var/run/crio/crio.sock

成功的話應該會看到類似以下畫面

kubeadm init 建立 主節點成功

請保存好這段，這是其他 k8s 節點要加入使用的

kubeadm join 192.168.99.61:6443 --token xxxxxx     --discovery-token-ca-cert-hash sha256:xxxx

5. K8S Master Node 驗證

從上個步驟我們看到類似的提示

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/configd

這個可以使用自己的慣用 user ( 非 root ) 來執行，然後這個 User 就可以使用 kubectl 指令了

我們可以透過

kubectl get nodes

來看看是否運作正常，應該會看到類似以下訊息

我們的 k8s node ready

然後執行一個 deployment 範例

kubectl create deployment hello-node --image=k8s.gcr.io/echoserver:1.4

在輸入 kubectl get pods 觀察狀態，會發現一直 pending
透過 kubectl describe 指令觀察該 pods 會發現類似錯誤訊息 :
( 主要為 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn’t tolerate. )

QoS Class:       BestEffortNode-Selectors:  <none>Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300snode.kubernetes.io/unreachable:NoExecute op=Exists for 300sEvents:Type     Reason            Age                From               Message----     ------            ----               ----               -------Warning  FailedScheduling  31s (x2 over 31s)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.

這時候需要輸入

kubectl taint nodes --all node-role.kubernetes.io/master-

來整理 nodes ( 參考 https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)

再觀察一次 pod 狀況 ( kubectl get pods )

k8s 自架 master 節點運作成功

可以看到運行成功，到此步驟為止就是一個可以(實驗)用的 kubernetes 了

關於其他節點的加入，我們下篇再說…

Kubernetes
Ubuntu
DevOps
Kubernetes Cluster

--

--

Seachaos
tree.rocks
Follow

Written by Seachaos

193 Followers
Editor for
Follow

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams