k8s-1.14-安装及部署
安装k8s
添加kubeadm的源，然后直接使用apt-get安装即可
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
$ cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
EOF
$ apt-get update
$ apt-get install -y docker.io kubeadm
部署Kubernetes的Master节点
使用yaml配置文件方式部署，kubeadm.yaml：
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
controllerManager:
extraArgs:
horizontal-pod-autoscaler-use-rest-clients: "true"
horizontal-pod-autoscaler-sync-period: "10s"
node-monitor-grace-period: "10s"
apiServer:
extraArgs:
runtime-config: "api/all=true"
kubernetesVersion: "stable-1.14"
- kube-controller-manager : 使自定义资源（Custom Metrics）进行自动水平扩展。
- stable-1.11 : kubeadm帮我们部署的k8s版本号。也可以直接指定一个版本，比如：kubernetesVersion: "v1.11.1"
此配置文件只针对kubeadm-1.14，如果安装的kubeadm版本不对可能会出错，安装命令默认安装最新版
执行：
root@ubuntu:~# kubeadm init --config kubeadm-1.14.yaml
I0506 13:57:25.433572 31873 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0506 13:57:25.433754 31873 version.go:97] falling back to the local client version: v1.14.1
[init] Using Kubernetes version: v1.14.1
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
如果开启了swap则需要关闭
root@ubuntu:~# swapoff -a
初始化kubeadm
root@ubuntu:~# kubeadm init --config kubeadm-1.14.yaml
I0506 13:59:25.925975 32329 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0506 13:59:25.926333 32329 version.go:97] falling back to the local client version: v1.14.1
[init] Using Kubernetes version: v1.14.1
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [ubuntu localhost] and IPs [10.1.100.230 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [ubuntu localhost] and IPs [10.1.100.230 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [ubuntu kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.1.100.230]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 23.880826 seconds
[upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --experimental-upload-certs
[mark-control-plane] Marking the node ubuntu as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node ubuntu as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: q8j7ne.mwe28mqq3lcdmft3
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.1.100.230:6443 --token q8j7ne.mwe28mqq3lcdmft3 \
--discovery-token-ca-cert-hash sha256:9e4bbfeaaba6ff8bcb8c11e2a9898940acd1dfea383f74770eab7a63dc5e3d39
这个kubeadm join命令，就是用来给这个Master节点添加更多工作节点（Worker）的命令。
此外还会提示我们第一次使用Kubernetes集群所需要的配置命令：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
需要这些配置的原因是：Kubernetes集群默认需要加密方式访问。这几条命令就是将刚刚部署生成的集群的安全配置文件，保存到当前用户的.kube目录下，kubectl默认会使用这个目录下的授权信息访问集群。
查看当前唯一一个节点状态：
root@ubuntu:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
ubuntu NotReady master 57m v1.14.1
查看节点状态：
root@ubuntu:~# kubectl describe node ubuntu
Name: ubuntu
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=ubuntu
kubernetes.io/os=linux
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 06 May 2019 14:03:10 +0800
Taints: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Mon, 06 May 2019 15:03:13 +0800 Mon, 06 May 2019 14:49:02 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Mon, 06 May 2019 15:03:13 +0800 Mon, 06 May 2019 14:49:02 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Mon, 06 May 2019 15:03:13 +0800 Mon, 06 May 2019 14:49:02 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Mon, 06 May 2019 15:03:13 +0800 Mon, 06 May 2019 14:49:02 +0800 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
InternalIP: 10.1.100.230
Hostname: ubuntu
Capacity:
cpu: 8
ephemeral-storage: 71130496Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 4015660Ki
pods: 110
Allocatable:
cpu: 8
ephemeral-storage: 65553865006
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3913260Ki
pods: 110
System Info:
Machine ID: 7a258a0c9378476b92bc4f9eb52e4819
System UUID: 50294D56-823B-007D-F27C-2EF31C62164A
Boot ID: a861216c-9311-4a91-9bbc-86cbd9a1b40e
Kernel Version: 4.15.0-47-generic
OS Image: Ubuntu 16.04.5 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.9.2
Kubelet Version: v1.14.1
Kube-Proxy Version: v1.14.1
Non-terminated Pods: (5 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system etcd-ubuntu 0 (0%) 0 (0%) 0 (0%) 0 (0%) 59m
kube-system kube-apiserver-ubuntu 250m (3%) 0 (0%) 0 (0%) 0 (0%) 59m
kube-system kube-controller-manager-ubuntu 200m (2%) 0 (0%) 0 (0%) 0 (0%) 59m
kube-system kube-proxy-57tpg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 60m
kube-system kube-scheduler-ubuntu 100m (1%) 0 (0%) 0 (0%) 0 (0%) 58m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 550m (6%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasSufficientMemory 60m (x8 over 60m) kubelet, ubuntu Node ubuntu status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 60m (x8 over 60m) kubelet, ubuntu Node ubuntu status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 60m (x7 over 60m) kubelet, ubuntu Node ubuntu status is now: NodeHasSufficientPID
Normal Starting 60m kube-proxy, ubuntu Starting kube-proxy.
Normal Starting 14m kubelet, ubuntu Starting kubelet.
Normal NodeHasSufficientMemory 14m (x2 over 14m) kubelet, ubuntu Node ubuntu status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 14m (x2 over 14m) kubelet, ubuntu Node ubuntu status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 14m (x2 over 14m) kubelet, ubuntu Node ubuntu status is now: NodeHasSufficientPID
Normal NodeNotReady 14m kubelet, ubuntu Node ubuntu status is now: NodeNotReady
Normal NodeAllocatableEnforced 14m kubelet, ubuntu Updated Node Allocatable limit across pods
可以看到Ready False，因为没部署任何网络插件。
还可以通过kubectl查看这个节点上各个系统Pod的状态，其中，kube-system是Kubernetes项目预留的系统Pod的工作空间。
root@ubuntu:~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-pfhsz 0/1 Pending 0 64m
coredns-fb8b8dccf-t952r 0/1 Pending 0 64m
etcd-ubuntu 1/1 Running 0 63m
kube-apiserver-ubuntu 1/1 Running 0 63m
kube-controller-manager-ubuntu 1/1 Running 0 64m
kube-proxy-57tpg 1/1 Running 0 64m
kube-scheduler-ubuntu 1/1 Running 0 63m
部署网络插件
一切皆容器，部署网络插件非常简单：
root@ubuntu:~# kubectl apply -f https://git.io/weave-kube-1.6
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.extensions/weave-net created
部署Worker节点
Worker节点跟Master节点几乎相同，都是运行着一个kubelet组件。唯一的区别是，在kubeadm init过程中，kubelet启动后，Master节点上还会自动运行kube-apiserver、kube-scheduler、kube-controller-manager这三个系统Pod。
1.在所有Worker节点上安装kubeadm，重复之前的安装步骤 2.执行部署Master节点时生成的kubeadm join命令
kubeadm join 10.168.0.2:6443 --token 00bwbx.uvnaa2ewjflwu1ry --discovery-token-ca-cert-hash sha256:00eb62a2a6020f94132e3fe1ab721349bbcd3e9b94da9654cfe15f2985ebd711
如果创建master节点后长时间（24h）没加入worker节点，则需要重新生成并替换token
[root@server ~]# kubeadm join 10.1.100.230:6443 --token q8j7ne.mwe28mqq3lcdmft3 --discovery-token-ca-cert-hash sha256:9e4bbfeaaba6ff8bcb8c11e2a9898940acd1dfea383f74770eab7a63dc5e3d39
[preflight] Running pre-flight checks
[WARNING Hostname]: hostname "server" could not be reached
[WARNING Hostname]: hostname "server": lookup server on 10.1.100.254:53: no such host
error execution phase preflight: couldn't validate the identity of the API Server: abort connecting to API servers after timeout of 5m0s
在master节点生成token
root@ubuntu:~/testk8s# kubeadm token create
kv1hb3.mumkf4ccx2v9fc5e
之后替换--token再次加入
[root@server ~]# kubeadm join 10.1.100.230:6443 --token kv1hb3.mumkf4ccx2v9fc5e --discovery-token-ca-cert-hash sha256:9e4bbfeaaba6ff8bcb8c11e2a9898940acd1dfea383f74770eab7a63dc5e3d39
[preflight] Running pre-flight checks
[WARNING Hostname]: hostname "server" could not be reached
[WARNING Hostname]: hostname "server": lookup server on 10.1.100.254:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
部署Dashboard可视化插件
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
部署完之后就可以查看对应的Pod状态了：
kubectl get pods -n kube-system
kubernetes-dashboard-6948bdb78-f67xk 1/1 Running 0 1m
Dashboard是一个Web Server，为防止安全隐患，默认只能通过Proxy的方式在本地访问官方文档
部署容器存储插件
容器的持久化存储，是用来保存容器存储状态的重要手段：存储插件会在容器里挂载一个基于网络或者其他机制的远程数据卷，使得在容器里创建的文件，实际上是保存在远程存储服务器上，或者以分布式的方式保存在多个节点上，而与当前宿主机没有任何绑定关系。
$ kubectl apply -f https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/ceph/operator.yaml
$ kubectl apply -f https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/ceph/cluster.yaml
部署完成后，你就可以看到Rook项目会将自己的Pod放置在由它自己管理的两个Namespace中：
kubectl get pods -n rook-ceph-system
NAME READY STATUS RESTARTS AGE
rook-ceph-agent-7cv62 1/1 Running 0 15s
rook-ceph-operator-78d498c68c-7fj72 1/1 Running 0 44s
rook-discover-2ctcv 1/1 Running 0 15s
$ kubectl get pods -n rook-ceph
NAME READY STATUS RESTARTS AGE
rook-ceph-mon0-kxnzh 1/1 Running 0 13s
rook-ceph-mon1-7dn2t 1/1 Running 0 2s
总结
到此已经部署了一个完整的Kubernetes集群，过程不那么繁琐主要得益于：
- kubeadm项目大大简化了部署的准备工作，尤其是配置文件、证书、二进制文件的准备和制作。
- Kubernetes本身"一切皆容器"的思想，加上良好的可扩展机制，使得部署非常方便。