Task:
Create a virtual cluster for practice and development
1. Create 5 Ubuntu 64-bit VMs, for a 5 node cluster.
Nodes:
- k8master1 k8master1.mindwatering.net (2 CPU / 8 GB RAM / 20 GB disk)
- k8master2 k8master2.mindwatering.net (2 CPU / 8 GB RAM / 20 GB disk)
- k8master3 k8master3.mindwatering.net (2 CPU / 8 GB RAM / 20 GB disk)
- k8worker1 k8worker1.mindwatering.net (2 CPU / 6 GB RAM / 20 GB disk)
- k8worker2 k8worker2.mindwatering.net (2 CPU / 6 GB RAM / 20 GB disk)
For the packages selection, leave the "standard system utilities" option selected, and add "OpenSSH server".
We will also have a generic master DNS address to manage the masters via the API. To start we will give it same IP as k8master1, but later point it through a haproxy for redundancy.
- k8master k8master.mindwatering.net
2. Perform OS prerequisites:
Disable UFW firewall on all hosts since not production.
Check status:
$ sudo ufw status
If enabled, disable with:
$ sudo ufw disable
If this is production, then it's a bit more complicated.
- You might want to allow all traffic within the cluster if you don' know all the apps you would be running, and which ports they would need
- You want to limit traffic externally to the cluster to the K8s ports and the ports of the apps exposed to the Internet.
At minimum, you'd have the following ports to consider
Master Nodes:
6443 - API servers
2379-2380 - etcd server client API (used by the API server internally on the masters, and across the master nodes)
10250 - kubelet API / health check port
10251 - kube-scheduler
10252 - kube-controller-manager
10255 - read-only kubelet API
Worker Nodes:
10250 - kubelet API
10255 - read-only kubelet API
30000-32767 app node-port services (also add to master if it is doing app services)
All traffic allowed on the calico nodes.
Disable swap:
$ sudo swapoff -a
and comment the swap line:
$ sudo vi /etc/fstab
...
# /dev/mapper/k8master1--vg-swap_1 none swap sw 0 0
<esc>:wq
Update the hostname for each node:
$ sudo hostnamectl set-hostname k8master
Edit /etc/hosts and add each master and worker IP, as needed.
$ sudo vi /etc/hosts
Note:
Don't forget to create a DNS entry for the k8master1 that is the generic k8master address. (If you forget, the control plane 40 sec time-out will occur and you'll need to do the init over. Hint: $sudo kubeadm reset )
Network Update:
$ sudo ifconfig
Note the config for the main ethernet port: e.g. en33.
Switch to static IP:
$ sudo vi /etc/netplan/01-network-manager-all.yaml
Update and add the following for the ethernet port:
(e.g. en33 and network 192.168.0.0/24, note that DNS is comma delineated. Note that all entries after ens33 are indented two spaces, and that addresses is indented 2 additional spaces under namesservers -- this is YAML, spacing matters. )
ens33:
dhcp4: no
addresses: [192.168.0.21/24]
gateway4: 192.168.0.1
nameservers:
addresses: [192.168.0.1,111.222.333.123]
<esc>:wq
Iptables bridging step/check:
$ sudo -i
# sudo modprobe br_netfilter
# cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
# exit
SSH Login Setup:
(Not required, but convenient if you don't want to do username/password)
On the k8master1 server, do:
$ cd /home/myadmin/tmp/
$ ssh-keygen
(skip the password as desired)
Copy the ID to the other masters and nodes:
$ ssh-copy-id myadmin@k8master2
<enter the current OS admin password>
$ ssh-copy-id myadmin@k8master3
<enter the current OS admin password>
$ ssh-copy-id myadmin@k8worker1
<enter the current OS admin password>
$ ssh-copy-id myadmin@k8worker2
<enter the current OS admin password>
Test the access to each of the other nodes:
$ ssh myadmin@k8master2
(No password should be needed.)
Install Keepalive:
$ sudo apt-get install keepalived
Check if the bridge-nf-call-iptables = 1 via sysctl:
$ sudo sysctl net.bridge.bridge-nf-call-iptables
If not correct, then:
$ sudo sysctl net.bridge.bridge-nf-call-iptables=1
Edit the kubeadm configuration file.
$ sudo vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Add another Environment string after the other Environment strings:
Environment=”cgroup-driver=systemd/cgroup-driver=cgroupfs”
<esc>:wq (to save)
--> Loop and repeat for each node.
3. Prep and perform prerequisites on all master and worker nodes.
Do system updates:
$ sudo apt-get update && apt-get upgrade
Add k8s repository:
$ sudo curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ sudo vi /etc/apt/sourses.list.d/kubernetes.list
<a>
deb http://apt.kubernetes.io/ kubernetes-xenial main
<esc> :wq
$ sudo apt-get update
Install docker:
$ sudo apt-get install docker.io
Repeat these steps on all the other master and worker nodes.
4. Install K8s on all the master and worker nodes:
Install kubeadm, kubectl, and kubelet:
$ sudo apt-get install kubeadm kubectl kubelet
Note: This will install the current release.
- apt-transport-https is a prerequisite, but already installed when using the Ubuntu server image.
- To see what the current release is: $ sudo apt-cache policy kubeadm
- To install a specific release, perform something similar to: $ sudo apt-get install kubeadm=1.16.12-00 kubeadm=1.16.12-00 kubeadm=1.16.12-00
- To hold to the release so you don't accidentally upgrade, do: sudo apt-mark hold kubelet kubeadm kubectl
On the master nodes only, add kubelet autocomplete.
Verify bash-completion is installed:
$ type _init_completion
If it is not installed, install it with:
$ sudo apt-get install bash-completion
Add kubelet to the bash completion, by either:
- adding the completion script source into the .bashrc file. The first command below adds to current session, the second copies it to the .bashrc and makes it permanent:
$ echo 'source <(kubectl completion bash)' >>~/.bashrc
$ . ~/.bashrc
or
- adding the completion script to the bash_completion.d folder:
$ sudo kubectl completion bash >/etc/bash_completion.d/kubectl
Next: Repeat these steps on all the master and worker nodes.
5. Initiate the cluster primary master and install the RBAC and Calico network components:
Get the less than 500 node configuration for Calico (currently 3.10) at:
- docs.projectcalico.org/v3.x/getting-started/kubernetes/installation/calico
Set the following flags to the node-controller-manager. Use with kubeadm init or a YAML script.
Calico says to set these:
--cluster-cidr=192.168.0.0/16 --allocate-node-cidrs=true
Note:
The Calico 1.17 and 1.18 instructions don't mention --allocate-node-cidrs. We are no longer using that flag.
However, the pod subnet and the control-plane-endpoint are missing. So instead we could do this kubeadm init:
$ sudo kubeadm init --cluster-cidr=192.168.0.0/16 --allocate-node-cidrs=true --control-plane-endpoint=k8master.mindwatering.net --pod-network-cidr-=192.168.0.0/16
The 192.168.0.0/16 network is the default intra-cluster network that will be set-up w/Calico.
Alternately, we can configure using a YAML file that adds the same components. Create kubeadminit.yaml.
Update the kubernetesVersion to the one you've installed.
$ mkdir working
$ cd working
$ vi kubeadminit.yaml
<a>
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.16.2
controlPlaneEndpoint: "k8master.mindwatering.net"
networking:
podSubnet: 192.168.0.0/16
<esc>:wq
Note: Watch white-space indentation. Update the kubernetesVersion, the controlPlaneEndpoint, and the networking podSubnet lines, as applicable.
Initiate the cluster primary master using the YAML file:
$ sudo kubeadm init --config=kubeadminit.yaml --upload-certs > kubeadm-results.txt
Do the post init config file copy:
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
Verify you can access the k8s environment:
Confirm you can authenticate and access the K8s cluster:
$ kubectl cluster-info
< the API will return two sentences showing where the master is running, and CoreDNS would be running. CoreDNS would not be running until Calico, Flannel, or another network is added. >
Confirm we get see the master:
$ kubectl get nodes
< the k8master node is returned >
$ kubectl get pods --all-namespaces
< list of pods is returned, with the core-dns ones showing Pending >
Note: All should be running except the core-dns pods. They will have a status of Pending because the network is not yet installed.
Install the Calico network:
Note:
The 3.3 version has both an RBAC file and a calico manifest file to install for the less and 50 nodes set-up. As of 2019/11/9, the 3.3 page gives a warning that 3.3 is not the current release, and that 3.10 is the current release. Therefore, we visit the most current page. It doesn't appear that the RBAC YAML is needed anymore, so we are downloading the following instead:
$ cd ~/working
$ curl https://docs.projectcalico.org/v3.10/manifests/calico.yaml -O
If you have any changes to make to calico.yaml, like custom networking, then make them, otherwise just apply the file after downloading:
$ kubectl apply -f calico.yaml
< wait or watch the configuration initialize >
$ kubectl get pods --all-namespaces
Note:
After about 30 seconds the new calico pods and the coredns pods will initialize and change eventually to a status of Running. At this point all pods should show a Running status. If not, there is something wrong, and it must be fixed before continuing.
Check the primary master node:
$ kubectl describe node k8master1
If you want to run application (worker) pods on the master, do the following:
$ kubectl taint node k8master1 node-role.kubernetes.io/master:NoSchedule-
<node/k8master1 untainted>
If the DNS pods (coredns-x-y) and the calico nodes (calico-kube-controller-x-y and calico-node-mh.jwg) are not running, or if the node says "Not Ready" check that you set the calico.yaml and the kubeadminit.yaml networks to the same subnet configuration. This will typically show up as a CNI failure message on the node. If not, update the kubeadmininit.yaml file and re-apply it.
e.g.
$ journalctl -u kubelet
< confirm error message and fix YAML file >
$ kubectl apply -f calico.yaml
Next: Proceed directly to the next steps, step 6 and step 7, as you want to complete the master and worker set-ups before the certs expire in 24 hours.
6. Set-up the secondary master, k8master2.
SSH into the primary master using the Mac client so we can use copy-and-paste of the join command.
(Mac - Terminal Window)
$ ssh myadminid@k8master1.mindwatering.net
< accept ssh key, and enter password>
$ cd ~/working/
$ less kubeadm-results.txt
< scroll down until you see the kubeadm join command>
< copy the command with all three parts to the clipboard >
The command will look like this:
kubeadm join k8master.mindwatering.net:6443 --token k1uazc.1h0mwabbc9fp3c10 \
--discovery-token-ca-cert-hash sha256:7a1a7e55f971cb151529777a48e0df777a91111aba9cb47317ae7ef77f641aa1 \
--control-plane --certificate-key d77d3a12ea19ca27b51c77732b775197e322551a85b27a55cbeca51b77ce3c55
Open another SSH window using the Mac client to k8master2, and login.
(Mac - Terminal Window 2)
$ ssh myadminid@k8master2.mindwatering.net
< accept ssh key, and enter password>
$ sudo su
< enter password>
Paste the command and run it:
$ kubeadm join k8master.mindwatering.net:6443 --token k1uazc.1h0mwabbc9fp3c10 \
--discovery-token-ca-cert-hash sha256:7a1a7e55f971cb151529777a48e0df777a91111aba9cb47317ae7ef77f641aa1 \
--control-plane --certificate-key d77d3a12ea19ca27b51c77732b775197e322551a85b27a55cbeca51b77ce3c55
< wait about 30 to 40 seconds >
Open a third Terminal window, and repeat these steps for the third master.
7. Add the worker nodes to the cluster.
Important: The join command is the same for workers, except we need to drop the --control-plan line. This is basically what tells the join whether this is a master or worker node addition to a cluster.
Open another SSH window using the Mac client to k8worker1, and login.
(Mac - Terminal Window 4)
$ ssh myadminid@k8worker1.mindwatering.net
< accept ssh key, and enter password>
$ sudo su
< enter password>
Paste the command and run it:
$ kubeadm join k8master.mindwatering.net:6443 --token k1uazc.1h0mwabbc9fp3c10 \
--discovery-token-ca-cert-hash sha256:7a1a7e55f971cb151529777a48e0df777a91111aba9cb47317ae7ef77f641aa1
< wait about 30 to 40 seconds >
Open a fifth Terminal window, and repeat these steps for the other worker.
Deploying apps/services:
- You can deploy by YAML locally or via a URL, or by the kubectl via the run or deploy command (verb).
e.g.
$ kubernetes-master:~$ kubectl run --image=nginx nginx-svr --port=80 --env="DOMAIN=cluster"
$ kubernetes-master:~$ kubectl expose deployment nginx-svr --port=80 --name=nginx-svr-http
Note:
In interesting "feature" is that you can specify a port other than -80, say 8080, and the deployment will show the custom port, but the actual http.conf.d configuration file will still show port 80. The port specification doesn't do anything inside the container. We have to use targetPort and Port to map the port from 80 to 8080 for external consumption in the cluster/world when we create the service. The kubectl run can specify a port, where the kubectl create cannot. Both can create YAML which then could be customized to add a port.
$ kubectl run nginx-svr --image=nginx --port=80 --dry-run -o yaml > kubectl-run.yaml
$ vi kubectl-run.yaml
. . .
$ kubectl create -f kubectl-run.yaml
Expose the service as NodePort:
$ kubectl expose pod nginx-svr --port=8080 --target-port=80 --type=NodePort --name=nginxsvr-svc
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 39d
nginx-one NodePort 10.110.255.202 <none> 80:30604/TCP 64m
nginxsvr-svc NodePort 10.107.147.185 <none> 8080:31624/TCP 97s
In the above example:
1. nginx-one runs with intra-cluster port 80, and NodePort 30604
2. nginxsvr-svc runs w/in the pod on port 80 (nginx default), but intra-cluster at port 8080, and NodePort 31624. You cannot see it here, but the /etc/http.conf.d config file still has the default port 80 set.
$ curl 10.110.255.202
< returns default NGINX welcome page >
$ curl 10.107.147.185
< times out >
$ curl 10.107.147.185:8080
< returns default NGINX welcome page >
- To view your new deployment on your worker nodes (hosts), either run on the master:
$ kubectl get pods --namespace=all
or, if on the worker node
$ sudo docker ps
To test the new nginx-server, get the Service IP created with the expose command using the k8master server API:
$ kubectl get svc
< note the cluster IP for nginx-svr-http service >
$ curl -I <IP>
previous page
|