Issue:
Following a VM reboot, the Node API does not become available on one of the Ubuntu-based nodes.
Cannot do kubectl get pods -n=kube-system.
DNS Resolution is actually in /etc/hosts for the k8master generic address to HAProxy. Tested changing host file DNS to k8master1 doesn't make a difference since port 6443 to the API server isn't responding.
Message:
$ kubectl get nodes
The connection to the server <myip>:6443 was refused - did you specify the right host or port?
$ sudo servicectl status kubelet
Nov 21 17:46:28 k8master1 kubelet[5843]: E1121 17:46:28.308226 5843 kubelet.go:2267] node "k8master1" not found
Nov 21 17:46:28 k8master1 kubelet[5843]: E1121 17:46:28.408792 5843 kubelet.go:2267] node "k8master1" not found
Nov 21 17:46:28 k8master1 kubelet[5843]: E1121 17:46:28.507814 5843 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://k8master.mindwatering.net:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192
Ping of k8master1 works, but there is not service listening on 6443.
Steps for Resolution:
Using the kublet service log, we found that etcd container isn't loading:
Nov 21 18:24:32 k8master1 kubelet[6990]: E1121 18:24:32.549627 6990 pod_workers.go:191] Error syncing pod ba4aa0298e806609aaafcfef39af5e0d ("etcd-k8master1_kube-system(ba4aa0298e806609aaafcfef39af5e0d)"), skipping: failed to "StartContainer" for "etcd" with CrashLoopBackOff: "back-off 2m40s restarting failed container=etcd pod=etcd-k8master1_kube-system(ba4aa0298e806609aaafcfef39af5e0d)"
Checked to see if k8master2 and k8master3 were running again. k8master3 was started, but ETCD container was not yet running either.
It appears that k8master1 needed k8master2, in order to come up. Once k8master1 started coming up, k8master3 started coming up.
The logs showed that k8master2 was the last one to be rebooted. Evidently, that is important?
We shutdown k8master3, first and then k8master2 and k8master1 at the "same" time. However, because #3 was shutdown a few seconds before the other two, we are now thinking that it's ETCD was likely too out of sync to be useful to k8master1 or k8master2 when they come up. So k8master1 had to wait for 8master2 to sync and it evidently takes a while.
End comments:
Shutting down 1 first was definitely a mistake. If we are going to shutdown all of cluster, it appears that we want to shutdown all VMs all at the same time.
Just wait.
By the time we looped back to k8master1, the kubectl was able to talk to the API service. The logs showed:Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.223733 6990 kuberuntime_manager.go:961] updating runtime config through cri with podcidr 192.168.0.0/24
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.252805 6990 docker_service.go:355] docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:192.168.0.0/24,},}
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.253943 6990 kubelet_network.go:77] Setting Pod CIDR: -> 192.168.0.0/24
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.306828 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/77552120-2024-4ce5-bcd1-292092624be7-xtables-lock") pod "kube-proxy-4295m" (UID: "77552120-2024-4ce5-bcd1-292092624be7")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.306853 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/77552120-2024-4ce5-bcd1-292092624be7-lib-modules") pod "kube-proxy-4295m" (UID: "77552120-2024-4ce5-bcd1-292092624be7")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.306869 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-zzrfb" (UniqueName: "kubernetes.io/secret/77552120-2024-4ce5-bcd1-292092624be7-kube-proxy-token-zzrfb") pod "kube-proxy-4295m" (UID: "77552120-2024-4ce5-bcd1-292092624be7")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.306890 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-p4wpf" (UniqueName: "kubernetes.io/secret/8df8e656-53f6-4a69-afee-034f30273ce2-coredns-token-p4wpf") pod "coredns-5644d7b6d9-ljkbf" (UID: "8df8e656-53f6-4a69-afee-034f30273ce2")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.306969 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "cni-bin-dir" (UniqueName: "kubernetes.io/host-path/1942f4a3-8c29-4320-b794-24cbbe8949fe-cni-bin-dir") pod "calico-node-p8r6g" (UID: "1942f4a3-8c29-4320-b794-24cbbe8949fe")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.306989 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "var-run-calico" (UniqueName: "kubernetes.io/host-path/1942f4a3-8c29-4320-b794-24cbbe8949fe-var-run-calico") pod "calico-node-p8r6g" (UID: "1942f4a3-8c29-4320-b794-24cbbe8949fe")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307002 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "host-local-net-dir" (UniqueName: "kubernetes.io/host-path/1942f4a3-8c29-4320-b794-24cbbe8949fe-host-local-net-dir") pod "calico-node-p8r6g" (UID: "1942f4a3-8c29-4320-b794-24cbbe8949fe")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307014 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/77552120-2024-4ce5-bcd1-292092624be7-kube-proxy") pod "kube-proxy-4295m" (UID: "77552120-2024-4ce5-bcd1-292092624be7")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307058 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "calico-kube-controllers-token-zjlwb" (UniqueName: "kubernetes.io/secret/ffdaa5bd-1891-4417-a180-e0380bd86783-calico-kube-controllers-token-zjlwb") pod "calico-kube-controllers-6b64bcd855-zj8s5" (UID: "ffdaa5bd-1891-4417-a180-e0380bd86783")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307072 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "var-lib-calico" (UniqueName: "kubernetes.io/host-path/1942f4a3-8c29-4320-b794-24cbbe8949fe-var-lib-calico") pod "calico-node-p8r6g" (UID: "1942f4a3-8c29-4320-b794-24cbbe8949fe")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307104 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/1942f4a3-8c29-4320-b794-24cbbe8949fe-xtables-lock") pod "calico-node-p8r6g" (UID: "1942f4a3-8c29-4320-b794-24cbbe8949fe")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307124 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvol-driver-host" (UniqueName: "kubernetes.io/host-path/1942f4a3-8c29-4320-b794-24cbbe8949fe-flexvol-driver-host") pod "calico-node-p8r6g" (UID: "1942f4a3-8c29-4320-b794-24cbbe8949fe")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307157 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-p4wpf" (UniqueName: "kubernetes.io/secret/7b796ddf-db95-49e0-b1dd-a09d62c69cb7-coredns-token-p4wpf") pod "coredns-5644d7b6d9-7tbrn" (UID: "7b796ddf-db95-49e0-b1dd-a09d62c69cb7")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307170 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/7b796ddf-db95-49e0-b1dd-a09d62c69cb7-config-volume") pod "coredns-5644d7b6d9-7tbrn" (UID: "7b796ddf-db95-49e0-b1dd-a09d62c69cb7")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307204 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/1942f4a3-8c29-4320-b794-24cbbe8949fe-lib-modules") pod "calico-node-p8r6g" (UID: "1942f4a3-8c29-4320-b794-24cbbe8949fe")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307220 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "cni-net-dir" (UniqueName: "kubernetes.io/host-path/1942f4a3-8c29-4320-b794-24cbbe8949fe-cni-net-dir") pod "calico-node-p8r6g" (UID: "1942f4a3-8c29-4320-b794-24cbbe8949fe")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307251 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "policysync" (UniqueName: "kubernetes.io/host-path/1942f4a3-8c29-4320-b794-24cbbe8949fe-policysync") pod "calico-node-p8r6g" (UID: "1942f4a3-8c29-4320-b794-24cbbe8949fe")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307301 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-79ffj" (UniqueName: "kubernetes.io/secret/375e130a-d242-4e28-8cbe-68b5f76c226d-default-token-79ffj") pod "nginx-demo-89fcbc74c-fx9rm" (UID: "375e130a-d242-4e28-8cbe-68b5f76c226d")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307335 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-7z79v" (UniqueName: "kubernetes.io/secret/539a9985-bd89-4f27-886e-844e70c7d1e0-default-token-7z79v") pod "ingress-default-backend-558fbc9b46-dq2ft" (UID: "539a9985-bd89-4f27-886e-844e70c7d1e0")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307353 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "calico-node-token-vm5d5" (UniqueName: "kubernetes.io/secret/1942f4a3-8c29-4320-b794-24cbbe8949fe-calico-node-token-vm5d5") pod "calico-node-p8r6g" (UID: "1942f4a3-8c29-4320-b794-24cbbe8949fe")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307476 6990 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/8df8e656-53f6-4a69-afee-034f30273ce2-config-volume") pod "coredns-5644d7b6d9-ljkbf" (UID: "8df8e656-53f6-4a69-afee-034f30273ce2")
Nov 21 18:34:58 k8master1 kubelet[6990]: I1121 18:34:58.307484 6990 reconciler.go:154] Reconciler: start to sync state
previous page
|