k8s故障排查 | 云深不知处

type

status

date

slug

summary

虚拟机重启后应用web页面无法打开-flannel问题

1. 现象

虚拟机重启后，pod自动启动，但是之前可以正常使用的web页面无法打开，就像服务死掉了。挑选一个pod查看日志，服务正常，但是看到网络超时的记录。

2. 排查

由于虚拟机环境的cni插件是flannel，故先看插件VTEP/flannel.1的状态，其中一台节点明显异常：只有mac地址，无ip地址


# ifconfig  flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::b8f8:a6ff:fec8:4027  prefixlen 64  scopeid 0x20<link>
        ether ba:f8:a6:c8:40:27  txqueuelen 0  (Ethernet)
        RX packets 54265  bytes 50594927 (48.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 30089  bytes 6922450 (6.6 MiB)
        TX errors 6  dropped 15 overruns 0  carrier 6  collisions 0

查看该节点的flannel pod日志


I0506 09:20:00.627283       1 main.go:435] Wrote subnet file to /run/flannel/subnet.env
I0506 09:20:00.627306       1 main.go:439] Running backend.
I0506 09:20:00.627677       1 iptables.go:274] generated 3 rules
I0506 09:20:00.627742       1 vxlan_network.go:62] watching for new subnet leases
E0506 09:20:00.628062       1 retry.go:29] #0: network is down
I0506 09:20:00.635400       1 main.go:460] Waiting for all goroutines to exit
I0506 09:20:00.651282       1 iptables.go:267] bootstrap done
I0506 09:20:00.656369       1 iptables.go:267] bootstrap done
E0506 09:20:00.810596       1 retry.go:29] #1: network is down
E0506 09:20:01.078769       1 retry.go:29] #2: network is down
E0506 09:20:01.515031       1 retry.go:29] #3: network is down
E0506 09:20:02.403103       1 retry.go:29] #4: network is down
E0506 09:20:04.053005       1 retry.go:29] #5: network is down
E0506 09:20:07.286808       1 retry.go:29] #6: network is down
E0506 09:20:13.721814       1 retry.go:29] #7: network is down
E0506 09:20:26.606154       1 retry.go:29] #8: network is down
E0506 09:20:52.287223       1 retry.go:29] #9: network is down
E0506 09:20:52.291654       1 vxlan_network.go:209] failed to add vxlanRoute (10.244.1.0/24 -> 10.244.1.0): All attempts fail:

说明该节点的flannel状态异常，导致分配路由出错。

3. 解决

删除pod，故障依旧。

ifconfig flannel.1 down 之后ifconfig flannel.1 up ，故障依旧。

重启该节点，再次查看ifconfig flannel.1 ，恢复正常。