Calico VXLAN 使用指南

发布时间：2026/6/23 1:22:22

这也是 Overlay 网络的核心利用真实的 IP 网络三层作为运输通道把原本只能在同一交换机下生效的二层 MAC 机制用隧道技术伪装成一个大的虚拟局域网。不同于传统二层依赖广播学习 MACOverlay 通常由控制平面主动下发映射来避免广播风暴。此处的控制平面指的是跑在每个节点上的 felixrootnetwork-demo:~# kubectl get pods -n kube-system | grep ‘calico-node’calico-node-696p5 1/1 Running 1 (5d5h ago) 7d3hcalico-node-dxz57 1/1 Running 1 (5d5h ago) 7d3hrootnetwork-demo:~# docker exec calico-vxlan-control-plane ss -nltup | grep -i calicotcp LISTEN 0 4096 127.0.0.1:9099 0.0.0.0:* users:((“calico-node”,pid1284,fd9))rootnetwork-demo:~# docker exec calico-vxlan-control-plane ps aux | grep ‘-felix’root 1284 0.8 0.2 2117200 64012 ? Sl Mar31 66:17 calico-node -felix使用场景场景是否推荐说明Azure 环境 ✅ Azure 封禁 IPIP 协议VXLAN 是唯一 overlay 选项IPv6 环境 ✅ IPIP 仅支持 IPv4VXLAN 是 IPv6 overlay 唯一选择启用 eBPF ✅ eBPF 模式不支持 IPIP必须用 VXLANAWS 多 AZ / 跨子网 ✅ 同 AZ 直通、跨 AZ 封装 → vxlanMode: CrossSubnet 性能更优公有云 ✅ 无法配 BGP 或底层路由 → vxlanMode: Always 开箱即用快速使用 ✅ 不折腾底层网络直接 overlay 起来可关闭 BGP 减少组件自建机房 ❌ 有完整网络控制权 → 无封装 BGP 是官方首选Pod IP 需集群外可达 ❌ Overlay 下 Pod IP 不可路由到外部 → BGP 纯路由模式部署流程通过 Kind 快速生成集群并部署 Calico VXLan 模式#!/bin/bashset -v1. prep NoCNI environmentcat EOF | HTTP_PROXY HTTPS_PROXY http_proxy https_proxy kind create cluster --namecalico-vxlan --imagekindest/node:v1.27.3 --config-kind: ClusterapiVersion: kind.x-k8s.io/v1alpha4networking:disableDefaultCNI: truenodes:- role: control-plane- role: workerEOF2. Remove taintscontroller_node_ipkubectl get node -o wide --no-headers | grep -E control-plane|bpf1 | awk -F {print $6}kubectl taint nodes $(kubectl get nodes -o name | grep control-plane) node-role.kubernetes.io/control-plane:NoSchedule-kubectl get nodes -o wide3. Collect startup messagecontroller_node_nameKaTeX parse error: Undefined control sequence: \n at position 68: …etadata.name}{\̲n̲}{end} | grep…controller_node_name ]; thentimeout 1 docker exec -t $controller_node_name bash -c ‘cat EOF /root/monitor_startup.sh#!/bin/baship -ts monitor all /root/startup_monitor.txt 21EOFchmod x /root/monitor_startup.sh /root/monitor_startup.sh’elseecho “No such controller_node!”fi4. Install CNI[Calico v3.23.2]kubectl apply -f calico.yamlcalico.yamlhttps://gitee.com/rowan-wcni/wcni-kind/blob/master/LabasCode/calico/03-calico-vxlan/calico.yaml创建测试 Pod实际就是 Nginx仅用于后续互访时抓包apiVersion: apps/v1kind: DaemonSetmetadata:labels:app: wluoname: wluospec:selector:matchLabels:app: wluotemplate:metadata:labels:app: wluospec:containers:- image: burlyluo/nettool:latestname: nettoolboxenv:- name: NETTOOL_NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeNamesecurityContext:privileged: true查看部署结果rootnetwork-demo:~# kubectl get pods -A -o wideNAMESPACE NAME READY STATUS RESTARTS AGE IP NODEdefault wluo-nkmtp 1/1 Running 1 47h 10.244.177.66 calico-vxlan-workerdefault wluo-qf5cc 1/1 Running 1 47h 10.244.110.74 calico-vxlan-control-planekube-system calico-kube-controllers-5b5ff6cb74-fp4r9 1/1 Running 1 47h 10.244.110.72 calico-vxlan-control-planekube-system calico-node-696p5 1/1 Running 1 47h 172.18.0.2 calico-vxlan-workerkube-system calico-node-dxz57 1/1 Running 1 47h 172.18.0.3 calico-vxlan-control-planekube-system coredns-5d78c9869d-w4d2g 1/1 Running 1 47h 10.244.110.71 calico-vxlan-control-planekube-system coredns-5d78c9869d-zl7qg 1/1 Running 1 47h 10.244.110.70 calico-vxlan-control-planekube-system etcd-calico-vxlan-control-plane 1/1 Running 1 47h 172.18.0.3 calico-vxlan-control-planekube-system kube-apiserver-calico-vxlan-control-plane 1/1 Running 1 47h 172.18.0.3 calico-vxlan-control-planekube-system kube-controller-manager-calico-vxlan-control-plane 1/1 Running 1 47h 172.18.0.3 calico-vxlan-control-planekube-system kube-proxy-rsf5n 1/1 Running 1 47h 172.18.0.2 calico-vxlan-workerkube-system kube-proxy-zn8vq 1/1 Running 1 47h 172.18.0.3 calico-vxlan-control-planekube-system kube-scheduler-calico-vxlan-control-plane 1/1 Running 1 47h 172.18.0.3 calico-vxlan-control-plane验证效果查询 Node 节点 VXLAN 设备、FDB、ARP、路由表信息节点信息rootnetwork-demo:~# kubectl get node -o wideNAME STATUS ROLES AGE VERSION INTERNAL-IPcalico-vxlan-control-plane Ready control-plane 47h v1.27.3 172.18.0.3calico-vxlan-worker Ready 47h v1.27.3 172.18.0.2Pod 信息rootnetwork-demo:~# kubectl get pods -o wideNAME READY STATUS RESTARTS AGE IP NODEwluo-nkmtp 1/1 Running 1 47h 10.244.177.66 calico-vxlan-workerwluo-qf5cc 1/1 Running 1 47h 10.244.110.74 calico-vxlan-control-plane1.查询 vxlan.calico 设备信息开头提到 vxlan 模式是把一个二层网络封装到 UDP 包里通过三层网络传输。那么 vxlan.calico 设备就是主机上的 VTEP 设备用来做封装/解封装。rootcalico-vxlan-control-plane:/# ip -d link show vxlan.calico6: vxlan.calico: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group defaultlink/ether 66:40:d4:7a:17:07 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535vxlan id 4096 local 172.18.0.3 dev eth0 srcport 0 0 dstport 4789 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535rootcalico-vxlan-control-plane:/# ip address show vxlan.calico6: vxlan.calico: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1450 qdisc noqueue state UNKNOWN group defaultlink/ether 66:40:d4:7a:17:07 brd ff:ff:ff:ff:ff:ffinet 10.244.110.64/32 scope global vxlan.calicovalid_lft forever preferred_lft forever2.查询节点路由表查询路由表发现访问对端 Pod 时通过 vxlan.calico 设备进行转发路由信息中 onlink 标志表示认为下一跳位于本地链路上即使该网关不在同一网段。因为实际可达性由 VXLAN 隧道保证而不是直接 L3 可达rootnetwork-demo:~# docker exec -it calico-vxlan-control-plane ip route show | grep ‘10.244.177.64’访问 10.244.177.66 走这条路由子网掩码 /26 包含具体看下面 ipcalc 结果10.244.177.64/26 via 10.244.177.64 dev vxlan.calico onlinkipcalc 10.244.177.64/26Address: 10.244.177.64 00001010.11110100.10110001.01 000000Netmask: 255.255.255.192 26 11111111.11111111.11111111.11 000000Wildcard: 0.0.0.63 00000000.00000000.00000000.00 111111Network: 10.244.177.64/26 00001010.11110100.10110001.01 000000HostMin: 10.244.177.65 00001010.11110100.10110001.01 000001HostMax: 10.244.177.126 00001010.11110100.10110001.01 111110Broadcast: 10.244.177.127 00001010.11110100.10110001.01 111111Hosts/Net: 62 Class A, Private Internet效果同 ip route showrootnetwork-demo:~# docker exec -it calico-vxlan-control-plane route -n | grep ‘10.244.177.64’Kernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface10.244.177.64 10.244.177.64 255.255.255.192 UG 0 0 0 vxlan.calico3.查询节点 ARP 表跨节点 Pod 流量走 VXLAN 隧道。每条远端 Pod IP → VTEP MAC 的映射由 FelixCalico agent 主动写入邻居表并标记为 PERMANENT。既然是远端这里的 66:aa:ae:c0:d0:7a MAC 肯定是对端 Pod Node VTEP MAC。rootnetwork-demo:~# docker exec calico-vxlan-control-plane ip neighbor show | grep ‘10.244.177.64’10.244.177.64 dev vxlan.calico lladdr 66:aa:ae:c0:d0:7a PERMANENTrootnetwork-demo:~# docker exec calico-vxlan-control-plane arp -n | grep ‘10.244.177.64’Address HWtype HWaddress Flags Mask Iface10.244.177.64 ether 66:aa:ae:c0:d0:7a CM vxlan.calico4.查询节点 FDB 表vxlan.calico 是 vxlan 设备请求经过他时会通过 FDB 表查询rootnetwork-demo:~# docker exec -it calico-vxlan-control-plane bridge fdb show | grep ‘172.18.0.2’如果要发 VXLAN 帧给这个 MAC封装后的外层目的 IP 是 172.18.0.266:aa:ae:c0:d0:7a dev vxlan.calico dst 172.18.0.2 self permanent查询 Pod 路由、ARP、网络设备信息验证路由规则、calico 生成的网络设备开启 proxy_arp 后的效果1.查询 Pod 网卡信息Pod 所处网段子网掩码为 32即该网段只存在 Pod IP/32 意味着 Pod 是孤岛任何其他 IP 都不在本地网段rootnetwork-demo:~# kubectl exec wluo-qf5cc – ip address show eth04: eth0if11: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1450 qdisc noqueue state UP group defaultlink/ether a2:7f:ae:c6:af:99 brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 10.244.110.74/32 scope global eth0valid_lft forever preferred_lft forever2.查询 Pod 路由与 Calico IPIP 模式一样容器默认网关的 IP 169.254.1.1 是什么其实无所谓。因为通过 scope link 配置后这条路由被标记为本地链路路由通信走的是二层转发依赖的是 MAC 地址而非 IP 地址。rootnetwork-demo:~# kubectl exec wluo-qf5cc – ip route showdefault via 169.254.1.1 dev eth0169.254.1.1 dev eth0 scope linkrootnetwork-demo:~# kubectl exec wluo-qf5cc – route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface0.0.0.0 169.254.1.1 0.0.0.0 UG 0 0 0 eth0169.254.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth03.查询 Pod ARP 信息rootnetwork-demo:~# kubectl exec wluo-qf5cc – ip neighbor show172.18.0.3 dev eth0 lladdr ee:ee:ee:ee:ee:ee STALE169.254.1.1 dev eth0 lladdr ee:ee:ee:ee:ee:ee STALErootnetwork-demo:~# kubectl exec wluo-qf5cc – arp -nAddress HWtype HWaddress Flags Mask Iface172.18.0.3 ether ee:ee:ee:ee:ee:ee C eth0169.254.1.1 ether ee:ee:ee:ee:ee:ee C eth0Pod 网卡处抓包rootnetwork-demo:~# kubectl exec wluo-qf5cc – curl -s 10.244.177.66PodName: wluo-nkmtp | PodIP: eth0 10.244.177.66/32imageimageNode 网卡处抓包