When a Pod is created, kubelet invokes CNI with a JSON config and environment variables. The plugin creates veth pairs, moves one end into the Pod namespace, assigns IPs, and returns a result to kubelet.

What kubelet passes (common env vars):

• CNI_COMMAND = ADD / DEL / CHECK
• CNI_CONTAINERID = container ID
• CNI_NETNS = path to Pod netns
• CNI_IFNAME = interface name (usually eth0)
• CNI_PATH = where plugin binaries live

Config locations:

• Binaries: /opt/cni/bin
• Configs: /etc/cni/net.d/*.conf or *.conflist

Plugin chain: main + IPAM + optional meta plugins#

Most real deployments use a chain:

• Main plugin: creates the interface and sets up connectivity (bridge, macvlan, vxlan, eBPF).
• IPAM plugin: allocates IPs (host-local, DHCP, cloud IPAM).
• Meta plugins: tuning, bandwidth, portmap.

Example: bridge + host-local IPAM#

This is a minimal CNI config that creates a Linux bridge and hands out Pod IPs from a local pool.

Data path: what CNI actually builds#

Most CNIs create a veth pair between the Pod namespace and the host. The host side is attached to a bridge or virtual switch, and routes are installed so Pods can reach each other.

Encapsulation and MTU (the hidden sharp edge)#

If your CNI overlays traffic (VXLAN/Geneve), it adds headers and reduces usable MTU. If MTU is too large, you see TCP stalls and intermittent failures.

Practical rule of thumb:

• Underlay MTU 1500, VXLAN overhead ~50 bytes -> set Pod MTU to ~1450

IPAM details that matter#

IP allocation is simple until it is not:

• host-local stores state on each node, easy to run out or leak
• cluster-wide IPAM (etcd, cloud IPAM, Cilium) avoids collisions
• Misaligned subnets cause overlap or blackholed routes

Policy and Service routing: CNI vs kube-proxy#

CNI is responsible for Pod interfaces and routing, but Service load balancing is often done by kube-proxy or eBPF dataplanes:

• iptables/IPVS (kube-proxy)
• eBPF (Cilium, kube-proxy replacement)

Think of it as:

• CNI: connect Pods to the network
• Service dataplane: translate ClusterIP to Pods

CNI deep dive: design philosophies and architectures#

Each popular CNI reflects a strong opinion about what “good cluster networking” looks like. The choice is often less about features and more about architecture and operational fit.

Calico: routable L3 and policy-first#

Philosophy: “Make pods first-class IPs.” Calico prefers pure L3 routing and strong policy enforcement instead of hiding networking behind overlays.

Architecture (high-level):

• Felix on every node programs routes and policy.
• BGP (via BIRD/GoBGP) distributes routes; can run in node-to-node mesh or with route reflectors.
• Dataplane: iptables or eBPF.
• IPAM: Calico IP pools; can run overlay (IP-in-IP/VXLAN) when needed.

Strengths:

• Excellent NetworkPolicy coverage and enterprise policy controls.
• Works well with routable infrastructure and bare metal.
• Flexible: pure routing, overlay, or hybrid.

Tradeoffs:

• BGP adds operational complexity.
• eBPF mode needs newer kernels and tooling.
• More moving parts than simple overlays.

Flannel: simplicity-first connectivity#

Philosophy: “Just connect the Pods.” Flannel focuses on the simplest possible way to make pod-to-pod communication work.

Architecture (high-level):

• flanneld allocates a subnet per node and stores it in etcd or the Kubernetes API.
• Data path options: VXLAN (default), host-gw, or UDP.

Strengths:

• Easy to deploy and understand.
• Minimal control plane and low operational overhead.

Tradeoffs:

• No NetworkPolicy by itself.
• Overlay overhead (MTU/encapsulation).
• Limited observability and advanced features.

Cilium: eBPF-native dataplane and observability#

Philosophy: “Policy and visibility should live in the kernel.” Cilium replaces large iptables rule sets with eBPF programs and BPF maps.

Architecture (high-level):

• cilium-agent on each node loads eBPF programs at XDP/TC and socket layers.
• BPF maps store service backends, identities, and conntrack state.
• cilium-operator manages IPAM and cluster-wide state.
• Optional kube-proxy replacement and Hubble for flow visibility.

Strengths:

• High performance at scale; fast service load balancing.
• Rich L3–L7 policy and deep observability.
• Great fit for modern, high-throughput clusters.

Tradeoffs:

• Kernel version requirements are stricter.
• Debugging requires BPF tooling expertise.
• Memory footprint can be higher due to map state.

Istio CNI: service-mesh plumbing (not primary networking)#

Philosophy: “Avoid privileged initContainers.” Istio CNI exists to move traffic redirection setup into a CNI chain step.

Architecture (high-level):

• Chained CNI that configures iptables rules in the Pod netns.
• Enables sidecar interception without NET_ADMIN in the pod.
• Not a full CNI: it depends on another CNI for IP allocation and routing.

Strengths:

• Improves security posture for service mesh deployments.
• Simplifies Pod security policies.

Tradeoffs:

• Adds CNI chain complexity and version alignment concerns.
• Does not provide the actual pod network.

Performance notes#

• iptables-heavy CNIs can slow down when rule counts are large.
• eBPF CNIs reduce rule churn and improve observability.
• Conntrack limits are a common bottleneck for high-connection services.

Troubleshooting checklist (practical)#

1) Validate CNI config and binaries

2) Check Pod netns and interfaces

3) Inspect routes and bridge

4) If using Cilium

Common failure patterns#

• Pod has no IP: IPAM failure or CNI config missing
• Pod can not reach cluster: routes missing on host or bridge down
• Intermittent timeouts: MTU mismatch or conntrack exhaustion
• Pod stuck in ContainerCreating: CNI binary missing or wrong CNI_PATH

Wrap-up#

CNI is the glue between Kubernetes and Linux networking. Once you see it as a contract and a lifecycle (ADD/DEL), debugging becomes much more straightforward.

Previous: Part 4 — Service Networking.

Series: Kubernetes Internals: How the Cluster Actually Works