The core flow (ClusterIP)#

Key moving parts:

• Service: defines virtual IP + ports
• Endpoints/EndpointSlice: list of ready backend Pod IPs
• kube-proxy: installs dataplane rules to select a backend

kube-proxy modes (iptables vs IPVS)#

In iptables mode, kube-proxy programs DNAT rules that map ClusterIP → Pod IP. In IPVS mode, kube-proxy creates virtual servers and real servers inside the kernel’s IPVS table.

eBPF dataplanes (Cilium and others)#

Modern CNIs can bypass kube-proxy with eBPF:

• Service lookups happen in BPF maps instead of iptables
• Faster updates and better observability
• Requires kernel support and CNI integration

How eBPF works in the kernel (short, practical view)#

At a high level, eBPF programs are small, verified bytecode programs that run safely inside the Linux kernel. They are attached to specific hook points (network ingress/egress, socket ops, tracepoints) and can:

• Read packet metadata
• Look up values in BPF maps (shared key/value data structures)
• Make forwarding decisions (drop/redirect/mark)

In a Service dataplane, the program typically:

1. receives a packet at a hook (XDP or TC)
2. looks up the Service VIP/port in a BPF map
3. selects a backend Pod IP
4. rewrites the packet and forwards it

XDP vs TC (where you hook matters)#

XDP runs at the earliest point (driver RX path) and can drop/redirect packets before the kernel allocates socket buffers. TC runs later (qdisc layer) with richer context but more overhead.

Rule of thumb: use XDP for ultra-fast filtering and TC when you need skb metadata or more complex actions.

XDP/TC program examples (minimal, real-ish C)#

Below is a deliberately small example that shows a map lookup and a simple rewrite. It is simplified and omits full parsing, checksum updates, and error handling.

Compile and attach (example only):

Tail calls (program chaining): Tail calls let one eBPF program jump to another without returning, enabling modular pipelines (e.g., parsing → service lookup → policy). They reduce verifier complexity and allow large logic to be split across programs.

Key constraints:

• Requires a BPF map of programs (PROG_ARRAY)
• Limited by MAX_TAIL_CALL_CNT (default 32)
• If the target program is missing, execution falls back

BPF maps: types and performance tradeoffs#

Maps are shared kernel data structures used for service backends, policy, and counters.

In service dataplanes, the hot path usually reads from hash/LRU maps and increments per-CPU counters.

Map pinning and bpffs (sharing state safely)#

By default, maps live only as long as the creating process. Pinning stores them in bpffs (usually /sys/fs/bpf) so they survive process exit and can be shared across programs.

Inspect programs and maps (bpftool and Cilium)#

If you are running a kube-proxy replacement (for example Cilium), you can verify actual programs and maps like this:

Cilium-specific shortcuts:

Verifier limits (why programs fail to load): The verifier guarantees safety and termination before a program is loaded. Common pitfalls:

• Too many instructions (program too large)
• Unbounded loops (loop not proven finite)
• Invalid pointer math (unsafe memory access)
• Stack size too large (512 bytes per frame by default)
• Map access without bounds checks

Practical tips:

• Break logic into smaller programs with tail calls
• Use maps instead of large stack objects
• Keep helper usage minimal in the hot path

BPF vs eBPF (what actually changed)#

Classic BPF (cBPF) was designed for packet filtering (tcpdump). eBPF is a major extension that makes BPF a general-purpose, safe kernel VM.

Key kernel pieces:

• Verifier: proves termination and memory safety before load
• JIT: turns bytecode into native instructions for speed
• Helpers: controlled access to kernel data/functions
• Maps: fast shared state (hash, LRU, array, etc.)

Service types in practice#

• ClusterIP: internal virtual IP (default)
• NodePort: exposes a port on every node
• LoadBalancer: cloud LB + ClusterIP + NodePort
• Headless: no ClusterIP; DNS returns Pod IPs directly

Service behavior details#

Traffic policy and client IP:

• externalTrafficPolicy: Cluster → load balance across nodes, client IP may be SNAT’d
• externalTrafficPolicy: Local → preserve client IP, but only nodes with local Pods receive traffic

Endpoint readiness and probe effects: Only Ready endpoints receive traffic. A failing readiness probe removes a Pod from Service backends without killing the container. This is why readiness governs routing and liveness governs restart.

Debug checklist (practical)#

1) Inspect Services and endpoints

2) Confirm kube-proxy mode

3) Inspect dataplane rules

• iptables:

• IPVS:

• eBPF (Cilium):

4) Validate DNS resolution

Common failure patterns#

• No endpoints: selector mismatch or Pods not Ready
• Traffic blackhole: kube-proxy not running / CNI issues
• Wrong backend: stale endpoints or CNI cache
• Client IP missing: externalTrafficPolicy set to Cluster

Wrap-up#

Kubernetes Service networking is just virtual IPs plus backend lists. The dataplane can be iptables/IPVS or eBPF, but the model stays the same: ClusterIP points to ready endpoints. Once you understand where that translation happens, Service debugging becomes straightforward.

Previous: Part 3 — How a Pod Actually Runs.

Series: Kubernetes Internals: How the Cluster Actually Works