> your AI agent picks dependencies from memory; give it dated facts — try starlog.dev ↗ vet your agent's deps ↗ vibe-coding is fine. vibe-importing isn’t. — try starlog.dev ↗ vibe-importing isn’t fine ↗ your agent has never seen your private packages — try starlog.dev ↗ facts for private packages ↗ a linter for the dependencies your AI agent picks — try starlog.dev ↗ a linter for agent deps ↗ whois is redacted, cdns mask the rest — get the real operator — try whoisgeni.us ↗ who really runs that domain ↗ domain attribution that shows its work — full evidence chain — try whoisgeni.us ↗ domain intel w/ evidence ↗

← Back to Articles

Inside Kubernetes: Why Reconciliation Loops Beat Event-Driven Orchestration

[ View on GitHub ]

Inside Kubernetes: Why Reconciliation Loops Beat Event-Driven Orchestration

Hook

Kubernetes doesn't react to events—it ignores them entirely. Every controller continuously reconciles actual state with desired state, treating events as mere optimization hints. This architectural choice is why your pods eventually start, even when half the control plane crashes.

Context

Before Kubernetes, container orchestration meant scripting deployment sequences: SSH into boxes, pull images, start containers, wire up networking, pray nothing fails mid-process. Tools like Capistrano and Fabric automated the steps but remained fundamentally imperative—you described how to deploy, not what should run. When something crashed, you got paged. These systems were edge-triggered: they responded to events (a container died! a node rebooted!) but had no memory, no way to know if the resulting state matched intent.

Google built Borg and Omega to manage containers at planetary scale, learning that declarative systems with reconciliation loops survive chaos better than event-driven ones. Kubernetes, released in 2014 as Borg's open-source descendant, brought this philosophy to the industry: you declare desired state ("run 3 nginx pods"), controllers continuously reconcile reality to match, and the system self-heals without operator intervention. With 128k+ GitHub stars and CNCF stewardship, Kubernetes became the de facto standard not because it's simple—it's notoriously complex—but because its architecture handles distributed systems failure modes that kill simpler orchestrators.

Technical Insight

Worker Node

Control Plane

Declarative Resources

Read/Write State

Watch Events

Watch Unscheduled Pods

Watch Pod Specs

Update Desired State

Bind Pod to Node

Report Status

Run Containers

kubectl/API Client

kube-apiserver

RESTful API Gateway

etcd

State Store

kube-controller-manager

Reconciliation Loops

kube-scheduler

Pod Assignment

kubelet

Node Agent

Container Runtime

CRI

System architecture — auto-generated

The reconciliation loop pattern is Kubernetes' architectural foundation. Unlike event-driven systems where controllers react to specific triggers (container_crashed, node_added), Kubernetes controllers continuously observe actual state and drive it toward desired state. The Deployment controller doesn't "respond" to pod deletion—it perpetually ensures the correct number of pods exist, discovering mismatches through watches and periodic resyncs.

Here's a simplified reconciliation loop from a custom controller using controller-runtime:

func (r *PodReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var pod corev1.Pod
    if err := r.Get(ctx, req.NamespacedName, &pod); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Desired state: pods should have monitoring label
    desiredLabels := map[string]string{"monitored": "true"}
    
    // Compare actual vs desired
    if !reflect.DeepEqual(pod.Labels, desiredLabels) {
        pod.Labels = desiredLabels
        if err := r.Update(ctx, &pod); err != nil {
            return ctrl.Result{}, err
        }
        // Requeue to verify the change stuck
        return ctrl.Result{Requeue: true}, nil
    }

    return ctrl.Result{}, nil
}

The Reconcile function receives a request (typically triggered by a watch event), fetches current state, compares it to desired state, and takes corrective action if they diverge. Critically, the function is idempotent and called repeatedly—if the update fails or another controller changes the pod, this loop runs again until convergence. The system doesn't care why state diverged; it only cares about fixing it.

The informer pattern solves the scalability problem inherent in watching resources. Naively, if 50 controllers each watch all pods in a 2000-node cluster, you'd establish 50 watch connections to the API server, each streaming potentially 10,000+ pod objects. Informers use shared caches: a single watch connection per resource type feeds an in-memory cache, and multiple controllers register event handlers against that cache. When a pod changes, the shared informer notifies all registered handlers without additional API calls.

// Shared informer factory creates cached informers
informerFactory := informers.NewSharedInformerFactory(clientset, 30*time.Second)
podInformer := informerFactory.Core().V1().Pods()

// Multiple controllers share the same informer/cache
podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc: func(obj interface{}) {
        pod := obj.(*corev1.Pod)
        workqueue.Add(pod.Name) // Queue for reconciliation
    },
    UpdateFunc: func(oldObj, newObj interface{}) {
        workqueue.Add(newObj.(*corev1.Pod).Name)
    },
    DeleteFunc: func(obj interface{}) {
        workqueue.Add(obj.(*corev1.Pod).Name)
    },
})

informerFactory.Start(stopCh)

The informer establishes a single watch connection, maintains an eventually-consistent in-memory cache, and delivers events to handler functions. Controllers receive events but treat them as hints—the reconciliation loop always fetches fresh state from the cache (or API) before acting. If events are lost, the periodic resync (30 seconds above) re-triggers reconciliation for all objects, ensuring eventual consistency even after network partitions.

Scheduling demonstrates how specialized reconciliation works. The kube-scheduler watches unscheduled pods (those without spec.nodeName), runs them through a two-phase filter/score pipeline, and binds the winning pod to a node by creating a Binding object. The scheduler doesn't track "pod created" events—it continuously queries for pods needing placement. If binding fails (node disappeared, resource exhaustion), the pod remains unscheduled, and the scheduler retries on the next loop iteration.

Optimistic concurrency prevents conflicts when multiple controllers modify the same object. Every resource has a metadata.resourceVersion field (etcd's MVCC revision number). Updates include this version; if another controller modified the object meanwhile, etcd rejects the write with a conflict error. The controller refetches current state (with updated resourceVersion) and retries its reconciliation logic. This pattern scales horizontally—multiple controller-manager replicas run identical reconciliation loops, but only writes with current resourceVersion succeed, naturally serializing modifications without distributed locks.

The staged repository structure (staging/src/k8s.io/*) deserves attention because it solves a brutal problem: publishing consumable libraries from a monorepo. The client-go library, apimachinery, and component-base live in staging/ directories with independent go.mod files. A bot synchronizes these to separate GitHub repos, allowing external tools to import k8s.io/client-go without depending on the entire Kubernetes monorepo. This architecture means contributors work in a unified codebase while consumers get clean module boundaries—genius engineering that more monorepos should copy.

Gotcha

The reconciliation model's biggest weakness is debuggability. When a pod won't schedule, you're debugging a distributed system failure across multiple controllers: the Deployment controller created a ReplicaSet, the ReplicaSet controller created a Pod, the scheduler hasn't bound it (why? resource constraints? taints? affinity rules?), and kubelet is waiting for a node assignment. Each controller logged its perspective, but correlating logs across components to understand why state won't converge requires deep system knowledge. Tools like kubectl describe help, but you're still reconstructing causality from independently reconciling loops that don't share context.

Eventual consistency bites during migrations or configuration changes. You update a Deployment's image tag, and pods start rolling... eventually. The Deployment controller creates a new ReplicaSet, the ReplicaSet controller creates pods, the scheduler places them, kubelet pulls images and starts containers. Each step is asynchronous reconciliation with no guaranteed latency bounds. In production, you'll write complex readiness checks and admission webhooks to enforce invariants because the core system makes no timing guarantees beyond "it'll converge." This is correct for distributed systems but maddening for operators used to synchronous deployments.

The monorepo build system is genuinely hostile to newcomers. Running make for the first time downloads gigabytes of dependencies, generates thousands of lines of code (clients, informers, deep-copy functions), and compiles for 10+ minutes on beefy hardware. The code generation toolchain (deepcopy-gen, client-gen, informer-gen, lister-gen) is fragile, poorly documented, and breaks in cryptic ways when you modify API types. Contributing a simple feature often requires understanding Bazel, code generation tags (// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object), and the staging sync process—a brutal onboarding experience that filters out casual contributors.

Verdict

Use if: You're running 20+ microservices with multiple teams needing self-service infrastructure, you require the ecosystem (Istio, Prometheus, cert-manager, external-secrets-operator), or you're building a platform where declarative APIs and GitOps workflows justify the operational complexity. Kubernetes shines when coordination overhead across teams exceeds Kubernetes' own complexity—typically around 50+ engineers. The reconciliation architecture genuinely delivers self-healing infrastructure if you have the expertise to operate it. Skip if: You're a small team (<10 engineers) shipping applications, not platforms, or you lack dedicated SRE resources to maintain the control plane. The operational burden—etcd backups, certificate rotation, version upgrades, CNI troubleshooting—will sink productivity. Use managed services (ECS Fargate, Cloud Run, Fly.io) or simpler orchestrators (Nomad) unless you're specifically hiring for Kubernetes expertise. The project's technical decisions are brilliant, but brilliance doesn't justify complexity for teams that don't need it.