Don’t Make Your Users Wait: Parallelism in Kubernetes Operators

Note: This is hand written blog and further refractored but not an AI Slop.

The Restaurant Example#

Now, here is the story. Let’s say you are in a restaurant which has only one counter to take the order and each order is taking 1 minute to finish. Let’s say there are 50 people in the queue. Now, the 50th person who is in the queue at 10:00 AM needs to wait till 10:50 AM for his order to be picked. Now, this is what it feels like as a customer who is very hungry :) and I can’t wait for so long, right.

What I will do is tell the owner/manager of that restaurant, “Hey, could you please increase the number of workers so that each customer can be served in parallel.”

Now, just relate the customer as the Custom Resource and the restaurant as the Kubernetes Operator.

What’s Actually Happening Inside an Operator?#

By default, most Kubernetes operators process one reconcile request at a time. The reconciler picks up a resource, does its work (calls an API, creates child resources, waits for a status update), and only then picks up the next one.

This is fine when you have 3 resources. It becomes a serious problem when you have 300.

The 50th Custom Resource sits in the queue while the operator dutifully works through every resource ahead of it -one at a time.

Without Parallelism vs With Parallelism#

More workers → shorter wait → happier users.

The Fix: MaxConcurrentReconciles#

In controller-runtime, you can increase the number of concurrent workers with a single field:

err = ctrl.NewControllerManagedBy(mgr).
    For(&myv1.MyResource{}).
    WithOptions(controller.Options{
        MaxConcurrentReconciles: 10,
    }).
    Complete(r)

That’s it. You’ve just hired 9 more cashiers.

But few things to be consider#

API server rate limits. More workers mean more simultaneous Get, Update, and Patch calls, use built-in rate limiting and exponential backoff.

Memory and CPU. More goroutines mean more resource usage, set the workers based on your preference.

Conclusion#

The default single-worker reconciler doesn’t scale. That queue becomes a waiting room and your users feel every second of it.

The change is one line of code. The impact can be enormous.