Skip to content
You've successfully subscribed to Amarinder Sidhu
Great! Next, complete checkout for full access to Amarinder Sidhu
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.

Production is Hell

Soul of an Engineer #3: Shipping in production is risky. "Go Lives" are tricky to navigate. Shipping in smaller batches reduces the risk.

AmarinderSidhu
AmarinderSidhu
1 min read
Production is Hell

“Production is Hell”. “Prototyping is easy and fun”.

It is easy to build software quickly these days. But building software systems that work, scale and perform as per user expectations is still very hard.

If you have experienced the dread (and existential reality!!) of software “go-lives”, you will know what I am talking about.

You want to ship on time, but almost always, you feel there isn’t enough time to ship what is needed. This risk of not being able to ship on time elicits irrational behavior. You feel that everything you see as problematic before shipping will never come to pass. The hell will just freeze over. It never does, though. More you ignore the hell before, the worse it feels when you are in it. The hell of botched “go-lives” certainly isn’t for the faint of heart. The worst of our tribal mindsets manifests.

The thing though is shipping without everything fully ironed out is also a rite of passage. “Learning” happens through “shipping” . Even if you feel nervous, you cannot not ship. The best recourse is to focus on reducing the risk. The things you can’t de-risk, you can only mitigate.

Bigger the batch, bigger the “go-live” hell. So, small batch iterations, focused on learning and eliminating the problems, is the way to go.

Ship, but reduce the batch size.


Related: The quote above comes from a nugget from this YouTube conversation between Elon Musk and Sandy Munro. The 1:18 mark is where they discuss the “Production Hell”. The whole video is awesome if you have time.


Afterthought: "In science, if you know what you are doing, you should not be doing it. In engineering, if you do not know what you are doing, you should not be doing it. Of course, you seldom, if ever, see either pure state."— Richard Hamming


Cover Art by Ybrayym Esenov on Unsplash

Newsletter