Soul of an Engineer #3

Production is Hell

“Production is Hell”. “Prototyping is easy and fun”.

It is easy to build software quickly these days. But building software systems that work, scale and perform as per user expectations is still very hard.

If you have experienced the dread (and existential reality!!) of software “go-lives”, you will know what I am talking about.

You want to ship on time, but almost always, you feel there isn’t enough time to ship what is needed. This risk of not being able to ship on time elicits irrational behavior. You feel that everything you see as problematic before shipping will never come to pass. The hell will just freeze over. It never does, though. More you ignore the hell before, the worse it feels when you are in it. The hell of botched “go-lives” certainly isn’t for the faint of heart. The worst of our tribal mindsets manifests.

The thing though is shipping without everything fully ironed out is also a rite of passage. “Learning” happens through “shipping” . Even if you feel nervous, you cannot not ship. The best recourse is to focus on reducing the risk. The things you can’t de-risk, you can only mitigate.

Bigger the batch, bigger the “go-live” hell. So, small batch iterations, focused on learning and eliminating the problems, is the way to go.

Ship, but reduce the batch size.

Related: The quote above comes from a nugget from this YouTube conversation between Elon Musk and Sandy Munro. The 1:18 mark is where they discuss the “Production Hell”. The whole video is awesome if you have time.

Afterthought: "In science, if you know what you are doing, you should not be doing it. In engineering, if you do not know what you are doing, you should not be doing it. Of course, you seldom, if ever, see either pure state."— Richard Hamming