Critical Systems

December 26, 2016 - Reading Time: 2 minutes, 6 seconds

A while ago there was an outage of a major airliner due to a computer system glitch.

Jetstars computer system went down. 1

I was pondering about how facebook never goes down yet alot of corporate networks always go down.

Why?

Why is it that non-critical systems such as Twitter, Facebook, Amazon, Ebay, Google never go down and real time critical systems go down often?

I believe that this comes down to two reasons;

Organisation Structures and Company Culture.

Org Structures are ok, they define where you sit on the companies structure... Usually the people writing the code are at the lower end of the org structure. One of the major cons of an org structure is, you could have Albert Einstein working for you and give CEO level advice, but it will be unheard simply due to his position on the companies org structure, the number of hops from the top he/her is.

I would have to guess that some engineer that works on the Jetstars system knew there was a major flaw due to the age of the system and when it was written. The problem with culture is its defined by usually the founders and first hires. These people usually grow to be at the high end of the org chart. The problem we have here is choice of tech used is defined by the people at the top of the org chart. If the people at the top are wrong, usually they will pick the wrong tech. So even if half their engineers agreed that switching the platform from x to y would result in z% improvement with 0 downtime - Its not their choice and the culture of the company isnt allowing for their reasoning to be heard.

Paul Graham writes about culture and technologies stacks in a great essay called 'Great Hackers' - 2

The companies that have online services that never go down understand this. They understand the power of culture and feedback from engineers.

Maybe the distance between reality and perception seems proportional to how far away from the bottom of the org chart you are.

Meaning the more levels on your org chart – the more likey the suits are disconnected from what is really important.

These thoughts are probably incorrect, they are simply my observations of the world from my view over time. I also fly jetstar all the time so no hard feelings there.