> The shared theme of all of the failure stories is missing the true cost of self-hosting: The hours spent getting the servers just right, managing the hosting, debating the best way to run things, and dealing with little issues add up but are easily lost in the noise if you’re not looking closely.
What the modern software business seems to have lost is the understanding that ops and dev are two different universes. DevOps was a reaction to the fact that even outsourcing ops to AWS doesn’t entirely solve all of your ops problems and the role is absolutely no substitute for a systems administrator. Having someone that helps derive the requirements for your infrastructure, then designs it, builds it , backs it up, maintains it, troubleshoots it, monitors performance, determines appropriate redundancy, etc. etc. etc. and then tells the developers how to work with it is the missing link. Hit-by-a-bus documentation, support and update procedures, security incident response… these are all problems we solved a long time ago, but sort of forgot about moving everything to cloud architecture.
> DevOps was a reaction to the fact that even outsourcing ops to AWS doesn’t entirely solve all of your ops problems and the role is absolutely no substitute for a systems administrator.
This is revisionist history. DevOps was a reaction to the fact that many/most software development organizations had a clear separation between "developers" and "sysadmins". Developers' responsibility ended when they compiled an EXE/JAR file/whatever, then they tossed it over the fence to the sysadmins who were responsible for running it. DevOps was the realization that, huh, software works between when the people responsible for building the software ("Dev") are also the same people responsible for keeping it running ("Ops").
It was very much this for me. I knew the hosting side of things because my second job as a programmer was at a small ISP that hosted custom websites. I got used to maintaining Linux web and email servers by hand over SSH. There were some common scripts, but for the most part the pattern was SSH into the server and make the changes you need to make. Most of my early startup career was like this. Closely working with hardware, the server installs, hosting configs as well as the code that actually powered things.
Jump to my first "enterprise" job and suddenly I can't fix things anymore. I have to submit tickets to other teams to look at why the thing I built isn't running as expected. That, to me, was pure insanity. The sysadmins knew fuck all about my app and as far as I was concerned barely knew how to admin systems. I knew a lot more in my 20's after all. But the friction of not running what I wrote was absolutely real and one of the main killers of productivity versus my startup days.
I also have seen this from most of the "enterprise" companies that do "DevOps" when really they just mean they have a sysadmin team who uses modern tools and IaC. The same exact friction and issues exist between dev and ops as before DevOps days. Those companies are explicitly doing DevOps wrong. When you look at the troubleshooting steps during an incident, it's identical. Bring in the devs and the ops team so we can figure out what's going on. I do think startups are more likely to get DevOps right because they aren't trying to force it on the only mental model they seem to be able to understand.
I've also found that dev teams who run and maintain their own stacks are better about automatic failure recovery and overall more reliable solutions. Whether that's due to better alignment between the app code and the app stack during development or because the dev team is now the first call when things aren't working I'm not entirely sure. Likely a mix of both.
> DevOps was a reaction to the fact that even outsourcing ops to AWS doesn’t entirely solve all of your ops problems
DevOps, conceptually, goes back to the 90s. I was using the term in 2001. If memory serves, AWS didn't really start to take off until the mid/late aughts, or at least not until they launched S3.
DevOps was a reaction to the software lifecycle problem and didn't have anything to do with AWS. If anything it's the other way around: AWS and cloud hosting gained popularity in part due to DevOps culture.
> What the modern software business seems to have lost is the understanding that ops and dev are two different universes.
This is a fascinating take, if you ask me, treating them as separate is the whole problem!
The point of being an engineer is to solve real world problems, not to live inside your own little specialist world.
Obviously there's a lot to be said for being really good at a specialized set of skills, but thats only relevant to the part where you're actually solving problems.
Read some old o’reilly books on systems administration. Solving problems in a business domain and operating the infrastructure to support that are, indeed, very different. Urban planners needn’t understand how to orchestrate a bunch of heavy construction equipment and civil engineers needn’t care about how the sight lines affect the revenue of the stores in the buildings. They’re all building the same exact thing.
The issue is that precious few devs are actually good at ops. There are a ton of abstractions that have sprung up that attempt to paper over this, and they all suck for various reasons.
If you think you need to actually know your programming language of choice to be a good dev, I have news for you about actually knowing how computers work to be good at ops.