i just use Hetzner.
Everything which cloud companies provide just cost so much, my own postgres running with HA setup and backup cost me 1/10th the price of RDS or CloudSQL service running in production over 10 years with no downtime.
i directly autoscales instances off of the Metrics harvested from graphana it works fine for us, we've autoscaler configured via webhooks. Very simple and never failed us.
i don't know why would i even ever use GCP or AWS anymore.
All my services are fully HA and backup works like charm everyday.
I founded a hosting company 25 years ago when User-Mode Linux was the hot new virtualisation tech. We aspired to just replicate the dedicated server experience because that was obviously how you deploy services with the most flexibility, and UML made it so cheap! Through the 2010s I (extremely wrongly) assumed that being metered on each little part of their stack was not something most developers would choose, for the sake of a little convenience.
Does a regular 20-something software engineer still know how to turn some eBay servers & routers into a platform for hosting a high-traffic web application? Because that is still a thing you can do! (I've done it last year to make a 50PiB+ data store). I'm genuinely curious how popular it is for medium-to-big projects.
And Hetzner gives you almost all of that economic upside while taking away much of the physical hassle! Why are they not kings of the hosting world, rather than turning over a modest €367M (2021).
I find it hard to believe that the knowledge to manage a bunch of dedicated servers is that arcane that people wouldn't choose it for this kind of gigantic saving.
> I find it hard to believe that the knowledge to manage a bunch of dedicated servers is that arcane that people wouldn't choose it for this kind of gigantic saving.
Managing servers is fine. Managing servers well is hard for the average person. Many hand-rolled hosting setups I've encountered includes fun gems such as:
- undocumented config drift.
- one unit of availability (downtime required for offline upgrades, resizing or maintenance)
- very out of date OS/libraries (usually due to the first two issues)
- generally awful security configurations. The easiest configuration being open ports for SSH and/or database connections, which probably have passwords (if they didn't you'd immediately be pwned)
Cloud architecture might be annoying and complex for many use-cases, but if you've ever been the person who had to pick up someone else's "pet" and start making changes or just maintaining it you'll know why the it can be nice to have cloud arch put some of their constraints on how infra is provisioned and be willing to pay for it.
Companies buy cloud services because they want to reduce in-house server management and operations, for them it's a trade-off with hiring the right people. But you are right, when you can find the right people doing it yourself can be a lot cheaper.
In some sense I'm starting to think it has more to do with accounting. Hardware, datacenters and software licenses (unless it's a subscription, which is probably is these days) are capital expenses, cloud is an operation expense. Management in a lot of companies hates capital expenditures, presumable because it forces long term thinking, i.e. three to five years for server hardware. Better to go the cloud route and have "room for manoeuvrability". I worked for a company that would hire consultants, because "you can fire those at two weeks notice, with no severance". Sure, but they've been here for five years now, at twice the cost of actual staff. Companies like that also loves the cloud.
Whether or not cloud is viable for a company is very individual. It's very hard to pin point a size or a use case that will always make cloud the "correct" choice.
Another point (but my common observation) is the responsibility. By going SaaS or using cloud - any kind of data protection, rules/responsibility etc is moved away. and in many ways it is better - Google, dropbox or Onedrive will have better PR to take the pain if something goes crazy. Tickbox compliance is easy.
Something I know nothing about is whether the depreciation on server hardware outpaces the value it creates for a business, creating a tax incentive to own your own metal.
Right... That's why the hire "AWS Certified specialist ninja"
I get the feeling that with LLMs in the mix, in-house server management can do a lot more than it used to.
The internet of 20 years ago was awash with info for running dedicated servers, fragmented and badly-written in places but it was all there. I can absolutely believe LLMs would enable more people to find that knowledge more easily.
Perhaps it saves some time looking through the docs, but do you really trust an LLM to do the actual work?
Yes and an LLM checks it as well. I am yet to find a sysadmin task that an LLM couldn't solve neatly.
A nice bonus is that sysadmin tasks tend to be light in terms of token usage, that’s very convenient given the increasingly strict usage limits these days.
Yes, with a lot of reviewing what its doing/asking questions, 100%
By this point? Absolutely. They still get stuck in rabbit holes and go down the wrong path sometimes, so it's not fully fire and forget, but if you aren't taking advantage of LLMs to perform generic sysadmin drudgery, you're wasting your time that could be better spent elsewhere.
Also using Hetzner.
But I came across Mythic Beasts (https://www.mythic-beasts.com/) yesterday, similar idea, UK based. Not used them yet but made the account for the next VPS.
This is way way more expensive than hetzner. Not even comparable?
Agree, I used to always use Heroku or Render style platforms for my own software, but nowadays I just have a Linux server with Docker Compose and a Cron job. The cron job every minute runs docker pull (downloads latest image) and docker up -d (switches to new version only if there is a new version). And put caddy in front for the HTTPS. This has been very cheap and reliable for years now.
What images are you running that you'd need the latest version up after just a minute?
I'm not the OP but I'd clarify the cron check for new versions is done every minute. So when new images are pushed they're picked up quickly.
OP is not saying they push new versions at such a high frequency they need checks every one minute.
The choice of one minute vs 15 minute is implementation detail and when architected like this costs nothing.
I hope that helps. Again this is my own take.
When I push new images via CI, I want it to go in production immediately. Like Heroku/Render/Dokku
One annoyance (I don't know if they've since fixed it) was that Docker Hub would count pulls that don't contain an update towards the rate limit. That ultimately prompted me to switch to alternate repositories.
one way is to host a manifest file (can host one on r2) and update it on each deploy and when manifest changes, new container image is pulled.
Especially these days you can SSH to a baremetal server and just tell Claude to set up Postgres. Job done. You don't need autoscaling because you can afford a server that's 5X faster from the start.
You just use docker.
It is like 4 lines of config for Postgres, the only line you need to change is on which path Postgres should store the data.
You also probably want the Postgres storage on a different (set) of disks.
Maybe change the filesystem?
Do you run containers? What orchestrator or deploy tool do you use?
I find it interesting that Hetzner was never a consideration, until... LLMs started recommending them.
Hetzner was raved about before AI was cool. I know since based on those good reviews I moved half of my apps from DigitalOcean to Hetzner. My DigitalOcean droplet was lacking in RAM and it was more expensive for me to grow it than move some stuff to another small VPS on Hetzner.
we've done both. Hetzner dedicated was genuinely fine, until a disk started throwing SMART warnings on a Sunday morning and we remembered why we pay 10x elsewhere for some things. probably less about the raw cost and more about which weekends you want back.
Well, you gotta take all that into consideration before your build out.
You can use block storage if data matters to you.
Many services do not need to care about data reliability or can use multiple nodes, network storage or many other HA setups.
Isn't this nature of every dedicated server? You also take on the hardware management burden - that's why they can be insanely cheap.
But there is middleground in form of VPS, where hardware is managed by the provider. It's still way way cheaper than some cloud magic service.
VPS comes at the cost of potential for oversubscription - even from more reputable vendors. You never really know if you're actually getting what you're paying for.
They also offer dedicated VPS with guaranteed resource allocation.
Honestly I like Hetzner a lot but lately it has been very unstable for us. https://status.hetzner.com/ this page always has couple of incidents happening at the same time. I really appreciate the services they provide but i wish they were more stable.
There are several things going on even now, 1 hour after your comment. But I appreciate that they list them. That hopefully means that they have a good culture of honesty, and they can improve.
I looked through the issues and basically only ongoing thing is that backup power is not working in one of the data centers (could be a problem). The rest are warnings about planned shutdown of some services and speed limitation of object storage in one location.
I am sure it's luck but we have few hetzner VPSes in both German locations and in last 5 years afaik they've never been down. On our http monitor service they have 100s of days uptime only because we restarted them ourselves.
Because if I have a government service with millions of users, I don’t want the cheap shitter servers to crap out on me.
An employee is going to cost anywhere between 8k and 50k per month. Hiring an employee to save 200/month on servers by using a shitty VPS provider is not saving you any money.
If you have millions of users, you absolutely need to have someone whose whole job is managing infrastructure. Expecting servers or cloud services to not crap out on you without someone with the skills and time to keep things running seems foolish.
[dead]