Are there any protections these days at the cloud provider level?
Like setting a maximum budget for a certain service (EC2, Aurora?) because downtime is preferable to this?
Are there any protections these days at the cloud provider level?
Like setting a maximum budget for a certain service (EC2, Aurora?) because downtime is preferable to this?
That's why I like VPS setups. You hit the monthly maximum, and it just stops working.
I host demos, not running a business, so it's less of an issue to get interrupted. Better an interruption than a $50,000 bill for forgetting to shut off a test database from last Wednesday.
Unless a startup has five+ nines service contracts with their customers already, a little bit of downtime once in a while is not the end of the world the cloud services want us to believe.
That's not comparable. With a VPS there is no monthly maximum, just a max load on a second by second basis. You can be hit with traffic of which 90% bounces because your server is down, get nowhere near your intended monthly maximum, and then the rest of the month is quiet.
You seem to be describing this as a bad thing instead of the objectively good thing that it is.
The ideal is obviously smoothed limits, such that you can absorb a big traffic spike if it still fits within your budget. Nobody seems to offer that.
How would you predict the smooth curve ahead of time?
Not _really_. AWS has a budget tool, but it doesn’t natively support shutting down services. Of course, you can ingest the alerts it sends any way you want, including feeding them into pipelines that disable services. There’s plenty of blueprints you can copy for this. More seriously - and this is a legitimate technical limitation - of course AWS doesn’t check each S3 request or Lambda invocation against your budget, instead, it consolidates periodically via background reporting processes. That means there’s some lag, and you are responsible for any costs incurred that go over budget between such reporting runs.
> of course AWS doesn’t check each S3 request or Lambda invocation against your budget
If it can bill them per-invocation, why can't it also check against a budget? I don't expect it to be synchronous, but a lag of minutes to respond is still better than nothing. Can you even opt-in to shutting down services from the budget tool, or is that still something you have to script by hand from Cloudwatch alarms?
You script it by hand.
I think figuring out how to do this faster is less trivial than it might sound. I agree that synchronous checks aren’t reasonable. But let’s take Lambdas. They can run for 15 minutes, and if you consolidate within five minutes after a resource has been billed, that gives you a twenty minute lag.
I’m not trying to make apologies for Amazon, mind you. Just saying that this isn’t exactly easy at scale, either. Sure, they bill by invocation, but that’s far from synchronous, too. In fact, getting alerts might very well be happening at the frequency of billing reconciliation, which might be an entirely reasonable thing to do. You could then argue that that process should happen more frequently, at Amazon’s cost.
> but it doesn’t natively support shutting down services [...] of course AWS doesn’t check each S3 request or Lambda invocation against your budget, instead, it consolidates periodically via background reporting processes
So, in other words, the vendor has provided substandard tooling with the explicit intent of forcing you to spend more money.
Just set alerts that are not really timely and homeroll your own kill scripts its easy. It doesn't really work but its not really any harder than just fucking self hosting.