I thought this would be about the horrors of hosting/developing/debugging on “Serverless” but it’s about pricing over-runs. I scrolled aimlessly through the site ignoring most posts (bandwidth usage bills aren’t super interesting) but I did see this one:
https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-c...
About how you make unauth’d API calls to an s3 bucket you don’t own to run up the costs. That was a new one for me.
> I thought this would be about the horrors of hosting/developing/debugging on “Serverless” but it’s about pricing over-runs.
Agreed about that. I was hired onto a team that inherited a large AWS Lambda backend and the opacity of the underlying platform (which is the value proposition of serverless!) has made it very painful when the going gets tough and you find bugs in your system down close to that layer (in our case, intermittent socket hangups trying to connect to the secrets extension). And since your local testing rig looks almost nothing like the deployed environment...
I have some toy stuff at home running on Google Cloud Functions and it works fine (and scale-to-zero is pretty handy for hiding in the free tier). But I struggle to imagine a scenario in a professional setting where I wouldn't prefer to just put an HTTP server/queue consumer in a container on ECS.
I've had similar experiences with Azures services. Black boxes impossible to troubleshoot. Very unexpected behavior people aren't necessarily aware of when they initially spin these things up. For anything important I just accept the pain of deploying to kubernetes. Developers actually wind up preferring it in most cases with flux and devsoace.
I recently had customer who had smart idea to protect Container Registry with firewall... Breaking pretty much everything in process. Now it kinda works after days of punching enough holes in... But I still have no idea where does something like Container registry pull stuff from, or App Service...
And does some of their suggested solutions actually work or not...
Convince them to add IPv6 and you’ll be set for life
They did!
But they network address translate (NAT) IPv6, entirely defeating the only purpose of this protocol.
It's just so, so painful that I have no words with which I can adequately express my disdain for this miserable excuse for "software engineering".
Every time I've done a cost benefit analysis of AWS Lambda vs running a tiny machine 24/7 to handle things, the math has come out in favor of just paying to keep a machine on all the time and spinning up more instances as load increase.
There are some workloads that are suitable for lambda but they are very rare compared to the # of people who just shove REST APIs on lambda "in case they need to scale."
Is that what people do is test/develop primarily with local mocks of the services? I assumed it was more like you deploy mini copies of the app to individual instances namespaced to developer or feature branch, so everyone is working on something that actually fairly closely approximates prod just without the loading characteristics and btw you have to be online so no working on an airplane.
There are many paths. Worst case, I've witnessed developers editing Lambda code in the AWS console because they had no way to recreate the environment locally.
If you can't run locally, productivity drops like a rock. Each "cloud deploy" wastes tons of time.
SST has the best dev experience but requires you be online. They deploy all the real services (namespaced to you) and then instead of your function code they deploy little proxy lambdas that pass the request/response down to your local machine.
It’s still not perfect because the code is running locally but it allows “instant” updates after you make local changes and it’s the best I’ve found.
Mocks usually don’t line up with how things run in prod. Most teams just make small branch or dev environments, or test in staging. Once you hit odd bugs, serverless stops feeling simple and just turns into a headache.
Yeah, I’ve never worked at one of those shops but it’s always sounded like a nightmare. I get very anxious when I don’t have a local representative environment where I can get detailed logs, attach a debugger, run strace, whatever.
[dead]
I believe they changed that shortly after that blog post went viral: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3...
I raised that exact same issue to AWS in ~2015 and even though we had an Enterprise support plan, AWS response was basically: well, you problem.
We then ended up deleting the S3 bucket entirely, as that appeared to be the only way to get rid of the charges, only for AWS to come back to use a few weeks later telling us there are charges for an S3 bucket we previously owned. After explaining to them (again) that this way our only option to get rid of the charges, we never heard back.
You have to wonder how many people quietly got burned by that in the 18 years between S3 launching and that viral post finally prompting a response.
Seems an interesting oversight. I can just imagine the roundtable, uhh guys who do we charge for 403? Who can we charge? But what if people hit random buckets as an attack? Great!
> Seems an interesting oversight. I can just imagine the roundtable, uhh guys who do we charge for 403? Who can we charge? But what if people hit random buckets as an attack? Great!
It is amazing, isn't it? Something starts as an oversight but by the time it reaches down to customer support, it becomes an edict from above as it is "expected behavior".
> AWS was kind enough to cancel my S3 bill. However, they emphasized that this was done as an exception.
The stench of this bovine excrement is so strong that it transcends space time somehow.
Even pooper is upset about the stench. Tech is fuckin dumb in the corps, the only logical explanation to me is kickbacks to the CTO or similar.
Pooping at the job is one thing but pooping at the job and trying to sell it as a favor to the customer is a whole different game.
you don't need kickbacks at this level. They're all judged by their 6-month outlook on revenue and their market shares.
This is just obfuscating grift justified by the "well, you own the severless functions!"
> I can just imagine the roundtable
That's the best part!
The devs probably never thought of it, the support people who were complained to were probably either unable to reach the devs, or time crunched enough to not be able to, and who as a project manager would want to say they told their Devs to fix an issue that will lose the company money!
BS, I can't imagine that not a single developer entertains this scenario and voices concerns.
How to destroy your competition. Love it. Also why i dislike AWS. Zero interest to protect their SMB customers from surprise bills. Azure isn't much better but at least they got a few more protections in place.
> I reported my findings to the maintainers of the vulnerable open-source tool. They quickly fixed the default configuration, although they can’t fix the existing deployments.
Anyone wanna guess which open source tool this was? I'm curious to know why they never detected this themselves. I'd like to avoid this software if possible as the developers seem very incompetent.
Same, I was hoping for tales of woe and cloud lock-in, of being forced to use Lambda and Dynamo for something that could easily run on a $20/month VPS with sqlite.
The webflow one at the top has an interesting detail about them not allowing you to offload images to a cheaper service. Which you can probably work around by using a different domain.
> Imagine you create an empty, private AWS S3 bucket in a region of your preference. [...] As it turns out, one of the popular open-source tools had a default configuration to store their backups in S3. And, as a placeholder for a bucket name, they used… the same name that I used for my bucket.
What are the odds?
(Not a rhetorical question. I don't know how the choice of names works.)