We self-host sentry in Hetzner, but with a high-end server. 96c, 512gb. It ends up only costing around $300 a month, however with the scale of events that it processes, the managed version would be in the 10's of thousands.
The overhead at low volume is pretty high, but in the higher volumes (25M transactions/24h) it's a massive cost saving for us.
Edit:
There were just some initial headaches with needing to increase kafka partitions and add replications to the transaction processors, otherwise we didn't quite leverage the available compute and the backpressure would fill Redis up until OOM.
Same here with the community maintained Helm chart. Not the easiest thing but quite reasonable for almost two years now. This is for 50M transactions per month and we're seeing massive cost savings compared to SaaS at this volume as well.
For those interested in only errors, the self-hosted version recently introduced errors-only mode which should cut down on the containers.
Yeah I fully get how that's a volume where going self-hosted Sentry makes perfect sense at the bottom line and including any upkeep you might have.
Bugsink's also quite scalable[0], but I wouldn't recommend it a 25M/day.
[0] https://www.bugsink.com/scalable-and-reliable/
> Bugsink's also quite scalable[0], but I wouldn't recommend it a 25M/day.
Well, your homepage disagrees with this statement:
> Bugsink can deal with millions of events per day on dirt cheap hardware
To me, "millions" usually means less than 10M. 25M falls into "tens of millions" to me.
But it's a very fuzzy way of quantifying something, and open to various interpretations.
Fair point, but if it can cope with millions on "dirt cheap hardware" it logically follows that it can do 25M on more expensive hardware.
I do have one major complaint though, in dotnet, the tracing/errors are always captured regardless of the sampling rate. So you end up with a lot more memory usage on high throughput/low memory services with no way to lower it.
There's a ticket now open to stop this, but it's still in progress.
It's open source, you guys could always create a PR to fix it. That's the power of open source!
there's no guarantee it will get merged though, even if a PR is created.
Forking has down sides that can't be hand waved away too, especially for a service like this.
It's just a client library, what's the alternative, put a proxy in front that drops 90% of the events
Any chance you can link me to the ticket?
Feel free to email - david at sentry
It looks like there was some motion a week ago.
https://github.com/getsentry/sentry-dotnet/issues/3636#event...
> It ends up only costing around $300 a month, however with the scale of events that it processes, the managed version would be in the 10's of thousands.
I think this is a repeated question but... are you considering the cost of the people managing the deployment, security oversight, dealing with downtime etc?
If you can keep the people doing all the things, they become cheaper over time. Because as your system settles and people become more competent, both downtime and effort required to mend these problems reduce dramatically, and you can give more responsibilities to the same people without overloading them.
Disclosure: I'm a sysadmin.
I wonder what is your managers take on this, given your incentives here.
Honestly asking, what my incentives are looking like from there?
You are incentivised to argue that it is good to keep employing sysadmins for self hosting, because that will keep you employed. You have a monetary incentive, thus you are a bit biased, in my opinion.
I think I didn't elaborate my point enough, so there's a misunderstanding.
What I said is true for places where they already have sysadmins for various tasks. For the job I do (it's easy to find), you have to employ system administrations to begin with.
So, at least for my job, working the way I described in my original comment is the modus operandi for the job itself.
If the company you're working in doesn't prefer self-hosting things, and doesn't need system administrators for anything, you might be true, but having a couple of capable sysadmins on board both enables self-hosting and allows this initiative to grow without much extra cost, because it gets cheaper as the sysadmins learn and understand what they're doing, so they can handle more things with the same/less effort.
See, system administrators are lazy people. They'd rather solve problems for once and for all and play PacMan in their spare time.
I have to imagine at that size they have an ops team already for all the other services so those are pretty amortized.
I am the person, it's occasionally I log in to delete a log file that I just haven't setup to rotate. About once a month, apart from that, no intervention needed (so far).
Am I reading correctly that your software generates 25 million error messages per day?
Sentry does a lot more than tracking errors - presumably most of those transactions are 'breadcrumb'-style events.
Nope, 25M transactions. In Sentry a transaction is more like an OTEL-trace. Errors are much lower ;)
sorry if this is a silly question, the quick google search didn’t give me clues.
Transactions like full user flows start to finish, or 1 transaction = 1 post/get and 1 response?
> Transactions like full user flows start to finish, or 1 transaction = 1 post/get and 1 response?
For most applications we are talking closer to 1 transportation 1 web request. Distributed tracing across microservices is possible, the level of extra effort required depends on your stack. But that's also the out of the box, plug and play stuff. With lower level APIs you define your own transactions, when they start and end, which is needed for tracing applications where there isn't a built in framework integration (e.x not a web application).
That is more likely performance traces or session replays.
> a high-end server. 96c, 512gb. It ends up only costing around $300 a month
Wow, that's really cheap. I'm seriously overpaying for my cloud provider and need to try Hetzner. I always assumed Hetzner was only European based.
Just be forewarned it doesn't seem to offer one iota of IAM, so whether or not one is "overpaying" for a cloud provider depends on what you're getting from them. If you mean "rent a machine," then likely. If you mean "have the machines heal themselves instead of Pagerduty waking me up" then reasonable people can differ about where that money is going
FWIW, https://lowendbox.com/ is good fun for the former set of things, too