This article resonates with similar struggles I've had at multiple companies. Its content is valuable. I wish the writing and the article itself had less of a slop odor.
That aside, what I don't understand (especially having worked on the side of the webhook sender, which is itself really tricky to get correct/performant/cheap) is why more companies which broadcast webhooks don't, say, provide direct access to Kafka topics, S3 buckets with ordered data objects landing, SQS queues, or any of the alternatives to those things.
"But it's irresponsible to expose an internal-use-only datastore directly to clients" goes one objection. But plenty of log-store systems have the notion of sharing a subpart of the log with a less-than-trusted external peer, so while exposing Kafka directly might be asking for the same kind of trouble as exposing your customer's SQL database for authenticated connection over the open internet (e.g. "we said you could issue reads, not that you could open/close TCP connections a million times a second! You just took out our message broker!"), exposing, say, an S3 bucket or Kinesis stream is much less risky because those systems have put some thought towards semi-trusted sharing.
"But everyone is used to getting HTTP webhooks and doesn't have the expertise to connect to something else"--that'd be true if, say, reading from a websocket or Postgres NOTIFY stream or Kafka topic or S3-change-notification stream were advanced techniques, but libraries around those things are so good nowadays that even the most web-tech-only low-skill developer can probably integrate with them with minimal hassle. Maybe it's just that a lot of shops literally only know how to run their code in a webserver, and have never deployed any other kind of application service/cronjob/queue worker? That seems unlikely to me, but I might be surprised.
"If we do something weird our competition will beat us on ease-of-use" goes another objection. But is it that hard given the libraries available? And can't you hedge back on the ease-of-use sell with "our data is fresher and more provably ordered and correct"?
I'm glad that SCROLL exists as a possible solution here. I'm just puzzled why more people haven't been using existing technologies to achieve this property.
Do most webhook senders literally not have a log store? Are they just firing webhooks in the middle of business event handlers and giving up synchronously if they can't be delievered?
Because if that's not the case (and I don't think it's the case), then it seems like the SCROLL API is ... basically just the Kafka consumer API. Or Kinesis. Or SQS. And so on.
> That seems unlikely to me, but I might be surprised.
I'm pretty sure you've hit the nail on the head. Especially now with LLM's you'd be surprised of how many people are running full on production apps that are coded solely by an agent, deployed with a mix of random providers (supabase + fly.io + whatever) and have no clue about what Kafka is. They are never going to ask for it because... they don't know they can, and why they should. They ask an agent "connect to stripe" and the agent codes your typical webhook ingestion mechanism, never proposes advocating for an event polling loop. Whenever there is a bug they will tell the agent to fix it and the agent will just write a webhook deduplication mechanism and that's it. You'd be surprised how little supposedly technical people care about the current state of technologies.
PS: Thanks for the feedback on the writing.
We (Svix, webhooks infra) actually help our customers directly write to Kafka topics, S3 buckets, SQS, etc. and have for a few years now. There are definitely people that adopt that, but receivers as well prefer the simplicity of webhooks.
> receivers as well prefer the simplicity of webhooks.
Do you see receive-side customers as prepared to outright reject paying for vendors that only offer non-webhook event streams if there's a webhook-ful competitor available? Or is that preference more of the "well, it's easier to add a webhook route to our existing webapp than it is to run a stream consumer/cron/whatever, but neither of those two is cost- or effort-preventative" situation, where customers don't consider event delivery systems to be the main differentiating factor?
I know (from our customers) that some of their customers make buying decisions based on the availability of webhooks. I'm unaware of anyone doing it based on the availability of our other functionality (but it could be I just don't have the data).
Though the data I do have: how many people adopt these advanced endpoints in practice (as we offer these), and it's less than webhooks.
Interesting. There’s probably a market for the consumer side of that, then: really easy-to-integrate-with-common-app-frameworks-and-hosting-platforms runtimes that provide a polling/socketful event consumer and sidestep all the concerns that come with scaling webhooks on the consumer side. Your company looks like it helps with those concerns a lot! But if HTTP webhooks are the lingua franca, there’s only so much I can do as a consumer of them per TFA, you know?