What kind of systems are generating enough useful analytics data to ever scale to petabytes in the first place?
It's difficult (for my bird brain, at least) to imagine a scenario where such volume of analytics data would ever be necessary.
What kind of systems are generating enough useful analytics data to ever scale to petabytes in the first place?
It's difficult (for my bird brain, at least) to imagine a scenario where such volume of analytics data would ever be necessary.
It's multiple records per user over time. See something like posthog. One interaction with a form can generate 10-15 events. eg enter page, click link with text "settings", change input, clicked input, etc. So active users can generate 15-150 events per session, each ~1kb to 2kb in size.
The reason to store that data is to be able to build analytics and funnels you didn't anticipate and construct them retrospectively. You can of course save massive amounts of storage by building rollups and aggregating, but that prevents you from being able to eg change the funnels and see the funnel in the past.
10M users × 100 events/day × 1KB ≈ 1TB/day. A couple years gets you into PB range.
The other obvious thing is logs. Being able to have a vulnerability then go back in time and see if you were exploited is super valuable. See eg log4shell: if you kept records, you probably could go back and see if you'd been exploited.