I would argue the described format is exactly the origin of the problem.

It tries to optimize on disk footprint by deduplication and resulting in way more complex file format with many possible footguns leading to things like write amplification while also making it less robust for the actual use cases of a persistent log.

In a way, it's using a file format more useful for aggregation layer, except it doesn't do that well either, compromising immediate needs at local level.