The traditional solution for the problem of the repetitive data included in logs is that every time when a log file grows over a certain size (or periodically in time), a new log file is created and the old file is compressed with some standard data compression algorithm, which eliminates the repetitions.
This optimally solves the problem of the space taken by logs on disk.
The only possible disadvantage is that any application that is used to scan the logs must decompress them, but in practice I have never seen any case when this caused any nuisance, even when using such a primitive solution like "zcat|grep", instead of a full-featured application.
Witty very large files using some form of indexing (graylog etc) is sensible.
If decompression is a pain though, change your logrotate so it doesn’t compress. Obviously costs more in disk space and less in compute.
You can also write directly compressed and flush (without resetting state) after every line. Let the compression state reset on reboot, it's not that important to preserve it.