In this context it means "text written before the advent of AI slop, so we know it at least came from humans and wasn't influenced by AI slop".
In this context it means "text written before the advent of AI slop, so we know it at least came from humans and wasn't influenced by AI slop".
understood. my question is more, how is the advent of inorganic online content impacting the scaling of pretraining? is there some mathematical model or science on this topic? these old books being scanned, is this just about more precisely modeling human thought, or it more about getting a finer-grained picture of the world and history? any more color like that would be really interesting.
> how is the advent of inorganic online content impacting the scaling of pretraining?
Thinking too hard about this question is incompatible with increasing shareholder value.