> We’re already ripping the spines out of old rare paper books (we playfully call these “low-background steel”)
can someone closer to pretraining elaborate?
> We’re already ripping the spines out of old rare paper books (we playfully call these “low-background steel”)
can someone closer to pretraining elaborate?
Not at all close, but this link might give you more to go on (includes more links):
https://www.snopes.com/fact-check/ai-companies-destroying-ra...
More here: https://en.wikipedia.org/wiki/Project_Panama
https://en.wikipedia.org/wiki/Low-background_steel
Replace "nuclear explosion" with "AI produced works"
In this context it means "text written before the advent of AI slop, so we know it at least came from humans and wasn't influenced by AI slop".
understood. my question is more, how is the advent of inorganic online content impacting the scaling of pretraining? is there some mathematical model or science on this topic? these old books being scanned, is this just about more precisely modeling human thought, or it more about getting a finer-grained picture of the world and history? any more color like that would be really interesting.
> how is the advent of inorganic online content impacting the scaling of pretraining?
Thinking too hard about this question is incompatible with increasing shareholder value.