The quadratic attention problem seems to be largely solved by practical algorithmic improvements. (Iterations on flash attention, etc.)
What's practically limiting context size IME is that results seem to get "muddy" and get off track when you have a giant context size. For a single-topic long session, I imagine you get a large number of places in the context which may be good matches for a given query, leading to ambiguous results.
I'm also not sure how much work is being put into reinforcement in extremely large context inference, as it's presumably quite expensive to do and hard to reliably test.
Indeed, filling the adversitsed context more than 1/4 full is a bad idea in general. 50k tokens is a fair bit, but works out to between 1 and 10k lines of code.
Perfect for a demo or work on a single self contained file.
Disastrous for a large code base with logic scattered all throughout it.
Right. It’s not practical to apply AI tools as they are today to existing, complex code bases and get reliable results.
Greenfield is easy (but it always was). Working on well-organised modules that are self contained and cleanly designed is easy - but that always was, too.