We have the weights, we have the papers, nothing preventing development on this foundation.
Except compute
Do you think we are near the theoretical maximums of how much model we cram into 20B?
I don't consider myself qualified to make that call, though I believe there's plenty of room for improvement. But I don't mean inference compute, which is fairly accessible for well-off enthusiasts even now. Training is a lot more expensive though.
Back in the day we had seti@home, prime95 and fold@home. We will make it work.
Except compute
Do you think we are near the theoretical maximums of how much model we cram into 20B?
I don't consider myself qualified to make that call, though I believe there's plenty of room for improvement. But I don't mean inference compute, which is fairly accessible for well-off enthusiasts even now. Training is a lot more expensive though.
Back in the day we had seti@home, prime95 and fold@home. We will make it work.