> Hm I don't think so. You might be thinking about the file size, which is ~64GB.
No, the numbers I put above is literally the VRAM usage I see when I load 120B with llama.cpp, it's a real-life number, not theoretical :)
> Hm I don't think so. You might be thinking about the file size, which is ~64GB.
No, the numbers I put above is literally the VRAM usage I see when I load 120B with llama.cpp, it's a real-life number, not theoretical :)