Hacker News

If you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's llama.cpp fork to get them to work, from https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-...

This should work:

  cd /tmp

  # Get the Prism macOS runtime
  curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz
  tar -xzf bonsai-runtime.tar.gz

  # Get the ~5.95 GB GGUF model:
  curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf

  # Run the server, I used port 8331
  ./llama-prism-b10685-7dffb15/llama-server \
    -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \
    --port 8331 -ngl 99 -fa on -c 32768

Then open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this:

  uvx llm openai endpoint http://127.0.0.1:8331/v1 \
    --model bonsai-2-27b --responses hi

That's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said "ggml_metal_device_init: - the tensor API is not supported in this environment - disabling".

./llama.cpp/build/bin/llama-cli -m Ternary-Bonsai-2-27B-PTQ1_0.gguf -ngl 99 -fa on -c 32768 -b 256 -ub 64 --temp 1.0 --top-p 0.95 --top-k 20 -ctk q8_0 -ctv q8_0 -p "hello world in x86 assembler" -n 256

simonw 13 hours ago [ - ]

I used that to Generate an SVG of a pelican riding a bicycle:

https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

It took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.

raylad 3 hours ago [ - ]

How does that compare with the bf16 version?

For my "Please recite Jabberwocky" test the bf16 almost passes but the ternary and even fp8 versions fail badly.

rahimnathwani 8 hours ago [ - ]

M1 Pro, same prompt, same cli options:

  32,706 tokens
  38min 19s
  14.22 t/s

11 hours ago [ - ]

[deleted]

kadoban 13 hours ago [ - ]

Honestly looks pretty good except whatever is going on with its booty. Is that an ass helmet? I cannot parse what's going on there.

tomcam 10 hours ago [ - ]

Like you've never worn an ass helmet

kadoban 6 hours ago [ - ]

Only because I hadn't previously thought of it xD Step up from the standard ass-hat for sure.

Forgeties79 13 hours ago [ - ]

I think it’s supposed to be a wing

bigwheels 12 hours ago [ - ]

I like the lens effect behind the rear tire.

rahimnathwani 12 hours ago [ - ]

If you want to download the gguf to your regular huggingface cache directory instead of to /tmp, you can download the model and run the server in one step:

  export HF_TOKEN=xxx # optional, speeds up the download
  
  ./llama-prism-b10685-7dffb15/llama serve \
    -hf prism-ml/Ternary-Bonsai-2-27B-gguf:PTQ1_0 \
    --port 8331 -ngl 99 -fa on -c 32768

wombat23 2 hours ago [ - ]

I managed to run it with RTX 3070 (8GB VRAM) following the "Quickstart" on HF model card with minor modifications (modify the architecture 86 for your own hardware):

  git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
  cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 && cmake --build build -j

Then downloaded & verified `Ternary-Bonsai-2-27B-PTQ1_0.gguf` from HF and ran

the parameters were suggested by gpt-5.6-luna to reduce memory footprint, as the defaults ran OOM on my gpu. result looks good:

  [ Prompt: 165.6 t/s | Generation: 40.6 t/s ]

would be nice if they upstreamed their changes so that it runs with the original llama.cpp

aktenlage 25 minutes ago [ - ]

Would it speed up prompt processing if you increased the -ub (and -b) parameters.

francisjp 10 hours ago [ - ]

Thanks for all of your exploration in public Simon.

Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.

Here’s a link to the diff if you want to try and update that fork to take advantage of the prefill gains afforded by the hardware: https://github.com/ggml-org/llama.cpp/pull/27461/changes

jb_briant 3 hours ago [ - ]

That kind of issue is exactly why Im so happy to have LLMs, let it take one hour or trial and error instead of me spending a day digging traces

refibrillator 13 hours ago [ - ]

Where did you get these instructions?

They have a demo repo with a setup.sh script:

https://github.com/PrismML-Eng/Bonsai-demo

The release tag and weight file you suggest doesn’t match what they wrote.

I figured them out, starting from the GGUF on Hugging Face.

If you have found better instructions and they work then use those instead!

Personally I prefer to download models directly rather than running some `./setup.sh` script where I need to then review what it does first.

refibrillator 12 hours ago [ - ]

Yeah just wanted to mention in case it explains the 2x lower throughout you are seeing on M5. To be fair their documentation is a bit inconsistent in some spots.

Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.

nikwen 12 hours ago [ - ]

It would be great to have upstream llama.cpp support for this!

iJohnDoe 11 hours ago [ - ]

Agreed. They always sound exciting to try out but are such a pain to get working.

Zetaphor 9 hours ago [ - ]

I always just throw an agent at it. Is this the RSI I keep hearing about

hedgehog 7 hours ago [ - ]

RSI saves you from RSI

13 hours ago [ - ]