Semi-off-topic, but I have a basic question:
I have exactly one (Windows) machine at home with a decent GPU. I want to run a local LLM on it and let it run various apps on my machine while taking reasonable security precautions. What am I supposed to do, exactly? Migrate all my files to a VM that can I give pass-through CUDA access to the host somehow? Or is firewalling it and remotely controlling it from a second machine the only reasonable way?
I'd guess that prompt injection is the biggest risk in this setup, dwarfing the risk of exploits against the inference engine. Personally, I run LLM agents only inside a Docker container that limits the LLM's access to sensitive information and the LLM's ability to take irreversible destructive actions.
See also: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
What about Docker Desktop with GPU passthrough to a container running the LLM? That way you can be explicit on which files you share through volume mapping and the LLM is contained in the container otherwise.
If you want it to both host the inference AND host the harness, then yes, you should firewall one from the other in some way, e.g. with VMs.
Delete your user profile, set it up a unique user account. Dont leave any websites logged in as yourself.
Restrict that users file permissions if necessary, don't add it to the administrators group.