There is a lot of "AI demand" that isn't just running inference on an LLM whose weights you downloaded.
I'm training a model using reinforcement learning with self-play. I can and do use vast.ai when scaling but for experiments it's far faster, and cheaper, to run it locally until the bugs are all figured out. Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes. It's zero locally.
Likewise. I have a huge demand personally to run AI noise-filtering models on many TB per month of raw video files. It takes about 3 days per file.
Apples ProRes codec is only licensed to run in high quality mode on a Mac, and so my Nvidia PC can’t do what I need. Thus, I own the beefiest Mac Studio you can currently buy. I would pay more for more TFlops.
I have done local LLM on there but it wasn’t interesting. Far worse performance and intelligence per dollar than the cloud boys.
There is no cloud offering for my video needs though.
Have you looked into those tinycorp nvidia drivers for thunderbolt egpus on Mac?
Out of curiosity, how would you transfer many TB to cloud and back if such service was available?
My guess is that it wouldn’t be TB sized files.
It’s probably files that, over the course of a month, add up to multiple TBs.
Which would suggest that a 1Gbit fibre connection would be adequate. For serious commercial usage, multi Gbit fibre is available in many places around the world.
Although being video files, they could easily be in the TB range. In which case, it would be interesting to know!
10mbps constant = ~3TiB a month, which is close to what I assume they're doing to acquire the video (that's about what a few SD x264 streams would run). So you don't even need a fast connection, per se.
[dead]
A C-level executive I know is getting a top-of-the-line new Mac simply to function as a personal build server and host for agentic coding instances - they are able to orchestrate so many parallel projects that they're hitting RAM limits from sessions and the builds and local test runs they're kicking off (largly unsupervised). Before AI, they'd only had a MacBook Air; this completely changes their workflows. They talk about how many other executives they've met are equally giddy at having gone from coding few to no projects themselves, to coding more projects in parallel than any of their respective pre-AI technical colleagues.
I'd suspect that agentic coding has birthed so many new effective engineers, that the entire dynamics of demand for high-end machines have been upended.
ram? coding instances? i suspect this is not doing any inference on the machine and i also suspect the ram use is due to a bagilion node/python processes running.
nothing like 600mb of for type check server another 500 for webpack and another 600 for the inevitable electron wrapper you didnt know about. python isnt as bad but still not great.
I’ve been happy training and running inference for small language models on my M4 Mac.
Inference with MLX is surprisingly zippy. I’m running a classification task on the entire HN comment dataset and it’s projected to take about two and a half days, which is not bad considering we’re talking about tens of millions of comments.
Yes, I could do it much more quickly by throwing Modal GPUs at it but this is low-priority work. I might as well throw my M4 a bone.
> Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes.
Modal significantly improves this. Highly recommend.
Is Modal at all similar to Vast.ai or just related because "It's for AI"? I looked at Modal's page for training, and it talks about using some SDK and other junk, can you not just get a beefy instance from Modal with tons of VRAM to do what you want with?
Modal is more like AWS Lambda in a way. Computing functions as a service instead of an whole container or whole instance / GPU. Much more granular.
Are you training something so big you need that much unified RAM though?
If you can fit it on a GPU, and especially for training, it is so much quicker than a Mac.
Same, but with vision models. Unfortunately, I might be at my limit locally. I have three models that I'm using to find and identify objects in pictures. The largest dataset and model now takes about 8 hours per epoch on my Mac M4 with 16G memory.
Yeah this was what got me to start doing short rentals of bigger gpus in the clouds, upload your parquet files and it takes a couple of hours for a thing that would have my mac at 100% for a couple of days
What are you training on using self play? Like alpha go? Curious what your setup is like .
Yes, basically like alphago. I’m teaching it to play magic: the gathering.
I had to start with some heuristic-based bots that played the decks very simply just to get to the point where the was some signal to learn from. I did behavioral cloning on the bots as a foundation, then self-play.
I’m doing the same!
Do you find that CoreML manages to fill up your drive with so many tiny files that a reboot takes hours to clean them up? I keep meaning to get my friends still inside the spaceship to file a radar about that.
What game are you building?