only time i had slowdowns running LLMs on my M3 Max was when i was hitting the context limit so the memory pressure was hitting high yellow / red.