because it turns out that optimizing performance of a programming language designed for use-cases where runtime performance doesn't matter ... doesn't matter
because it turns out that optimizing performance of a programming language designed for use-cases where runtime performance doesn't matter ... doesn't matter
There's currently talk of adding gigawatts of data center capacity to the grid just for use cases where python dominates development. While a lot of that will be compiled into optimized kernels on CPU or GPU, it only takes a little bit of 1000x slower code to add up to a significant chunk of processing time at training or inference time.
What percentage of the CPU cycles are actually spent running Python though? My impression is _very_ low in production LLM workloads. I think significantly less than 1%. There are almost certainly better places to spend the effort, and if it did matter, I think they would replace Python with something like C++ or Rust.
Might as well take the work that's already done though? I can't think of a logical reason why you'd want to run it at potentially half the speed (depending on the hot code specifics how much, if any, speedup you get of course)