Hot take of the day:
Computer scientists are in denial about it, but CS is a branch of theoretical physics, not mathematics. You can point to this or that model of computation, such as lambda calculus or mu-recursive functions and try to claim its abstracted well beyond the particular laws of physics for some specific universe, but they all have some kind of rate limit built into them... and where does the motivation for this idea, that it takes something (time, space, work) to compute something ultimately come from? That's right - from underlying physics itself[1] - from the Bekenstein bound or Bremermann's limit or the like.
Even apparently non-physically-realizable models of computation like non-deterministic Turing machines are ultimately informed by and motivated by concepts in physics... otherwise they would just be examples of chmess[2] and of no interest to anyone. Computer science is of course somewhat abstracted from the details, but no more so than, say, thermodynamics, where concepts like entropy or Gibbs free energy can be studied in the abstract without reference to whether we are talking about a gas of non-interacting molecules or the spins of a bunch of electrons trapped in a lattice.
So, it's of no surprise whatsoever that the fundamental problems of distributed computing are ultimately the same as those found in the relativity of simultaneity[3]. You've all been studying the same things all along, just with different tools and at different levels of abstraction.
[1]: https://en.wikipedia.org/wiki/Limits_of_computation
[2]: https://link.springer.com/article/10.1007/s11245-006-0005-2
[3]: https://en.wikipedia.org/wiki/Relativity_of_simultaneity
Related hot take
Maybe thats why one of the not-so-bad ideas how to go about distributed systems came from a guy who was trained as a physicist and used to complain to his fellow programmers that “a lot of systems actually break the laws of physics”[1]?
> In distributed systems there is no real shared state (imagine one machine in the USA another in Sweden) where is the shared state? In the middle of the Atlantic? - shared state breaks laws of physics. State changes are propagated at the speed of light - we always know how things were at a remote site not how they are now. What we know is what they last told us. If you make a software abstraction that ignores this fact you’ll be in trouble.[2]
[1]: “The Mess We’re In”, 2014 https://www.youtube.com/watch?v=lKXe3HUG2l4 [2]: https://news.ycombinator.com/item?id=19708900
And yet we have CRDTs.
CRDTs are eventually consistent data structures
A thought provoking leading question I have is (1) a space heater throws power into a coil and generates heat and nothing more in the world (2) an LLM into a GPU throws power into the GPU and generates heat and organized information.
Is (2) generating less heat so that the laws of thermodynamics account for the increased amount of "useful stuff" generated (the organized information)? Switch (2) for a bitcoin miner if you prefer.
Is the act of increasing order in the data, somehow costing some energy? Is there a fundamental minimum cost to flipping bits? Something about the switch of energy to matter and back, seems involved here.
I think the nature of computation is holding some secrets about the meaning of the universe, if there is any.
> Is there a fundamental minimum cost to flipping bits?
Yes:
https://en.wikipedia.org/wiki/Landauer%27s_principle
But modern computers are nowhere near this theoretical limit, nor any of the other limits I mentioned above. Nevertheless, most of heat generated from modern CPUs does come from bits turning on and off. Each transistor is a tiny capacitor, that holds a charge when its ON. When it switches OFF, it dumps that charge down the drain, creating waste heat. This is a limitation of our technolgy, not a fundamental limit of physics.
Could be worse, though; early chips would disipate heat even when they weren't doing anything. CMOS improved this enormously by pairing up "complementary" transistors so current only flows when something changes.
Still, from the universe's point of view, what we consider a super advanced computer is a lot closer to a space heater than anything that pushes up against its computational limits. Consider, for example, that quarks operate on time scales of 10^23 Hz, and the universe is happy to run three of those in every proton in every star in the universe. In fact, it runs 10^24 of them for one CPU, and that same CPU can't even simulate the quarks of one proton in real time.
Let's face it: we're like kids in Minecraft who think it's cool watch a calculation of 2+2 trickle through a redstone computer in a minute, while the GPU is rendering a billion triangles every second to give them that view.
I think you should consider the gigawatts of heat generated to create the LLM model to make it a more fair comparison.
I suspect this is old news for you, but just in case you haven't heard of it, check out Feynman's Lectures on Computation. A surprising amount of the book is still relevant, and it's fun how much he always brings everything back to the physics.