The Upcoming Challenges of AI Training
This is fascinating
So OpenAI is working on GPT-6 already with Microsoft hand in hand. And according to various articles I’ve read recently, they are hitting some truly fascinating limitations.
For instance, they need more than 200,000 H100 AI chips (basically GPUs) to train the model. The problem is the limit of an entire power plant is somewhere around 70,000 GPUs on average running at full power.
So obviously the solution is multiple data centers attached to different power plants, right? Well, that creates a problem.
See, AI training moves a lot of data and requires incredibly synchronized work across GPUs and CPUs. We’re talking hundreds of gigabits a second, probably terabits a second moving all across a datacenter. This is mostly done via a system called Infiniband which basically allows to different servers to communicate so fast they can basically act as one single computer.
The problem is Infiniband uses so many interconnections and is so fast it has two unique features. The first is the number of wires. Infiniband, in my limited knowledge, uses huge cables with dozens to hundreds of individual wires. This is necessary for the tremendous amount of data being sent and received.
The second is distance. Most infiniband connections are a few feet long since you need the CPUs and GPUs sharing data in real time. Once you spread them out by even a few miles, the latency cause by the speed of light makes this impossible.
So basically training the next generations of AI models has a fascinating problem. How do you get multiple data centers working in parallel and/or how do you train a model without all of the compute power working in parallel?
Microsoft and Bill Gates actually know the answer to this and have for a while: fusion. Both are heavily invested in dozens of fusion power startups desperately trying new reactor designs to basically create infinite, cheap power.
So the bad news is that AI companies, Microsoft in particular, have a fascinating problem on their hands. The good news is I’m a firm believer that fusion is the future. It’s limitless, clean, and inordinately safe. The massive investments being made into fusion so that AI can continue on its trajectory are also a great contribution to the human species and its survival on this planet.
“Combined with a form of fusion, the machines have found all the energy they would ever need.”
Morpheus, The Matrix