Inside the AI Efficiency Revolution: Why AI Is Getting Cheaper to Run

Training is the process of building an AI model — running billions of examples through the network and adjusting weights over weeks or months on thousands of GPUs. Inference is serving the trained model to end users. Training is done once (or periodically for fine-tuning); inference runs continuously at commercial scale. Australia's 5.4 GW pipeline is sized for inference demand, not domestic model training. The efficiency revolution described in this report — MoE, quantisation, KV-cache compression — primarily reduces inference energy cost. Training remains dense and energy-hungry. Since Australia is not building domestic model training capability (as Venture Insights' Beyond the Green Data Centre analysis notes), the pipeline is exposed precisely to the category of workload most targeted by efficiency advances.

No — and this is the counterintuitive insight at the core of MoE. The model retains all its learned parameters; it simply doesn't use all of them for every token. Different experts specialise in different types of content: some become specialists in code syntax, others in legal reasoning, others in numerical calculation. For a given token, the router selects the most relevant specialists. Total capacity is preserved; wasteful computation is eliminated. The Switch Transformer paper demonstrates this explicitly: trillion-parameter sparse models outperform smaller dense models on task benchmarks despite using the same per-token compute as a far smaller dense model.

Quantisation (including LLM.int8()) can be applied post-training to existing model checkpoints — Dettmers et al. explicitly demonstrated that 'a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation.' KV-cache compression operates at inference time and requires no model changes. MoE, however, is an architectural choice made at training time — existing dense models cannot be converted to MoE after the fact. The practical implication is that quantisation and KV-cache compression offer immediate efficiency gains on deployed infrastructure, while the larger MoE gains require new model training runs.

Export controls restricting access to Nvidia's most advanced chips (A100, H100 and their successors) forced Chinese AI labs to achieve equivalent performance with fewer, less powerful chips. This is the same dynamic that historically drove Japanese automotive efficiency innovation under oil price shocks: constraint produces ingenuity. US hyperscalers faced with abundant chip supply have less incentive to optimise aggressively at the algorithm level — buying more hardware is cheaper than engineering around the need for it. The efficiency techniques are not exclusively Chinese innovations (Switch Transformers was Google Research; LLM.int8() was University of Washington / Meta), but their aggressive production application at frontier scale has been driven by necessity in China.