Every 0 or 1 that a computer writes replaces the one that was there, and erasing the old one always releases a little heat that no machine can avoid. The finished model is about 6.5 trillion 0s and 1s, and if the least energy needed to store them is one unit, training the model took about 5,400 billion billion units.
Training one of today’s largest language models consumes roughly the energy that a large nuclear reactor produces in a day, and no one can yet say how much of that energy the capability itself requires and how much is the cost of the method used to train it.
READY.
Doll 1 · Limits
Three known limits
Heat engines, communication channels and the erasure of information each have a limit that no design can pass, derived by Carnot in 1824, by Shannon in 1948 and by Landauer in 1961. None of these limits showed how to build a better machine, but each showed engineers how far their machines stood from the best possible, and so where further work could still improve them.
The most work from heat
Sadi Carnot, a French army engineer, showed at the age of 28 that the fraction of its heat that an engine can convert into work is fixed by the temperatures of its hot and cold sides, whatever its fuel, materials or design.
η ≤ 1 − Tcold / Thot
With the cold side at 20 °C, no engine can convert more than 66% of its heat into work.
Carnot died of cholera in 1832 at the age of 36, when his book had few readers, and its importance was recognised after his death.
The most data through a channel
Claude Shannon proved that information can be sent reliably through every noisy channel, whether a telephone wire or a radio link, at any rate up to its capacity, and that no method of coding can exceed that capacity.
C = B log2(1 + S/N)
A telephone line passes frequencies between about 300 and 3,400 hertz, so at this signal strength no modem can send more than 30,900 bits a second over it.
By the same formula, the rate of 33,600 bits a second that dial-up modems of the mid-1990s reached on good lines requires a signal about 2,000 times stronger than the noise. In the forty-five years between Shannon’s paper and 1993, when codes that come close to the limit on channels in general were found, engineers knew how much improvement remained possible.
The least energy to erase a bit
Rolf Landauer, a physicist at IBM, showed that erasing one bit of information must release at least a certain amount of heat, and that this minimum depends only on the temperature.
E ≥ kBT ln 2
At 300 K (27 °C), erasing one bit releases at least 2.9 × 10−21 joules of heat.
A colder computer releases less heat for each bit it erases, but that heat must then be pumped out into the 27 °C room, and Carnot’s limit sets the least work the pumping requires. With that work included, the total cannot fall below the room-temperature minimum of 2.9 × 10−21 joules.
The limit was first confirmed experimentally in 2012.
????
Not measured
The least computation for a capability
No one can say, for any machine that has been built or proposed, how much computation it takes at minimum to learn a capability such as translating between languages or recognising faces.
Capability????
Doll 2 · Cost
What training costs today
The amount of text, energy and computation used to train Llama 3.1 405B, which Meta has published, can be set against estimates of the same quantities for a child learning language between birth and the age of 13.
Measure
Llama 3.1 405B
A child, birth to 13
Words
1.2 × 101315.6 trillion tokens of text
under 108everything heard in those years
Energy
about 1014 Jroughly a day’s output of a large nuclear reactor
8.2 × 109 Jabout three months of an American home’s electricity
Power
about 15 MW16,384 chips, with servers and cooling
20 Wthe brain
Counted this way, the model used about 12,000 times the child’s energy and read at least 120,000 times as many words.
Because a brain spends most of its energy on functions other than language, the child does not mark the minimum, but since the minimum must hold for any machine, a brain included, the comparison places it far below the present cost of training.
Meta’s figures are 3.8 × 1025 operations, 15.6 trillion tokens and 30.84 million hours on H100 chips rated at 700 W. The figure for the whole data centre adds an estimated 30% for servers, networking and cooling. Between birth and 13 a child hears fewer than 100 million words, and its body uses about 70 W on average.
Doll 3 · Frameworks
Four frameworks
Scaling laws, learning theory, Kolmogorov complexity and Landauer’s limit all bear on the least computation a capability needs, although each of them measures a different quantity.
Framework 1 · Kaplan 2020 · Hoffmann 2022
Scaling laws describe one method
Scaling laws predict how a model’s error falls as its number of parameters (N) and its number of training tokens (D) increase. Training costs about six operations per parameter per token.
C ≈ 6ND
L(N, D) = E + A/Nα + B/Dβ
The laws are fitted to Transformer networks trained by gradient descent to predict the next token. Within that single method they show how to divide a budget between N and D, but they say nothing about what another method would need. Since the least computation a capability needs cannot fall, and the compute needed to reach a fixed level of performance halved roughly every eight months between 2012 and 2023, these measurements describe the cost of a method.
Compute needed for the same result, relative to 2012
If the compute kept halving every 8 months, the trend would reach a minimum set at 10−12 in 2039, but since no theory yet fixes the minimum, the date could lie decades earlier or later.
The solid line is the trend that Ho and colleagues (2024) measured for language models between 2012 and 2023, shown with the 95% range of its halving time, from 5 to 14 months. The dotted line after 2023 is an extrapolation. With a mouse, the assumed minimum can also be set by dragging on the chart.
Framework 2 · PAC · VC · Statistical queries
Learning theory bounds a class of learners
Learning theory proves bounds on the data and computation that learning requires, but the general bounds assume nothing about the data, and for large networks they come out far larger than the amount of data from which those networks in fact generalise. Sharper lower bounds hold for particular classes of learner, and the standard task for them is sparse parity. Each example is a string of d bits, labelled by whether an odd number of k particular bits, whose positions are hidden from the learner, are set to 1, and the learner must find which k bits they are.
The hidden rule8 OF 32 BITS · HIDDEN
Gaussian elimination
Gaussian elimination treats each example as an equation in the hidden bits and solves all the equations together, by the procedure used for simultaneous equations in school algebra.
40 EXAMPLES · READY
Averages only
A learner that sees only averages over many examples, as gradient descent does, learns almost nothing about one candidate rule from testing another, and so must in effect test the candidates one at a time.
10,518,300 POSSIBLE RULES · READY
Cost of the two methods on larger problems, for k = 8, on a computer performing a billion steps a second
Bits, d
Averages only
Elimination
32
1.1 × 107 rules11 milliseconds
32,768 operationsunder a millisecond
100
1.9 × 1011 rules3.1 minutes
1.0 × 106 operations1 millisecond
1,000
2.4 × 1019 rules760 years
1.0 × 109 operations1 second
Elimination needs about d3 steps however many bits are hidden, whereas a learner restricted to averages needs work on the order of the number of possible rules.
The lower bound for learners restricted to averages therefore belongs to gradient descent and related methods and not to the task, since elimination, which works with the exact bits of each example, solves the same task quickly. The rule itself is a circuit of 7 XOR gates.
Framework 3 · Kolmogorov 1965
The shortest description cannot be computed
The Kolmogorov complexity K of a sequence is the length of the shortest program that produces it. The digits of π pass the usual statistical tests for randomness, yet the 293-character program in the listing prints them one after another for as long as it is left to run.
These digits are computed in the browser as the page is read.
Program · 293 characters
// Gibbons’ spigot for π
let q = 1n, r = 0n, t = 1n, k = 1n, n = 3n, l = 3n;
while (true) {
if (4n*q + r - t < n*t) {
print(n);
[q, r, n] = [10n*q, 10n*(r - n*t), 10n*(3n*q + r)/t - 10n*n];
} else {
[q, r, t, k, n, l] = [q*k, (2n*q + r)*l, t*l, k + 1n,
(q*(7n*k + 2n) + r*l)/(t*l), l + 2n];
}
}
Written out1,000 characters
Compressed with zipabout 460 bytes
Shortest known program293 characters
Shortest possible, KCannot be computed
Zip finds no pattern in π and stores each digit in about 3.7 bits, as it would any sequence of digits, whereas the program stays at 293 characters for any number of digits and becomes shorter than zip’s output at around 600 digits.
Every program that prints a sequence gives an upper bound on its K, as does every compressor, but no algorithm can compute K itself for every input. In particular, no general method can tell from the digits alone whether a sequence such as π’s has a short program.
K measures the size of the answer and not the work of finding it. With present methods that work is done by training, which for a model of N parameters at the compute-optimal rate of about 20 tokens per parameter takes about 120N2 operations. Recent research on epiplexity, a measure of the structure that a learner with limited compute can extract, has begun to ask how much less work would suffice to find the same answer.
Framework 4 · Landauer 1961
The physical limit is too low to guide engineering
For Llama 3.1 405B, whose weights store about 6.5 trillion bits, the minimum that physics gives depends only on that number of bits. Since writing a bit into memory erases the bit that was there before, storing all of these bits at Landauer’s limit would take about 2 × 10−8 joules, roughly the energy a coarse grain of sand releases in falling a millimetre, whereas the training run used 1014 joules. Because this minimum lies so far below what training uses, it shows only that training uses more energy than it must, without showing whether the excess is due to the hardware or to the method.
Doll 4 · The gap
The size of the gap
Training spends more than 1021 times the energy that physics requires to store its result. Better hardware could remove a factor of about a billion from that, but the remaining factor of several trillion belongs to the method.
Joules spent ÷ joules required
≈ 5,400,000,000,000,000,000,000
The three settings correspond to the 16 bits at which each weight is trained, the 8-bit version that Meta released for running the model, and the finding by Allen-Zhu and Li that language models store at most about 2 bits of knowledge per weight.
With these settings the ratio is 5.4 × 1021, while every combination of settings gives a ratio between 4 × 1021 and 4 × 1022.
Energy in joules, on a scale on which each step is a factor of ten
The hardware’s share≈ 9 × 108
The method’s share≈ 6 × 1012not measured, because there is nothing yet to compare it with
The training runroughly a day’s output of a large nuclear reactor
A child’s brain, birth to 13about three months of an American home’s electricity
The same run on perfect hardwareeach operation at the price of erasing one bit, about the energy in a sugar cube and a half
Lifting an apple one metre
Writing what the model keeps, at Landauer’s limitabout what a coarse grain of sand releases in falling a millimetre
Erasing one bit2.9 × 10−21 J at room temperature
Hardware · 9.2E+08
Energy per operation
On current chips each operation costs about a billion times the energy of erasing one bit. Chips have been approaching the known limit of this factor for decades.
Method · 5.9E+12
Operations per bit kept
This part of the gap, which perfect hardware would leave unchanged, amounts to about 5.9 × 1012 operations for every bit the model stores.
Doll 5 · Middle
The missing middle
None of the four frameworks gives a minimum between the measured cost of present methods and the physical limit.
Framework
What it measures
Why it is not the minimum
Scaling laws
Error against compute, for Transformers trained by gradient descent
It measures the current method rather than the task.
Learning theory
Data in the worst case, or the queries a gradient-like learner needs on simple tasks
It is too pessimistic for real data, or tied to one class of learner.
Kolmogorov complexity
The shortest program that produces a behaviour
It cannot be computed, and it sizes the answer rather than the search.
Landauer’s limit
The least heat needed to erase a bit
It lies more than 1021 below training, too low to guide engineering.
Not measured
A practical theory of the minimum
Such a theory would give, for a specific capability, the least computation needed to learn it and to run it, and it would have to hold on any hardware, as Carnot’s limit holds for every engine whatever its design.
Doll 6 · Matruska
How Matruska measures it
For some tasks the smallest possible circuit is known, and on those tasks the cost of a trained network can be compared with the minimum directly.
In 1969 Volker Strassen showed that two matrices, each divided into four blocks, can be multiplied with seven multiplications of blocks instead of eight, and that the saving can be repeated inside each block. Matruska applies the same procedure to learning, searching within each computation for a smaller one that computes the same function and counting the operations of both.
Step 1
Choose tasks whose smallest circuit is known
Parity, modular arithmetic and exact state routing have smallest circuits whose size is either proven or, at small sizes, found by exhaustive search; the parity of n bits, for instance, needs exactly n − 1 two-input gates.
Step 2
Measure the network
Standard Transformers are trained by gradient descent with every operation counted, both those spent finding the rule and those spent running it.
Step 3
Build the minimal circuit
The smallest network of two-input logic gates that computes the same function is constructed and its gate operations counted.
Step 4
Take the ratio
The Algorithmic Compression for the task, the network’s operations divided by the circuit’s, measures how far the usual method lies from the minimum.
A worked example
Network
A network with 1 million parametersIt performs at least 2 million operations for each answer, two per parameter.
Minimal circuit
Parity of 8 bitsIt needs 7 XOR gates, and no circuit of two-input gates can use fewer.
Algorithmic compression
≈ 290,000 : 1
This ratio covers the work of a single answer, whereas finding the rule during training costs far more.
This is an illustration from public formulas, not a Matruska measurement, and its count of gate operations rests on a rough figure of a thousand logic gates for each 16-bit multiply-and-add.
Enough ratios measured by Matruska, across tasks and sizes, would show how the distance between the cost of the present method and the minimum depends on the task and on its size, and a theory of the minimum could then be built on that evidence.
READY.
Sources
Where the numbers come from
Each figure comes from a published source or from the arithmetic listed here, and the controls change only the assumptions.
Arithmetic
Training energy30.84 million H100 hours at 700 W come to 7.8 × 1013 J for the chips. An estimated 30% for servers, networking and cooling raises this to about 1014 J, or about 15 MW across 16,384 chips.
What the model keeps405 billion weights at 16 bits make 6.5 × 1012 bits (3.2 × 1012 at 8 bits and 8.1 × 1011 at 2 bits). At kBT ln 2 = 2.87 × 10−21 J (300 K), writing the 16-bit weights costs 1.9 × 10−8 J.
The gap1014 J ÷ 1.9 × 10−8 J ≈ 5.4 × 1021; the other settings give between 4.2 × 1021 and 4.3 × 1022.
Hardware’s share1014 J ÷ 3.8 × 1025 operations is 2.6 × 10−12 J per operation, about 9 × 108 times kBT ln 2.
Method’s share3.8 × 1025 operations ÷ 6.5 × 1012 bits ≈ 5.9 × 1012; at one bit’s price per operation the run would need 1.1 × 105 J, while a sugar cube holds about 6.7 × 104 J.
The childA brain at 20 W for 13 years uses 8.2 × 109 J. The whole body averages about 1,400 kcal a day from birth to 13, about 70 W, which comes to 2.9 × 1010 J; an average American home uses about 3.9 × 1010 J of electricity a year.
Reading speed15.6 × 1012 tokens over 30.84 million hours on 16,384 chips is about 2.3 million tokens a second, or 108 words in about a minute.
Carnot, ShannonTemperatures are in kelvin, with the cold side at 20 °C (293 K). A telephone line passes about 3,100 Hz, so 33,600 bits a second needs a signal-to-noise ratio of about 2,000.
The fridgeA computer at Tc pays kBTc ln 2 per bit, and pumping that heat to a room at Tr costs at least kB(Tr − Tc) ln 2, so the total is at least kBTr ln 2.
Training costCompute-optimal training reads about 20 tokens per parameter, so C ≈ 6N × 20N = 120N2.
ParityThere are C(d, k) possible rules, and statistical-query learners need work on that order, while elimination needs about d3 operations on about d examples. The demonstration tests each rule against 64 examples and confirms a match on 1,024 more. The XOR of k bits needs k − 1 two-input gates.
ZipSizes come from the deflate compressor built into the browser, and the random digits from its cryptographic random number generator.
UnitsThe opening counts energy in units of the least energy needed to store the 16-bit weights, 1.9 × 10−8 J, so that the training run comes to about 5.4 × 1021 units, and each of the 166 steps of its dial multiplies the count by 0.74, which brings it to about one.
Sources
Carnot, S. (1824). Réflexions sur la puissance motrice du feu.
Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal.
ITU-T (1998). Recommendation V.34, the modem standard for 33,600 bits a second.
Berrou, C., Glavieux, A. and Thitimajshima, P. (1993). Near Shannon limit error-correcting coding and decoding: turbo-codes.
Landauer, R. (1961). Irreversibility and heat generation in the computing process. IBM Journal of Research and Development.
Bérut, A. et al. (2012). Experimental verification of Landauer’s principle linking information and thermodynamics. Nature 483.