LOAD"THE SMALLEST DOLL",8,1

The least computation a capability needs

5,400,000,000,000,000,000,000

Every 0 or 1 that a computer writes replaces the one that was there, and erasing the old one always releases a little heat that no machine can avoid. The finished model is about 6.5 trillion 0s and 1s, and if the least energy needed to store them is one unit, training the model took about 5,400 billion billion units.

Training one of today’s largest language models consumes roughly the energy that a large nuclear reactor produces in a day, and no one can yet say how much of that energy the capability itself requires and how much is the cost of the method used to train it.

Doll 1 · Limits

Three known limits

Heat engines, communication channels and the erasure of information each have a limit that no design can pass, derived by Carnot in 1824, by Shannon in 1948 and by Landauer in 1961. None of these limits showed how to build a better machine, but each showed engineers how far their machines stood from the best possible, and so where further work could still improve them.

The most work from heat

Sadi Carnot, a French army engineer, showed at the age of 28 that the fraction of its heat that an engine can convert into work is fixed by the temperatures of its hot and cold sides, whatever its fuel, materials or design.

η ≤ 1 − Tcold / Thot

600 °C

With the cold side at 20 °C, no engine can convert more than 66% of its heat into work.

Carnot died of cholera in 1832 at the age of 36, when his book had few readers, and its importance was recognised after his death.

The most data through a channel

Claude Shannon proved that information can be sent reliably through every noisy channel, whether a telephone wire or a radio link, at any rate up to its capacity, and that no method of coding can exceed that capacity.

C = B log2(1 + S/N)

1,000×

A telephone line passes frequencies between about 300 and 3,400 hertz, so at this signal strength no modem can send more than 30,900 bits a second over it.

By the same formula, the rate of 33,600 bits a second that dial-up modems of the mid-1990s reached on good lines requires a signal about 2,000 times stronger than the noise. In the forty-five years between Shannon’s paper and 1993, when codes that come close to the limit on channels in general were found, engineers knew how much improvement remained possible.

The least energy to erase a bit

Rolf Landauer, a physicist at IBM, showed that erasing one bit of information must release at least a certain amount of heat, and that this minimum depends only on the temperature.

E ≥ kBT ln 2

27 °C

At 300 K (27 °C), erasing one bit releases at least 2.9 × 10−21 joules of heat.

The limit was first confirmed experimentally in 2012.

Not measured

The least computation for a capability

No one can say, for any machine that has been built or proposed, how much computation it takes at minimum to learn a capability such as translating between languages or recognising faces.

Doll 2 · Cost

What training costs today

The amount of text, energy and computation used to train Llama 3.1 405B, which Meta has published, can be set against estimates of the same quantities for a child learning language between birth and the age of 13.

For the model, count
For the child, count
MeasureLlama 3.1 405BA child, birth to 13
Words1.2 × 101315.6 trillion tokens of textunder 108everything heard in those years
Energyabout 1014 Jroughly a day’s output of a large nuclear reactor8.2 × 109 Jabout three months of an American home’s electricity
Powerabout 15 MW16,384 chips, with servers and cooling20 Wthe brain

Counted this way, the model used about 12,000 times the child’s energy and read at least 120,000 times as many words.

Because a brain spends most of its energy on functions other than language, the child does not mark the minimum, but since the minimum must hold for any machine, a brain included, the comparison places it far below the present cost of training.

Meta’s figures are 3.8 × 1025 operations, 15.6 trillion tokens and 30.84 million hours on H100 chips rated at 700 W. The figure for the whole data centre adds an estimated 30% for servers, networking and cooling. Between birth and 13 a child hears fewer than 100 million words, and its body uses about 70 W on average.

Doll 3 · Frameworks

Four frameworks

Scaling laws, learning theory, Kolmogorov complexity and Landauer’s limit all bear on the least computation a capability needs, although each of them measures a different quantity.

Framework 1 · Kaplan 2020 · Hoffmann 2022

Scaling laws describe one method

Scaling laws predict how a model’s error falls as its number of parameters (N) and its number of training tokens (D) increase. Training costs about six operations per parameter per token.

C ≈ 6ND

L(N, D) = E + A/Nα + B/Dβ

The laws are fitted to Transformer networks trained by gradient descent to predict the next token. Within that single method they show how to divide a budget between N and D, but they say nothing about what another method would need. Since the least computation a capability needs cannot fall, and the compute needed to reach a fixed level of performance halved roughly every eight months between 2012 and 2023, these measurements describe the cost of a method.

Compute needed for the same result, relative to 2012
8 months
10−12

If the compute kept halving every 8 months, the trend would reach a minimum set at 10−12 in 2039, but since no theory yet fixes the minimum, the date could lie decades earlier or later.

The solid line is the trend that Ho and colleagues (2024) measured for language models between 2012 and 2023, shown with the 95% range of its halving time, from 5 to 14 months. The dotted line after 2023 is an extrapolation. With a mouse, the assumed minimum can also be set by dragging on the chart.

Framework 2 · PAC · VC · Statistical queries

Learning theory bounds a class of learners

Learning theory proves bounds on the data and computation that learning requires, but the general bounds assume nothing about the data, and for large networks they come out far larger than the amount of data from which those networks in fact generalise. Sharper lower bounds hold for particular classes of learner, and the standard task for them is sparse parity. Each example is a string of d bits, labelled by whether an odd number of k particular bits, whose positions are hidden from the learner, are set to 1, and the learner must find which k bits they are.

32
8
Labels flipped at random
The hidden rule 8 OF 32 BITS · HIDDEN

Gaussian elimination

Gaussian elimination treats each example as an equation in the hidden bits and solves all the equations together, by the procedure used for simultaneous equations in school algebra.

40 EXAMPLES · READY

Cost of the two methods on larger problems, for k = 8, on a computer performing a billion steps a second
Bits, dAverages onlyElimination
321.1 × 107 rules11 milliseconds32,768 operationsunder a millisecond
1001.9 × 1011 rules3.1 minutes1.0 × 106 operations1 millisecond
1,0002.4 × 1019 rules760 years1.0 × 109 operations1 second

Elimination needs about d3 steps however many bits are hidden, whereas a learner restricted to averages needs work on the order of the number of possible rules.

The lower bound for learners restricted to averages therefore belongs to gradient descent and related methods and not to the task, since elimination, which works with the exact bits of each example, solves the same task quickly. The rule itself is a circuit of 7 XOR gates.

Framework 3 · Kolmogorov 1965

The shortest description cannot be computed

The Kolmogorov complexity K of a sequence is the length of the shortest program that produces it. The digits of π pass the usual statistical tests for randomness, yet the 293-character program in the listing prints them one after another for as long as it is left to run.

Digits to describe

Output · live

These digits are computed in the browser as the page is read.

Program · 293 characters

// Gibbons’ spigot for π
let q = 1n, r = 0n, t = 1n, k = 1n, n = 3n, l = 3n;
while (true) {
  if (4n*q + r - t < n*t) {
    print(n);
    [q, r, n] = [10n*q, 10n*(r - n*t), 10n*(3n*q + r)/t - 10n*n];
  } else {
    [q, r, t, k, n, l] = [q*k, (2n*q + r)*l, t*l, k + 1n,
      (q*(7n*k + 2n) + r*l)/(t*l), l + 2n];
  }
}
1,000
Written out1,000 characters
Compressed with zipabout 460 bytes
Shortest known program293 characters
Shortest possible, KCannot be computed

Zip finds no pattern in π and stores each digit in about 3.7 bits, as it would any sequence of digits, whereas the program stays at 293 characters for any number of digits and becomes shorter than zip’s output at around 600 digits.

Every program that prints a sequence gives an upper bound on its K, as does every compressor, but no algorithm can compute K itself for every input. In particular, no general method can tell from the digits alone whether a sequence such as π’s has a short program.

K measures the size of the answer and not the work of finding it. With present methods that work is done by training, which for a model of N parameters at the compute-optimal rate of about 20 tokens per parameter takes about 120N2 operations. Recent research on epiplexity, a measure of the structure that a learner with limited compute can extract, has begun to ask how much less work would suffice to find the same answer.

Framework 4 · Landauer 1961

The physical limit is too low to guide engineering

For Llama 3.1 405B, whose weights store about 6.5 trillion bits, the minimum that physics gives depends only on that number of bits. Since writing a bit into memory erases the bit that was there before, storing all of these bits at Landauer’s limit would take about 2 × 10−8 joules, roughly the energy a coarse grain of sand releases in falling a millimetre, whereas the training run used 1014 joules. Because this minimum lies so far below what training uses, it shows only that training uses more energy than it must, without showing whether the excess is due to the hardware or to the method.

Doll 4 · The gap

The size of the gap

Training spends more than 1021 times the energy that physics requires to store its result. Better hardware could remove a factor of about a billion from that, but the remaining factor of several trillion belongs to the method.

Joules spent ÷ joules required

≈ 5,400,000,000,000,000,000,000

For the run, count
What the model keeps, per weight

The three settings correspond to the 16 bits at which each weight is trained, the 8-bit version that Meta released for running the model, and the finding by Allen-Zhu and Li that language models store at most about 2 bits of knowledge per weight.

With these settings the ratio is 5.4 × 1021, while every combination of settings gives a ratio between 4 × 1021 and 4 × 1022.

Energy in joules, on a scale on which each step is a factor of ten
The hardware’s share≈ 9 × 108
The method’s share≈ 6 × 1012not measured, because there is nothing yet to compare it with
  1. The training runroughly a day’s output of a large nuclear reactor
  2. A child’s brain, birth to 13about three months of an American home’s electricity
  3. The same run on perfect hardwareeach operation at the price of erasing one bit, about the energy in a sugar cube and a half
  4. Lifting an apple one metre
  5. Writing what the model keeps, at Landauer’s limitabout what a coarse grain of sand releases in falling a millimetre
  6. Erasing one bit2.9 × 10−21 J at room temperature

Hardware · 9.2E+08

Energy per operation

On current chips each operation costs about a billion times the energy of erasing one bit. Chips have been approaching the known limit of this factor for decades.

Method · 5.9E+12

Operations per bit kept

This part of the gap, which perfect hardware would leave unchanged, amounts to about 5.9 × 1012 operations for every bit the model stores.

Doll 5 · Middle

The missing middle

None of the four frameworks gives a minimum between the measured cost of present methods and the physical limit.

FrameworkWhat it measuresWhy it is not the minimum
Scaling lawsError against compute, for Transformers trained by gradient descentIt measures the current method rather than the task.
Learning theoryData in the worst case, or the queries a gradient-like learner needs on simple tasksIt is too pessimistic for real data, or tied to one class of learner.
Kolmogorov complexityThe shortest program that produces a behaviourIt cannot be computed, and it sizes the answer rather than the search.
Landauer’s limitThe least heat needed to erase a bitIt lies more than 1021 below training, too low to guide engineering.

Not measured

A practical theory of the minimum

Such a theory would give, for a specific capability, the least computation needed to learn it and to run it, and it would have to hold on any hardware, as Carnot’s limit holds for every engine whatever its design.

Doll 6 · Matruska

How Matruska measures it

For some tasks the smallest possible circuit is known, and on those tasks the cost of a trained network can be compared with the minimum directly.

In 1969 Volker Strassen showed that two matrices, each divided into four blocks, can be multiplied with seven multiplications of blocks instead of eight, and that the saving can be repeated inside each block. Matruska applies the same procedure to learning, searching within each computation for a smaller one that computes the same function and counting the operations of both.

  1. Step 1

    Choose tasks whose smallest circuit is known

    Parity, modular arithmetic and exact state routing have smallest circuits whose size is either proven or, at small sizes, found by exhaustive search; the parity of n bits, for instance, needs exactly n − 1 two-input gates.

  2. Step 2

    Measure the network

    Standard Transformers are trained by gradient descent with every operation counted, both those spent finding the rule and those spent running it.

  3. Step 3

    Build the minimal circuit

    The smallest network of two-input logic gates that computes the same function is constructed and its gate operations counted.

  4. Step 4

    Take the ratio

    The Algorithmic Compression for the task, the network’s operations divided by the circuit’s, measures how far the usual method lies from the minimum.

A worked example

8
1 million
Count the network’s work in

Network

A network with 1 million parametersIt performs at least 2 million operations for each answer, two per parameter.

Minimal circuit

Parity of 8 bitsIt needs 7 XOR gates, and no circuit of two-input gates can use fewer.

Algorithmic compression

≈ 290,000 : 1

This ratio covers the work of a single answer, whereas finding the rule during training costs far more.

This is an illustration from public formulas, not a Matruska measurement, and its count of gate operations rests on a rough figure of a thousand logic gates for each 16-bit multiply-and-add.

Enough ratios measured by Matruska, across tasks and sizes, would show how the distance between the cost of the present method and the minimum depends on the task and on its size, and a theory of the minimum could then be built on that evidence.

Sources

Where the numbers come from

Each figure comes from a published source or from the arithmetic listed here, and the controls change only the assumptions.

Arithmetic

  • Training energy30.84 million H100 hours at 700 W come to 7.8 × 1013 J for the chips. An estimated 30% for servers, networking and cooling raises this to about 1014 J, or about 15 MW across 16,384 chips.
  • What the model keeps405 billion weights at 16 bits make 6.5 × 1012 bits (3.2 × 1012 at 8 bits and 8.1 × 1011 at 2 bits). At kBT ln 2 = 2.87 × 10−21 J (300 K), writing the 16-bit weights costs 1.9 × 10−8 J.
  • The gap1014 J ÷ 1.9 × 10−8 J ≈ 5.4 × 1021; the other settings give between 4.2 × 1021 and 4.3 × 1022.
  • Hardware’s share1014 J ÷ 3.8 × 1025 operations is 2.6 × 10−12 J per operation, about 9 × 108 times kBT ln 2.
  • Method’s share3.8 × 1025 operations ÷ 6.5 × 1012 bits ≈ 5.9 × 1012; at one bit’s price per operation the run would need 1.1 × 105 J, while a sugar cube holds about 6.7 × 104 J.
  • The childA brain at 20 W for 13 years uses 8.2 × 109 J. The whole body averages about 1,400 kcal a day from birth to 13, about 70 W, which comes to 2.9 × 1010 J; an average American home uses about 3.9 × 1010 J of electricity a year.
  • Reading speed15.6 × 1012 tokens over 30.84 million hours on 16,384 chips is about 2.3 million tokens a second, or 108 words in about a minute.
  • Carnot, ShannonTemperatures are in kelvin, with the cold side at 20 °C (293 K). A telephone line passes about 3,100 Hz, so 33,600 bits a second needs a signal-to-noise ratio of about 2,000.
  • The fridgeA computer at Tc pays kBTc ln 2 per bit, and pumping that heat to a room at Tr costs at least kB(Tr − Tc) ln 2, so the total is at least kBTr ln 2.
  • Training costCompute-optimal training reads about 20 tokens per parameter, so C ≈ 6N × 20N = 120N2.
  • ParityThere are C(d, k) possible rules, and statistical-query learners need work on that order, while elimination needs about d3 operations on about d examples. The demonstration tests each rule against 64 examples and confirms a match on 1,024 more. The XOR of k bits needs k − 1 two-input gates.
  • ZipSizes come from the deflate compressor built into the browser, and the random digits from its cryptographic random number generator.
  • UnitsThe opening counts energy in units of the least energy needed to store the 16-bit weights, 1.9 × 10−8 J, so that the training run comes to about 5.4 × 1021 units, and each of the 166 steps of its dial multiplies the count by 0.74, which brings it to about one.

Sources