Two programs can pass the same tests and still differ greatly in how hard they are to change. The difference may only become clear when someone asks them to do something new.
A similar problem appears in mathematics. A proof can establish that a claim is true. Some proofs also introduce a method or a way of describing the problem that helps with other work. In his conversation with Dwarkesh Patel, Grant Sanderson discusses this difference between solving a problem and developing useful ideas.
To judge that second kind of value, we need to look at what people can do with the idea afterward.
Galois studied how the roots of an equation fit together
The roots of an equation are the values that make it true. For example, both 2 and -2 are roots of x² - 4 = 0.
The quadratic formula gives the roots of a quadratic equation using arithmetic and a square root. Mathematicians also found formulas for cubic and quartic equations, which have degree three and four. These formulas use radicals: square roots, cube roots, or higher roots. Niels Henrik Abel proved that no such formula could solve every equation of degree five, although some individual fifth-degree equations can be solved that way.
Évariste Galois helped explain which equations can be solved by radicals. He studied how their roots could exchange places while keeping the same algebraic relationships.
Consider x² - 2 = 0. Its roots are √2 and -√2. Both square to 2, their sum is zero, and their product is -2. Exchanging them preserves those facts. It also preserves every algebraic relation formed using rational numbers, which are the numbers being held fixed in this example.
A rearrangement of roots is called a permutation. The permitted permutations form a group: combining two permitted rearrangements, or undoing one, gives another permitted rearrangement. Galois connected the structure of this group to whether the equation can be solved by radicals.
The numbers held fixed matter. If √2 is itself one of the fixed numbers, exchanging it with -√2 is no longer allowed. The fixed numbers form what mathematicians call the base field. Changing that field can change the group.
This is a different way to describe the original problem. Instead of looking only for a formula for the roots, mathematicians can study the group of allowed rearrangements. Galois's work helped establish group theory, though many mathematicians developed the subject and the explanation here uses modern language.
The value became clearer through later work
Galois's contemporaries struggled to understand his manuscripts. In a prison preface, he complained that Poisson had not understood his work and urged mathematicians to classify operations by their structure.
Galois died in 1832. In 1843, Joseph Liouville announced that he had found deep results in the manuscripts, and he published them in 1846. Camille Jordan's 1870 book later helped make these structures widely used in mathematics. Understanding and extending the work took decades.
This does not give us a test for deciding whether a difficult manuscript contains a great idea. It shows one limitation of judging an idea only by its first result. Later work can reveal uses that were not apparent at the time.
Test what happens when the requirements change
In software, this problem can be studied more directly. Consider two programs that read five file formats. One processes each format separately. The other converts them into a shared representation, so the same code can process the data after it has been read.
Both programs may pass all their tests. When a sixth format arrives, the shared representation might make it easy to reuse the existing code. It might also make the change harder if the new format does not fit its assumptions. Passing the original tests cannot tell us which will happen.
A useful comparison would start with two implementations that pass the same tests. Keep later requests hidden while the implementations are built. Then give each codebase to a fresh instance of the same AI agent, with the same tools, context limits, and time budget. Ask both to complete the same sequence of changes.
Using the same agent matters because we want to compare the starting designs. If different agents make the later changes, differences in their ability could affect the result.
For each change, check the new behavior and run the earlier tests again. Record whether the agent finishes, how many attempts it makes, how long it takes, and whether it breaks anything that used to work. Repeat each sequence several times, because the same agent can make different choices on different attempts. Also compare several sequences of changes. A design that suits one extension may be awkward for another.
The amount of code changed can be useful evidence too, but a small change is not always easy to understand or correct. The stronger evidence is that later work takes less effort and causes fewer problems.
Judge the design through the work it makes possible
A good boundary between parts of a program can let someone change one part without disturbing the rest. A useful shared representation can let several file formats use the same processing code. These benefits can be observed when the program changes, even if the first version gives no clear advantage.
The comparison still depends on the follow-up tasks we choose. Agents could become good at that test without handling other kinds of change well. I do not know how to remove that problem. The proposed test would give us evidence about the selected tasks, rather than a complete measure of design quality.
Galois described mathematical elegance as grasping many operations "with a single glance." In software, a shared representation has practical value when it makes several real problems easier to understand or solve. We need to see it used before we can judge that value well.