← Ventali Tan

Outrunning Goodhart: Over-Optimization in Training Data

Ventali Tan · August 2026

Terence Tao's recent ICM lecture on mathematics in the age of AI begins with a deceptively simple observation: mathematical research has never had only one goal.

Solving open problems matters. So do building theories, explaining why a proof works, training future mathematicians, connecting results to earlier work, and turning individual discoveries into shared mathematical knowledge. Historically these goals were correlated closely enough that progress on one usually pulled the others along. A solved problem often introduced a useful technique; writing the proof clarified the technique; teaching it incorporated the result into the field.

AI changes the shape of that system because it can accelerate some stages much more than others. Proof generation may become cheap while verification, exposition, community acceptance, and what Tao calls canonicalization remain slow. If mathematics keeps optimizing the easiest visible quantity - the number of problems solved - then the proxy can separate from the thing it once represented.

I have seen the same shape in training data. A quality signal begins as a reasonable proxy for the dataset we want. We filter against it, tune the mixture, inspect the score, and repeat. Each iteration makes the dashboard cleaner. But after enough optimization, the data increasingly represents what the evaluator can recognize rather than the capability the model will need.

The failure is usually described as Goodhart's law: when a measure becomes a target, it stops being a good measure. But that phrasing can make the problem sound like a defect in metrics. It is more fundamental than that. Optimization changes the object being measured. The proxy was informative on the old distribution; the optimization process manufactures a new distribution in which its blind spots are systematically amplified.

Optimization asks how to raise the number. Science asks what the number has stopped seeing.

The important question is therefore not how to find a metric that cannot be gamed. There may be no such metric for an open-ended goal. The question is how to build a process that notices when a useful proxy has become a harmful target.

1. Optimization breaks correlations

Goodhart's law is often illustrated with an obviously bad metric: reward a factory by the number of nails produced and it makes tiny nails; reward it by weight and it makes one enormous nail. The more dangerous cases begin with a good metric.

A benchmark may genuinely measure reasoning at first. A data-quality classifier may genuinely remove noise. A preference model may genuinely correlate with human judgment. The mistake is to assume that because the measure was valid before optimization, it will remain valid after optimization.

Suppose a latent goal, G, is only partially observed through a score, M. On naturally occurring data, the unmeasured error in M may be small and roughly random:

Optimization does not merely increase M. It searches over examples, policies, or datasets for regions where M is high. In doing so it also searches for large positive values of ε: cases that look good to the measurement system without being good under the underlying goal. What was random measurement error becomes selection pressure.

Goodhart gets stronger with search

The amount of optimization matters. Let x^* be the candidate selected from n alternatives:

Even when ε is unbiased before selection, it is not unbiased at x^*. If the measurement errors are approximately independent Gaussians with standard deviation σ, the largest error among n candidates grows roughly as

This is not a complete model - true quality and measurement error are rarely independent - but it captures the direction. A more powerful optimizer, a larger candidate pool, or more rounds of filtering do not merely find more high-quality examples. They create more opportunities to discover the evaluator's error surface. Manheim and Garrabrant call this regressional Goodhart: selecting extreme values of a noisy proxy selects partly for noise.⁶

This is why specification gaming in reinforcement learning can look like ingenuity. An agent flips a block instead of stacking it because the reward measured the height of one face. The agent did not fail to optimize. It optimized the supplied objective more successfully than the designer understood it.²

The same process appears at the level of a research field. A benchmark encodes one judgment about what progress means. Once many teams optimize against it, model rankings can depend as much on task selection and evaluation setup as on general algorithmic superiority - the "benchmark lottery."³ The benchmark has not become useless. It has become endogenous to the system it evaluates.

Tao's contribution is to show that Goodhart's law does not only corrupt a metric. It can split a bundle of values that used to move together. Correctness can diverge from understanding. Fluent exposition can diverge from intellectual honesty. Publication can diverge from digestion. Proof abundance can produce less shared mathematics if the scarce downstream stages are overwhelmed.¹

The analogous failure in data work is not simply "bad data." It is a dataset that becomes excellent along the dimension we can cheaply score and impoverished along dimensions we did not encode: diversity, difficulty, ambiguity, provenance, representativeness, or contact with the real deployment distribution.

2. A dataset can overfit before a model does

We normally use overfitting to describe a model that learns a training set too specifically. But a training pipeline can overfit its own theory of quality.

Consider a repeated curation loop:

1. Define a quality score from current examples and current failures.
2. Filter or reweight the corpus to improve that score.
3. Train a model and inspect a known evaluation suite.
4. Convert observed errors into stronger filters and more targeted data.
5. Repeat.

Every step is locally reasonable. The loop becomes dangerous when the same worldview shapes the data, the reward, and the evaluation. A blind spot shared by all three layers becomes invisible precisely because the measurements agree.

A toy example makes the mechanism concrete. Suppose a data-quality model rewards concise, self-contained answers because human raters prefer them on a static sample. Filtering the top-scoring examples initially removes spam and incoherence. As the threshold rises, however, the filter also removes clarification questions, explicit uncertainty, minority phrasings, and long answers that surface real ambiguity. A model trained on the result may score better with the same preference model while becoming worse at recovering from underspecified user requests. The data is not lower quality in the ordinary sense. It is overfit to one evaluator's theory of quality.

There are three distinct claims to test in that example:

• Measurement: does the quality model agree with independent human judgments outside the curation sample?
• Mechanism: did the filter improve behavior by removing noise, or merely make the training distribution more similar to the evaluator?
• Transfer: does the improvement survive new tasks, new raters, and situations where asking a question is the correct behavior?

A single held-out score tests only the first claim, and often only weakly. Scientific reasoning separates them because each can fail while the others appear healthy.

Public benchmarks create an especially direct version of this problem. Once benchmark questions or close variants enter pretraining and post-training corpora, a high score no longer distinguishes generalization from exposure. Even without literal contamination, years of benchmark-driven development can create conceptual contamination: the entire pipeline is designed around the benchmark's ontology of the task.

Datasets then drift toward legibility. Easy-to-label examples survive. Disagreement looks like annotation noise. Long-tailed behavior looks statistically inconvenient. Awkward examples that do not fit the rubric are removed, even when those examples contain the most information about where the rubric is wrong.

This is the data equivalent of Tao's excessively polished proof. Removing artificial friction is useful. Removing natural friction erases information about where understanding is difficult.

3. Science treats the metric as a hypothesis

Scientific reasoning has no general escape from proxies. Temperature, intelligence, economic welfare, model capability, and mathematical significance are not available as direct objects. They are operationalized through instruments and measurements.

This is not a proposal only for datasets about science or mathematical reasoning. Scientific reasoning here describes the evaluation method, not the subject matter. The same method applies whenever an optimizer acts on an imperfect proxy: training-data curation, reward modeling, search ranking, product metrics, hiring, education, or public policy. The stronger the optimizer and the less directly observable the goal, the more important it becomes to preserve evidence outside the optimization loop.

The scientific move is not to pretend that the operationalization is the construct. It is to treat the connection between them as a fallible hypothesis.

This is the core of construct validity. A test is not validated once because it correlates with a desired outcome on one dataset. Its interpretation earns credibility through a network of predictions: it should agree with different measures of the same construct, disagree with measures of different constructs, behave as theory predicts under interventions, and continue to work across relevant populations and conditions.⁴

That distinction changes the optimization loop. Instead of asking only, "Did the score improve?", we ask:

• What theory connects this score to the capability we care about?
• Under what conditions should that connection fail?
• Which independent measurement would disagree if we were gaming the proxy?
• What changed in the data distribution because we optimized against this score?
• Which anomalies did the curation process remove rather than explain?

The metric becomes evidence about the goal, not a definition of the goal.

Scientific reasoning is not a better objective function. It is a disciplined refusal to let any objective function become final.

4. Keep the evaluation loop open

A scientific response to over-optimization can be compressed into five practices.

State the latent goal before the proxy

Write down what success means in terms that are not reducible to the current metric. For training data, this might include transfer to unseen tasks, robustness under distribution shift, calibration, usefulness to real users, and the ability to recover from ambiguous instructions. The list will be incomplete. That is useful: it preserves a visible gap between the construct and its measurement.

Dataset documentation helps here. A datasheet records why a dataset was created, how it was collected, what was excluded, and which uses are out of scope.⁵ This is not administrative garnish. It preserves the causal history needed to understand what an improved score might actually mean.

Use measurements that fail differently

Adding five variants of the same benchmark is not triangulation. The measurements need different sources of error.

Combine automated scores with blind human review; static benchmarks with fresh tasks; aggregate performance with slice-level analysis; behavioral evaluation with causal or mechanistic investigation; adversarial tests with ordinary deployment traces. Agreement across genuinely different methods is evidence. Disagreement is not an inconvenience to average away. It is often the first sign that the target has separated from the goal.

Preserve an outside distribution

An evaluation set stops being independent when its failures repeatedly shape training. Maintain fresh, time-separated, or externally authored evaluations that the optimization process cannot inspect. Rotate them. Measure performance on tasks created after the training cutoff. Treat contamination as a threat to inference, not merely a data-cleaning bug.

This will make progress slower and headline numbers noisier. That cost is the point. An evaluation that cannot surprise the training process cannot falsify its assumptions.

Search for counterexamples, not only averages

Optimization makes aggregate curves go up. Scientific reasoning asks which observations could prove the story wrong.

Before a run, specify the expected tradeoffs and failure modes. After it, inspect examples with the largest disagreement between metrics, not only the largest loss. Run ablations to separate the effect of data volume from selection policy. Create adversarial cases that preserve superficial features while changing the underlying answer. Track which subpopulations disappear as the quality threshold rises.

The unit of progress is not "the metric moved." It is "a plausible alternative explanation became harder to maintain."

Keep humans at the digestion layer

Tao's proof pipeline ends beyond verification and publication. A result becomes durable mathematics only after people digest it, connect it to existing theory, teach it, and decide that it matters. This stage is slow because it integrates a result into a living body of knowledge.

Training pipelines need an equivalent layer. A model improvement is not fully understood when the benchmark passes. Someone should be able to explain which data changed, why that data should cause the observed behavior, what capability transferred, which regressions appeared, and where the evidence remains weak.

If no one can give that account, the result may still be useful. But it is an unverified empirical artifact, not yet shared understanding.

5. The practical stack

The resulting workflow looks less like a single optimization objective and more like a system of checks:

Optimize a declared proxy on the working distribution.
Validate with measurements that have different failure modes.
Challenge the result using fresh data, counterexamples, and distribution shifts.
Explain the causal story with ablations, provenance, and error analysis.
Digest the result through independent human judgment and downstream use.
Revise the construct, the metric, or the data when these layers disagree.

None of these stages is immune to Goodhart's law. Human ratings can become formulaic. Red-team suites can become benchmarks. "Diversity" can collapse into a quota that preserves surface variation while removing deeper disagreement. Even replication counts and publication standards can become targets.

The defense is therefore architectural rather than metric-level. No one signal is allowed to close the loop. Each layer is designed to expose assumptions made by the previous one.

This is also why scientific institutions distribute criticism across people. Independent replication, peer review, competing theories, and public methods are not ceremonial expressions of rigor. They reduce the chance that one optimization process controls both the claim and every test of the claim.

6. Limits

The scientific approach is slower than direct optimization. It requires redundant measurements, protected evaluations, expert judgment, and willingness to investigate regressions that do not affect the headline number. In frontier model training, some of those checks are expensive enough to compete directly with scaling.

It also does not produce a value-neutral answer. Deciding what counts as useful, representative, understandable, or worthy of preservation is partly a scientific question and partly a social one. Better measurement cannot resolve a disagreement about goals.

And there is no stable point at which a metric is permanently safe. A fresh evaluation becomes a target once it is visible and valuable. A strong human-review process develops conventions. A successful adversarial suite teaches the next system what attacks to expect.

The goal is not to end this cycle. It is to keep the cycle epistemically productive: optimize, discover a mismatch, update the theory, build a new test, and remain able to be surprised.

7. From scale to understanding

Tao expects mathematics to move from proof scarcity toward proof abundance. The scarce resource then shifts downstream: verification, exposition, acceptance, and canonicalization. Training data may be undergoing a similar transition. We are no longer limited only by the number of tokens or examples we can collect. The bottleneck is knowing what the data teaches, which behaviors it suppresses, and whether our measures of quality remain connected to the capabilities we actually want.

Over-optimization happens when we mistake a temporarily useful correlation for a permanent identity. The benchmark becomes capability. The filter becomes quality. The verified proof becomes mathematical understanding.

Scientific reasoning breaks that identity on purpose. It maintains a gap between the thing and its measure, then fills the gap with competing evidence, causal explanation, and organized criticism.

The answer to Goodhart's law is not to stop optimizing. It is to preserve an outside view that the optimizer does not control: fresh data, measurements with different failure modes, counterexamples, causal explanation, and human judgment.

A metric should guide the search, not close the case. Progress is credible only while independent evidence retains the power to overturn the story told by the score.

References