Language model harnesses are compositional generalizers

Alex L. Zhang and Omar Khattab wrote the article “Language model harnesses are compositional generalizers”; this source set also includes a response from swyx. The article argues that an RLM harness can transfer learned decomposition strategies across longer tasks and different domains by keeping each root-model call structurally familiar.

> “Training on only short tasks generalizes to held-out tasks 8–32x longer, with roughly 10x the eval lift with the same train lift over training the underlying Transformer directly.”

Zhang and Khattab argue that generalization can live in the program around a language model. Their harness breaks an unfamiliar task into smaller observations that resemble tasks the model already learned.

They test the idea by training Qwen3-30B-A3B-Instruct-2507 inside a Recursive Language Model. Training on short tasks transferred to held-out tasks 8–32 times longer. Across the authors’ experiments, the RLM produced roughly ten times the evaluation lift of training the Transformer directly for the same training lift. Training in one domain also transferred more strongly to domains with a similar decomposition strategy.

The authors describe this as a harness-induced equivalence relation. Two tasks can look unrelated at the input level yet produce nearly the same root-model trajectory after task-specific details are offloaded. These are author-reported results from one model and training recipe, not an independent replication or a claim that every harness generalizes.

> “This is the issue that modern post-training attempts to bandage with targeted environments based on perceived applications.”

Modern post-training often builds a new environment for each desired capability. That approach gets expensive when a model cannot reuse a procedure outside the domain where it learned it.

Compositional generalization means solving an unseen problem by combining familiar concepts and procedures. Zhang and Khattab argue that language tasks often have short, describable decompositions even when their inputs and outputs are long. Reusing one learned decomposition across tasks could reduce how many new environments post-training needs.

The argument does not reject data scaling. It says data produces weaker returns when the surrounding system does not help learned procedures transfer.

> “meaning structurally similar tasks fall under the same harness-induced equivalence class and produce similar sets of observations for the neural network involved.”

A harness sits between the environment and the neural network. It selects model-visible state, calls models and tools, and maps their outputs to the next environment action.

The article calls an observation locally in-distribution when an individual model call has a familiar shape relative to training, even if the complete task is new. Standard agent loops often append instructions, reasoning, tool output, and environment feedback into one growing context. The authors argue that this history drifts away from familiar inputs and contributes to context rot.

Their alternative is to make structurally similar tasks look similar to the root model. This is an operational goal, not a directly measured fact about the model’s unknown training distribution. Similar token trajectories also do not prove identical internal reasoning.

> “The Recursive Language Model (RLM) harness is designed around abstracting tasks by their decomposition and deferring input-specific information to sub-calls.”

The RLM uses two mechanisms. First, context offloading stores input-specific material in an external variable instead of placing it in the root prompt. Second, programmatic sub-agent calls treat models and tools as functions inside a REPL. Their outputs can stay in variables and feed later calls without flooding the root context.

This lets the root model work mainly with the decomposition. A math task and an essay task may share a search-and-filter procedure even though their content differs. The sub-calls handle those domain-specific details.

“Isomorphic” describes the ideal case where equivalent tasks produce nearly identical root trajectories. Offloading does not remove domain dependence from sub-calls, and recursive calls add latency, memory use, and more places for errors to propagate.

> “On MRCRv2, GraphWalks, OOLONG, and OOLONG-Pairs, the trained Qwen3-30B-A3B-Instruct-2507 RLM approaches or exceeds an RLM with a frontier model, GPT-5.5 on the long eval, while far exceeding the base Transformer.”

The length experiment uses six environments: MRCRv2, GraphWalks, LongBench Pro, OOLONG, OOLONG-Pairs, and Ada-LEval. The model trains only on short splits and evaluates on substantially longer ones, spanning changes in input length, output length, and instruction count.

The trained RLM improved on the long splits across all six tasks. On MRCRv2, GraphWalks, OOLONG, and OOLONG-Pairs, it approached or exceeded an RLM using the stronger GPT-5.5 comparison model. The base Transformer’s training reward rose while its long-split evaluation result often stayed flat.

The harness still has to learn the right strategy. On short inputs, it can take a shortcut by sending the whole task to one sub-call. That strategy breaks at longer lengths. A brief instruction to decompose helped on MRCRv2, showing that supervision or hints may be needed for sample-efficient learning.

The 8–32× result applies to these selected splits. It does not mean RLMs solve tasks of arbitrary length or difficulty.

> “We similarly find that the RLM exhibits clear generalization capabilities on completely different domains than the one it trained on, while the base Transformer struggles to meaningfully improve.”

The authors also train and evaluate on different domains that share an intended strategy. One experiment moves from aggregating TREC categories to aggregating spam labels. Two OBLIQ-Bench experiments move from matching writing styles to matching math-solution patterns, and from finding stances in tweets to finding error types in WildChat conversations.

The RLM transferred across all three pairs. The base Transformer showed little or negative evaluation lift. The result supports the authors’ claim that the root model can learn a reusable operation while sub-calls absorb the domain-specific language.

The pairs were chosen because they share a decomposition. They do not establish arbitrary domain transfer. The metrics also differ: OBLIQ uses nDCG@10, while OOLONG uses task-specific accuracy.

The RLM is more expensive per training step. The article reports roughly 7–13 times more combined root-and-subcall tokens than direct Transformer training. The proposed advantage is broader transfer from each environment, not free compute.

> “In the Appendix, we discuss some proxies for determining whether two trajectories fall under the same equivalence class.”

The authors cannot directly prove that a prompt is in-distribution for the model. Instead, they compare each evaluation trajectory with the closest root trajectory seen earlier in training.

Token-for-token identical root sequences would make two tasks identical under this measurement. Real trajectories rarely match exactly, so the appendix measures degrees of similarity. This turns the isomorphism claim into a testable proxy, but the result depends on how distance is defined.

> “And as things stand today, it seems that the returns on scaling Transformers and related neural architectures are just not budging in terms of being poor compositional generalizers.”

Zhang and Khattab do not propose replacing data scaling with fixed human decompositions. Data remains the main driver of progress. Their claim is that the machinery receiving that data changes how efficiently learning transfers.

Harness design can be searched, optimized, and trained like other parts of an AI system. A useful harness should reduce unfamiliar problems to familiar ones without hard-coding a separate solution for each domain. The current experiments are early evidence for that direction, not a comparison of long-run scaling curves across many harness families.

> “We plot across 5 different distance metrics to highlight how much closer RLM trajectories on unseen tasks look than an appended context baseline like a base Transformer.”

The appendix measures distance from each evaluation trajectory to its nearest earlier training trajectory in five ways: normalized token edit distance, word 3-gram containment, word 3-gram Jaccard, weighted Jaccard, and length ratio.

Across five of the six length experiments with a base comparison and all three strategy-transfer experiments, the RLM’s evaluation trajectories looked much closer to prior training trajectories than the Transformer’s appended contexts. Figure 8 attributes much of that gap to context offloading.

The metrics are limited. Length ratio ignores content. Edit and n-gram measures reward surface overlap and miss paraphrased semantic equivalence. The authors state that a stronger metric should combine token and semantic similarity. The chart shows greater surface similarity; it does not show that the RLM chose the same decomposition.

> “It happens to support the finding that RLMs can generalize to unseen tasks that share latent structure observed in training.”

swyx uses the appendix to discuss benchmark lookalike training without exact test leakage. Developers can target an evaluation by training on structurally similar tasks. When training datasets and reinforcement-learning environments are not released, outsiders cannot easily separate memorization, task-specific practice, and genuine transfer.

Research on “training on the test task” supports the distinction between exact data contamination and exposure to task-relevant examples. It does not support swyx’s rhetorical claims that this happens “99% of the time” or can produce almost any desired score. Those remain his characterization of frontier training, not a measured finding in Zhang and Khattab’s work.

The article’s trajectory analysis also does not audit frontier-model training corpora. It compares trajectories produced inside the authors’ controlled RLM experiments.

> “We find that well-designed harnesses form a quotient set over task trajectories, meaning their individual LLM calls can see structurally “similar” tasks as near-identical, token-for-token!”

swyx attaches screenshots of the article’s distance-metric table and Figure 8. They show the nearest-training-trajectory setup and the RLM/Transformer similarity gap he wants readers to notice.

The screenshots reproduce the primary article. They add emphasis, not independent evidence. The method and results remain attributable to Zhang and Khattab.

> “Harnesses can effectively generalize for the Transformer during training, without relying on any intrinsic generalization capability from the model.”

The response quotes Zhang’s launch post. Alex summarizes the quotient-set claim, the 8–32× length experiments, and transfer between unrelated surface domains that share a decomposition.

That post explains why swyx focused on hidden trajectories, but it is promotional author framing rather than corroboration. The article supplies the fuller conditions: selected benchmark pairs, one base model, additional RLM compute, shortcut failures, and surface-level similarity metrics.