Core Automation says transformers cannot continually learn and is automating kernel generation to find a replacement

This Recap covers a Sequoia Capital interview with Core Automation founders Jerry Tworek and Rohan Anil, hosted by Sonya Huang and Pat Grady. It is about the founders' claim that transformer architecture now limits further progress because deployed models cannot continually learn from real work. It matters because Core Automation is building an automated research lab to search for a replacement architecture, starting with the GPU kernels needed to run new ideas efficiently.

Tworek says the field has mastered large-scale pre-training and reinforcement learning. Most architecture work now makes transformers cheaper rather than more capable.

Current coding agents automate parts of his work but still need a person to choose, direct, and judge tasks. He believes more attention and mixture-of-experts scaling will not remove that dependence.

Tworek expected scaled reinforcement learning to close the gap to AGI. Asked in 2024, he would have predicted AGI in 2025.

Benchmark scores improved without covering the full range of real tasks. He says training tasks and evaluations resemble each other, while production work is wider and messier.

Tworek wants models to learn from users and their real tasks during deployment. In-context learning is short-lived; he says he often has to compact Codex after about 20 minutes. Continual fine-tuning can retain more data, but he says it is inefficient and causes catastrophic forgetting.

He wants a learned mechanism at the architecture level that can keep adapting over much longer periods. He also says alternatives may need substantial compute before their advantages appear, so small experiments can reject them too early.

Tworek says frontier labs are competing on current products and coding agents. A profitable architecture gets resources that a possible replacement may not.

Anil treats architecture as a decision about how to spend computation. Autoregressive generation adds one token at a time. Core Automation intends to work across architecture, training, optimization, kernels, and hardware because a better model is useless if it cannot run efficiently.

Anil says chain-of-thought reasoning adds computational depth through a longer token sequence. Stronger results therefore require more serial generation.

Tworek says pre-training can compress existing knowledge and reinforcement learning can install known workflows. The model still depends on a lab to retrain it as events, tools, codebases, and tasks change.

Tworek treats pre-training as compression and imitation. Reinforcement learning lets a model generate experience and learn from outcomes.

He expects other methods because physical practice and conceptual learning use experience differently. Both are learning, but they do not follow the same process.

Anil says pre-training perplexity does not measure the final system after reinforcement learning changes it. He wants both stages optimized together.

A stronger optimizer can train architectures that a weaker one cannot. Anil also calls reinforcement learning inefficient because a long generated trace may deliver little feedback to the model.

Tworek defines automation as giving each researcher more agency. Core Automation is organizing a small team and its processes around coding agents so it can test ideas faster.

His threshold for AGI is a model that can improve itself without humans in the loop. He says human-model teams work well today, while models alone have not taken over complete work loops.

Anil says the first target is kernel generation. New architecture operations must run efficiently on GPUs before researchers can test them at scale.

He describes a QR-kernel result that required rare human skill, four weeks, and about $100,000 in coding-agent spend. He says it reached a 60-times speedup for the target case, which current models could not produce alone.

Core Automation wants to automate the path from an architecture idea to an efficient implementation, then search for systems that continually learn. Its test is whether the systems get better at doing the scientists' work from one day to the next.