Loop. Loop. Loop. If 2024 asked how big your model is and 2025 asked how long it can think, 2026 asks what your loop is closed around. "An LLM in a while loop" started as a dismissal and ended up a job title; there are loop-engineering manifestos now. Underneath the branding, a loop is a search process, and a search process is only as good as the signal it climbs. At NeurIPS 2025, months before the loop had a fan club, my brother Chin and I won the Embodied Agent Interface challenge by closing ours around the strongest signal available: the benchmark's own evaluator.
The challenge measures how well language models plan household-robot tasks in two simulators, BEHAVIOR and VirtualHome. During the development phase, the official evaluator does more than score a submission — it explains precisely why a plan failed. We built our pipeline around that feedback: prompt a frontier model, evaluate its answer, feed the errors into the next attempt, and repeat. Every plan that survives the process becomes verified training data.
The loop earns its keep because of where capable models fail. They produce plans a human would readily accept, and the simulator rejects them for omitting details no person would think to mention — the new identifier an object receives after being sliced, or an OPEN action that seems implied. Conventions like these are nearly impossible to anticipate in a prompt, but they are easy to learn from error logs, and the loop collects them automatically.
Why this is a good idea
The data makes itself. There are no human labels anywhere in the pipeline. The loop converts inference calls into verified training examples, so the cost of data scales with compute rather than with annotator time.
Hard tasks teach the most. A task that takes ten attempts to pass contributes ten error logs. The dataset naturally over-samples whatever the model finds hardest, which is exactly the curriculum you would want to design by hand.
Even the judge can be distilled. The official evaluator is withheld during the test phase, so we trained a model to imitate its feedback and used that imitation to review our answers before submission. The loop keeps working after the oracle is gone.
What it bought us
We distilled the loop's output into small Qwen3 models. On BEHAVIOR, every module we submitted was a 0.6-billion-parameter specialist, and every one of them outscored the frontier baseline:
| Module (BEHAVIOR) | gpt-5-mini baseline | Qwen3-0.6B fine-tuned (ours) |
|---|---|---|
| Goal interpretation | 78.6 | 99.6 |
| Subgoal decomposition | 50.0 | 97.0 |
| Action sequencing | 68.0 | 98.0 |
| Transition modeling | 80.0 | 99.5 |
Official BEHAVIOR scores. VirtualHome shows the same pattern with larger models; full tables in the report. Overall: 90.09, against 84.32 for the second-place team.
Why robotics should care
Robotics is unusually rich in the one ingredient this recipe needs. The field runs on simulators, and every simulator is a free evaluator: the goal state is reached or it is not, at zero labeling cost. Closing a loop between a frontier model and that signal turns any simulated environment into a training-data factory. "Overfitting to the simulator" is usually said with a wince, but here it is the point — the loop exhaustively learns whatever the environment actually rewards, without a single human label.
The sharper lesson is where competence has to live. The model in our loop knew nothing special about robotics — it stumbled into BEHAVIOR's conventions attempt by attempt, because the evaluator could always say what was wrong. Machine learning has a name for the asymmetry that makes this work: the generator–verifier gap. Checking a plan is far easier than producing one, so a model too weak to write expert answers on demand can still search its way to them, as long as the verdict is real. This is the same verifiable reward the reasoning-model boom runs on — and robotics is the field where it comes free. If frontier models are going to become the robot brains, as I argue in GPT-7 Will Have Arms, they won't need to arrive knowing robotics. They need loops closed around real evaluators, and robotics has more of those than any other field in AI.
Team AxisTilted2: Chin Pradeep (NYU Neuroinformatics Lab) and San Kala (independent); equal contribution. Full methods are in the technical report and winners' presentation. The benchmark: Embodied Agent Interface.