Paper · Working manuscript · preliminary, not peer reviewed · PDF, 5 pages
When more self-play makes an agent worse
An AI that trains by playing itself got better, then worse, even though its training numbers kept improving.
October 2026
In a MuZero-style agent trained in a custom combat environment, head-to-head performance commonly peaks after roughly 24,000–32,000 self-play games and then declines, even while training loss keeps improving. Lower learning rates, wider models and longer training do not consistently prevent it; continuing a strong checkpoint against a pool of historical opponents is a preliminary stabilization result.
Working manuscript — preliminary, not peer reviewed. It does not establish a causal mechanism.
More papers