A learner model developed the ability to predict images, text and speech by training exclusively on synthetic byte sequences rather than natural datasets. The "Self-Play Pretraining" method employs two models that start from random initialization: a generator that writes programs for a universal Turing machine and a learner that predicts the resulting outputs. As the generator produces increasingly challenging sequences to drive improvement, the learner's zero-shot validation loss on real-world audio, melodies and text decreases as training compute increases.

The research, co-led by Michael Yli, Aditya Cowsik and Kfir Dolev of Stanford NLP, demonstrates that general inductive biases can act as a "Universal Grammar" for learning about the world. The model acquired in-context learning capabilities without exposure to any human-generated examples, proving that general patterns can emerge through self-play. This approach suggests a potential shift away from the current dependency on massive, human-curated datasets for AI pretraining.