GPT trained from scratch solves 15 puzzle but fails easiest boards
Start here
Aatish Bhatia trained a GPT from scratch on a million 15-puzzle solutions written as token sequences. The model solved 99.9% of randomly scrambled boards, matching its teacher's exact moves in 2,996 of 3,000 test puzzles. The same model failed on the easiest boards, including one that was already solved, and gave its wrong opening moves probabilities above 97%. Bhatia attributes this to imitation of well-scrambled training data and to shortcuts that do not generalize outside that distribution.
Generated from reports · updated 2 h ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AIA simple 15 puzzle shows how AI's jagged intelligence emerges
Aatish Bhatia trained a GPT from scratch on a million 15-puzzle solutions written as token sequences, and it solved 99.9 percent of randomly scrambled boards, matching the teacher's exact moves in 2,996 of 3,000 test puzzles. The same model failed on the easiest boards, including one already solved, and assigned its wrong opening moves probabilities above 97 percent. The author attributes this to imitation of well-scrambled training data and to shortcuts that do not generalize out of distribution.
Heat for this event
Not enough continuous observations yet to draw a trend.