A simple 15 puzzle shows how AI's jagged intelligence emerges

Aatish Bhatia trained a GPT from scratch on a million 15-puzzle solutions written as token sequences, and it solved 99.9 percent of randomly scrambled boards, matching the teacher's exact moves in 2,996 of 3,000 test puzzles. The same model failed on the easiest boards, including one already solved, and assigned its wrong opening moves probabilities above 97 percent. The author attributes this to imitation of well-scrambled training data and to shortcuts that do not generalize out of distribution.