Finding structure in time

Jeffrey Elman was a linguist, and the question in his 1990 paper was whether a network could learn anything about language from exposure alone. His network was simple: a standard hidden layer whose activations were copied, at each step, into a set of context units that fed back in at the next step, so that the network's state carried a memory of what it had seen. He trained it to predict the next word in sentences generated from a small grammar.

It could not predict the exact word, because that is not predictable. What it learned instead was the structure. The hidden states clustered nouns apart from verbs, animate from inanimate, and the network's predictions respected agreement across intervening words, without anyone telling it what a noun was. The model had discovered categories from the statistics of sequence.

The paper is the ancestor of every language model that followed, and its title is the programme. Next-word prediction as a task, learned representations as the product, and grammar emerging rather than being written: GPT is Elman's network with a hundred billion times the parameters and attention instead of recurrence. The weakness he found, that memory decayed over long sequences, was named the vanishing gradient the next year and solved by the LSTM in 1997.