A neural probabilistic language model

Language models in 2003 counted. An n-gram model estimated the probability of the next word from how often the preceding two or three words had been followed by it in a corpus, and it could not generalise: a sentence it had never seen was as unlikely as one that made no sense. Yoshua Bengio's group in Montréal proposed, in a paper first presented at NIPS in 2000 and published in full in 2003, to give every word a learned vector of a few dozen numbers and to predict the next word with a neural network from the vectors of the previous ones. Similar words would get similar vectors, and what the model learned about "cat" would transfer to "dog".

The paper introduced the word embedding, the idea that meaning could be a position in a space learned from prediction, and it beat the best n-gram models. It also took days to train on a few million words and was regarded as an elegant impracticality for most of a decade.

Word2vec in 2013 made the embeddings cheap and famous; the language models of the 2020s are this architecture with attention in place of the fixed window and a hundred billion parameters in place of a few hundred thousand. The task, predict the next word, has not changed since this paper.