A mathematical theory of communication

In two parts published in July and October 1948, Claude Shannon gave information a unit and a mathematics. A message carries information in proportion to how surprising it is. The average surprise of a source is its entropy, measured in bits. A noisy channel has a capacity, and below that capacity you can send messages with as few errors as you like. Communication engineering had been a craft; after Shannon it was a science with theorems.

The paper also contains, almost as a diversion, the first statistical language models. Shannon shows how to generate English-looking text by sampling letters, then letter pairs, then words, according to their measured frequencies, and prints the increasingly plausible gibberish that results. He estimates the entropy of English at roughly one bit per letter. That is the training objective of every large language model, stated seventy years early: predict the next symbol, and measure how surprised you are.

Cross-entropy loss, perplexity, the idea that compression and prediction are the same task, all of it comes from these pages. When a modern model is trained, the number that goes down is Shannon's.