Attention

The encoder–decoder models of 2014 compressed a whole source sentence into one vector and then generated the translation from it. That worked for short sentences and degraded for long ones, because a fixed vector cannot hold forty words. Dzmitry Bahdanau's fix, posted on 1 September 2014 with Kyunghyun Cho and Yoshua Bengio, was to let the decoder look back. At each output word the model computes a weight for every source word, takes a weighted average of their encodings, and uses that as its context. The weights are learned, and when you plot them, they draw the alignment between the two languages.

The paper called it soft alignment; the community called it attention, and the name stuck. It was the first mechanism that let a network choose what to look at, and it removed the bottleneck that had limited sequence models since Elman.

Three years later, Google researchers asked what happened if attention was the only mechanism, with no recurrence at all, and the transformer was the answer. Every large language model computes, at every layer, a version of the weighted average Bahdanau introduced here. The instrument on this site shows the weights a real model computes over a sentence you type.