0 XP

Attention · Why words need neighbours

The problem

Some words can’t be understood alone

Give every word a fixed position in a space of meaning and you hit a wall almost immediately. “Bank” gets one position — but a river bank and a savings bank have nothing to do with each other. The word alone simply doesn’t contain the answer.

The attention patterns here are hand-authored illustrations of what trained models reliably do — not weights read out of a running model. the original paper