The Machine Learning “Advent Calendar” Day 24: Transformers for Text in Excel

The self-attention mechanism in Transformer models is explained using a simplified example with the word 'mouse' in different contexts. Initially, 'mouse' has an ambiguous embedding, but through self-attention, it develops a context-specific representation. The process involves calculating dot-product scores between word embeddings, scaling them, and applying a softmax function to create an attention matrix. This matrix then determines how to create new, context-aware output embeddings by taking a weighted average of the input embeddings. The article concludes by introducing the learned weight matrices for Queries, Keys, and Values (Q, K, V) that allow the model to refine this process during training.