What is the core structure of Transformer?

The Transformer's core structure consists of an encoder-decoder architecture built using stacked layers. The defining feature is the multi-head self-attention mechanism, enabling the model to weigh the importance of all words in the input sequence relative to each other for any given position, regardless of distance.

The encoder and decoder layers share key components. Each contains a multi-head self-attention sub-layer and a position-wise feed-forward neural network sub-layer. Residual connections surround each sub-layer, followed by layer normalization, significantly aiding training stability and convergence. The decoder includes an additional encoder-decoder attention sub-layer that allows it to focus on relevant parts of the encoder's output. Positional encodings are added to the input embeddings to inject information about the order of tokens since the model lacks inherent recurrence or convolution.

This structure revolutionized natural language processing. The Transformer's efficient parallel processing and ability to capture long-range dependencies make it foundational for modern large language models. Its core principles enable state-of-the-art performance in sequence-to-sequence tasks like machine translation, text summarization, question answering, and text generation. The attention mechanism is central to understanding relationships within data.