Freaks City

Model Structure The basic unit consists of token embedding with positional encoding, encoder, and decoder. Encoder: Self-attention layer with skip connections and layer normalization, followed by a feed-forward network (FFN) with skip connections and layer normalization. Decoder: Self-attention layer with skip connections and layer normalizati ...

Attention Mechanisms Multi-Head Attention The attention mechanism computes: The scaling factor \(\sqrt{d_k}\) prevents large inner product values that could cause gradient instability. Assuming Q and K elements have mean 0 and variance \(\sigma^2\), the variance of \(QK^T\) grows with \(d_k\). Scaling by \(\sqrt{d_k}\) maintains stable varianc ...

Attention Mechanisms and Transformers The attention mechanism addresses a fundamental challenge in deep learning: transforming variable-dimensional inputs into fixed-dimensional outputs through a weighted aggregation process. This capability proves essential when dealing with sequences or sets of varying sizes, where traditional fixed-parameter ...

MobileFormer introduces a novel architecture that synergistically combines the strengths of convolutional neural networks (CNNs) and Transformers to achieve high efficiency with minimal computational overhead. By leveraging a lightweight bidirectional bridge between a mobile backbone and a compact Transformer, it enables effective exchange of l ...