Embedding layers serve as fundamental components in neural network architectures that process textual data. These layers transform discrete tokens into continuous vector representations that machines can effectively process.

Concept of Token Embedding

Token embedding represents the transformation of symbolic text into numerical vectors. This conversion enables computational systems to work with language data through mathematical operations. The embedding process essentially creates a lookup table where each unique token maps to a specific vector representation.

The embedding vectors capture semantic relationships between tokens, allowing similar words to have similar vector representations in the high-dimensional space.

PyTorch Implementation

The torch.nn.Embedding module accepts index sequences as input and proudces corresponding embedded vectors:

torch.nn.Embedding(vocab_size, vector_dimension, pad_index=None,
                   maximum_norm=None, normalization_type=2.0, 
                   frequency_scaling=False, 
                   sparse_gradients=False, weight_matrix=None)

Configuration Parameters

• vocab_size (int): Size of the vocabulary, representing the total number of unique tokens available for embedding
• vector_dimension (int): Dimensionality of the output embedding vectors
• pad_index (int, optional): Index reserved for padding tokens, typically set to zero during initialization
• maximum_norm (float, optional): Maximum allowed norm value; embeddings exceeding this threshold undergo renormalization
• normalization_type (float, optional): Type of norm used for calculations, defaults to L2 norm
• frequency_scaling (boolean, optional): When enabled, scales gradients based on token frequency within mini-batches
• sparse_gradients (bool, optional): Controls whether gradient updates use sparse tensor representations

Implementation Example

import torch
import numpy as np

# Initialize embedding layer: vocabulary of 15 tokens, 4-dimensional vectors
token_embedding = torch.nn.Embedding(15, 4)

# Sample sentences: ['She runs fast', 'They play games', 'She reads books']
# Original: [['she','runs','fast'],['they','play','games'],['she','reads','books']]

# Create vocabulary mapping with indices
# Vocabulary: {'pad': 0, 'eos': 1, 'she': 2, 'runs': 3, 'fast': 4, 
#              'they': 5, 'play': 6, 'games': 7, 'reads': 8, 'books': 9}

# Convert sentences to index sequences
sentences_indices = [
    [2, 3, 4, 1],  # she runs fast + EOS
    [5, 6, 7, 1],  # they play games + EOS
    [2, 8, 9, 1]   # she reads books + EOS
]

# Determine actual sequence lengths
sequence_lengths = [4, 4, 4]

# Apply padding using index 0 for shorter sequences
padded_sequences = [
    [2, 3, 4, 1],
    [5, 6, 7, 1], 
    [2, 8, 9, 1]
]

# Transform to time-major format [sequence_length, batch_size]
time_major_format = np.transpose(padded_sequences)

# Convert to LongTensor for embedding lookup
input_tensor = torch.LongTensor(time_major_format)

# Generate embedded representations
embedded_output = token_embedding(input_tensor)

print(embedded_output)

The resulting tensor contains embedded represetnations for each token in the input sequence, organized according to the specified dimensions and vocabulary configuration.