Processing variable-length sequences through Recurrent Neural Networks (RNNs) requires special handling. PyTorch provides two key functions for this: pack_padded_sequence and pad_packed_sequence. The core concept involves transforming padded batch tensors into a packed format for efficient RNN computation and then restoring the batch structure.

Function: torch.nn.utils.rnn.pack_padded_sequence

This function converts a batch of padded sequences into a PackedSequence object.

torch.nn.utils.rnn.pack_padded_sequence(
    input, 
    lengths, 
    batch_first=False, 
    enforce_sorted=True
)

Parameters:

• input: Batch of sequences. Shape is [S, B, *] (if batch_first=False) or [B, S, *] (if batch_first=True), where S is max sequence length, B is batch size.
• lengths: List or 1D Tensor containing the actual length of each sequence in the batch.
• batch_first: Boolean indicating if the batch dimension is first in input.
• enforce_sorted: If True, expects lengths and input to be sorted in descendign order of length.

Returns a PackedSequence object.

Function: torch.nn.utils.rnn.pad_packed_sequence

This is the inverse operation, converting a PackedSequence back to a padded tensor.

torch.nn.utils.rnn.pad_packed_sequence(
    sequence, 
    batch_first=False, 
    padding_value=0.0, 
    total_length=None
)

Parameters:

• sequence: The PackedSequence object to unpack.
• batch_first: Boolean. If True, output tensor shape is [B, S, *].
• padding_value: Value used for filling padded elements.
• total_length: If set, pads the output to this langth.

Returns a tuple: (padded_sequence, lengths) where lengths is a tensor of sequence lengths.

Practical Implementation Example

This example demonstrates preparing data, packing it, passing it through a RNN layer, and unpacking the results.

import torch
import torch.nn as nn

# Sample data: A batch of three sequences with varying lengths
original_sequences = [
    [23, 14, 56, 71, 9],
    [42, 8, 33],
    [19, 5, 27, 88, 64, 12]
]
actual_lengths = [5, 3, 6]  # Corresponding valid lengths
batch_count = 3
max_seq_len = 6
feat_dim = 10
hidden_dim = 20
vocab_size = 100

# 1. Sort sequences by length (descending order)
sorted_pairs = sorted(zip(actual_lengths, original_sequences), reverse=True)
sorted_lengths, sorted_seqs = zip(*sorted_pairs)

print("Sorted Sequences:", sorted_seqs)
print("Sorted Lengths:", sorted_lengths)

# 2. Pad the sorted sequences
def apply_padding(seq, target_len, pad_idx=0):
    return seq + [pad_idx] * (target_len - len(seq))

padded_batch = [
    apply_padding(s, max_seq_len) for s in sorted_seqs
]
print("Padded Batch:", padded_batch)

# 3. Create embedding layer and RNN
embed_layer = nn.Embedding(vocab_size, feat_dim, padding_idx=0)
rnn_cell = nn.GRU(feat_dim, hidden_dim, batch_first=True)

# Convert to tensor
input_tensor = torch.tensor(padded_batch)
embedded = embed_layer(input_tensor)  # Shape: [B, S, feat_dim]

# 4. Pack the padded sequences
packed_input = nn.utils.rnn.pack_padded_sequence(
    embedded,
    torch.tensor(sorted_lengths),
    batch_first=True,
    enforce_sorted=True  # Because we sorted
)

# 5. Pass through the RNN
rnn_output_packed, final_hidden = rnn_cell(packed_input)

# 6. Pad the packed output back to a standard tensor
rnn_output_padded, output_lengths = nn.utils.rnn.pad_packed_sequence(
    rnn_output_packed,
    batch_first=True
)

print("\nPadded RNN Output Shape:", rnn_output_padded.shape)
print("Output Lengths:", output_lengths)

Output:

Sorted Sequences: ([19, 5, 27, 88, 64, 12], [23, 14, 56, 71, 9], [42, 8, 33])
Sorted Lengths: (6, 5, 3)
Padded Batch: [[19, 5, 27, 88, 64, 12], [23, 14, 56, 71, 9, 0], [42, 8, 33, 0, 0, 0]]
Padded RNN Output Shape: torch.Size([3, 6, 20])
Output Lengths: tensor([6, 5, 3])

The packed_input removes the padding timesteps, allowing the RNN (GRU in this case) to compute only over the actual data, improving efficiency. The final rnn_output_padded restores the batch dimensions with zeros in the padded positions, matching the original padded input structure. The output_lengths tensor confirms the lengths of the valid sequences in the result.