Encoder

The encoder interface accepts variable-length sequences as input X. Any model that inherits from the base Encoder class implements the actual encoding logic.

from torch import nn

class Encoder(nn.Module):
    """Base encoder interface for encoder-decoder architecture"""
    def __init__(self, **kwargs):
        super(Encoder, self).__init__(**kwargs)

    def forward(self, X):
        raise NotImplementedError

Decoder

The decoder interface includes an additional init_state method that transforms encoder outputs (enc_outputs) into encoded states. This step may require supplementary inputs such as sequence lengths. To generate varible-length token sequences step by step, the decoder maps enputs (such as tokens generated in previous timesteps) along with encoded states to produce outputs at each time step.

class Decoder(nn.Module):
    """Base decoder interface for encoder-decoder architecture"""
    def __init__(self, **kwargs):
        super(Decoder, self).__init__(**kwargs)
    
    def init_state(self, enc_outputs, *args):
        raise NotImplementedError
    
    def forward(self, X, state):
        raise NotImplementedError

Combining Encoder and Decoder

The encoder-decoder architecture consists of a encoder, a decoder, and optional additional parameters. During forward propagation, the encoder's output generates encoded states, which are then fed to the decoder as part of its input.

class EncoderDecoder(nn.Module):
    def __init__(self, encoder, decoder, **kwargs):
        super(EncoderDecoder, self).__init__(**kwargs)
        self.encoder = encoder
        self.decoder = decoder

    def forward(self, enc_x, dec_x, *args):
        enc_outputs = self.encoder(enc_x, *args)
        dec_state = self.decoder.init_state(enc_outputs, *args)
        return self.decoder(dec_x, dec_state)

The use of the term "state" in this architecture motivates stateful neural network implementations for such structures.