To address the excesive parameter counts incurred by fully connected networks when processing images—where even small images can lead to hundreds of millions of parameters—convolutional neural networks (CNNs) were introduced. CNNs exploit two key structural inductive biases: translation invariance and locality. These enable_sparse, weight-shared computations via convolutional layers, followed often by pooling layers to reduce spatial resolution and improve robustness to small geometric variations.

6. From Fully Connected to Convolutional Layers

6.1.1 The Limitations of Fully Connected Layers

A fully connected layer maps a flattened image vector $\mathbf{x} \in \mathbb{R}^{n_h n_w}$ to a hidden representation $\mathbf{h} \in \mathbb{R}^{m_h m_w}$ using a weight matrix $\mathbf{W} \in \mathbb{R}^{m_h m_w \times n_h n_w}$. To preserve spatial structure, we reinterpret indices as 2D positions and define a 4D weight tensor $\mathsf{W}_{i,j,k,l}$ that connects input position $(k,l)$ to output position $(i,j)$:

$$ h_{i,j} = u_{i,j} + \sum_{k,l} W_{i,j,k,l} x_{k,l}. $$

Reparameterizing with relative offsets $(a = i - k, b = j - l)$ yields:

$$ h_{i,j} = u_{i,j} + \sum_{a,b} V_{i,j,a,b} x_{i-a,j-b}. $$

Imposing translation invariance requires the mapping from offsets to weights to be independent of $(i,j)$:

$$ h_{i,j} = u + \sum_{a,b} v_{a,b} x_{i+a,j+b}, $$

which is the foundational form of a convolutional operation.

Further enforcing locality restricts $a, b$ to a bounded window (e.g., $|a|, |b| \leq \Delta$), yielding:

$$ h_{i,j} = u + \sum_{a=-\Delta}^{\Delta} \sum_{b=-\Delta}^{\Delta} v_{a,b} x_{i+a,j+b}. $$

This local filter—referred to as a convolution kernel or filter—ensures each neuron in the layer responds to only a small neighborhood of the input, drastically reducing parameter count while limiting representational scope.

6.1.2 Channels and Multi-channel Convolution

Real-world images contain multiple channels (e.g., RGB). Thus an input $X \in \mathbb{R}^{c_i \times n_h \times n_w}$, where $c_i$ is the number of input channels. A convolutional layer with $c_o$ output channels uses a 4D kernel tensor $V \in \mathbb{R}^{c_o \times c_i \times k_h \times k_w}$, where each output channel $d$ is computed as:

$$ h_{i,j,d} = \sum_{a=-\Delta}^{\Delta} \sum_{b=-\Delta}^{\Delta} \sum_{c=1}^{c_i} v_{a,b,c,d} , x_{i+a,j+b,c}. $$

Each channel in the output (also called a feature map) captures distinct spatial patterns (e.g., edges, textures) learned across input channels.

7. Convolutoin as Computation

7.1 Cross-Correlation vs. Convolution

Mathematically, continuous convolution is defined as:

$$ (f * g)(x) = \int f(z) g(x - z),dz, $$

while discrete 2D convolution is:

$$ (f * g)(i, j) = \sum_a \sum_b f(a, b) , g(i - a, j - b). $$

However, deep learning frameworks implement cross-correlation, not true convolution:

$$ (f \otimes g)(i, j) = \sum_a \sum_b f(i + a, j + b) , g(a, b). $$

The distinction lies in the sign of the offset applied to the kernel—during training, this difference is irrelevant, so implementations swaps kernel indexing.

7.2 Implementing Cross-Correlation

Below is a minimal implementation of 2D cross-correlation:

def corr2d(X, K):
    h, w = K.shape
    Y = torch.zeros(X.shape[0] - h + 1, X.shape[1] - w + 1)
    for i in range(Y.shape[0]):
        for j in range(Y.shape[1]):
            Y[i, j] = (X[i:i+h, j:j+w] * K).sum()
    return Y

This computes a sliding-window dot product between the kernel and input.

7.3 Convolutional Layers

A learnable convolutional layer includes a kernel tensor and optional scalar offset (bias):

class Conv2d(nn.Module):
    def __init__(self, kernel_size):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(*kernel_size))
        self.bias = nn.Parameter(torch.zeros(1))
    
    def forward(self, x):
        return corr2d(x, self.weight) + self.bias

Modern frameworks like PyTorch provide nn.Conv2d, where kernel size, padding, stride, and channels are configurable.

7.4 Edge Detection as a Convolutive Task

A simple non-learned kernel can detect vertical edges via cross-correlation. Consider a binary image with a vertical stripe of black pixels (0) amidst white (1):

X = torch.ones(6, 8)
X[:, 2:6] = 0  # Insert black bar from column 2 to 5

Using kernel $K = [[1, -1]]$:

K = torch.tensor([[1.0, -1.0]])
Y = corr2d(X, K)

The response highlights transitions: positive values indicate white-to-black edges, negative for black-to-white.

7.5 Learning an Edge Detector

We can train a convolutional layer to recover the same kernel. With squared-error loss between predictions and target $Y$:

conv2d = nn.Conv2d(1, 1, kernel_size=(1, 2), bias=False)
X = X.reshape(1, 1, 6, 8)     # (batch, channel, H, W)
Y = Y.reshape(1, 1, 6, 7)
optimizer = torch.optim.SGD(conv2d.parameters(), lr=0.03)

for epoch in range(10):
    optimizer.zero_grad()
    Y_hat = conv2d(X)
    l = (Y - Y_hat).pow(2).sum()
    l.backward()
    optimizer.step()

After convergence, learned kernel weights approach $[1.0, -1.0]$.

7.6 Receptive Field

The receptive field of an output element is the region in the input that influenced it. For an initial convolution with $2\times2$ kernel, each output point depends on $4$ input pixels. If followed by another $2\times2$ kernel, the resulting output depends on up to $9$ input pixels—the receptive field grows combinatorially with depth.

8. Padding and Stride

8.1 Padding

To preserve spatial resolution across layers, zero-padding is commonly applied. For input size $n_h \times n_w$, kernel size $k_h \times k_w$, and padding $(p_h, p_w)$, output size becomes:

$$ (n_h - k_h + p_h + 1) \times (n_w - k_w + p_w + 1). $$

Setting $p_h = k_h - 1$, $p_w = k_w - 1$ ensures output height/width match the input, a configuration often used in practice. PyTorch supports this via padding=(ph, pw).

8.2 Stride

Stride controls how far the kernel advances per step. With vertical/horizontal strides $s_h, s_w$, output shape becomes:

$$ \left\lfloor \frac{n_h - k_h + p_h}{s_h} \right\rfloor + 1 \quad \times \quad \left\lfloor \frac{n_w - k_w + p_w}{s_w} \right\rfloor + 1. $$

This enables efficient downsampling. In PyTorch, use stride=(sh, sw).

9. Multi-Channel Inputs and Outputs

9.1 Multi-Input Channels

For $c_i$ input channels, the kernel must have matching depth. Each channel is convolved independently, then results summed:

def corr2d_multi_in(X, K):
    return sum(d2l.corr2d(x, k) for x, k in zip(X, K))

9.2 Multi-Output Channels

Multiple output channels require multiple kernels (one per output channel), each applied to the full multi-channel input. The output becomes a 3D tensor: channels × height × width.

def corr2d_multi_in_out(X, K):
    return torch.stack([corr2d_multi_in(X, k) for k in K], 0)

9.3 $1 \times 1$ Convolution

A $1 \times 1$ convolution ignores spatial structure and performs per-pixel mixing across channels—equivalent to a learned linear transformation at each spatial location. This is effectively a dense layer applied uniquely at every pixel:

def corr2d_multi_in_out_1x1(X, K):
    c_i, h, w = X.shape
    c_o = K.shape[0]
    X = X.reshape(c_i, h * w)
    K = K.reshape(c_o, c_i)
    Y = K @ X
    return Y.reshape(c_o, h, w)

Here, the weight matrix has shape $c_o \times c_i$, analogous to a fully connected layer on the channel dimension.

10. Pooling Layers

Pooling layers aggregate local regions to reduce sensitivity to translation and decrease spatial dimensions—promoting translation invariance and computational efficiency.

• Max pooling: output = $\max$ of window values.
• Average pooling: output = mean of window values.

No learnable parameters are involved—only aggregation strategy matters.

Implementation sketch:

def pool2d(X, pool_size, mode='max'):
    p_h, p_w = pool_size
    Y = torch.zeros(X.shape[0] - p_h + 1, X.shape[1] - p_w + 1)
    for i in range(Y.shape[0]):
        for j in range(Y.shape[1]):
            region = X[i:i+p_h, j:j+p_w]
            Y[i, j] = region.max() if mode == 'max' else region.mean()
    return Y

Like convolutions, pooling supports padding and stride. For multi-channel inputs, pooling is applied independently to each channel, preserving the channel count.

11. LeNet-5

LeNet-5, proposed by Yann LeCun (1998), was among the first CNNs applied to handwritten digitt recognition.

Architecture:

1. Convolutional block 1:

• 6 output channels, $5\times5$ kernel, padding=2
• Sigmoid activation
• $2\times2$ average pooling (stride=2)

2. Convolutional block 2:

• 16 output channels, $5\times5$ kernel
• Sigmoid activation
• $2\times2$ average pooling (stride=2)

3. Flatten + dense layers:

• $120$ → $84$ → $10$ units, with Sigmoid nonlinearity
• Final output represents probabilities over digits 0–9

PyTorch realization:

net = nn.Sequential(
    nn.Conv2d(1, 6, kernel_size=5, padding=2), nn.Sigmoid(),
    nn.AvgPool2d(kernel_size=2, stride=2),
    nn.Conv2d(6, 16, kernel_size=5), nn.Sigmoid(),
    nn.AvgPool2d(kernel_size=2, stride=2),
    nn.Flatten(),
    nn.Linear(16 * 5 * 5, 120), nn.Sigmoid(),
    nn.Linear(120, 84), nn.Sigmoid(),
    nn.Linear(84, 10))

Training uses cross-entropy loss, Xavier initialization, and stochastic gradient descent. On Fashion-MNIST (256-batch, 10 epochs), this model achieves ~82% accuracy at over 37k images/sec on a GPU.