Machine learning forms the foundation of many revolutionary AI applications, from natural language processing to image recognition.

Machine learning relies on algorithms, statistical models, and neural networks. Deep learning is a subfield of machine learning that focuses on neural networks.

A key component of any neural network is the activation function. Understanding why they are essential for any neural network system is a common question, but also one that can be challenging to answer.

This article explains why activation functions are necessary, using simple explanations and analogies.

By understanding this, you will gain insight into how AI models process information.

Before that, we will explore neural networks in AI. We will also discuss the most commonly used activation functions.

We will also analyze a very simple PyTorch neural network code example line by line.

This article covers the following topics:

• The rise of artificial intelligence and deep learning
• Understanding activation functions: simplifying neural network mechanisms
• Simple analogy: the necessity of activation functions
• What happens without activation functions?
• PyTorch activation function code example
• Conclusion: the unsung heroes of AI neural networks

This article does not cover dropout or other regularization techniques, hyperparameter optimization, complex architectures like CNNs, or detailed differences in gradient descent variants.

I just want to show why activation functions are needed and what happens when they are not applied to a neural network.

The Rise of Artificial Intelligence and Deep Learning

What is deep learning in artificial intelligence?

Image showing the connection between society and artificial intelligence

Deep learning is a subfield of artificial intelligence. It uses neural networks to process complex patterns, much like a sports team uses strategies to win a game.

The larger the neural network, the more impressive things it can do—like ChatGPT, which uses natural language processing to answer questions and interact with users.

To truly understand the basics of neural networks—what every individual AI model has in common that allows it to work—we need to understand the activation layer.

Deep learning = training neural networks

Simple neural network

At the core of deep learning is training neural networks.

This basically means using data to obtain the correct weight values sothat predictions can be made.

Neural networks consist of neurons organized in layers. Each layer extracts unique features from the data.

This layered structure enables deep learning models to analyze and interpret complex data.

Understanding Activation Functions: Simplifying Neural Network Mechanisms

Leaky ReLU activation function

Activation functions help neural networks handle complex data. They change the value of neurons based on the data received.

Almost every neuron has a similar filter before sending its value to the next neuron.

Essentially, activation functions control the flow of information within the neural network—they determine which data is relevant and which is not.

This helps prevent vanishing gradients, ensuring the network learns correctly.

Vanishing gradient problems occur when the learning signal is too weak, making it difficult for the weights to change. This makes learning from data very challenging.

Simple Analogy: Why Are Activation Functions Necessary?

Football player thinking

In a football match, players decide whether to pass, dribble, or shoot.

These decisions are based on the current game situation, just like neurons in a neural network process data.

In this case, activation functions play a role in the decision-making process.

Without them, neurons would pass data without discrimination—like players kicking the ball without thinking, regardless of the game context.

This way, activation functions introduce complexity into neural networks, allowing them to learn complex patterns.

What Happens Without Activation Functions?

Player running

To understand what happens without activation functions, let's first imagine what would happen if players kicked the ball without thinking during a football match.

They might lose the game because there is no team decision-making process. The ball would still be kicked somewhere—but most of the time, it wouldn't go where it was intended.

This is similar to what happens with a neural network without activation functions: the neural network won't make good predictions because neurons simply pass data randomly to each other.

We would still get a prediction result. Just not the one we want, or useful one.

This severely limits the capabilities of both the football team and the neural network.

Intuitive Explanation of Activation Functions

Now let's look at an example to help you understand this intuitively.

ReLU activation function

Let's start with the most widely used activation function in deep learning (and also one of the simplest).

This is a ReLU activation function. It acts as a filter before the neuron sends its value to the next neuron.

This filter contains two conditions:

• If the weight value is negative, it becomes 0
• If the weight value is positive, it remains unchanged

By doing this, we add a decision-making process to each neuron. It decides what data to send and what to not send.

Now let's look at some other examples of activation functions.

Sigmoid Activation Function

This activation function converts the input value into a value between 0 and 1. Sigmoid is widely used in binary classification problems in the final neuron.

Sigmoid activation function

However, the Sigmoid activation function has issues. Consider the output values from a linear transformation:

• 0.00000003
• 0.99999992
• 0.00000247
• 0.99993320

We can ask some questions about these values:

• Are values like 0.00000003 and 0.000002 really important? Can they just be 0, which would reduce the computational load? Remember, many models today have millions of weights. Millions of 0.00000003 and 0.000002 cannot be 0, can they?
• How does it differentiate between large values and very large values? For example, in 0.99993320 and 0.99999992, where are the input values like 7 and 13 or 7 and 55? 0.99993320 and 0.99999992 don't accurately describe their input values.

How do we distinguish small differences in the output to maintain accuracy?

This is what the ReLU activation function solves: setting negative values to zero while keeping positive values, which enhances the computational efficiency of the neural network.

Tanh (Hyperbolic Tangent) Activation Function

tanh activation function

The output values of these activation functions range between -1 and 1, similar to Sigmoid.

They are often used in recurrent neural networks (RNNs) and long short-term memory networks (LSTMs).

Tanh is also used because it is centered around zero. This means the average of the output values is approximately zero. This characteristic helps address the vanishing gradient problem.

Leaky ReLU

Leaky ReLU activation function

The Leaky ReLU activation function doesn't ignore negative values, but instead allows a small negative value.

This way, negative values are used during the training of the neural network.

With the ReLU activation function, neurons with negative values are inactive and do not contribute to the learning process.

With the Leaky ReLU activation function, neurons with negative values are active and contribute to the learning process.

This decision-making process is implemented by the activation function. Without it, it would simply pass the data to the next neuron (like a player kicking the ball without thinking).

Mathematical Explanation of Activation Functions

Power of mathematical transformations

Neurons perform two tasks:

• They perform a linear transformation using the weight values from previous neurons
• They use the activation function to filter certain values to selectively pass them along.

Without an activation function, the neural network only performs one task: linear transformation.

If it only performs linear transformation, it is a linear system.

If it is a linear system, in simple terms, the superposition theorem tells us that any combination of two or more linear transformations can be simplified into a single transformation.

Essentially, this means that without an activation function, this complex neural network:

Neural network without activation function

Is equivalent to this simple neural network:

Short neural network without activation function

Because each layer is a product of a linear transformation of the previous layer in matrix form.

According to the theorem, since any combination of two or more linear transformations can be simplified into a single transformation, any combination of hidden layers (the layers between the input and output of the neurons) in the neural network can be reduced to a single layer.

What does this mean?

It means that it can only model data in a linear way. But in real life, systems using real data are nonlinear. So we need activation functions.

We introduce non-linearity into the neural network so that it can learn non-linear patterns.

PyTorch Activation Function Code Example

In this section, we will train the following neural network:

Simple feedforward neural network

This is a simple neural network AI model with four layers:

• An input layer with 10 neurons
• Two hidden layers, each with 18 neurons
• A hidden layer with 18 neurons
• An output layer with 1 neuron

In the code, we can choose any of the four activation functions mentioned in this tutorial.

Here is the complete code - we will go through it step by step:

import torch
import torch.nn as nn
import torch.optim as optim

# Choose the activation function to use in the code
defined_activation_function = 'relu'

activation_functions = {
    'relu': nn.ReLU(),
    'sigmoid': nn.Sigmoid(),
    'tanh': nn.Tanh(),
    'leaky_relu': nn.LeakyReLU()
}

# Initialize hyperparameters
num_samples = 100
batch_size = 10
num_epochs = 150
learning_rate = 0.001

# Define a simple synthetic dataset
def generate_data(num_samples):
    X = torch.randn(num_samples, 10)
    y = torch.randn(num_samples, 1)
    return X, y

# Generate synthetic data
X, y = generate_data(num_samples)

class SimpleModel(nn.Module):
    def __init__(self, activation=defined_activation_function):
        super(SimpleModel, self).__init__()
        self.fc1 = nn.Linear(in_features=10, out_features=18)
        self.fc2 = nn.Linear(in_features=18, out_features=18)
        self.fc3 = nn.Linear(in_features=18, out_features=4)
        self.fc4 = nn.Linear(in_features=4, out_features=1)
        self.activation = activation_functions[activation]

    def forward(self, x):
        x = self.fc1(x)
        x = self.activation(x)
        x = self.fc2(x) 
        x = self.activation(x)
        x = self.fc3(x) 
        x = self.activation(x)  
        x = self.fc4(x) 
        return x

# Initialize model, define loss function, and optimizer
model = SimpleModel(activation=defined_activation_function)
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

# Training loop
for epoch in range(num_epochs):
    for i in range(0, num_samples, batch_size):
        # Get mini-batch data
        inputs = X[i:i+batch_size]
        labels = y[i:i+batch_size]

        # Zero gradients
        optimizer.zero_grad()

        # Forward pass
        outputs = model(inputs)

        # Calculate loss
        loss = criterion(outputs, labels)

        # Backward pass and optimization
        loss.backward()
        optimizer.step()

    print(f'Epoch {epoch+1}/{num_epochs}, Loss: {loss}')

print("Training complete.")

It looks a bit much, right? Don't worry - we'll go through it step by step.

1: Import libraries and define activation functions

import torch
import torch.nn as nn
import torch.optim as optim

# Choose the activation function to use in the code
defined_activation_function = 'relu'

activation_functions = {
    'relu': nn.ReLU(),
    'sigmoid': nn.Sigmoid(),
    'tanh': nn.Tanh(),
    'leaky_relu': nn.LeakyReLU()
}

Import libraries and define a dictionary containing activation functions

In this code:

• import torch imports the PyTorch library.
• import torch.nn as nn imports the neural network module from PyTorch.
• import torch.optim as optim imports the optimization module from PyTorch.

The variables and dictionary above help you easily define the activation function to use for this deep learning model.

2: Define hyperparameters and generate dataset

# Initialize hyperparameters
num_samples = 100
batch_size = 10
num_epochs = 150
learning_rate = 0.001

# Define a simple synthetic dataset
def generate_data(num_samples):
    X = torch.randn(num_samples, 10)
    y = torch.randn(num_samples, 1)
    return X, y

# Generate synthetic data
X, y = generate_data(num_samples)

Initialize hyperparameters and create a synthetic dataset using a function

In this code:

• num_samples is the number of samples in the synthetic dataset.
• batch_size is the size of each mini-batch during training.
• num_epochs is the number of times the entire dataset is iterated over during training.
• learning_rate is the learning rate used by the optimization algorithm.

Additionally, we define a generate_data function to create two tensors with random values. Then we call this function to generate two tensors with random values for X and y.

3: Create the deep learning model

class SimpleModel(nn.Module):
    def __init__(self, activation=defined_activation_function):
        super(SimpleModel, self).__init__()
        self.fc1 = nn.Linear(in_features=10, out_features=18)
        self.fc2 = nn.Linear(in_features=18, out_features=18)
        self.fc3 = nn.Linear(in_features=18, out_features=4)
        self.fc4 = nn.Linear(in_features=4, out_features=1)
        self.activation = activation_functions[activation]

    def forward(self, x):
        x = self.fc1(x)
        x = self.activation(x)
        x = self.fc2(x) 
        x = self.activation(x)
        x = self.fc3(x) 
        x = self.activation(x)  
        x = self.fc4(x) 
        return x

A simple feedforward neural network deep learning model

The __init__ method in the SimpleModel clas initializes the neural network architecture. It initializes four fully connected layers and defines the activation function we will use.

We use nn.Linear to create each layer, and the forward method defines how the data flows through the neural network.

4: Initialize the model and define the loss function and optimizer

model = SimpleModel(activation=defined_activation_function)
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

Define the activation function, loss function, and the gradient descent variant to use

In this code:

1. model = SimpleModel(activation=defined_activation_function) creates a neural network model with the specified activation function.
2. criterion = nn.MSELoss() defines the mean squared error (MSE) loss function.
3. optimizer = optim.Adam(model.parameters(), lr=learning_rate) sets up the Adam optimizer to update the model parameters during training and specifies the learning rate.

5: Train the deep learning model

for epoch in range(num_epochs):
    for i in range(0, num_samples, batch_size):
        # Get mini-batch data
        inputs = X[i:i+batch_size]
        labels = y[i:i+batch_size]

        # Zero gradients
        optimizer.zero_grad()

        # Forward pass
        outputs = model(inputs)

        # Calculate loss
        loss = criterion(outputs, labels)

        # Backward pass and optimization
        loss.backward()
        optimizer.step()

    print(f'Epoch {epoch+1}/{num_epochs}, Loss: {loss}')

Train the model

• The outer loop controls how many times the entire dataset is processed based on num_epochs (number of iterations).
• The inner loop uses the range function to divide the dataset into mini-batches.

In each inner loop:

1. Using the inputs and labels, we get the mini-batch data to process.
2. We use optimizer.zero_grad() to clear the gradients from the previous mini-batch iteration — this is important to prevent mixing gradient information between mini-batches.
3. We get the model's predicted values (outputs) through forward propagation and calculate the loss using the specified loss function (criterion).
4. We compute the gradients of the weights using loss.backward().
5. Finally, optimizer.step() updates the model's weights based on these gradients to minimize the loss function.

This is the complete code for training a very simple deep learning model on a very simple dataset.

It does not include more advanced content, such as convolutional neural networks.

Conclusion: The Unsung Heroes of AI Neural Networks

Activation functions act like goalkeepers. By limiting the flow of information, the neural network can learn better.

Activation functions are like people learning or football players deciding how to handle the ball.

These functions give the neural network the ability to learn and make accurate predictions.

Mathematically, activation functions are crucial for the neural network to approximate any linear or nonlinear function correctly. Without them, the neural network can only approximate linear functions.