After completing "Plain Deep Learning and TensorFlow," I gained a basic understanding of fundamental neural network architectures like BP networks, CNNs, and RNNs. The underlying algorithms aren't particularly complex; for instance, BP networks can be understood with basic calculus and probability theory. CNNs incorporate well-established theories from digital image processing such as convolution and pooling. RNNs require more advanced mathematical tools that need further study. I'm currently studying whitepapers to thoroughly understand these theories mathematically.

However, theoretical knowledge alone isn't sufficient. Practice is essential to reinforce understanding of BP networks. Below are three beginner-level projects that can help in comprehending neural networks.

Environment: Windows 10 WSL Ubuntu 18.04

1. Linear Regression

Beginner: TensorFlow Linear Regression

Code:

from __future__ import print_function
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

# Parameters
learning_rate = 0.01
training_epochs = 1000
display_step = 50
save_step = 500

# Training Data
train_X = np.array([3.3, 4.4, 5.5, 6.71, 6.93, 4.168, 9.779, 6.182, 7.59, 2.167,
                    7.042, 10.791, 5.313, 7.997, 5.654, 9.27, 3.1])
train_Y = np.array([3, 4.7, 5, 7.21, 6.93, 4.168, 9.779, 6.182, 7.59, 2.167,
                    7.042, 10.291, 5.813, 7.997, 5.654, 9.17, 3.2])

# Define graph input
X = tf.placeholder(tf.float32, name="X")
Y = tf.placeholder(tf.float32, name="Y")

with tf.variable_scope("linear_model"):
    # Set model parameters
    weight = tf.get_variable("weight", initializer=np.random.randn())
    bias = tf.get_variable("bias", initializer=np.random.randn())
    
    # Construct a linear model
    predictions = tf.add(tf.multiply(X, weight), name="predictions")
    
# Mean squared error loss
with tf.variable_scope("loss"):
    error = tf.reduce_mean(tf.square(predictions - Y))
    
# Optimizer
optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(error)

# Initialize variables
init = tf.global_variables_initializer()

# Checkpoint path
checkpoint_path = './checkpoints/linear_model.ckpt'

# Create saver
saver = tf.train.Saver()

# Summary path
summary_path = './summaries/'

# Create summaries for TensorBoard
tf.summary.scalar('weight', weight)
tf.summary.scalar('bias', bias)
tf.summary.scalar('loss', error)
merged_summaries = tf.summary.merge_all()

# Start training
with tf.Session() as sess:
    summary_writer = tf.summary.FileWriter(summary_path, sess.graph)
    sess.run(init)
    
    # Training loop
    for epoch in range(training_epochs):
        for (x, y) in zip(train_X, train_Y):
            sess.run(training_op, feed_dict={X: x, Y: y})
            
        if (epoch + 1) % save_step == 0:
            save_path = saver.save(sess, checkpoint_path, global_step=epoch)
            print(f"Model saved in file: {save_path}")
            
        if (epoch + 1) % display_step == 0:
            current_loss = sess.run(error, feed_dict={X: train_X, Y: train_Y})
            current_weight = sess.run(weight)
            current_bias = sess.run(bias)
            print(f"Epoch: {epoch+1:04d}, Loss: {current_loss:.9f}, "
                  f"Weight: {current_weight:.4f}, Bias: {current_bias:.4f}")
            
            # Add summary
            summary = sess.run(merged_summaries, feed_dict={X: train_X, Y: train_Y})
            summary_writer.add_summary(summary, global_step=epoch)
    
    print("Optimization finished")
    
    # Calculate final metrics
    final_loss = sess.run(error, feed_dict={X: train_X, Y: train_Y})
    final_weight = sess.run(weight)
    final_bias = sess.run(bias)
    print(f"Final Loss: {final_loss:.4f}, Weight: {final_weight:.4f}, Bias: {final_bias:.4f}")
    
    # Plot results
    plt.figure(figsize=(10, 6))
    plt.scatter(train_X, train_Y, label='Original Data')
    plt.plot(train_X, final_weight * train_X + final_bias, 'r-', label='Fitted Line')
    plt.legend()
    plt.title('Linear Regression Results')
    plt.show()
    
    summary_writer.close()

To use TensorBoard:

tensorboard --logdir=./summaries/

Then open a browser and navigate to:

localhost:6006

During training, I encountered issues where the loss increased instead of decreasing, indicating a problematic learning rate selection.

Besides the learning rate, training effectiveness also depends on the choice of optimization algorithm. Initially, I used gradient descent:

optimizer = tf.train.GradientDescentOptimizer(learning_rate)

However, the results were poor. After switching to the Adam optimizer:

optimizer = tf.train.AdamOptimizer(learning_rate)

The performance improved significantly, with the loss being more than half of what was achieved with gradient descent.

For information about different gradient algorithms, refer to: https://ruder.io/optimizing-gradient-descent/index.htmlhttps://zhuanlan.zhihu.com/p/22252270

Disadvantages of SGD (Stochastic Gradient Descent):

• Uses a uniform learning rate throughout, making it difficult to select an appropriate rate
• Prone to getting stuck in saddle points (can be mitigated with proper initialization and step size)

The Adam algorithm introduces first and second moment estimates of gradients, incorporating properties similar to momentum and friction, making training adaptive and often more effective than plain gradient descent.

Using TensorBoard to visualize the loss, bias, and weight changes is quite convenient. The graph visualization feature is also very useful.

Loading a model saved with checkpoints:

1. Define an identical graph, then import it. The process of creating the graph is the same as in the linear regression code. After creating the graph, the key step is to create a tf.train.Saver() and call its restore method with the session and checkpoint file path as parameters.
2. Directly import the meta file, which allows importing both the graph and parameters.

Code example 1:

from __future__ import print_function
import tensorflow as tf
import numpy as np

# Define the graph
X = tf.placeholder(tf.float32, name="X")
Y = tf.placeholder(tf.float32, name="Y")

with tf.variable_scope("linear_model"):
    weight = tf.Variable(np.random.randn(), name="weight")
    bias = tf.Variable(np.random.randn(), name="bias")
    predictions = tf.add(tf.multiply(X, weight), bias, name="predictions")

saver = tf.train.Saver()
checkpoint_path = './checkpoints/linear_model.ckpt-999'

with tf.Session() as sess:
    saver.restore(sess, checkpoint_path)
    print(f'Restored values: Weight={sess.run(weight):.4f}, Bias={sess.run(bias):.4f}')

Code example 2: Using import_meta_graph to import the graph. When creating the session, set the config parameter.

from __future__ import print_function
import tensorflow as tf
import numpy as np

config = tf.ConfigProto(allow_soft_placement=True)
checkpoint_path = './checkpoints/linear_model.ckpt-999'

with tf.Session(config=config) as sess:
    saver = tf.train.import_meta_graph(checkpoint_path + '.meta')
    saver.restore(sess, checkpoint_path)
    graph = sess.graph
    
    X = graph.get_tensor_by_name("X:0")
    predictions = graph.get_tensor_by_name("linear_model/predictions:0")
    
    weight = graph.get_tensor_by_name("linear_model/weight:0")
    bias = graph.get_tensor_by_name("linear_model/bias:0")
    
    print(f'Restored values: Weight={sess.run(weight):.4f}, Bias={sess.run(bias):.4f}')

2. Nonlinear Regression

In linear regression, we used tf.multiply(). However, in practical applications, matrix multiplication is more common, so tf.matmul() is frequently used.

Difference between tf.matmul() and tf.multiply():

1. tf.multiply(): Multiplies corresponding elements of two matrices. Format: tf.multiply(x, y, name=None) Parameters:

   • x: A tensor of type half, float32, float64, uint8, int8, uint16, int16, int32, int64, complex64, complex128.
   • y: A tensor with the same type as x. Returns: x * y element-wise.

2. tf.matmul(): Multiplies matrix a by matrix b, producing a * b. Format: tf.matmul(a, b, transpose_a=False, transpose_b=False, adjoint_a=False, adjoint_b=False, a_is_sparse=False, b_is_sparse=False, name=None) Parameters:

   • a: A tensor with rank > 1.
   • b: A tensor with the same type as a.
   • transpose_a: If True, a is transposed before multiplication.
   • transpose_b: If True, b is transposed before multiplication.
   • adjoint_a: If True, a is conjugated and transposed before multiplication.
   • adjoint_b: If True, b is conjugated and transposed before multiplication.
   • a_is_sparse: If True, a is treated as a sparse matrix.
   • b_is_sparse: If True, b is treated as a sparse matrix. Returns: A tensor of the same type as a and b, with inner matrices being the product of corresponding matrices.

Code:

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

# Parameters
learning_rate = 0.1
training_epochs = 20000
display_step = 500

# Generate data
x_data = np.linspace(-1, 1, 200).reshape((200, 1))
noise = np.random.normal(0, 0.05, x_data.shape)
y_data = np.square(x_data) + noise

# Define input placeholders
x = tf.placeholder(tf.float32, [None, 1])
y = tf.placeholder(tf.float32, [None, 1])

# Hidden layer
hidden_weights = tf.Variable(tf.random_normal([1, 10]))
hidden_biases = tf.Variable(tf.zeros([1, 10]))
hidden_layer = tf.nn.tanh(tf.matmul(x, hidden_weights) + hidden_biases)

# Output layer
output_weights = tf.Variable(tf.random_normal([10, 1]))
output_biases = tf.Variable(tf.zeros([1, 1]))
predictions = tf.nn.tanh(tf.matmul(hidden_layer, output_weights) + output_biases)

# Loss function
loss = tf.reduce_mean(tf.square(y - predictions))

# Optimizer
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
training_op = optimizer.minimize(loss)

# Initialize variables
init = tf.global_variables_initializer()

# Training
with tf.Session() as sess:
    sess.run(init)
    
    for epoch in range(training_epochs):
        sess.run(training_op, feed_dict={x: x_data, y: y_data})
        
        if (epoch + 1) % display_step == 0:
            current_loss = sess.run(loss, feed_dict={x: x_data, y: y_data})
            print(f"Epoch: {epoch+1:05d}, Loss: {current_loss:.9f}")
    
    # Get final predictions
    final_predictions = sess.run(predictions, feed_dict={x: x_data})
    
    # Plot results
    plt.figure(figsize=(10, 6))
    plt.scatter(x_data, y_data, label='Original Data')
    plt.plot(x_data, final_predictions, 'r-', lw=3, label='Model Prediction')
    plt.title('Nonlinear Regression Results')
    plt.legend()
    plt.show()

The loss decreased significantly from 0.0076 to 0.0027. The Gradient Descent results closely follow the quadratic curve distribution.

When using Adam optimizer, although the loss was smaller than with Gradient Descent, there was noticeable overfitting.

The activation function used here was tanh. When replaced with sigmoid, the performance was worse than tanh because the derivative value of tanh is greater than sigmoid, leading to faster convergence. However, after doubling the training epochs to 40000, I found that the training ceiling for sigmoid was around 0.004, while using ReLU caused the model to diverge and fail to converge.

ReLU is more suitable as an activation function in deep networks, but performs poorly in shallow networks like this one (its original purpose was to address gradient explosion in deep networks).

3. Fully Connected Network for MNIST Classification

The implementation is divided into mnist_model.py for network construction and mnist_train.py for training, which is a common approach for most neural networks.

One-dimensional vectors can be represented as [1, dimension] or [None, dimension].

Since labels use one-hot encoding (1-10), the cross-entropy loss function is used, defined as (p is the true distribution, q is the predicted distribution):

loss = -tf.reduce_sum(true_labels * tf.log(predictions + 1e-10))

Adding 1e-10 prevents numerical overflow when predictions equal 0.

About feed_dict: The feed_dict in sess.run() is essentially for temporary assignment.

Model code:

import tensorflow as tf

class NeuralNetwork:
    def __init__(self):
        # Learning rate
        self.learning_rate = 0.01
        
        # Input tensor
        self.inputs = tf.placeholder(tf.float32, [None, 784])
        
        # Label tensor
        self.labels = tf.placeholder(tf.float32, [None, 10])
        
        # Weights and biases
        self.weights = tf.Variable(tf.random_normal([784, 10]))
        self.biases = tf.Variable(tf.random_normal([10]))
        
        # Output with softmax activation
        self.logits = tf.matmul(self.inputs, self.weights) + self.biases
        self.predictions = tf.nn.softmax(self.logits)
        
        # Loss function (cross-entropy)
        self.loss = tf.reduce_mean(
            tf.nn.softmax_cross_entropy_with_logits_v2(logits=self.logits, labels=self.labels)
        )
        
        # Training operation
        self.optimizer = tf.train.GradientDescentOptimizer(self.learning_rate)
        self.training = self.optimizer.minimize(self.loss)
        
        # Accuracy calculation
        correct_predictions = tf.equal(
            tf.argmax(self.labels, axis=1), 
            tf.argmax(self.predictions, axis=1)
        )
        self.accuracy = tf.reduce_mean(tf.cast(correct_predictions, tf.float32))

Training code:

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
from mnist_model import NeuralNetwork

class Trainer:
    def __init__(self):
        self.network = NeuralNetwork()
        self.session = tf.Session()
        self.session.run(tf.global_variables_initializer())
        
        # Load MNIST data
        self.mnist = input_data.read_data_sets('/path/to/mnist/data', one_hot=True)
        
        # Model saver
        self.saver = tf.train.Saver()
        
    def train(self):
        batch_size = 64
        total_epochs = 10000
        save_interval = 1000
        display_interval = 100
        
        checkpoint_path = './checkpoints/mnist_model.ckpt'
        
        for step in range(total_epochs):
            # Get next batch
            batch_images, batch_labels = self.mnist.train.next_batch(batch_size)
            
            # Training step
            self.session.run(self.network.training, 
                           feed_dict={self.network.inputs: batch_images, 
                                    self.network.labels: batch_labels})
            
            if (step + 1) % display_interval == 0:
                # Calculate and display loss
                current_loss = self.session.run(
                    self.network.loss,
                    feed_dict={self.network.inputs: batch_images, 
                             self.network.labels: batch_labels}
                )
                print(f"Step: {step+1}, Loss: {current_loss:.4f}")
            
            if (step + 1) % save_interval == 0:
                # Save model checkpoint
                save_path = self.saver.save(
                    self.session, 
                    checkpoint_path, 
                    global_step=step
                )
                print(f"Model saved to: {save_path}")
    
    def evaluate(self):
        # Test the model on test data
        test_images = self.mnist.test.images
        test_labels = self.mnist.test.labels
        
        accuracy = self.session.run(
            self.network.accuracy,
            feed_dict={self.network.inputs: test_images, 
                     self.network.labels: test_labels}
        )
        print(f"Accuracy: {accuracy:.4f} on {len(test_labels)} test images")

if __name__ == "__main__":
    trainer = Trainer()
    trainer.train()
    trainer.evaluate()

The training process saves the model, which we can then load in a prediction script to classify new samples.

import numpy as np
import tensorflow as tf
from PIL import Image
from mnist_model import NeuralNetwork

class DigitRecognizer:
    def __init__(self):
        # Initialize the network
        self.network = NeuralNetwork()
        self.session = tf.Session()
        self.session.run(tf.global_variables_initializer())
        self.load_model()
        
    def load_model(self):
        checkpoint_path = './checkpoints/mnist_model.ckpt-9999'
        saver = tf.train.Saver()
        saver.restore(self.session, checkpoint_path)
        
    def predict(self, image_path):
        # Load and preprocess image
        img = Image.open(image_path).convert('L')
        img = img.resize((28, 28))  # MNIST images are 28x28
        img_array = np.array(img)
        img_array = 1 - (img_array / 255.0)  # Invert colors and normalize
        img_array = img_array.reshape(1, 784)  # Flatten to 1D array
        
        # Make prediction
        prediction = self.session.run(
            self.network.predictions,
            feed_dict={self.network.inputs: img_array}
        )
        
        # Get the predicted digit
        predicted_digit = np.argmax(prediction[0])
        
        print(f"Image: {image_path}")
        print(f"Predicted digit: {predicted_digit}")
        
        return predicted_digit

if __name__ == "__main__":
    recognizer = DigitRecognizer()
    
    # Test with sample images
    test_images = ['./test_images/0.png', './test_images/1.png', './test_images/4.png']
    for img_path in test_images:
        recognizer.predict(img_path)

When loading the model, we can also add error handling:

def load_model(self):
    saver = tf.train.Saver()
    checkpoint_dir = './checkpoints/'
    
    checkpoint = tf.train.get_checkpoint_state(checkpoint_dir)
    if checkpoint and checkpoint.model_checkpoint_path:
        saver.restore(self.session, checkpoint.model_checkpoint_path)
        print("Model loaded successfully")
    else:
        raise FileNotFoundError("No saved model found in the checkpoint directory")

We could also add TensorBoard summaries for visualization, though the training went smoothly in this case. Overall, the process was quite successful.