Dimensionality Reduction Through Averaging

High-dimentional data presents computational challenges in machine learning. Consider an image of size 32×32 with RGB channels—it contains 32×32×3 = 3072 values. Processing such data across millions of samples demands efficient operations. Averaging provides a straightforward mechanism to reduce dimensionality while preserving essential statistical properties.

Mathematically, averaging transforms a collection of values into a single representative number. For vectors and matrices, averaging can occur along different dimensions: row-wise, column-wise, or across the entire structure. TensorFlow's tf.reduce_mean function implements this functionality for n-dimensional tensors.

Function Signature

tf.reduce_mean(input_tensor, axis=None, keep_dims=False, name=None, reduction_indices=None)

Parameters:

Returns: A tensor containing the computed mean value(s).

Practical Examples

import tensorflow as tf

# Initialize a 3x3 data matrix
data = tf.Variable([[1.0, 2.0, 3.0], 
                   [4.0, 5.0, 6.0], 
                   [7.0, 8.0, 9.0]])

init = tf.global_variables_initializer()

with tf.Session() as session:
    session.run(init)
    
    # Global mean across all elements
    global_avg = tf.reduce_mean(data)
    
    # Column-wise mean (along axis 0)
    col_avg_keep = tf.reduce_mean(data, axis=0, keep_dims=True)
    col_avg_drop = tf.reduce_mean(data, axis=0, keep_dims=False)
    
    # Row-wise mean (along axis 1)
    row_avg = tf.reduce_mean(data, axis=1)
    
    print("Input matrix:")
    print(data.eval())
    print("\nGlobal mean:", global_avg.eval())
    print("Column means (keep dimensions):", col_avg_keep.eval())
    print("Column means (drop dimensions):", col_avg_drop.eval())
    print("Row means:", row_avg.eval())

Output:

Input matrix:
[[1. 2. 3.]
 [4. 5. 6.]
 [7. 8. 9.]]

Global mean: 5.0
Column means (keep dimensions): [[4. 5. 6.]]
Column means (drop dimensions): [4. 5. 6.]
Row means: [2. 3. 4.]

Understanding the Axis Parameter

The axis parameter controls the direction of reduction:

• axis=None: Computes the mean across all elements, producing a scalar.
• axis=0: Reduces along the first dimension (columns in a 2D matrix), producing one mean per column.
• axis=1: Reduces along the second dimension (rows in a 2D matrix), producing one mean per row.

For tensors with higher dimensions, additional axis values (2, 3, etc.) become available.

The keep_dims Effect

When keep_dims=True, the output tensor maintains the same number of dimensions as the input, with reduced dimensions set to size 1. This preserves broadcasting compatibility with the original tensor shape:

# Without keep_dims: shape changes from (3,3) to (3,)
col_avg_drop = tf.reduce_mean(data, axis=0, keep_dims=False)  # shape: (3,)

# With keep_dims: shape changes from (3,3) to (1,3)
col_avg_keep = tf.reduce_mean(data, axis=0, keep_dims=True)   # shape: (1, 3)

This distinction becomes critical when performing element-wise operations between the averaged result and the original tensor.