Modern NVIDIA GPUs are engineered with a hierarchical hardware structure designed to maximize parallel processing throughput. At the highest level, an architecture like the Ampere A100 consists of multiple Graphics Processing Clusters (GPCs). These clusters interface with high-bandwidth memory (HBM2) via memory controllers and share a large L2 cache to facilitate high-speed data synchronization.

Within each GPC, the hardware is subdivided into Texture Processing Clusters (TPCs), which contain one or more Streaming Multiprocessors (SMs). The SM is the fundamental unit of execution in the GPU. It houses various specialized processing units:

• CUDA Cores: Designed for standard mathematical operations. In modern architectures, these are split into dedicated FP32 (floating-point) and INT32 (integer) paths, allowing for concurrent execution of different instruction types.
• Tensor Cores: Specialized hardware for high-speed matrix multiplications and deep learning operations, supporting formats like FP16, BF16, and INT8.
• Special Function Units (SFUs): Handle complex transcendental operations such as sines, cosines, and square roots.
• Warp Schedulers: Manage groups of 32 threads, known as Warps. The Warp is the smallest shceduling unit in the hardware, where all threads execute using the Single Instruction Multiple Thread (SIMT) model.

The CUDA Software Abstraction

CUDA (Compute Unified Device Architecture) provides the bridge between standard C++ code and the GPU hardware. It introduces a "Host-Device" paradigm where the CPU (Host) manages memory and control flow, while the GPU (Device) handles data-parallel computational kernels.

Software execution is organized into a three-level hierarchy:

1. Thread: The basic logical unit of execution.
2. Block: A collection of threads that can collaborate via shared memory and synchronization primitives. Each block is assigned to a single SM.
3. Grid: A collection of blocks launched by a single kernel execution. All blocks in a grid share the same global memory space.

Implementation Examples

To illustrate the difference between serial CPU execution and parallel GPU execution, consider a simple vector addition task.

Standard CPU Implementation

#include <iostream>
#include <vector>

void computeVectorSum(size_t length, const float* vecA, const float* vecB, float* result) {
    for (size_t i = 0; i < length; ++i) {
        result[i] = vecA[i] + vecB[i];
    }
}

int main() {
    const size_t ARRAY_SIZE = 1 << 20;
    std::vector<float> a(ARRAY_SIZE, 1.5f);
    std::vector<float> b(ARRAY_SIZE, 2.5f);
    std::vector<float> res(ARRAY_SIZE);

    computeVectorSum(ARRAY_SIZE, a.data(), b.data(), res.data());

    return 0;
}

CUDA Unified Memory Implementation

In the GPU version, the __global__ specifier indicates a kernel. Using cudaMallocManaged allows memory to be accessible by both the CPU and GPU automatically.

#include <iostream>

__global__ 
void vectorSumKernel(size_t n, float* a, float* b) {
    // Map global thread ID to array index
    size_t index = blockIdx.x * blockDim.x + threadIdx.x;
    if (index < n) {
        a[index] = a[index] + b[index];
    }
}

int main() {
    const size_t ELEMENT_COUNT = 1 << 20;
    float *dataA, *dataB;

    cudaMallocManaged(&dataA, ELEMENT_COUNT * sizeof(float));
    cudaMallocManaged(&dataB, ELEMENT_COUNT * sizeof(float));

    for (size_t i = 0; i < ELEMENT_COUNT; ++i) {
        dataA[i] = 1.5f;
        dataB[i] = 2.5f;
    }

    // Define execution configuration: 256 threads per block
    int threadsPerBlock = 256;
    int blocksPerGrid = (ELEMENT_COUNT + threadsPerBlock - 1) / threadsPerBlock;

    vectorSumKernel<<<blocksPerGrid, threadsPerBlock>>>(ELEMENT_COUNT, dataA, dataB);

    // Synchronize to ensure GPU work is finished
    cudaDeviceSynchronize();

    cudaFree(dataA);
    cudaFree(dataB);
    return 0;
}

Calculating Peak Theoretical Performance

Understanding the relationship between hardware units and clock speeds is essential for calculating the peak throughput of a GPU, typically measured in FLOPS (Floating Point Operations Per Second). The theoretical peak can be determined using the following formula:

$$\text{Theoretical Peak} = \text{Clock Frequency} \times \text{SM Count} \times \text{Operations per Clock per SM}$$

For a NVIDIA A100 GPU (SXM4 version) using Tensor Cores for FP32 processing:

• Clock Speed: ~1.41 GHz
• SM Count: 108
• Operations per Clock per SM: 1024 (for FP32 Tensor operations)

The calculation for FP32 Tensor Core peak performance is:

$$1.41 \times 10^9 \times 108 \times 1024 \approx 156 \text{ TFLOPS}$$

When utilizing hardware-level features like structural sparsity, these throughput figures can effectively double. These metrics are critical benchmarks for evaluating hardware efficiency when training large-scale artificial intelligence models.