Overview

Cambricon Technologies, founded in March 2016, specializes in AI chip design. The company name combines "Cambrian" (referencing the evolutionary explosion) with "Silicon." Their product portfolio encompasses cloud, edge, and endpoint solutions with unified software ecosystem, supporting both training and inference workloads.

Product Portfolio

Endpoint IP Cores

Cambricon entered the market through terminal intelligent processor IPs, including the 1A, 1H, and 1M series. These IP cores deliver 0.5 to 1 TOPS of computational power and serve emebdded AI applications.

Edge and Cloud Accelerators

Based on these foundational IPs, Cambricon developed multiple generations of accelerators:

MLU100 Series: Early-generation board products, largely superseded.

MLU200 Series: Comprises three distinct form factors:

• MLU220: Edge inference
• MLU270: Cloud inference
• MLU290: Cloud training

MLU300 Series: MLU370 serves both inference and training scenarios, representing the current flagship platform.

Endpoint IP	Edge	Cloud Inference	Cloud Training
1A, 1H, 1M	MLU220	MLU370, MLU270	MLU290, MLU-X1000, MLU370

MLU370-X8 Accelerator

The MLU370-X8 represents a full-featured data center training-and-inference accelerator built on the latest Siyuan 370 chip. Key specifications include:

• PCIe 4.0 x16 interface (full-height, full-length, dual-slot form factor)
• 250W power consumption
• Dual Siyuan 370 chips per board with MLU-Link interconnect
• 200 GB/s bidirectional MLU-Link inter-board bandwidth
• LPDDR5 memory

Chiplet Integration

Chiplet technology enables combining dies with different functions and process nodes through advanced packaging. This approach offers an alternative pathway for semiconductor advancement, potentially circumventing traditional scaling limitations.

Cloud-Edge-Endpoint Integration Strategy

Different deployment scenarios impose varying requirements for computational throughput and power efficiency. Cambricon's unified platform software (Cambricon Neuware) enables AI applications to run seamlessly across cloud, edge, and endpoint hardware, simplifying model migration between platforms.

MLU03 Architecture

Hardware Organization

MLU hardware employs domain-specific architecture optimized for AI workloads. The architecture differentiates between various memory access patterns and computational patterns through dedicated data paths and processing units.

MLU Core (IPU): The fundamental computational unit containing complete processing, I/O, and control capabilities. Each MLU Core can execute independent tasks or collaborate with other cores.

Cluster: Four MLU Cores form a Cluster, which additionally contains one Memory Core and shared SRAM. Memory Cores handle data transfers between SRAM and external memory rather than performing computation.

MLU03 Core Microarchitecture

The MLU03 architecture organizes computation around four IPUs and one MPU per Cluster. Each IPU incorporates:

Control Unit: Handles instruction fetch, decode, and dispatch to execution units.

Execution Queues:

• ALU-PIPE: Scalar arithmetic and logic operations
• XFU-PIPE: Vector and tensor operations
• DMA-PIPE: Data movement (IO-DMA and Move-DMA streams operating simultaneously)

Memory Structures:

• I-Cache: 64KB instruction cache (1024 × 512-bit entries)
• VA-Cache: Vector addressing cache for efficient scatter-gather operations
• Neural-RAM: 768KB (16-byte aligned), stores activations and feature maps
• Weight-RAM: 1024KB (8-byte aligned), stores convolution kernels and optimizer states

Processing Elements:

• ALU: Scalar computation unit
• VFU/TFU: Vector and tensor functional units
• GPR: 48-bit general-purpose registers for scalar operands
• SREG: 32-bit special registers for hardware attributes

MPU (Memory Processing Unit)

The MPU manages on-chip SRAM and inter-cluster communication:

• Cluster-DMA: Transfers data between shared SRAM banks across clusters
• Global-DMA: Manages transfers between GPR, external DRAM, and shared SRAM
• Shared-RAM: 4MB L1-equivalent cache per cluster

Note: MLUv02 and subsequent architectures introduced the MPU and Cluster concepts. Earlier products lack these structures.

Software Ecosystem

BANG C Programming Language

Cambricon BANG C extends C/C++ for heterogeneous MLU programming. Key characteristics include:

• Additional syntax for parallel task decomposition
• Compute primitives optimized for MLU architecture
• Unified programming across varying MLU Core and Cluster configurations

A BANG C program comprises host-side code (managing device selection, memory allocation, kernel launches via CNRT/CNDrv) and device-side kernel functions (exploiting MLU-specific primitives).

CNNL Library

The Cambricon CNNL provides optimized operators for common AI operations:

Convolution Operations: Forward convolution, convolution gradients for inputs and filters

Pooling: Forward and backward pooling operations

Activation Functions: ReLU, Sigmoid, Tanh with corresponding gradient operators

Normalization: BatchNorm and LayerNorm (forward and backward)

Matrix Operations: Matrix multiplication, element-wise tensor operations

Tensor Manipulations: Transpose, Split, Slice, Concat

Recurrent Layers: LSTM, GRU implementations

Loss Functions: Embedding, NLLLoss

MagicMind Inference Engine

MagicMind converts models from TensorFlow, PyTorch, Caffe, or ONNX into optimized inference graphs. The engine delivers:

• Optimal performance through model optimization
• Accuracy preservation
• Memory efficiency
• Flexible customization options

Supported inference scenarios include image classification/detection/segmentation, video processing, NLP, pose estimation, and recommendation systems.

CNCL Communication Library

CNCL provides collective communication primitives for multi-device/multi-node MLU deployments. Features include:

• Support for PCIe, MLU-Link, RoCE, InfiniBand, and socket transports
• Automatic topology-aware communication path selection
• Collective operations across communication domains

Communication Entities abstract memory spaces, MLU devices, and execution queues. Communicators associate a device with a specific queue, while Communication Cliques define groups of communicating entities.

AI Framework Integration

PyTorch Integration: Cambricon PyTorch extends the native framework through a Python package that registers MLU operators. The integration provides:

• Automatic operator fallback to CPU for unsupported operations
• Consistent API semantics with GPU/CPU backends

TensorFlow Integration: Cambricon TensorFlow enables native TensorFlow API usage while utilizing MLU acceleration, providing development experience comparable to CPU/GPU.

Additional Components

DeepSpeed: Distributed training framework supporting large-scale model parallelism with MLU devices.

CNCodec: Hardware-accelerated video and image codec library optimized for MLU.

CNStream: Modular streaming pipeline framework for video processing and inference integration.

CNServing: High-performance model serving system for production deployments.

Memory Hierarchy

The MLU architecture provides multiple storage tiers:

Tier	Scope	Capacity	Purpose
GPR	Per-core private	48-bit × n	Scalar operands, addresses
NRAM	Per-core private	768KB	Activations, intermediate results
WRAM	Per-core private	1024KB	Convolution kernels
SRAM	Per-cluster shared	4MB	Intermediate results, inter-core data sharing
L2 Cache	Global shared	-	Instructions, kernel parameters, read-only data
LDRAM	Per-core private	Configurable	Off-chip private storage
GDRAM	Global shared	-	Host-device data transfer

GPR Constraints: All scalar data (integer or floating-point) must load into GPR before computation. Data narrower than 48 bits zero-extends automatically.

NRAM Usage: Ideal for frequently-accessed data due to low latency and high bandwidth. Allocate strategically given the limited capacity.

WRAM Layout: Stores convolution kernels with specialized data layouts optimized for convolution operations.

SRAM Sharing: All cores within a cluster access shared SRAM for data exchange. The prefix "S" denotes "Shared" rather than "Static."

Programming Example

Matrix multiplication demonstrates efficient MLU kernel development:

A naive approach using only scalar operations fails to exploit vector/tensor units:

#define HEIGHT 256
#define WIDTH 256
#define DEPTH 256

__nram__ float output_buf[HEIGHT * WIDTH];
__nram__ int16_t input_buf_a[HEIGHT * DEPTH];
__nram__ int8_t input_buf_b[DEPTH * WIDTH];

for (int r = 0; r < HEIGHT; r++) {
  for (int c = 0; c < WIDTH; c++) {
    float accumulator = 0.0f;
    for (int k = 0; k < DEPTH; k++) {
      accumulator += input_buf_a[r * DEPTH + k] * input_buf_b[c * DEPTH + k];
    }
    output_buf[r * WIDTH + c] = accumulator;
  }
}

The optimized version utilizes the __bang_matmul primitive with appropriate data layout (row-major for left operand, column-major for right operand):

#define HEIGHT 256
#define WIDTH 256
#define DEPTH 256
#define PADDING_OFFSET 0
#define INPUT_A_ELEMENTS (HEIGHT * DEPTH)
#define INPUT_B_ELEMENTS (DEPTH * WIDTH)
#define OUTPUT_ELEMENTS (HEIGHT * WIDTH)

__mlu_global__ void MatrixMultiplyKernel(
    float* output,
    int16_t* matrix_a,
    int8_t* matrix_b
) {
  __nram__ float result_buffer[OUTPUT_ELEMENTS];
  __nram__ int16_t operand_a_buffer[INPUT_A_ELEMENTS];
  __wram__ int8_t operand_b_buffer[INPUT_B_ELEMENTS];
  
  __memcpy(operand_a_buffer, matrix_a, 
           INPUT_A_ELEMENTS * sizeof(int16_t), GDRAM2NRAM);
  __memcpy(operand_b_buffer, matrix_b, 
           INPUT_B_ELEMENTS * sizeof(int8_t), GDRAM2WRAM);
  
  __bang_matmul(result_buffer, operand_a_buffer, operand_b_buffer,
                HEIGHT, DEPTH, WIDTH, PADDING_OFFSET);
  
  __memcpy(output, result_buffer,
           OUTPUT_ELEMENTS * sizeof(float), NRAM2GDRAM);
}

This optimized implementation leverages hardware matrix multiplication units, dramatically improving throughput compared to scalar-based approaches.