Introduction
Modern computing systems encompass various interconnection structures within CPUs, SoCs, boards, and between boards. These are predominant bus-based architectures. To understand how "consistency" and "coherence" relate to internal CPU interconnects, categorizing these structures becomes essential.
Components in such systems are interconnected through various buses. While they operate harmoniously, conflicts arise when inconsistencies occur—indicating bugs.
Overview of Interconnect Types
When discussing interconnects, we distinguish among several levels:
On-Chip Interconnects:
- Dedicated Bus
- Crossbar
- Network-on-Chip (NoC)
Within-SOC Interconnects:
- Bus-based (suitable for up to ten masters)
- Non-x86 SoCs: AMBA/AXI/AHB/APB
- x86 CPUs: Internal processor-specific buses (not ISA/PCI/PCIe)
- Example: Intel Pentium D
- Crossbar (several to dozens of nodes)
- Example: NVIDIA Tesla GPU
- Point-to-Point (several to dozens of nodes)
- x86 CPUs: QPI (Intel) / HT (AMD)
- Example: Intel Xeon Processor
- Network-on-Chip (tens to hundreds of nodes)
- Example: Apple A15 Bionic SoC
- Example: ARM CMN (Coherent Mesh Network)
- CMN supports customizable mesh topologies using AMBA5 CHI
- CHI is a bus standard; CMN connects up to 128 cores
- CMN-600 evolved from CCN-500 series
On-Board Interconnects:
- ISA (x86 only)
- PCI (x86 only)
- PCIe (x86 & ARM servers)
- Other dedicated buses: I2C, SPI, UART
Inter-Board Interconnects:
- Short-range: PCIe
- Long-range: Network
Consistency and coherence primarily concern on-chip interconnects.
Interconnect Levels
| Level | x86 | ARM/RISC-V | Notes |
|---|---|---|---|
| L1 | Internal CPU | Core-level | Connects core units, private caches, local memory |
| L2 | System interconnect (QPI) | On-chip (AMBA) | Connects cores, DMA, shared cache, bridges, AMBA controllers; trending towards NoC |
| L3-1 | ISA/PCI/PCIe/CXL | PCIe/CXL/USB/I2C/SPI/UART | Bus topology connecting functional devices or expansion modules |
| L3-2 | Memory bus | Memory bus | Connected via memory controller on L2 bus to main memory |
| L4-1 | USB/I2C/SPI/UART | USB/I2C/SPI/UART | Controlled by PCIe/CXL devices |
| L4-2 | Memory bus | Memory bus | Controlled by CXL memory controller |
Network-on-Chip (NoC) Resources
As of 2023, NoC has seen widespread adoption in AI accelerators, mostly utilizing mesh or folded torus topologies with XY routing. Advanced routing schemes remain less common.
ARM's NoC Implementation
Multi-cluster Systems:
- Based on AMBA4 ACE protocol: CCI Controller
- Used in mobile SoCs
Multi-cluster Systems (Server-grade):
- Based on AMBA5 CHI protocol: CCN Controller
- Replaced by CMN
Latest High-performance Architecture:
- Based on AMBA5 CHI protocol: CMN Controller
- Designed for large-scale multi-core systems and data centers
- Examples: CMN-600 (2017), CMN-700 (2020)
Historical Evolution:
- CCI: CCI-400 → CCI-500 → CCI-550
- CCN: CCN-502 → CCN-504 → CCN-508 → CCN-512
- CMN: CMN-600 → CMN-700
Note: As of 2024, modern mobile SoCs typically use CCI rather than CCN or CMN due to power efficiency considerations.
PCIe Interconnect
In existing SoC/CPU designs, PCIe allows extending new controllers:
x86 CPUs:
- Do not integrate IO controllers (e.g., I2C); rely on PCIe for extension
- PCIe inherits PCI probing capabilities, enabling generic Linux images across x86 CPUs
ARM Embedded SoCs:
- Integrate IO controllers (e.g., I2C)
- Lack dynamic probing, leading to unique device trees per SoC
- Linux images require specific Device Tree Sources (DTS) per board
ARM Server SoCs:
- Adhere to standardized protocols for broader compatibility
- Ubuntu Server for ARM supports several ARM server SoCs
- Full support often requires collaboration between manufacturers and Ubuntu community
Kernel interconnects define address domains:
- System bus addresses belong to one domain
- PCIe packet addresses belong to another
These domains are isolated but can communicate through PCIe controllers.
Architecture Comparison
Intel x86:
- QPI: Internal interconnect (evolving towards NoC)
- PCIe: External interconnect (standard)
ARM Server:
- AMBA: Internal interconnect (evolving towards NoC)
- PCIe: External interconnect (standard)
ARM Embedded:
- AMBA: Internal interconnect (basic bus structures suffice)
- External interconnects vary (PCIe/I2C/SPI/SATA/UART); PCIe not dominant