Skip to content

Argonne Leadership Computing Facility

Aurora System Overview

Aurora is a 10,624 node HPE Cray-Ex based system. It has 166 racks with 21,248 CPUs and 63,744 GPUs. Each node consists of 2 Intel Xeon CPU Max Series (codename Sapphire Rapids or SPR) with on-package HBM and 6 Intel Data Center GPU Max Series (codename Ponte Vecchio or PVC). Each Xeon has 52 physical cores supporting 2 hardware threads per core and 64GB of HBM. Each CPU has 512 GB of DDR5. The GPUs are connected all-to-all with Intel XeLink interfaces. Each node has 8 HPE Slingshot-11 NICs, and the system is connected in a dragonfly topology. The GPUs may send messages directly to the NIC via PCIe, without the need to copy into CPU memory.

Intel Data Center GPU Max Series is based on Xe Core. Each Xe core consist of 8 vector engines and 8 matrix engines with 512 kb of L1 cache that can be configured as cache or Shared Local Memroy (SLM). 16 Xe cores are grouped together to form a slice. 4 slicess are combined along with a large L2 cache, 4 HBM2E memory controllers to form s stack or tile. One or more stacks/tiles can then be combined on a socket to form a GPU. More detailed information about node architecture can be found here:

Aurora Compute Node

Processor 2000 MHz 2 21,248
Cores/Threads Intel Xeon CPU Max 9470C Series 104/208 1,104,896/2,209,792
CPU HBM HBM2e 64x2 GiB 1.328 PiB
CPU DRAM DDR5 512x2 GiB 10.375 PiB
GPUS Intel Data Center Max 1550 Series 6 63,744
GPU HBM HBM2e 768 GiB 7.968 PiB

Aurora PVC GPU Components

Stack a.k.a. Tile 2
Xe Vector Engine a.k.a. EU (execution unit) 512 per Stack (4?? active) 8 threads, 512b SIMD
Xe Matrix Engine a.k.a systolic part of EU 512 per Stack (4?? active)
Register 512 bit register 128 per thread
Xe Core a.k.a. subslice; unit of 8 EUs 64 per Stack 128 per GPU
L1 cache 128 KiB
Last Level cache a.k.a. RAMBO cache 384 MiB per GPU