The ORNL Summit supercomputer has just become operational and is now pushing all of its data-crunching power towards scientific research. Designed with NVIDIA's Volta GPUs and IBM's Power9 CPUs, the Summit supercomputer is currently the fastest supercomputer on the planet, trumping its predecessor with 100 times more performance.
Oak Ridge National Laboratory's Summit Supercomputer Comes To Life - Built With More Than 27,000 NVIDIA Tesla V100 GPUs and Over 9,000 IBM Power9 CPUs, Breaks The Exascale Barrier
The Oak Ridge National Laboratory's Summit supercomputer has been a long time in the making. A juggernaut of a machine which took years to complete and the last NVIDIA Volta GPUs made their way to the machine just a little while ago. With the last batch of GPUs delivered, the Summit super-computer is now completely operational and can now pull all of its power towards the purpose it was built for, and that purpose is research into finding a cure for cancer, helping doctors identifying patterns of disease and addiction and accelerate fusion energy development. These are some of the main goals behind creating the Summit supercomputer. While the machine has a purpose, the technology that was used in building Summit is also fascinating.
This speed will let today’s generation of scientists accomplish wonders. The Oak Ridge National Laboratory is already a playground for cutting-edge science, and its campus is a hub for scientists eager to harness its machines to do their best work.
That’s why Summit already has a full schedule, accelerating projects including:
- Cancer Research: The DOE and National Cancer Institute are working on a program called CANcer Distributed Learning Environment (CANDLE). Their aim is to develop tools that can automatically extract, analyze and sort health data to reveal previously hidden relationships between disease factors such as genes, biological markers and the environment.
- Fusion Energy: Fusion, the energy source powering the sun, has long been touted for its promise of clean, abundant energy. Summit will be able to model a fusion reactor and its magnetically confined plasma, hastening commercial development.
- Disease and Addiction: Researchers will use AI to identify patterns in the function and evolution of human proteins and cellular systems. These patterns can help us better understand Alzheimer’s, heart disease or addiction, and inform the drug discovery process.
“Summit is fast, but what Summit means is even more remarkable,” NVIDIA CEO Jensen Huang said at Summit’s debut Friday (read his full speech here). “Summit is a magnificent scientific instrument that will attract the world’s great scientists.”
“Summit is a new breed of computer,” Huang said. “Summit is the world’s largest AI supercomputer, a machine that learns. Its software will write software — amazing software that no human can write.”
“To my friends and colleagues at Oak Ridge National Labs, a huge congratulation for reaching this Summit,” Huang said. “From here, we can all jump to the next peak.” via NVIDIA
Powered by the most complex arrays of GPUs and CPUs, the Summit supercomputer is a marvel of engineering. There are a total of 27,648 NVIDIA Tesla V100 GPUs and 9,216 IBM Power CPUs within the Summit. Just to put things into perspective, the Tesla V100 has 5120 CUDA cores, the Summit supercomputer has 141,557,760 CUDA cores in total. The system is arranged in 4,608 Nodes where each Node is configured with dual IBM Power9 CPUs and 6 Tesla V100 GPUs.
The system also has a 250 Petabyte storage system that can transfer up to 25 GB/s of data and a total of 10 Petabyte of system memory that is a combination HBM2 (16 GB per Tesla V100), DDR4 DIMMs and NVDIMMs. The Summit supercomputer sips in 13 MW of power compared to 9 MW on the Titan.
Looking at the performance metrics, the Summit is a millennium ahead of Titan. First of all, 95% of the Summit's total horsepower is driven by the powerful Tesla V100 GPUs. It's able to calculate up to 3.3 Exaops or 3 Billion Billion AI calculations per second with the Volta Tensor core architecture. It is also 100 times faster than the Titan. It is also able to do 200 PetaFLOPs (215 PFLOPs to be exact) of double precision and over 400 PetaFlops (430 PFlops to be exact) of single precision / high precision compute.
Those exaflops are not just theoretical either. According to ORNL director Thomas Zacharia, even before the machine was fully built, researchers had run a comparative genomics code at 1.88 exaflops using the Tensor Core capability of the GPUs. The application was rummaging through genomes looking for patterns indicative of certain conditions. “This is the first time anyone has broken the exascale barrier,” noted Zacharia. via TOP500
The NVIDIA powered Summit is an impeccable machine that is doing good for humanity itself. It's the first machine to break the exascale barrier and is the benchmark for next machines to come.
Summit Supercomputer Specifications:
| Supercomputer | Titan | Summit |
|---|---|---|
| Number of Nodes | 18688 | 4608 |
| Processors | 1 Opteron 1 Kepler K20X | 2 IBM Power9 6 NVIDIA Tesla V100 |
| GPUs | 18688 NVIDIA Tesla K20X | 27648 NVIDIA Telsa V100 |
| CPUs | 18688 Opteron CPUs | 9216 Power9 CPUs |
| Node Performance | 1.44 TF | 49 TF |
| Peak Performance | 27 PF | 200 PF |
| Peak OPs (Tensor) | N/A | 3.3 ExaOps |
| Memory Per Node | 38 GB DDR3 + 6 GB GDDR5 | 512 GB DDR4 + HBM2 (16/32 GB) + NVDIMM |
| NV Memory Per Node | 0 | 800 GB (Flash based) |
| Total System Memory | 710 TB | 10 PB |
| System Interconnect | Gemini (6.4 GB/s) PCIe 8 GB/s | Dual Rail EDR-IB (23 GB/s) / Dual Rail HDR-IB (48 GB/s) NVLINK 300 GB/s |
| Interconnect Topology | 3D Tours | Non-Blocking Fat Free |
| File System | 32 PB, 1 TB/s Lustre | 250 PC, 2.5 TB/s, GPFS |
| Peak Power Input | 9 MW | 13 MW |
