Banner

OVERVIEW

AADITYA is 790+ TeraFlops High Performance Computing System IBM iDataPlex cluster, which features 38,144 Intel Sandy Bridge processors and 149 TB of memory.

System Configuration Details

AADITYA is an IBM iDataPlex Supercomputer. The login and compute nodes are populated with two Intel Sandy Bridge 8-core processors. AADITYA uses the FDR 10 Infiniband interconnect in a Fat Tree configuration as its high-speed network for MPI messages and IO traffic. AADITYA uses IBM's General Parallel File System (GPFS) to manage its parallel file system. AADITYA has 2,384 compute nodes that share memory only on the node; memory is not shared across the nodes. Each compute node has two 8-core processors (16 cores) with its own Red Hat Enterprise Linux OS, sharing 64 GBytes of memory, with no user-accessible swap space. AADITYA is rated at 800 peak TFLOPS and has 6 PBytes (formatted) of disk storage.

AADITYA is intended to be used as a batch-scheduled HPC system. Its login nodes are not to be used for large computational (memory, IO, long executions) work. All executions that require large amounts of system resources must be sent to the compute nodes by batch job submission.

______________________________________________________________________________________________

System overview

______________________________________________________________________________________________

Datacenter Overview

Datacenter

______________________________________________________________________________________________

Hardware

38,144 processor cores 2.6-GHz Intel Xeon E5-2670 (Sandy Bridge) processors
2,384 computation nodes IBM dx360 M4, dual socket, 8 cores per socket
11 login nodes IBM dx360 M4, dual socket, 8 cores per socket
149 TB total system memory 4 GB/core, 64 GB/node, DDR3-1600 DIMMs
FDR Mellanox InfiniBand interconnect Mellanox 36-Port Managed FDR14 IB Switch

______________________________________________________________________________________________

Software Stack

Description Product
Operating System RHEL v6.x
System Management xCAT, IBM Mellanox OFED
File System IBM GPFS
Resource Management/Scheduler IBM Platform LSF Standard
MPI Library IBM P.E. Runtime Ed.
System Monitoring IBM PAC, Mellanox OFED
Compilers Intel Composer XE, GNU GCC
Libraries Intel MKL
Backup Software IBM Tivoli Storage Manager

______________________________________________________________________________________________

Details on processors

AADITYA uses 2.6-GHz Intel Sandy Bridge processors on its login and compute nodes. There are 2 processors per node, each with 8 cores, for a total of 16 cores per node. In addition, these processors have 8x256 KBytes of L2 cache, and 12x6 MBytes of L3 cache.

______________________________________________________________________________________________

Details on Memory

AADITYA uses both shared and distributed memory models. Memory is shared among all the cores on a node, but is not shared among the nodes across the cluster.

Each login node contains 128 GBytes of main memory. All memory and cores on the node are shared among all users who are logged in. Therefore, users should not submit memmory intensive jobs on login nodes.
2384 compute nodes contain 64 GBytes of memory.

______________________________________________________________________________________________

Operating System

The operating system on AADITYA is RedHat Enterprise Linux Server. The operating system supports 64-bit software.

______________________________________________________________________________________________

Architecture Difference

Type Prithvi Aaditya
Machine IBM P575 IBM IdataPlex dx360 m4
Processor Power 6 Intel® Xeon® Processor E5-2670
Core 32 cores per node 16 cores per node
RAM 128 GB per node 64 GB per node
OS IBM AIX 5.3 TL 12(On disks) RHEL v6.x (On RAM)
Usable Memory 128 GB per node 60 GB per node (4 GB used by OS)
Capacity 602 GFLOPS per node 303 GFLOPS per node
Compiler IBM INTEL

______________________________________________________________________________________________

Utility Node configuration

______________________________________________________________________________________________