Scientific Computing Facilities Technical Summary
September, 2007 Configuration
Table of contents
Blue Gene
Hardware Configuration
Each node contains two processors so the number of processors is twice the number of nodes listed.
| Machine Name |
Role in Cluster |
# Nodes |
Cache |
Memory (per node) |
Scratch Disk |
Network |
| Levi and Lee |
Front End |
2 x 1.5 GHz Power5 |
see below
|
4 GB |
72 GB |
1Gbps
Ethernet |
| |
Compute Nodes |
1024 x 700 MHz PPC440 |
see below
|
512 MB |
none |
Torus, Tree, Global Interrupt |
| |
IO Nodes |
128 x 700 MHz PPC440 |
see below
|
512 MB |
none |
1 Gbps
Ethernet, Tree |
Note on front end caches: Each processor has a 64 KB instruction, 32 KB data L1 cache. Each node has a shared 1.9MB L2 and 36 MB L3 cache.
Note on compute and IO node caches: Each node has a 32 KB L1 cache, 2 KB L2 cache, and a 4 MB L3 cache.
Batch System and Usage
The IBM Blue Gene's batch system is IBM's LoadLeveler. The current limitation is that all jobs must use a partition of exactly 32, 128, 512 or 1024 (the entire machine) nodes and no job may run for more than 5 hours of wall-clock time. 1024-node jobs are only allowed to run in off-hours.
Each CPU hour (counted by wall clock time) used on the Blue Gene will use 0.25 SUs of a project's allocation.
pSeries
Hardware Configuration
| Host Name |
Model |
# Processors |
Cache |
Memory (Aggregate) |
Scratch Disk |
Network |
| Kite |
IBM pSeries 690 |
32 x 1.3 GHz Power4 |
see below
|
32 GB |
36 GB |
1 Gbps
Ethernet |
| Frisbee |
IBM pSeries 690 |
32 x 1.3 GHz Power4 |
see below
|
32 GB |
36 GB |
1 Gbps
Ethernet |
| Pogo |
IBM pSeries 690 |
32 x 1.3 GHz Power4 |
see below
|
32 GB |
36 GB |
1 Gbps
Ethernet |
| Domino1 |
IBM pSeries 690 |
16 x 1.3 GHz Power4 |
see below
|
16 GB |
36 GB |
1 Gbps
Ethernet |
| Twister |
IBM pSeries 655 |
8 x 1.1 GHz Power4 |
see below
|
16 GB |
36 GB |
1 Gbps
Ethernet |
| Scrabble |
IBM pSeries 655 |
8 x 1.1 GHz Power4 |
see below
|
16 GB |
36 GB |
1 Gbps
Ethernet |
| Marbles |
IBM pSeries 655 |
8 x 1.1 GHz Power4 |
see below
|
16 GB |
36 GB |
1 Gbps
Ethernet |
| Crayon |
IBM pSeries 655 |
8 x 1.1 GHz Power4 |
see below
|
16 GB |
36 GB |
1 Gbps
Ethernet |
| Litebrite |
IBM pSeries 655 |
8 x 1.1 GHz Power4 |
see below
|
16 GB |
36 GB |
1 Gbps
Ethernet |
| Hotwheels |
IBM pSeries 655 |
8 x 1.1 GHz Power4 |
see below
|
16 GB |
36 GB |
1Gbps
Ethernet |
| Jacks2 |
IBM pSeries 655 |
8 x 1.7 GHz Power4 |
see below
|
8 GB |
72 GB |
1 Gbps
Ethernet |
| Playdoh2 |
IBM pSeries 655 |
8 x 1.7 GHz Power4 |
see below
|
8 GB |
72 GB |
1 Gbps
Ethernet |
| Slinky2 |
IBM pSeries 655 |
8 x 1.7 GHz Power4 |
see below
|
8 GB |
72 GB |
1 Gbps
Ethernet |
Note on 6xx caches: 32 KB L1 cache on each processor, 1.41 MB L2 cache shared by each pair of processors, 128 MB L3 cache shared by each set of eight processors.
Batch System and Usage
The IBM pSeries machines have a detailed queue structure and certain machines dedicated to certain job types as indicated below. The batch system used to manage the queues is the Load Sharing Facility (LSF).
| Host Name |
Function |
Service Level |
Batch Queues |
| Kite |
MP batch |
Production |
p4-mp16 |
| Frisbee |
MP batch |
Production |
p4-mp16 |
| Pogo |
MP batch |
Production |
p4-mp32 |
| Domino1 |
MP batch |
Production1 |
p4-ibmsur-mp16 |
| Twister |
Interactive |
Production |
none |
| Scrabble |
SP batch
MP batch |
Production |
p4-short
p4-verylong
p4-mp4 |
| Marbles |
SP batch |
Production |
p4-long |
| Crayon |
SP batch |
Production |
p4-long |
| Litebrite |
MP batch |
Production |
p4-mp8 |
| Hotwheels |
MP batch |
Production |
p4-mp8 |
| Jacks |
MP batch/Interactive2 |
Production |
p4-cism-mp8 |
| Playdoh |
MP batch/Interactive2 |
Production |
p4-cism-mp8 |
| Slinky |
MP batch/Interactive2 |
Production |
p4-cism-mp8 |
| Barbie |
Visualization |
Production |
N/A |
[Notes: SP=Single Processor; MP=Multiple Processor]
| Queue Name |
# processors |
CPU Limit
(in hours) |
Wall Clock Limit
(in hours) |
Run Window |
Slots |
| p4-short |
1 |
2 |
2.5 |
always |
2 |
| p4-long |
1 |
32 |
40 |
always |
16 |
| p4-verylong |
1 |
64 |
80 |
always |
2 |
| p4-mp4 |
4 |
16 |
5 |
always |
1 |
| p4-mp8 |
8 |
32 |
5 |
always |
2 |
| p4-cism-mp8 |
8 |
32 |
5 |
always |
3 |
| p4-mp16 |
16 |
64 |
5 |
always |
4 |
| p4-mp32 |
32 |
128 |
5 |
always |
1 |
| p4-ibmsur-mp161 |
16 |
128 |
9 |
always |
1 |
Note: In general, all of the LSF queues will have dedicated access to all of the processors they may access ("# processors" column above) as none of the machines are allowed to be oversubscribed.
Footnotes
1: The machine domino and the queue which runs on it, p4-ibmsur-mp16, is accessible only to users of a limited number of projects.
2: Users in the CISM group are given higher priority on the machines jacks, playdoh, and slinky and are the only people who can log in to those machines. Also, non-CISM users using these machines are charged 1.31 SUs per CPU hour they use simply due to these machines having faster processors.
SUs used per processor hour on the pSeries machines.
| Processor Type and Speed |
SUs used per CPU hour |
| 1.3 Ghz p690 |
1.0 |
| 1.1 Ghz p655 |
0.85 |
| 1.7 Ghz p655 |
1.31 |
Katana Cluster
Hardware Configuration
Each blade center node contains two dual-core 2.6 GHz AMD Opteron 2218HE processors processors so the number of processors is four times the number of nodes listed.
| Machine Name |
Role in Cluster |
# Nodes |
Cache |
Memory (per node) |
Scratch Disk |
Network |
| Katana |
Front End |
1 x 2.6 GHz AMD Opteron 2218HE |
64 KB L1 and
1 MB L2
|
8 GB |
50 GB |
1 Gbps Ethernet/
10 Gbps 4X Infiniband |
| |
Compute Nodes |
21 x 2.6 GHz AMD Opteron 2218HE |
64 KB L1 and
1 MB L2
|
8 GB |
50 GB |
1 Gbps Ethernet/
10 Gbps 4X Infiniband |
Batch System and Usage
Users are restricted to using a maximum of 16 processors and running for 24 hours of wall-clock time (but note that the default limit is 2 hrs, to run for longer you must request that). The batch system on the Katana Cluster is the Sun Grid Engine.
Accounting charges on the Katana Cluster went into effect in February, 2008. In July, 2008 8 additional nodes went into production bringing the total number of nodes up to 22.
Linux Cluster
Hardware Configuration
Each node contains two processors so the number of processors is twice the number of nodes listed.
| Machine Name |
Role in Cluster |
# Nodes |
Cache |
Memory (per node) |
Scratch Disk |
Network |
| Skate and Cootie |
Front End |
2 x 1.261 GHz Pentium III |
16 KB L1 and
512 KB L2
|
1 GB |
36 GB |
100 Mbps Ethernet/
2.2 Gbps Myrinet |
| |
Compute Nodes |
52 x 1.261 GHz Pentium III |
16 KB L1 and
512 KB L2
|
1 GB |
36 GB |
100 Mbps Ethernet/
2.2 Gbps Myrinet |
Batch System and Usage
Users are restricted to using a maximum of 24 nodes (48 processors) and running for 48 hours of wall-clock time. The number of nodes that can be used simultaneously is an aggregate, so it is fine to run 24 separate single-processor jobs (one per node) at the same time. Each such job would be subject to the 48 hour wall-clock limit. The batch system on the Linux Cluster is PBS.
Each CPU hour used on the Linux Cluster will use 0.3 SUs of a project's allocation.
|
