Lattice QuickStart Guide

About this QuickStart Guide

This QuickStart guide gives a brief overview of the WestGrid Lattice facility, highlighting some of the features that distinguish it from other WestGrid resources. It is intended to be read by new WestGrid account holders and by current users considering whether to move to the Lattice system. For more detailed information about the Lattice hardware and performance characteristics, available software, usage policies and how to log in and run jobs, follow the links given below.

Introduction

The Lattice cluster is intended for large-scale parallel applications that can take advantage of its InfiniBand interconnect.

Lattice is an HP high-density cluster with 4096 cores. See the hardware section below for details.

Here is a picture of Lattice during installation in July 2010 (click for a larger image in a new window).

Lattice during installation 2010-07-07

 

Request for access

Unlike most WestGrid systems, a separate request is required to obtain a WestGrid account on Lattice.  If you think the software you would like to run is appropriate for the Lattice cluster, please write to accounts@westgrid.ca with a subject line of the form "Lattice account request (your_username)" with a request for an account and a mention of the software you propose to use.

Hardware

Processors

Along with a login node and other machines for job management and file serving, the main part of the Lattice cluster consists of multicore compute nodes.

The Lattice cluster is comprised of 512 8-core nodes, each having 2 sockets. Each socket has an Intel Xeon L5520  (Nehalem) quad-core processor, running at 2.27 GHz. The 8 cores associated with one of the individual nodes share 12 GB of RAM.

Interconnect

As mentioned in the introduction, the Lattice cluster is intended for multi-node jobs that make use of the low-latency interconnect. Lattice uses an InfiniBand 4X QDR (Quad Data Rate) 40 Gbit/s switched fabric, with a two to one blocking factor (to reduce purchase costs).

Storage

There are three types of storage locations: home directories, global scratch and local scratch.  The home directories and global scratch are shared among the Breezy, Lattice and Parallel systems.

Home directories

There is approximately 6.5 TB of disk space allocated for home directories. There is a storage quota of 50 GB, with a 200,000-file limit, for each individual home directory.  Use your home directory for files that you want to save for longer than 30 days.  Run jobs from your directory in /global/scratch.

You can check the status of the quota for your home directory with

/usr/local/ibrix/bin/ibrix_quota -f /home

Global scratch directories

A total of about 154 TB is used for global scratch.  A directory /global/scratch/user_name has been set up for each user. The default quota in /global/scratch for individual users is 450 GB, with a 200,000-file limit. If you need an increased quota, please write to WestGrid support

You can check the status of the quota for your /global/scratch directory with

/usr/local/ibrix/bin/ibrix_quota -f /global/scratch

30-day policy: Please note that /global/scratch is intended only for files associated with running jobs or waiting for post-processing. Files older than 30 days are subject to deletion.

Local scratch directories

Each compute node has a local scratch directory, /tmp, with approximately 115 GB of storage space.  If you have an I/O intensive program you should consider using /tmp for temporary files associated with your running jobs. Delete files in /tmp at the end of your runs. Note that these local scratch partitions are shared among all users of a given node, so, you are not guaranteed that all the space will be available for any given run.

Software

See the main WestGrid software page for tables showing the installed application software on Lattice and other WestGrid systems, as well as information about the operating system, compilers, and mathematical and graphical libraries.

Please write to WestGrid support if there is additional software that you would like installed.

Using Lattice

Getting started

To log in to Lattice, connect to lattice.westgrid.ca using an ssh (secure shell) client. For more information about connecting and setting up your environment, see the QuickStart Guide for New Users.

Batch job policies

As on other WestGrid systems batch jobs are handled by a combination of TORQUE and Moab software. For more information about submitting jobs, see Running Jobs

Unlike most other WestGrid systems, we prefer that the syntax "-l nodes=xx,ppn=8" be used rather than "-l procs=yyy" when requesting processor resources on Lattice.  Lattice is used almost exclusively for large parallel jobs that use whole nodes.  This has the potential of improving the performance of some jobs and minimizes the impact of a misbehaving job or hardware failure. Since there are 8 cores per node on Lattice, a  ppn (processors per node) parameter of 8 will request that all the processors on a node be used.  Also, it is recommended that you ask for 10-11 GB of memory per node requested, using the mem parameter.  So, a typical job submission on Lattice would look like:

qsub -l nodes=4:ppn=8,mem=40gb,walltime=72:00:00 parallel_diffuse.pbs

The following limits are in place for batch jobs submitted to the default queue (that is, if no queue is specified on qsub command):

 

Resource
Policy or limit
Maximum walltime, but, see below for other comments related to walltime. 168 hours
Suggested maximum memory resource request, mem. 11 GB
Maximum number of running jobs for a single user 64
Maximum cores (sum for all jobs) for a single user 1024
Maximum jobs in Eligible (for scheduling) section of the input queue 8

 

Six to eight nodes are generally reserved for short batch jobs, some with a maximum walltime limit of 3 hours and some with a maximum of 24 hours.

Interactive jobs

Two to four nodes are reserved for interactive. These can be accessed by specifying the interactive queue and a walltime of less than or equal to three hours on your qsub command line.  If you require exclusive access to a node, you can add naccesspolicy=singlejob, as shown here:

qsub -q interactive -l walltime=03:00:00,naccesspolicy=singlejob job_script.pbs

See the Working Interactively section of the Running Jobs page for an alternate method to reserve processors for interactive use, which can be used in cases where you need more than two nodes or a walltime longer than 3 hours. 

The login node can be used for short testing and debugging sessions, but, a virtual memory limit of 6 GB per process has been imposed to reduce the chance of a user making the login node unusable for others. If you need to test a program that has a process requiring more virtual memory than that you could one the compute nodes.


Updated 2013-01-28.