Breezy QuickStart Guide

About this QuickStart Guide

This QuickStart guide gives a brief overview of the WestGrid Breezy facility, highlighting some of the features that distinguish it from other WestGrid resources. It is intended to be read by new WestGrid account holders and by current users considering whether to move to the Breezy system. For more detailed information about the Breezy hardware and performance characteristics, available software, usage policies and how to log in and run jobs, follow the links given below.

Introduction

Breezy is an Linux AMD Istanbul cluster with large-memory nodes connected by Infiniband. Breezy is intended for jobs that need more memory per node than can be obtained on other WestGrid clusters (for example, the 48 GB of memory on a Grex node).   Large OpenMP-based parallel programs are the expected type of Breezy workload.

Request for access

Unlike most WestGrid systems, a separate request is required to obtain a WestGrid account on Breezy.  If you think the software you would like to run is appropriate for Breezy, please write to accounts@westgrid.ca with a subject line of the form "Breezy account request (your_username)" with a request for an account and a mention of the software you propose to use.

Hardware

Processors

Breezy specifications:
  * Quad-socket, 6-core AMD Istanbul processors (24 cores @ 2.4 GHz) per node
  * 256 GB memory per compute node (64 GB on login node).  Do not try to use all memory on a node - see Batch Job Policies below.
  * 16 nodes conected with 4X DDR InfiniBand at 20 Gbits/s and Gigabit Ethernet

Interconnect

* 16 nodes conected with 4X DDR InfiniBand at 20 Gbits/s and Gigabit Ethernet

Storage

  Breezy shares IBRIX filesystems with lattice.westgrid.ca

  * Quotas are in effect: 450 GB for /global/scratch
                                    50 GB for /home

You can check the status of the quota for your home directory with

/usr/local/ibrix/bin/ibrix_quota -f /home

 and for your /global/scratch directory with

/usr/local/ibrix/bin/ibrix_quota -f /global/scratch

Software

See the main WestGrid software page for tables showing the installed software on Breezy and other WestGrid systems, including information about the operating system and compilers.

Using Breezy

Connecting for the first time

To log in to Breezy, connect to breezy.westgrid.ca using an ssh (secure shell) client. For more information about connecting and setting up your environment, see the QuickStart Guide for New Users.

Interactive use

The Breezy login node may be used for short interactive runs during development.  Production runs should be submitted as batch jobs.

Batch job policies

Batch jobs are handled by a combination of TORQUE and Moab software. For more information about submitting jobs, see the general Running Jobs page.

The maximum walltime limit for Breezy jobs is 3 days.

Since Breezy is intended for applications requiring large amounts of memory, one will often be expected to specify a TORQUE mem parameter on the qsub command line (or in #PBS directives in the batch job script).  Although the memory per node is nominally 256 GB, there is not quite that much available.  Do not specify more than 250 GB for the mem or pmem resource requests or your job will get stuck in input queue waiting for memory that will never be available.

Another intended use for Breezy is for multi-threaded single-node applications. For such cases, use a resource request of the form -l nodes=1:ppn=24,mem=250gb, where ppn, the processors per node, is the number of cores required and the memory needed is specified with the mem parameter. Since Breezy compute nodes have 24 cores, that is maximum number you can specify for ppn.  You can use smaller values for ppn and mem as appropriate for your calculation.  However, if you are using less than 24 cores, it is important to limit the number of threads used by your application to the number of cores requested, so as not to interfere with other users' jobs, which may be assigned to the same node.  Often this can be accomplished by setting the OMP_NUM_THREADS variable.  See the example script in the OpenMP section of Checkers programming guide for an example.

Please do not use the ncpus or procs parameters when requesting processors on Breezy.  In the rare cases in which multiple nodes are used for a single job, use the -l nodes=n:ppn=24 format to request multiple nodes, where n is the number required.

 


Updated 2013-01-28.