Programming on WestGrid Systems

Table of Contents



This page gives a general overview of programming on WestGrid systems, with links to more detailed documentation on the compiling and running programs on the individual machines.

The material presented here does not include tutorials designed to teach you programming in any particular language. The intention is to present basic usage instructions for compilers and the debugging and optimization tools available on the WestGrid systems.


The main languages used for scientific programming, Fortran (77 and 90), C and C++ are available on all the WestGrid systems. Other languages, such as Java, or MATLAB can be used, but, are not recommended for performance reasons and may not be as well supported. Scripting, whether using shell commands, or higher level languages such as Perl or Python, is often used to automate pre- or post-processing tasks, but, will not be covered here.

Editors and the X Window System

A number of text editors are available for use in developing programs. See the table on the WestGrid software page for a list.

Nedit may be of particular interest to users coming from a Microsoft Windows PC or Apple Macintosh background, as the keyboard shortcuts are the same as found in word-processing programs on those types of computer. It also has options to number each line and show the column position of the cursor. These are simple, but, helpful features when trying to relate the source code to an error message after a program has crashed.

We do not currently offer an integrated environment, such as Microsoft's Visual Studio, for programming on WestGrid systems. Many users will use simple character-based terminal windows for creating and editing program files. However, there are significant advantages to using a workstation on which an X Window display server program has been installed. Most UNIX and Linux environments will already have this support included. For MacOS X computers, a program called X11 can be downloaded from Apple. For Microsoft Windows PCs, a free alternative is Cygwin-X. Commercial programs, such as X-Win32, WinaXe and Exceed are also available.

When using a workstation that supports X Windows, the emacs and nedit editors support syntax highlighting, with language keywords marked in colour or boldface, for example. Comment lines are also indicated. Exactly which features are highlighted depends on the language being used.

Programming tools

Besides the basic editors, compilers, debuggers and performance programming tools, there are a few other programming aids available on the system. Among these are make (streamlines the edit and compile cycle), xdiff (comparing differences between files)) and ftnchek (Fortran syntax checking). See the programming section of the WestGrid software page for help in locating these utilities.

For those already familiar with make, please note that make is sometimes aliased to gmake (GNU make), which is slightly different than the make commonly available on UNIX systems.

Compiling Serial Code


See the table on the WestGrid software page for a comparison of the compilers available on the various WestGrid computers. To get detailed information about the options available for a particular compiler, use the man command. For example:

man g++


For examples of compiling and running serial code on specific WestGrid systems, follow the links here:

Parallel Programming


On WestGrid systems there are several ways to shorten the time it takes for a calculation by using two or more processors in parallel. The simplest way, which is often the most efficient, is to submit several independent serial jobs at the same time (perhaps running the same program with different input data sets). This will reduce the total time for a series of calculations, but, will not speed up an individual job.

If the turnaround time for a single job is of paramount importance or if a problem is too big to be run on a single machine, parallel programming techniques are needed to allow multiple processors to work simultaneously on that single job. Among the approaches one can use, the two that are discussed here are those based on message passing, as implemented by MPI (Message Passing Interface) and those based on shared memory using the OpenMP interface.

The preparation and submission of parallel jobs depends on which technique you select and the target platform. If your application is suited to the submission of multiple independent small jobs, this would be suitable for the glacier cluster. At the other extreme, OpenMP jobs can be used for large-memory jobs using many processors, such as the 256-processor machine accessible through the arcturus queue at OpenMP jobs are restricted to processors sharing the same memory space. This restricts its use on the Glacier cluster to just 2 per job, but, up to 8 could be used on the Bugaboo, Checkers or Orcinus clusters. Using MPI is more flexible, as one can run MPI jobs using relatively large numbers of processors at all the WestGrid computational sites, but, generally requires more programming effort.

In practice, the number of processors one can realistically use depends on other factors, such as how fast efficiency drops off as the number of processors is increased, policies in place at the various WestGrid sites, on system load and how long you are willing to have a job sit in an input queue waiting for processors to become available.

There are numerous on-line tutorials for parallel programming, such as listed at the MPICH and OpenMP home sites.

Message Passing Interface (MPI)

To use MPI in a batch mode on many of the WestGrid systems, you need to:

  • modify your source code to add calls to MPI routines,
  • compile your code, linking with the MPI libraries,
  • prepare a Torque (PBS) job submission script that references your program, and
  • submit the Torque (PBS) script with qsub.

The details of the above steps depend on the language, compiler and WestGrid site being used. Follow the links below for specific information about compiling and running MPI jobs on the various WestGrid machines.

For examples of compiling and running parallel MPI code on specific WestGrid systems, follow the links here. Note that the Checkers examples (with the exception of the commands for running interactively) will also apply to most of the other WestGrid clusters (such as Bugaboo, Grex, Hermes/Nestor, Lattice and Orcinus).


To use OpenMP in a batch mode on many of the WestGrid systems, you need to:

  • modify your source code to add OpenMP directives,
  • compile your code, linking with appropriate flags,
  • prepare a Torque (PBS) job submission script that references your program, and
  • submit the Torque (PBS) script with qsub.

The details of the above steps depend on the language, compiler and WestGrid site being used. For examples of compiling and running OpenMP code on specific WestGrid systems, follow the links here.  Note that the Checkers examples will also apply to most of the other WestGrid clusters.



One of the strengths of the WestGrid environment for debugging is the wide variety of compilers and debuggers available. Some users get their code running on one system and then tend to blame the compiler if it doesn't behave the same way on a different platform. However, the compilers get a lot more careful scrutiny than the typical user's code. It is almost always hidden errors such as non-standard usage, uninitialized variables, array bounds problems, type mismatches, or a failure to understand the numerical limitations in the researcher's program, rather than compiler problems that leads to inconsistency in behaviour across systems. So, you are encouraged to try your code on more than one of the WestGrid systems and with more than one type of compiler.

Here are some other general guidelines for code development and debugging.

  • Keep a baseline version of your code, along with both input and output in a separate directory from the development version on which you are currently working.
  • Make changes in relatively small steps.
  • Use diff, xdiff, or xxdiff to compare versions of files.
  • Use IMPLICIT NONE in Fortran.
  • Depending on the language, make liberal use of WRITE, printf or cout statements. Saving debugging data to one or more files may encourage you to keep the output statements in your code, where they may come in handy a little later on. Comment out the debugging statements rather than deleting them.
  • Use visualization tools to look at more than just a small sample of the output.
  • Don't be shy about showing your code to someone else, such as the WestGrid support analysts. Write to .

Regardless of the system being used, adding a -g flag to the compilation is a minimum prerequisite for using a debugger.

See the section above on serial code development for links to specific information about the debuggers available on the various WestGrid systems.

Improving Performance


For many systems, one uses a -pg compiler flag to instrument code to collect statistics for later analysis with the gprof utility.

For profiling MPI code, one can use the MPE profiling libraries.

For a discussion of performance profiling for OpenMP programs using the ssrun command on nexus, see the program development tools section of the course notes developed at the University of Alberta. The ssrun and perfex commands for SGI systems are also discussed in more recent workshop notes.


Updated 2011-11-09.