Efficient file transfers with globus-url-copy

How can I move my files quickly between WestGrid centres?


If you need to move large files between WestGrid sites, globus-url-copy is an order of magnitude faster than scp. Before you can use globus-url-copy, you need to initialize a proxy.

In this example, we will move the file large.dat from our home directory on lattice to the /tmp directory on our SFU storage site, gridstore.


> globus-url-copy file:/home/users/ssharpe/large.dat gsiftp://gridstore/tmp/large.dat

There are options we can use to boost the speed of copying;

  • The -tcp-bs 4194304 option requests a buffer size of 4MB, which we have found is a good size for fast transfers at WestGrid.
  • Another pair of options which has proven successful at WestGrid is -tcp-bs 2097152 -p 2. This runs two parallel transfer streams using 2MB buffers.
Putting this all together;

> globus-url-copy -tcp-bs 2097152 -p 2 file:/home/users/ssharpe/large.dat gsiftp://gridstore/tmp/large.dat

The globus-url-copy command can currently copy only files and not directories. If you need to move a directory, you might can use the tar or gzip commands to wrap it all up into a file.

For more detail, use globus-url-copy -help.

See also gcp


Updated 2009-01-30.