Efficient file transfers with gcp

Introduction

gcp is a grid enabled version of the scp copy command.

The gcp command is meant to emulate as closely as possible the scp command while using globus-url-copy for file transfers for performance reasons. gcp allows recursive transfers of directories and wildcards.

Since gcp is a grid tool using globus-url-copy it needs to use grid certificates for authentication. This is accomplished by first checking whether a valid grid proxy certificate exists. If that is not the case, gcp will attempt to run grid-proxy-init. Failing that gcp will contact the myproxy.westgrid.ca proxy server in order to obtain a proxy certificate. If that fails as well, gcp issues an error, explaining how to upload certificates to the myproxy server. The whole authentication process is setup such that it is not required that the user has knowledge of grid authentication.

In most cases the syntax for gcp is the same as for scp.

Some useful options:

  • -r : recursive transfer (obviously you only need this if you want to descent into directories)

    -v : verbose output; gcp is giving detailed information on what it is doing.

    -i : gcp will print a list of source files and destination files and (in the case of -r) destination directories that it will create on the destination host. It then will ask whether you want to proceed at which point you can answer 'n' and no files will be transfered.

Detailed information about gcp is available through its man page: type man gcp.

Examples

General examples

  • gcp file1 glacier:document

    • will transfer the file file1 in the current directory to glacier under the name document in your glacier home directory.

    gcp -r dir1 gridstore:data

    • transfer the directory dir1 and all its files, subdirectories to gridstore into the directory data/dir1 on gridstore.

    gcp -r docs* programs/*.exe progdir gridstore:vault/new

    • copy recursively all files and directories that start with docs, all files that match programs/*.exe, and the directory progdir into the directory vault/new in your home directory on gridstore.

Example of how to move your data to Silo

This command will copy your whole directory to Silo.

gcp -r dirname silo.westgrid.ca:/dirname

This command will copy your data from silo to the system you are logged into.

gcp -r silo.westgrid.ca:/dirname dirname

 


Updated 2011-11-09