File Transfer

Data Transfer to and from the cmslpc cluster

At Fermilab, access to User Analysis Farm (UAF) goes through cmslpc-el9.fnal.gov. This can be accessed using Secure Copy (scp) or sftp.

The following storage areas on NFS are available for users, be sure to reference the appropriate directory and use your username, and the first letter (here u) of your username:

/uscms/homes/u/username
/uscms_data/d1/username


To transfer a file to the cmslpc cluster:

Usage: scp file_name username@cmslpc-el9.fnal.gov:/uscms/homes/u/username

e.g.

Other Computer $ scp zprime705.jdf username@cmslpc-el9.fnal.gov:/uscms/homes/u/username
zprime705.jdf        100% |*****************************|   286       00:00

To transfer a file from the cmslpc cluster:

Usage: scp username@cmslpc-el9.fnal.gov:/uscms/homes/u/username/source destination

e.g.

Other Computer $ scp username@cmslpc-el9.fnal.gov:/uscms/homes/u/username/zprime_jj_10001.root .
zprime_jj_10001.root 100% |*****************************|  8880 KB    00:01

Copy files on the EOS T3_US_FNALLPC SE

For instructions how to copy files from/to the EOS Storage Element, follow the instructions on the Using EOS at the LPC website.

Copy files using gfal-copy from another site's Storage Element

Instructions are on the CRAB3 FAQ on how to use gfal tools to find and copy files from another site's Storage Element

• Note: if you have errors with gfal-copy, you may wish to do the gfal-copy from a clean environment. You can do that on the command line by starting with this: env -i X509_USER_PROXY=${X509_USER_PROXY}, or eval `scram unsetenv -sh`. This also works for xrdcp. Examples are shown for WebDAV (production default transfer method) and root. If you need to create the target directory in webdav, use the -p option. Examples below:
  • env -i X509_USER_PROXY=${X509_USER_PROXY} gfal-copy -pf filename davs://cmseos.fnal.gov:9000//eos/uscms/store/user/username/newdirectory/filename
  • eval `scram unsetenv -sh` gfal-copy -pf filename davs://cmseos.fnal.gov:9000//eos/uscms/store/user/username/newdirectory/filename
  • eval `scram unsetenv -sh` gfal-copy -pf filename root://cmseos.fnal.gov//eos/uscms/store/user/username/newdirectory/filename
• Note the error for trying this in a CMSSW environment is: /usr/bin/gfal-copy: line 28: /usr/bin/python: No such file or directory
• Alternately: The copyfiles.py script can be used to copy single files or a directory of files using gfal-copy or xrdcp from another site to T3_US_FNALLPC (note: gsiftp is depreciated)
• Note: The GetSiteInfo.py script can be useful to get the information of the site's endpoint to obtain a single file through gfal-copy, it is used by the copyfiles.py script above

File access with Xrootd

The WorkBookXrootdService twiki describes how to access files using Xrootd