U.S. CMS
Search
uscms.org  www 

As of 2017, these methods below are no longer working, in 10 seconds: redirecting to File transfer page for up to date methods

Note, contents below last editing March 2014. DataViewer is decommissioned, and many of the scripts below depend on old SL5 setup scripts which no longer exist.

FTS User tools

Purpose

The purpose of the ftsuser tools is to facilitate the transfer of files between dCache and EOS using the FTS system. We have set up a special FTS machine dedicated to users for these transfers. If you have large amounts of data to move, you may also contact cms-t1 for help. Transfers are to be done from the cmslpc machines at FNAL.

With the changes to our storage and tape systems, the new system has less files from tape already staged to disk and the space to maintain those files for long periods of time. It is recommended that if you want to guarantee availability of files while working, you should move them to EOS and work from there.

Files on dCache resilient are also no longer going to be available when the system changes are complete. To preserve files in this space, we recommend moving them to EOS.

Requirements

You will need a cmslpc account because the tools are only available from the cmslpc interactive nodes.

You will need a valid proxy to do the transfers.

You may also want to check your quota on EOS. See the quota section below for details and how to view your current quota.

Steps

1) Create a proxy before initiating the transfers. It is required to have a valid cms proxy:

voms-proxy-init -voms cms -valid 192:0

2) Create the delegation. This is required to do the transfers:

ftsuser-delegation

Delegates a new proxy to the FTS server for transfers. Returns the delegation id and expiration time for the proxy.

Usage: /usr/local/bin/ftsuser-delegation
-h, --help: shows this help message


3) Prepare a list of the LFNs of the dcache and dos space to provide to the transfer commands. Run the following command to get a list of files in the dCache area:

ftsuser-build-xfer-list -p resilient -o transfer-list.txt username substring dest-path

Usage: ftsuser-build-xfer-list [options] username substring dest-path
username: your username (i.e. burt)
substring: word to filter files by (i.e. ZMuMu )
dest-path: destination prefix path under /eos/uscms/store/user/username (i.e. the 'dest-path' part of /store/user/burt/dest-path)

options:
-h, --help show this help message and exit
-o OUTPUT, --output=OUTPUT
write output to file
-p PREFIX, --prefix=PREFIX
read from prefix (choices: resilient, storeuser)
(default: storeuser)

This will create a text file of the files you actually want to transfer from the output above. You should have one line for each file. The first item in the line is the FROM location on dCache and the second item in the line is the TO destination location on EOS.

4) Submit the transfer:


ftsuser-transfer-submit-list

Submits all files listed in a given file to FTS to transfer from EOS or dCache. If no destination is provided, the destination will match the source.

Usage: /usr/local/bin/ftsuser-transfer-submit-list FROMDCACHE|FROMEOS list-of-LFNs
-h, --help: shows this help message
example usage: ftsuser-transfer-submit-list FROMDCACHE mylist
example usage: ftsuser-transfer-submit-list FROMEOS mylist
where mylist is a file containing either sourceLFNs or pairs of sourceLFN destinationLFN
example for mylist: /resilient/catalind/A.txt /store/user/catalind/test52342.txt

Other useful commands to manage transfers

ftsuser-transfer-cancel

Cancels a file transfer job.

Usage: /usr/local/bin/ftsuser-transfer-cancel fts-job-id
-h, --help: shows this help message
example usage: user-transfer-cancel 3282b87c-4be0-11e3-ab7c-782bcb3c8ee8


ftsuser-transfer-list

Lists current FTS jobs with their status. Can also get status for a given job id.

Usage: /usr/local/bin/ftsuser-transfer-list [fts-job-id]
-h, --help: shows this help message
example usage: ftsuser-transfer-list 3282b87c-4be0-11e3-ab7c-782bcb3c8ee8


ftsuser-whoami

Shows proxy info if a valid proxy is found.

Usage: /usr/local/bin/ftsuser-whoami
-h, --help: shows this help message

ftsuser-transfer-submit-file

Submits a file transfer to FTS either from FNAL EOS or dCache. Destination LFN is optional, if not specified will default to the source LFN. The destination filename will match the source filename.

Usage: /usr/local/bin/ftsuser-transfer-submit-file FROMDCACHE|FROMEOS source-LFN [destination-LFN]
-h, --help: shows this help message
example usage: ftsuser-transfer-submit FROMDCACHE /resilient/catalind/A.txt /store/user/catalind/test52342.txt
example usage: ftsuser-transfer-submit FROMEOS /store/user/catalind/test52342.txt

ftsuser-transfer-submit-dir

Submits all files in a directory to FTS to transfer from EOS or dCache. This is not a recursive action, only the files in the immediate directory will be transferred. If you have a set of nested files and directories, each directory will have to be transferred separately.

If no destination is provided, the destination will match the source.

Usage: /usr/local/bin/ftsuser-transfer-submit-dir FROMDCACHE|FROMEOS source-LFN [destination-LFN]
-h, --help: shows this help message
example usage: ftsuser-transfer-submit-directory FROMDCACHE /resilient/catalind/testdir /store/user/catalind/
example usage: ftsuser-transfer-submit-directory FROMEOS /store/user/catalind/testdir

Monitoring

This system offers a web based interface to monitor transfers:

https://cmsfts3-users.fnal.gov:8449/fts3/ftsmon/

To filter which transfers are displayed on the monitoring pages, click the Filter button. To see only your transfers, select your DN from the dropdown list. You may also need to modify the Time window based on when you did your transfers. By default it only shows transfers in the last 12 hours.

Quotas

Quotas from dCache aren't necessarily the same on EOS. By default, all users are given 2TB of space on EOS unless they request an increase:

NOTE: If you have space on dCache you will already have space on EOS but the quotas do not transfer. It will be difficult to get large quotas on EOS that you may have had on dCache. Please only transfer and maintain the files you need at a given time to EOS.


An Example

Say I want to copy some of my data out of my resilient area into EOS. It lives in /pnfs/cms/WAX/resilient/dmason/eleventhgrid. I want to put it into /eos/uscms/store/dmason/examplecopy. I do this then:

[dmason@cmslpc30 ~]$ voms-proxy-init -voms cms -hours 200
Enter GRID pass phrase:
Your identity: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=dmason/CN=664436/CN=David Alexander Mason
Creating temporary proxy ........................................ Done
Contacting lcg-voms.cern.ch:15002 [/DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch] "cms" Done
Creating proxy ........................................................ Done
Your proxy is valid until Fri Jan 24 22:28:19 2014
[dmason@cmslpc30 ~]$ ftsuser-delegation
Using proxy at /tmp/x509up_u1234
Delegation id 58d662372cf46db5:

Renewing now:
-----BEGIN CERTIFICATE REQUEST-----
MIHJMHUCAQAwEDEOMAwGA1UEChMFRHVtbXkwXDANBgkqhkiG9w0BAQEFAANLADBI
AkEA7g/QcT+sHiT6FolpCqvHJmUBTNy5qHWdVWLsSPe/pod2kuIyX6/q9ZjiDKgU
D5gXHx2e5IQOtHasdsfa2O8HMrQIDAQABoAAwDQYJKoZIhvcNAQEEBQADQQCHkimX
xoGLM9dtrW49Kn9hxKHf8fiwK1/a90NloFiSsqAHauyi0HqsjNtLk8zpumLWa3pS
S3WB57QgcIqb4EvR
-----END CERTIFICATE REQUEST-----

{
"termination_time": "2014-01-25T04:28:19"
}

Ongoing transfers can be monitored through the web interface:
https://cmsfts3-users.fnal.gov:8449/fts3/ftsmon/

Then we make the file to feed the FTS system:

ftsuser-build-xfer-list -o testcopy -p resilient dmason eleventhgrid examplecopy

This will make a file with a pair of from and to files -- you should check this:

more testcopy
/resilient/dmason/eleventhgrid_noPU/reco/reco_750_1400_225_3501.root /store/user
/dmason/examplecopy/eleventhgrid_noPU/reco/reco_750_1400_225_3501.root
/resilient/dmason/eleventhgrid_noPU/tree/tree_750_400_375_1501.root /store/user/
dmason/examplecopy/eleventhgrid_noPU/tree/tree_750_400_375_1501.root

etc etc.

Note that it isn't the full path -- these will be the LFN's FTS wants to see. But look that beyond /store/user/username, etc. is where you actually wanted to put your files, and make sure that the contents of the list actually contain what you copy. It will literally only copy what is listed.

You then feed FTS:

ftsuser-transfer-submit-list FROMDCACHE testcopy
Using proxy at /tmp/x509up_u7178


"cred_id": "58d662372cf46db5",
"job_id": "066e468a-7eed-11e3-8607-782bcb2fb7b4",
"job_state": "SUBMITTED",
"user_dn": "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=dmason/CN=664436/CN=David Alexander Mason",
"vo_name": "cms",
"voms_cred": "/cms/Role=NULL/Capability=NULL /cms/integration/Role=NULL/Capability=NULL /cms/production/Role=NULL/Capability=NULL /cms/uscms/Role=NULL/Capability=NULL"


Ongoing transfers can be monitored through the web interface:
https://cmsfts3-users.fnal.gov:8449/fts3/ftsmon/
And after a bit I'm noticing:
[dmason@cmssrv168 filecopy]$ ls /eos/uscms/store/user/dmason/examplecopy
eleventhgrid_Flat10PU eleventhgrid_noPU

Webmaster | Last modified: Wednesday, 09-Aug-2017 09:32:46 CDT