uscms.org  www 

User Space at US CMS Tier-2 Centers and the FNAL LPC


When running jobs on large amounts of data, many users will need to write output files of substantial size -- larger than can be accommodated by the grid output sandbox. Additionally, it can be very useful to write this output in a way that it can be read back in easily by subsequent grid jobs. CMS has a mechanism for this. Computing sites will support a /store/user area in their storage systems to host user-generated data. CRAB can be instructed to write job outputs to this area through the SRM interface. Through the magic of the grid, no matter where your job actually runs, and no matter where the output is created, the output can be automatically transferred to whatever system is hosting your own /store/user area. Then, CRAB can also publish this data into a local-scope DBS database. Subsequent CRAB jobs can then reference the local-scope DBS, find the user-created data, and send jobs to the site that is hosting it.

US CMS has provisioned the Tier-2 computing systems and the FNAL LPC cluster to provide a 1 TB quota of /store/user space to every US-based user who is actively involved in data analysis. Note that this space is not backed up in any way, so use it with care. This page contains information on how users can make use of this space.

Complete documentation on using the distributed computing system can be found in Chapter 5 of the CMS Workbook.

Assignment of users to computing sites

Which site (Storage Element) hosts your own /store/user area? Users are grouped by institute, and then institutes are matched up with a Tier-2 site or the LPC on the basis of geography and capacity. The LPC has about 42% of the total available resources, and thus hosts the largest number of users. Each of the seven Tier-2 sites hosts about 8% of the remaining users. Leaders of the institutes are responsible for contacting the sites to establish accounts. Note that your institute may also have a Tier3 with associated storage space, contact your institute leader for more information. Institute assignments are as follows:

  • Caltech (Contact: Steven Lo): Caltech, Livermore, UC Davis, UCLA
  • Florida (Contact: Bockjoo Kim): FIU, Florida State, Florida Tech, Florida, Puerto Rico
  • MIT (Contact: Max Goncharov): Boston U., Brown, Fairfield, MIT, Northeastern
  • Nebraska (Contact: Ken Bloom): Colorado, Iowa, Kansas State, Kansas, Nebraska
  • Purdue (Contact: Fengping Hu): Carnegie-Mellon, Ohio State, Purdue, SUNY-Buffalo, Vanderbilt
  • UCSD (Contact: T2 support team): UC Riverside, UCSB, UCSD
  • Wisconsin (Contact: Ajit Mohapatra): Minnesota, Rochester, Wayne State, Wisconsin
  • FNAL (Get FNAL account; Get EOS area): All remaining institutes

To get /store/user space at their associated site, users must provide a) their Hypernews (HN) user name, which will be used as the name of their directory in /store/user and b) the distinguished name (DN) from their grid certificate, which will be used to verify that the user is allowed to write into that area. The DN can most easily be obtained by issuing the voms-proxy-info command from wherever you do your grid submissions.

Writing to your /store/user space from a CRAB job

Consult the SWGuideCrab Workbook for the most up-to-date documentation on techniques to write output from a CRAB job

You must specify config.Site.storageSite, which will depend on which center is hosting your area, and user_remote_dir which is the subdirectory of /store/user/ you want to write to.

  • Caltech storage_element = T2_US_Caltech
  • Florida storage_element = T2_US_Florida
  • MIT storage_element = T2_US_MIT
  • Nebraska storage_element = T2_US_Nebraska
  • Purdue storage_element = T2_US_Purdue
  • UCSD storage_element = T2_US_UCSD
  • Wisconsin storage_element = T2_US_Wisconsin
  • FNAL storage_element = T1_US_FNAL_Buffer

Accessing /store/user files through CRAB

This is described fairly well in a twiki page on publishing data using CRAB.

Then, to run on our output files with a new CRAB job, you need to specify the new dataset name, and the URL for the local-scope DBS.

Accessing /store/user through Xrootd

This is perhaps the easiest way to access your files in /store/user for interactive use, including reading them directly into ROOT. You don't even need to know the physical location of your file, just the path in the /store/user area. See this page in the CMS WorkBook for all the details.

Accessing /store/user by other (less convenient) means

You can copy files to your local machine via gfal-copy. Note that you need a valid CMS grid proxy to do this operation. More about this technique is at the data transfer page.

Webmaster | Last modified: Tuesday, 29-Aug-2017 10:42:43 CDT