U.S. CMS
Search
uscms.org  www 

Using EOS at the LPC

Users with computing accounts at the LPC have (or using Service Now, can request) an EOS area to store analysis files. Despite EOS being mounted on the LPC interactive nodes, users must interact with files on EOS using either EOS-dedicated or xrootd commands, depending on the needs. This page contains most of the information relevant to the use of EOS at the LPC.


How do I check if I have an EOS area?

Login to cmslpc-sl6.fnal.gov and issue the following command (in blue) - note that you need to use your CERNUsername for CRAB jobs (it may be different from your FNAL username):

[username@cmslpc37 ~]$ eosls -d /store/user/CERNUsername
username

a single line with your own username will indicate that you indeed have an EOS area. If your output instead looks like:

[username@cmslpc37 ~]$ eosls -d /store/user/CERNUsername
Unable to stat /eos/uscms/store/user/CERNUsername; No such file or directory (errc=2)
(No such file or directory)

then your EOS area is not configured to write to your CERNusername. Note that FNALUsername areas are made by default, but you may still need to have your grid certificate properly mapped in GUMS (the cmslpc cluster grid authentication system).

You can also test your EOS area with a crab checkwrite:

How to get an EOS area created or linked to CERNusername

You can request an EOS area by filling a simple form called "CMS Storage Space Request" on Service Now, using your services account to login (Show me how - [β]). Select "Create", and fill out the form, your DN is the line following "identity" as the result of voms-proxy-info after authenticating your grid certificate on a linux system. Your "hypernews name" is your CERNusername. Note: Once your ServiceNow request is done, if it required mapping your grid certificate, it may take 2-3 hours to propagate to all nodes.


How to change a grid certificate mapped to existing EOS area

Presuming you have an EOS area created with the correct CERNusername, but your grid certificate is not allowed to write to it you can request a GUMS mapping. GUMS is the cmslpc grid certificate authentication system. Note that the ServiceNow GUMS mapping ticket category is broken. For this we use the "CMS Storage Space Request" ticket on Service Now, using your services account to login (Show me how - [β]). Select "Create", and fill out the form, your DN is the line following "identity" as the result of voms-proxy-info after authenticating your grid certificate on a linux system. Your "hypernews name" is your CERNusername. Note: Once your ServiceNow request is done, if it required mapping your grid certificate, it may take 2-3 hours to propagate to all nodes.


EOS command error notes:

  • Certain versions of CMSSW environment (CMSSW_9_2_4) may cause problems with the commands, giving an error such as:
    [username@cmslpc25 ~]$ eosls /store/user/CERNusername
    eos: symbol lookup error: eos: undefined symbol: _ZN5XrdCl4File4OpenERKSsNS_9OpenFlags5FlagsENS_6Access4ModeEt
    • Solve this error by doing the commands in a new terminal without a CMSSW environment. This should not affect xrdcp commands.
  • If you changed shell by hand, the eos aliases (eosls, eosquota, etc.) are not loaded.

  • Condor batch jobs at the cmslpc do not have access to the interactive /usr/bin/eos commands, so commands such as xrdcp root://cmseos.fnal.gov or xrdfs root://cmseos.fnal.gov ls should be used within batch jobs

  • CMSSW_7_4_7 gives the following when trying to xrdcp at the end of a condor job to EOS:
    • xrdcp: symbol lookup error: xrdcp: undefined symbol: _ZN14EnvInitializerC1Ev
    • The solution is to move to a newer CMSSW for the xrdcp file transfer step, such as CMSSW_8_0_25 or CMSSW_9_2_3

How much space do I have on EOS?

EOS areas are created with a default 2 TB logical quota. However, files stored on EOS are by default replicated and because of that, the physical quota, or the real amount of space needed for files on EOS, is twice as much, or 4 TB by default.

However, although files are replicated by default, some areas can be setup in such way that the content in them have no replica. The total amount of space needed to store user's files will then be calculated as the amount of space used by non-replicated files plus twice the amount of space used by replicated ones (one for the original and one of the replica).




The system will look at the total amount of space needed for all the files owned by you stored on personal areas and compare that against the physical quota to decide whether there is still space available or not. The amount of space needed for the combination of replicated and non-replicated files on EOS can never go beyond the physical quota. As you are reviewing files and usage, there are a number of advanced EOS commands documented on a second web page, including scripts to find space usage, commands to find files older than a year, and similar tools.

EOS quota extensions can be requested by filling the ServiceNow form: "CMS Storage Space Request"(for creating an EOS area, getting your grid certificate properly mapped, or increase in EOS quota) - using your services account to login (Show me how - [β]).


How do I check the status of the space of EOS areas?

Personal areas

To check the space status of your personal EOS area, login to cmslpc-sl6.fnal.gov and issue the following command (in blue):

[jjesus@cmslpc37 ~]$ eosquota

==> Quota Node: /eos/uscms/store/user/
user used bytes logi bytes used files aval bytes aval logib aval files filled[%] vol-status ino-status
jjesus 17.88 GB 8.94 GB 98.06 k- 4.00 TB 2.00 TB 500.00 k- 0.45 ok ok
==> Quota Node: /eos/uscms/store/user/lpctau/
user used bytes logi bytes used files aval bytes aval logib aval files filled[%] vol-status ino-status
jjesus 13 B 13 B 1 - 0 B 0 B 0 - 100 ignored ignored

Note that the output has been formatted for clarity (lines for columns have been added, click here for an example of the real output you will see).

You are interested in the line in boldface (third row on the first table), as that is reporting the information corresponding to /eos/uscms/store/user/, which is your personal EOS area. As long as used bytes (second column) remains below aval bytes (fifth column), you can still add more files to your area. However, it is also important to note that there is also a limit on the number of files that can be stored. The maximum number of files is reported as aval files (seventh column) and the number of files already stored is shown under used files (fourth column). A quick summary is shown in the last two columns vol-status and ino-status, representing the current situation for space and number of files with the following meanings:

  • ok: current usage is below 90%,
  • warning: current usage is between 90% and below the quota, and
  • exceeded: current usage has reached the maximum limit and no more information can be added.

A user can only check his/her personal status and not other's from the command line.

LPC Physics Group areas

Similarly, to check the space status of a LPC physics group EOS area, login to cmslpc-sl6.fnal.gov and issue the following command (in blue):

[jjesus@cmslpc37 ~]$ eosgrpquota lpctau

==> Quota Node: /eos/uscms/store/user/lpctau/
user used bytes logi bytes used files aval bytes aval logib aval files filled[%] vol-status ino-status
jjesus 13 B 13 B 1 - 0 B 0 B 0 - 100 ignored ignored
==> Quota Node: /eos/uscms/store/user/lpctau/
group used bytes logi bytes used files aval bytes aval logib aval files filled[%] vol-status ino-status
us_cms 3.88 TB 1.94 TB 326.23 k- 20.00 TB 10.00 TB 1.00 M- 19.40 ok ok

Again, note that the output has been formatted for clarity (lines for columns have been added, click here for an example of the real output you will see).

In this case, you are interested in the line in boldface (third row of the second table), as that is reporting the information corresponding to /eos/uscms/store/user/lpctau/, which is the EOS area for that specific LPC physics group.

LPC Physics Group areas have essentially the same functionality as personal areas, so this table should be interpreted just as the one for a personal area (see explanation on Personal areas above). For more about joining LPC Physics Groups and understanding members, see this link at the NFS disks documentation

There is one basic, but important difference though. Information stored on LPC physics group areas do not count against personal space.


Note that the LPC physics group areas are accessed through /store/user/lpcgroupname. There is a soft link existing for CRAB jobs to /store/group/ (use config.Data.outLFNDirBase = '/store/group/lpcgroupname'). If for some reason the /store/group/lpcgroupname area is not functioning and the group is within quota, contact LPC Computing Support to get this fixed.


All users and group usage on EOS

The EOS usage of all users and groups can be found at this web page (authenticate with your grid certificate). Note that the calculation is somewhat different, in that it will report 3.725 TB actual (which is TiB) instead of 4TB actual quota which the command line will report.


Interacting with files on EOS

Depending on the command being used, files or directories on EOS have to be referred to by using, in some cases, their LFN (Logical File Name, or the part of the absolute path starting with /store/) or in other cases their corresponding URL's, but in general never by their absolute paths. The URL is formed by prepending root://cmseos.fnal.gov/ to the LFN. Click on the image below to enlarge.

Each action covered on this section include in the description what to use to refer to a file or directory on EOS.

Listing

There is a dedicated EOS command to list files or directories stored on EOS areas, it uses the LFN of the file or directory of interest to find it (see the image above if you're not sure what a LFN is):

[jjesus@cmslpc37 ~]$ eosls /store/user/jjesus/
EOSFile.txt
MinBias
README
basic.C
testArea
triggers.py

The default behavior of this command produces a single-column standard output (one file or directory per line, which mimics the behavior of ls -1 of the local file system, note that the flag here correspond to the number 1 and not letter l). what This EOS command can take arguments to modify its basic fuctionality. For example, adding the -l flag will produce a long listing format:

[jjesus@cmslpc37 ~]$ eosls -l /store/user/jjesus/
-rw-r--r--   2 jjesus   us_cms             13 Dec 16 10:24 EOSFile.txt
drwxr-sr-+   1 jjesus   us_cms              1 Jan 08 14:59 MinBias
-rw-r--r--   2 jjesus   us_cms            433 Dec 02 14:36 README
-rw-r--r--   2 jjesus   us_cms            265 Dec 03 13:53 basic.C
drwxr-sr-+   1 jjesus   us_cms              1 Dec 03 10:48 testArea
-rw-r--r--   2 jjesus   us_cms            842 Dec 09 09:42 triggers.py

while adding the -d flag will list directory entries instead of their content:

[jjesus@cmslpc37 ~]$ eosls -d /store/user/jjesus/
jjesus

The output of this command is just the name of the directory and nothing else. Adding the -a flag will show hidden entries as well:

[jjesus@cmslpc37 ~]$ eosls -a /store/user/jjesus/
.
..
EOSFile.txt
MinBias
README
basic.C
testArea
triggers.py

Any combination of the above flags is allowed. The use of multiple flags together (-ld), or separate (-l -d) yields the same result.

Listing by using simple ls with the absolute path to files or directories on EOS is absolutely discouraged as it can significantly reduce the performance of the interactive node. Furthermore, doing so from a batch job is a BIG NO.

There are few things you need to be aware when listing files on EOS areas.

  • Wildcards. The use of wildcards is not recognized by EOS commands. A '*' included somewhere will be interpreted in general as a part of the name and therefore will return an error. For example:
  • [jjesus@cmslpc37 ~]$ eosls /store/user/jjesus/*
    eos: No match.
    

    The same is true for '?'.
  • Tab completion. The EOS listing command uses the LFN, which starts with /store. Tab completion is not possible as the local filesystem will misinterpret the LFN as a local path starting with /store which does not exist.
  • The eos commands are only available on the cmslpc interactive nodes. For condor batch jobs use commands such as xrdcp outputfile.root root://cmseos.fnal.gov//store/username/outputfile.root or xrdfs root://cmseos.fnal.gov ls /store/username

Other tools for listing

EOS is a storage filesystem built on the XRootD framework. Because of that, some of the XRootD commands can be used when dealing with files on EOS, and there are some useful ways to list things with XRootD commands in a way that EOS dedicated commands are not able to do. Consult the CMS WorkBook on Xrootd for more on Xrootd.

To get directory listing on XRootD we use xrdfs (which is its file and directory meta-data utility), together with the ls command:

[jjesus@cmslpc37 ~]$ xrdfsls /store/user/jjesus/
[jjesus@cmslpc37 ~]$ xrdfs root://cmseos.fnal.gov ls /store/user/jjesus/
/store/user/jjesus/EOSFile.txt
/store/user/jjesus/MinBias
/store/user/jjesus/README
/store/user/jjesus/basic.C
/store/user/jjesus/testArea
/store/user/jjesus/triggers.py

The default behavior of this command will produce a single-column with one LFN per row, while adding the -u flag:

[jjesus@cmslpc37 ~]$ xrdfsls -u /store/user/jjesus/
[jjesus@cmslpc37 ~]$ xrdfs root://cmseos.fnal.gov ls -u /store/user/jjesus/
root://131.225.207.127:1094//store/user/jjesus/EOSFile.txt
root://131.225.207.127:1094//store/user/jjesus/MinBias
root://131.225.207.127:1094//store/user/jjesus/README
root://131.225.207.127:1094//store/user/jjesus/basic.C
root://131.225.207.127:1094//store/user/jjesus/testArea
root://131.225.207.127:1094//store/user/jjesus/triggers.py

will print URL's instead. The two xrdfs options for listing can be useful in workflows where either the LFN or the URL of a file or directory on EOS is needed rather than just the simple name. The xrdfsls command is an alias available on the interactive but not condor batch envrionment.

Copying

In order to copy files into, out-of or between EOS areas you need to use another XRootD command: xrdcp. In this case, EOS locations need to be included as URL's. There are three possible scenarios:

  • Local file to EOS:

[jjesus@cmslpc37 ~]$ xrdcp ~/test.txt root://cmseos.fnal.gov//store/user/jjesus/copyOfLocalFile.txt
[13B/13B][100%][==================================================][0B/s]

  • EOS to local file:

[jjesus@cmslpc37 ~]$ xrdcp root://cmseos.fnal.gov//store/user/jjesus/EOSFile.txt ~/copyOfEOSFile.txt
[13B/13B][100%][==================================================][0B/s]

  • and finally, EOS to EOS:

[jjesus@cmslpc37 ~]$ xrdcp root://cmseos.fnal.gov//store/user/jjesus/EOSFile.txt \
? root://cmseos.fnal.gov//store/user/jjesus/EOSFile1.txt
[13B/13B][100%][==================================================][13B/s]
    

xrdcp will assume by default that the destination file does not exist. If the destination file already exist, xrdcp will produce an error message like:

[jjesus@cmslpc37 ~]$ xrdcp ~/test.txt root://cmseos.fnal.gov//store/user/jjesus/copyOfLocalFile.txt
[0B/0B][100%][==================================================][0B/s]
Run: [ERROR] Server responded with an error: [3006] Unable to create file
/eos/uscms/store/user/jjesus/copyOfLocalFile.txt; File exists

Overwriting an existing file can be achieved by adding the -f flag to xrdcp:

[jjesus@cmslpc37 ~]$ xrdcp -f ~/test.txt root://cmseos.fnal.gov//store/user/jjesus/copyOfLocalFile.txt
[13B/13B][100%][==================================================][0B/s]

By now you may have noticed that xrdcp will show by default a text version of a progress bar: [====] (it may be more noticeable on big files because of the time it takes to transfer them). While this is a good way to visualize the progress of this operation, there are some situations in which you may not want such information. Here is an example: a HTCondor job using xrdcp to transfer a file will include the updates on the progress bar in the stderr file for the job, generating non-zero size error files even when the job is completely ok. Here is a typical stderr for a file that is exactly 1,000,000,000 B copied using xrdcp inside a HTCondor job that doesn't give any error:

[jjesus@cmslpc37 myWorkArea]$ more myJob_cluster_process.stderr
[96MB/953.7MB][ 10%][=====>                                            ][19.2MB/s 
[128MB/953.7MB][ 13%][======>                                           ][21.33MB/s
[160MB/953.7MB][ 16%][========>                                         ][22.86MB/s
[176MB/953.7MB][ 18%][=========>                                        ][22MB/s]  
[208MB/953.7MB][ 21%][==========>                                       ][23.11MB/s
[240MB/953.7MB][ 25%][============>                                     ][24MB/s]  
[256MB/953.7MB][ 26%][=============>                                    ][23.27MB/s
[288MB/953.7MB][ 30%][===============>                                  ][24MB/s]  
[304MB/953.7MB][ 31%][===============>                                  ][23.38MB/s
[320MB/953.7MB][ 33%][================>                                 ][22.86MB/s
[352MB/953.7MB][ 36%][==================>                               ][23.47MB/s
[384MB/953.7MB][ 40%][====================>                             ][24MB/s]  
[400MB/953.7MB][ 41%][====================>                             ][23.53MB/s
[416MB/953.7MB][ 43%][=====================>                            ][23.11MB/s
[432MB/953.7MB][ 45%][======================>                           ][22.74MB/s
[448MB/953.7MB][ 46%][=======================>                          ][22.4MB/s]
[464MB/953.7MB][ 48%][========================>                         ][22.1MB/s]
[496MB/953.7MB][ 52%][==========================>                       ][22.55MB/s
[528MB/953.7MB][ 55%][===========================>                      ][22.96MB/s
[544MB/953.7MB][ 57%][============================>                     ][22.67MB/s
[560MB/953.7MB][ 58%][=============================>                    ][22.4MB/s]
[576MB/953.7MB][ 60%][==============================>                   ][22.15MB/s
[592MB/953.7MB][ 62%][===============================>                  ][21.93MB/s
[608MB/953.7MB][ 63%][===============================>                  ][21.71MB/s
[640MB/953.7MB][ 67%][=================================>                ][22.07MB/s
[672MB/953.7MB][ 70%][===================================>              ][22.4MB/s]
[688MB/953.7MB][ 72%][====================================>             ][22.19MB/s
[720MB/953.7MB][ 75%][=====================================>            ][22.5MB/s]
[736MB/953.7MB][ 77%][======================================>           ][22.3MB/s]
[768MB/953.7MB][ 80%][========================================>         ][22.59MB/s
[800MB/953.7MB][ 83%][=========================================>        ][22.86MB/s
[816MB/953.7MB][ 85%][==========================================>       ][22.67MB/s
[832MB/953.7MB][ 87%][===========================================>      ][22.49MB/s
[864MB/953.7MB][ 90%][=============================================>    ][22.74MB/s
[896MB/953.7MB][ 93%][==============================================>   ][22.97MB/s
[912MB/953.7MB][ 95%][===============================================>  ][22.8MB/s]
[928MB/953.7MB][ 97%][================================================> ][22.63MB/s
[944MB/953.7MB][ 98%][=================================================>][22.48MB/s
[953.7MB/953.7MB][100%][==================================================][22.71MB
[953.7MB/953.7MB][100%][==================================================][22.71MB/s]

To avoid this, remember to add the -s flag to require xrdcp to be silent, for example:

[jjesus@cmslpc37 myWorkArea]$ xrdcp -s root://cmseos.fnal.gov//store/user/jjesus/test.txt .

copies the file from EOS to a local area without showing anything.

You can combine these flags into a single one, -fs, or use them separate, -f -s, with the same result.

Creating directories

In order to create a directory on an EOS area interactively, you need to use the dedicated EOS aliased command eosmkdir and the LFN of the directory to be created:

[jjesus@cmslpc37 ~]$ eosmkdir /store/user/jjesus/newDir

In case of success, this command doesn't produce any standard output. Note: all parent directories leading to the new one should exist for this command to succeed.

Adding the -p flag will make parent directories as needed:

[jjesus@cmslpc37 ~]$ eosmkdir -p /store/user/jjesus/newDir1/newDir2/newDir3

In case of success, this command doesn't produce any standard output and no error is shown if the parents already exist.

In condor batch jobs, the eos commands and aliases are not available and you will need to use xrdfs root://cmseos.fnal.gov type commands instead.

Removing files and directories

First thing to remember when removing files from EOS: a file removed from EOS is gone for good. Second thing to remember when removing files from EOS: a replica is not the same as a backup, the concept of backup does not exist on EOS. Knowing that can make you understand a bit more why wildcards can't be used on EOS, it can be seen as an extra layer of protection against accidental removal of files in bulk. You've been warned.

In order to remove files from an EOS area, you need to use the dedicated EOS command rm and the LFN of the file to be removed:

[jjesus@cmslpc37 ~]$ eosrm /store/user/jjesus/EOSFile.txt

This command doesn't produce any standard output.

There is a dedicated EOS command to remove directories, but it only works when the directory is empty. Instead of using that, you can use the extension of the EOS command to remove files that allows the removal of directories whether they are empty or not. To do so, you have to add the -r flag:

[jjesus@cmslpc37 ~]$ eosrm -r /store/user/jjesus/dir

As in the case for files, this command doesn't produce any standard output. Trying to remove a directory without the -r flag, will result in an error like:

[jjesus@cmslpc37 ~]$ eosrm /store/user/jjesus/dir
error: unable to remove file/directory '/eos/uscms/store/user/jjesus/dir'
(errc=21) (Is a directory)


Working with ROOT files on EOS

Access to ROOT files on EOS should be done in general by using a redirector, for example: root://cmsxrootd.fnal.gov. Xrootd file access via cmsxrootd.fnal.gov should include getting your grid certificate proxy first. If you do not have a grid certificate installed, consult this link to obtain and install a grid certificate. Authenticate your proxy:

[username@cmslpc37 ~]$ voms-proxy-init -voms cms

The redirector will look for the file at Fermilab first, if not found, it will continue looking at other sites until the file is found (provided it actually exists on a valid CMS grid Storage Element).

The string /eos/uscms should never appear while opening a ROOT file stored on EOS. If it does, the file will open using the FUSE mount, which in general has a negative impact on the performance of the interactive node, degrading the experience collectively for all the users connected to the same node. On the worker nodes, /eos/uscms won't be mounted any more, therefore, jobs making use of the FUSE mount to open ROOT files on EOS will simply fail.

Interactive access

From the command line on any interactive LPC node, (on any location on the local filesystem, usually somewhere inside /uscms_data, never from /eos/uscms) a ROOT file can be directly open after setting the proper environment in the following way:

[jjesus@cmslpc37 ~]$ root -l root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root
root [0]
Attaching file root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root as _file0...
(class TFile *) 0x3edd370

Once open, you can navigate through your file and work with it as usual, for example:

root [1] .ls();
TNetXNGFile**		root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root
 TNetXNGFile*		root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root
root [2] TH1F *theHist = (TH1F*)_file0->Get("nPU"); 
root [3] TCanvas *theCanvas = new TCanvas("theCanvas", "Pile-Up", 640, 480);
root [4] theHist->Draw();
root [5] theCanvas->SaveAs("thePlot.png");
Info in < TCanvas::Print >: png file thePlot.png has been created
root [6] .q

to load a histogram from the ROOT file previously open, and create the following PNG file out of it:

If you need to open a file from inside an already initiated ROOT session, you can do the folloiwng:

[jjesus@cmslpc37 ~]$ root -l
root [0]
.
Some ROOT directives
.
root [10] TFile *theFile = TFile::Open("root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root");
root [11] .ls();
TNetXNGFile**		root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root
 TNetXNGFile*		root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root

Hint: If you ever need a quick reminder about the proper way to open a ROOT file after you have initiated an interactive ROOT session, a quick way is just to open another file (on a different terminal or your local machine, for example). Once inside ROOT, hit the upwards arrow on your keyboard, and you will be presented with the line that ROOT used to open the file you indicated from the command line, ROOT appends that by default to the history of the current session (Show me how - [β]).

Inside a script

Existing EOS files that need to be open as readable inside scripts, should be open with the constructor shown in the previous section and using their URL's. For writing EOS Files from batch jobs, see below.

Inside a C++-like script

On a C++-like script this would look just like accessing the file interactively:

#include <iostream>
.
.
#include "TFile.h"
.
.
int main(int argc, const char * argv[]) {
    Some code here
    //  Proper way to open an EOS ROOT file:
    TFile *theFile = TFile::Open("root://cmsxrootd.fnal.gov//store/user/username/rootFile.root");
    Rest of code
    return 0;
}

Inside a pyROOT script

On a pyROOT script this would look just like accessing the file interactively through xrootd (note the .Open is needed):

import ROOT
import sys
# Some code here
#  Proper way to open an EOS ROOT file:
f = ROOT.TFile.Open("root://cmsxrootd.fnal.gov//store/user/username/rootFile.root")
# Rest of code

Inside a CMSSW python config file

On a CMSSW python config file this would be easiest letting CMSSW use /store/user:

process.source = cms.Source("PoolSource",
fileNames = cms.untracked.vstring('/store/user/username/myfile.root')
)

Or specify the xrootd redirector specifically root://cmsxrootd.fnal.gov/:

process.source = cms.Source("PoolSource",
fileNames = cms.untracked.vstring('root://cmsxrootd.fnal.gov//store/user/username/myfile.root')
)

Writing EOS files

Do not write directly to EOS from ROOT, or keep a file open writing to it. Instead, write to local storage, whether NFS disk interactively, or the local machine your job is running on (condor batch). Once completed, xrdcp the output.

The preferred methods to write to EOS from a batch job are:

  • Through crab3 jobs, which stage the output automatically to EOS at the end of the job when configured to output to T3_US_FNALLPC
  • Within condor batch jobs, transferring to EOS from the local worker machine at the end of the job, through xrdcp transfers, described earlier on this page.

Merge files with histograms and trees

Multiple ROOT files stored on EOS can be merged from the command line on any interactive LPC node (from any location on the local filesystem, usually somewhere inside /uscms_data, never from /eos/uscms). Note: CMSSW EDM files cannot be hadded, one should use the CMSSW CopyPickMerge tool instead. For root ntuples: after setting the proper environment, hadd can be used by referring to the list of files to be merged by their corresponding URLs. The list can be easily obtained by using the -u option of the xrdfs tool for listing (as explained above). For example, from a CMSSW relase area located inside /uscms_data/d3/jjesus/, after cmsenv one can do:

[jjesus@cmslpc37 ~/nobackup]$ hadd myTarget.root `xrdfsls -u /store/user/jjesus/rootFiles | grep '\.root'`
hadd Target file: myTarget.root
hadd Source file 1: root://131.225.207.127:1094//store/user/jjesus/rootFiles/rootFile0.root
hadd Source file 2: root://131.225.207.127:1094//store/user/jjesus/rootFiles/rootFile1.root
hadd Source file 3: root://131.225.207.127:1094//store/user/jjesus/rootFiles/rootFile2.root
hadd Target path: myTarget.root:/
hadd Target path: myTarget.root:/2h_cone_eta_totalpT_nPU
hadd Target path: myTarget.root:/2h_cone_eta_totalpT_nPU/all
hadd Target path: myTarget.root:/2h_cone_eta_totalpT_nPU/h
hadd Target path: myTarget.root:/2h_cone_eta_totalpT_nPU/e
.
.
.
hadd Target path: myTarget.root:/2p_cone_eta_nPU_totalpT_nPV/h_HF
hadd Target path: myTarget.root:/2p_cone_eta_nPU_totalpT_nPV/egamma_HF
hadd Target path: myTarget.root:/2p_cone_eta_nPU_totalpT_nPV/chs

which merges the three files into a single one named myTarget.root.

There are multiple parts to the line responsible for merging the files. First xrdfsls is used to obtain the list of files on the LFN given as argument, the output will be formatted as URLs instead of just LFNs by adding the -u option. Then, the result of this command is filtered by piping to a grep command which will look just for the .root file extension. This is an important step, files to be merged need to be ROOT files and have the same structure. If the directory where the the files are stored contain other files, trying to use hadd without filtering will fail when trying to initialize a non-ROOT file, exiting due to the error. Note: if the ROOT files don't have the same structure, hadd will still create a target file, it will most likely show warning and error messages and the final product will probably be not precisely what it is supposed to be. Finally, as this command is enclosed inside grave accent marks (`), this will be executed first and the output will be put in place at the end of the hadd command as its last argument.

You can xrdcp your output file from NFS ~/nobackup disk back to EOS:

[username@cmslpc38 ~/nobackup]$ xrdcp myTarget.root root://cmseos.fnal.gov//store/user/username/myTarget.root

[3.267GB/3.267GB][100%][==================================================][101.4MB/s] 

Things you shouldn't do

When listing

Do not use regular ls to list EOS areas over the FUSE mount:

                     - - - DON'T DO THIS - - -
[jjesus@cmslpc37 ~]$ ls /eos/uscms/store/.../
                     - - - DON'T DO THIS - - -

This is particularly bad for areas with large number of files. This is particularly bad when using wildcards: *, ?, etc. Do not do it on any interactive node and NEVER do that in a job. See Interacting with files on EOS on this page to find the correct way to do this.

When copying

Do not use regular cp to copy EOS files or directories over the FUSE mount:

                     - - - - - - - - - - - DON'T DO ANY OF THIS - - - - - - - - - - -
[jjesus@cmslpc37 ~]$ cp /eos/uscms/store/.../sourceFile /some/local/area/destFile
[jjesus@cmslpc37 ~]$ cp /some/local/area/sourceFile /eos/uscms/store/.../destFile
[jjesus@cmslpc37 ~]$ cp /eos/uscms/store/.../sourceFile /eos/uscms/store/.../destFile
                     - - - - - - - - - - - DON'T DO ANY OF THIS - - - - - - - - - - -

This is particularly bad when using wildcards, trying to copy large number of files or recursively. Do not do it on any interactive node and NEVER do that in a job. See Interacting with files on EOS on this page to find the correct way to do this.

When removing

Do not use regular rm to remove files or directories over the FUSE mount:

                     - - - - - - - - - - - - - DON'T DO THIS - - - - - - - - - - - - -
[jjesus@cmslpc37 ~]$ rm /eos/uscms/store/.../fileToRemove
                     - - - - - - - - - - - - - DON'T DO THIS - - - - - - - - - - - - -

When reading files

Do not run a script interactively from eos over the FUSE mount (copy it to NFS disk and run it from there):

                     - - - - - - - - - - - - - DON'T DO THIS - - - - - - - - - - - - -
[username@cmslpc37 ~]$ python /eos/uscms/store/username/myCoolScript.py
                     - - - - - - - - - - - - - DON'T DO THIS - - - - - - - - - - - - -

Do not use /eos/uscms

The string /eos/uscms should never appear while opening a ROOT file stored on EOS. If it does, the file will open using the FUSE mount, which in general has a negative impact on the performance of the interactive node, degrading the experience collectively for all the users connected to the same node. On the worker nodes, /eos/uscms won't be mounted any more, therefore, jobs making use of the FUSE mount to open ROOT files on EOS will simply fail.

Do not write to EOS without xrdcp or crab3

Do not keep a file open while writing to EOS, for example: piping output to EOS, (RE)CREATE file in ROOT, or hadd directly to EOS.

Do copy to EOS once your file is made locally using xrdcp or crab3.

Advanced/Additional EOS commands and links


This second web page on Advanced EOS commands at the CMS LPC gives some useful EOS links, scripts, and more advanced command line options to help you find, count, and organize your files on EOS.

Webmaster | Last modified: Monday, 30-Oct-2017 16:03:56 CDT