U.S. CMS
Search
uscms.org  www 

Using EOS at the LPC

Users with computing accounts at the LPC have (or using Service Now, can request) an EOS area to store analysis files. Despite EOS being mounted on the LPC interactive nodes, users must interact with files on EOS using either EOS-dedicated or xrootd commands, depending on the needs. This page contains most of the information relevant to the use of EOS at the LPC.


How do I check if I have an EOS area?

Login to cmslpc-sl6.fnal.gov and issue the following command (in blue):

[jjesus@cmslpc37 ~]$ eosls -d /store/user/`whoami`
jjesus

a single line with your own username will indicate that you indeed have an EOS area. If your output instead looks like:

[jjesus@cmslpc37 ~]$ eosls -d /store/user/`whoami`
Unable to stat /eos/uscms/store/user/jjesus; No such file or directory (errc=2)
(No such file or directory)

then you can request an EOS area by filling a simple form on Service Now, using your services account to login (Show me how - [β]).


How much space do I have on EOS?

EOS areas are created with a default 2 TB logical quota. However, files stored on EOS are by default replicated and because of that, the physical quota, or the real amount of space needed for files on EOS, is twice as much, or 4 TB by default.

However, although files are replicated by default, some areas can be setup in such way that the content in them have no replica. The total amount of space needed to store user's files will then be calculated as the amount of space used by non-replicated files plus twice the amount of space used by replicated ones (one for the original and one of the replica).




The system will look at the total amount of space needed for all the files owned by you stored on personal areas and compare that against the physical quota to decide whether there is still space available or not. The amount of space needed for the combination of replicated and non-replicated files on EOS can never go beyond the physical quota.

EOS quota extensions can be requested by filling the same form to request an EOS area on Service Now (using your services account to login) (Show me how - [β]).


How do I check the status of the space of EOS areas?

Personal areas

To check the space status of your personal EOS area, login to cmslpc-sl6.fnal.gov and issue the following command (in blue):

[jjesus@cmslpc37 ~]$ eosquota

==> Quota Node: /eos/uscms/store/user/
user used bytes logi bytes used files aval bytes aval logib aval files filled[%] vol-status ino-status
jjesus 17.88 GB 8.94 GB 98.06 k- 4.00 TB 2.00 TB 500.00 k- 0.45 ok ok
==> Quota Node: /eos/uscms/store/user/lpctau/
user used bytes logi bytes used files aval bytes aval logib aval files filled[%] vol-status ino-status
jjesus 13 B 13 B 1 - 0 B 0 B 0 - 100 ignored ignored

Note that the output has been formatted for clarity (lines for columns have been added, click here for an example of the real output you will see).

You are interested in the line in boldface (third row on the first table), as that is reporting the information corresponding to /eos/uscms/store/user/, which is your personal EOS area. As long as used bytes (second column) remains below aval bytes (fifth column), you can still add more files to your area. However, it is also important to note that there is also a limit on the number of files that can be stored. The maximum number of files is reported as aval files (seventh column) and the number of files already stored is shown under used files (fourth column). A quick summary is shown in the last two columns vol-status and ino-status, representing the current situation for space and number of files with the following meanings:

  • ok: current usage is below 90%,
  • warning: current usage is between 90% and below the quota, and
  • exceeded: current usage has reached the maximum limit and no more information can be added.

A user can only check his/her personal status and not other's from the command line.

Group areas

Similarly, to check the space status of a group EOS area, login to cmslpc-sl6.fnal.gov and issue the following command (in blue):

[jjesus@cmslpc37 ~]$ eosgrpquota lpctau

==> Quota Node: /eos/uscms/store/user/lpctau/
user used bytes logi bytes used files aval bytes aval logib aval files filled[%] vol-status ino-status
jjesus 13 B 13 B 1 - 0 B 0 B 0 - 100 ignored ignored
==> Quota Node: /eos/uscms/store/user/lpctau/
group used bytes logi bytes used files aval bytes aval logib aval files filled[%] vol-status ino-status
us_cms 3.88 TB 1.94 TB 326.23 k- 20.00 TB 10.00 TB 1.00 M- 19.40 ok ok

Again, note that the output has been formatted for clarity (lines for columns have been added, click here for an example of the real output you will see).

In this case, you are interested in the line in boldface (third row of the second table), as that is reporting the information corresponding to /eos/uscms/store/user/lpctau/, which is the EOS area for that specific group.

Group areas have essentially the same functionality as personal areas, so this table should be interpreted just as the one for a personal area (see explanation on Personal areas above).

There is one basic, but important difference though. Information stored on group areas do not count against personal space.


Interacting with files on EOS

Depending on the command being used, files or directories on EOS have to be referred to by using, in some cases, their LFN (Logical File Name, or the part of the absolute path starting with /store/) or in other cases their corresponding URL's, but in general never by their absolute paths. The URL is formed by prepending root://cmseos.fnal.gov/ to the LFN. Click on the image below to enlarge.

Each action covered on this section include in the description what to use to refer to a file or directory on EOS.

Listing

There is a dedicated EOS command to list files or directories stored on EOS areas, it uses the LFN of the file or directory of interest to find it (see the image above if you're not sure what a LFN is):

[jjesus@cmslpc37 ~]$ eosls /store/user/jjesus/
EOSFile.txt
MinBias
README
basic.C
testArea
triggers.py

The default behavior of this command produces a single-column standard output (one file or directory per line, which mimics the behavior of ls -1 of the local file system, note that the flag here correspond to the number 1 and not letter l). what This EOS command can take arguments to modify its basic fuctionality. For example, adding the -l flag will produce a long listing format:

[jjesus@cmslpc37 ~]$ eosls -l /store/user/jjesus/
-rw-r--r--   2 jjesus   us_cms             13 Dec 16 10:24 EOSFile.txt
drwxr-sr-+   1 jjesus   us_cms              1 Jan 08 14:59 MinBias
-rw-r--r--   2 jjesus   us_cms            433 Dec 02 14:36 README
-rw-r--r--   2 jjesus   us_cms            265 Dec 03 13:53 basic.C
drwxr-sr-+   1 jjesus   us_cms              1 Dec 03 10:48 testArea
-rw-r--r--   2 jjesus   us_cms            842 Dec 09 09:42 triggers.py

while adding the -d flag will list directory entries instead of their content:

[jjesus@cmslpc37 ~]$ eosls -d /store/user/jjesus/
jjesus

The output of this command is just the name of the directory and nothing else. Adding the -a flag will show hidden entries as well:

[jjesus@cmslpc37 ~]$ eosls -a /store/user/jjesus/
.
..
EOSFile.txt
MinBias
README
basic.C
testArea
triggers.py

Any combination of the above flags is allowed. The use of multiple flags together (-ld), or separate (-l -d) yields the same result.

Listing by using simple ls with the absolute path to files or directories on EOS is absolutely discouraged as it can significantly reduce the performance of the interactive node. Furthermore, doing so from a job is a BIG NO.

There are few things you need to be aware when listing files on EOS areas.

  • Wildcards. The use of wildcards is not recognized by EOS commands. A '*' included somewhere will be interpreted in general as a part of the name and therefore will return an error. For example:
  • [jjesus@cmslpc37 ~]$ eosls /store/user/jjesus/*
    eos: No match.
    

    The same is true for '?'.
  • Tab completion. The EOS listing command uses the LFN, which starts with /store. Tab completion is not possible as the local filesystem will misinterpret the LFN as a local path starting with /store which does not exist.

Other tools for listing

EOS is a storage filesystem built on the XRootD framework. Because of that, some of the XRootD commands can be used when dealing with files on EOS, and there are some useful ways to list things with XRootD commands in a way that EOS dedicated commands are not able to do.

To get directory listing on XRootD we use xrdfs (which is its file and directory meta-data utility), together with the ls command:

[jjesus@cmslpc37 ~]$ xrdfsls /store/user/jjesus/
[jjesus@cmslpc37 ~]$ xrdfs root://cmseos.fnal.gov ls /store/user/jjesus/
/store/user/jjesus/EOSFile.txt
/store/user/jjesus/MinBias
/store/user/jjesus/README
/store/user/jjesus/basic.C
/store/user/jjesus/testArea
/store/user/jjesus/triggers.py

The default behavior of this command will produce a single-column with one LFN per row, while adding the -u flag:

[jjesus@cmslpc37 ~]$ xrdfsls -u /store/user/jjesus/
[jjesus@cmslpc37 ~]$ xrdfs root://cmseos.fnal.gov ls -u /store/user/jjesus/
root://131.225.207.127:1094//store/user/jjesus/EOSFile.txt
root://131.225.207.127:1094//store/user/jjesus/MinBias
root://131.225.207.127:1094//store/user/jjesus/README
root://131.225.207.127:1094//store/user/jjesus/basic.C
root://131.225.207.127:1094//store/user/jjesus/testArea
root://131.225.207.127:1094//store/user/jjesus/triggers.py

will print URL's instead. The two xrdfs options for listing can be useful in workflows where either the LFN or the URL of a file or directory on EOS is needed rather than just the simple name.

Copying

In order to copy files into, out-of or between EOS areas you need to use another XRootD command: xrdcp. In this case, EOS locations need to be included as URL's. There are three possible scenarios:

  • Local file to EOS:

[jjesus@cmslpc37 ~]$ xrdcp ~/test.txt root://cmseos.fnal.gov//store/user/jjesus/copyOfLocalFile.txt
[13B/13B][100%][==================================================][0B/s]

  • EOS to local file:

[jjesus@cmslpc37 ~]$ xrdcp root://cmseos.fnal.gov//store/user/jjesus/EOSFile.txt ~/copyOfEOSFile.txt
[13B/13B][100%][==================================================][0B/s]

  • and finally, EOS to EOS:

[jjesus@cmslpc37 ~]$ xrdcp root://cmseos.fnal.gov//store/user/jjesus/EOSFile.txt \
? root://cmseos.fnal.gov//store/user/jjesus/EOSFile1.txt
[13B/13B][100%][==================================================][13B/s]
    

xrdcp will assume by default that the destination file does not exist. If the destination file already exist, xrdcp will produce an error message like:

[jjesus@cmslpc37 ~]$ xrdcp ~/test.txt root://cmseos.fnal.gov//store/user/jjesus/copyOfLocalFile.txt
[0B/0B][100%][==================================================][0B/s]
Run: [ERROR] Server responded with an error: [3006] Unable to create file
/eos/uscms/store/user/jjesus/copyOfLocalFile.txt; File exists

Overwriting an existing file can be achieved by adding the -f flag to xrdcp:

[jjesus@cmslpc37 ~]$ xrdcp -f ~/test.txt root://cmseos.fnal.gov//store/user/jjesus/copyOfLocalFile.txt
[13B/13B][100%][==================================================][0B/s]

By now you may have noticed that xrdcp will show by default a text version of a progress bar: [====] (it may be more noticeable on big files because of the time it takes to transfer them). While this is a good way to visualize the progress of this operation, there are some situations in which you may not want such information. Here is an example: a HTCondor job using xrdcp to transfer a file will include the updates on the progress bar in the stderr file for the job, generating non-zero size error files even when the job is completely ok. Here is a typical stderr for a file that is exactly 1,000,000,000 B copied using xrdcp inside a HTCondor job that doesn't give any error:

[jjesus@cmslpc37 myWorkArea]$ more myJob_cluster_process.stderr
[96MB/953.7MB][ 10%][=====>                                            ][19.2MB/s 
[128MB/953.7MB][ 13%][======>                                           ][21.33MB/s
[160MB/953.7MB][ 16%][========>                                         ][22.86MB/s
[176MB/953.7MB][ 18%][=========>                                        ][22MB/s]  
[208MB/953.7MB][ 21%][==========>                                       ][23.11MB/s
[240MB/953.7MB][ 25%][============>                                     ][24MB/s]  
[256MB/953.7MB][ 26%][=============>                                    ][23.27MB/s
[288MB/953.7MB][ 30%][===============>                                  ][24MB/s]  
[304MB/953.7MB][ 31%][===============>                                  ][23.38MB/s
[320MB/953.7MB][ 33%][================>                                 ][22.86MB/s
[352MB/953.7MB][ 36%][==================>                               ][23.47MB/s
[384MB/953.7MB][ 40%][====================>                             ][24MB/s]  
[400MB/953.7MB][ 41%][====================>                             ][23.53MB/s
[416MB/953.7MB][ 43%][=====================>                            ][23.11MB/s
[432MB/953.7MB][ 45%][======================>                           ][22.74MB/s
[448MB/953.7MB][ 46%][=======================>                          ][22.4MB/s]
[464MB/953.7MB][ 48%][========================>                         ][22.1MB/s]
[496MB/953.7MB][ 52%][==========================>                       ][22.55MB/s
[528MB/953.7MB][ 55%][===========================>                      ][22.96MB/s
[544MB/953.7MB][ 57%][============================>                     ][22.67MB/s
[560MB/953.7MB][ 58%][=============================>                    ][22.4MB/s]
[576MB/953.7MB][ 60%][==============================>                   ][22.15MB/s
[592MB/953.7MB][ 62%][===============================>                  ][21.93MB/s
[608MB/953.7MB][ 63%][===============================>                  ][21.71MB/s
[640MB/953.7MB][ 67%][=================================>                ][22.07MB/s
[672MB/953.7MB][ 70%][===================================>              ][22.4MB/s]
[688MB/953.7MB][ 72%][====================================>             ][22.19MB/s
[720MB/953.7MB][ 75%][=====================================>            ][22.5MB/s]
[736MB/953.7MB][ 77%][======================================>           ][22.3MB/s]
[768MB/953.7MB][ 80%][========================================>         ][22.59MB/s
[800MB/953.7MB][ 83%][=========================================>        ][22.86MB/s
[816MB/953.7MB][ 85%][==========================================>       ][22.67MB/s
[832MB/953.7MB][ 87%][===========================================>      ][22.49MB/s
[864MB/953.7MB][ 90%][=============================================>    ][22.74MB/s
[896MB/953.7MB][ 93%][==============================================>   ][22.97MB/s
[912MB/953.7MB][ 95%][===============================================>  ][22.8MB/s]
[928MB/953.7MB][ 97%][================================================> ][22.63MB/s
[944MB/953.7MB][ 98%][=================================================>][22.48MB/s
[953.7MB/953.7MB][100%][==================================================][22.71MB
[953.7MB/953.7MB][100%][==================================================][22.71MB/s]

To avoid this, remember to add the -s flag to require xrdcp to be silent, for example:

[jjesus@cmslpc37 myWorkArea]$ xrdcp -s root://cmseos.fnal.gov//store/user/jjesus/test.txt .

copies the file from EOS to a local area without showing anything.

You can combine these flags into a single one, -fs, or use them separate, -f -s, with the same result.

Creating directories

In order to create a directory on an EOS area, you need to use the dedicated EOS command mkdir and the LFN of the directory to be created:

[jjesus@cmslpc37 ~]$ eosmkdir /store/user/jjesus/newDir

In case of success, this command doesn't produce any standard output. Note: all parent directories leading to the new one should exist for this command to succeed.

Adding the -p flag will make parent directories as needed:

[jjesus@cmslpc37 ~]$ eosmkdir -p /store/user/jjesus/newDir1/newDir2/newDir3

In case of success, this command doesn't produce any standard output and no error is shown if the parents already exist.

Removing files and directories

First thing to remember when removing files from EOS: a file removed from EOS is gone for good. Second thing to remember when removing files from EOS: a replica is not the same as a backup, the concept of backup does not exist on EOS. Knowing that can make you understand a bit more why wildcards can't be used on EOS, it can be seen as an extra layer of protection against accidental removal of files in bulk. You've been warned.

In order to remove files from an EOS area, you need to use the dedicated EOS command rm and the LFN of the file to be removed:

[jjesus@cmslpc37 ~]$ eosrm /store/user/jjesus/EOSFile.txt

This command doesn't produce any standard output.

There is a dedicated EOS command to remove directories, but it only works when the directory is empty. Instead of using that, you can use the extension of the EOS command to remove files that allows the removal of directories whether they are empty or not. To do so, you have to add the -r flag:

[jjesus@cmslpc37 ~]$ eosrm -r /store/user/jjesus/dir

As in the case for files, this command doesn't produce any standard output. Trying to remove a directory without the -r flag, will result in an error like:

[jjesus@cmslpc37 ~]$ eosrm /store/user/jjesus/dir
error: unable to remove file/directory '/eos/uscms/store/user/jjesus/dir'
(errc=21) (Is a directory)


Working with ROOT files on EOS

Access to ROOT files on EOS should be done in general by using a redirector, for example: root://cmsxrootd.fnal.gov.

The redirector will look for the file at Fermilab first, if not found, it will continue looking at other sites until the file is found (provided it actually exist).

The string /eos/uscms should never appear while opening a ROOT file stored on EOS. If it does, the file will open using the FUSE mount, which in general have a negative impact on the performance of the interactive node, degrading the experience collectivelly for all the users connected to the same node. On the worker nodes, /eos/uscms won't be mounted any more, therefore, jobs making use of the FUSE mount to open ROOT files on EOS will simply fail.

Interactive access

From the command line on any interactive LPC node, (on any location on the local filesystem, usually somewhere inside /uscms_data, never from /eos/uscms) a ROOT file can be directly open after setting the proper environment in the following way:

[jjesus@cmslpc37 ~]$ root -l root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root
root [0]
Attaching file root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root as _file0...
(class TFile *) 0x3edd370

Once open, you can navigate through your file and work with it as usual, for example:

root [1] .ls();
TNetXNGFile**		root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root
 TNetXNGFile*		root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root
root [2] TH1F *theHist = (TH1F*)_file0->Get("nPU"); 
root [3] TCanvas *theCanvas = new TCanvas("theCanvas", "Pile-Up", 640, 480);
root [4] theHist->Draw();
root [5] theCanvas->SaveAs("thePlot.png");
Info in < TCanvas::Print >: png file thePlot.png has been created
root [6] .q

to load a histogram from the ROOT file previously open, and create the following PNG file out of it:

If you need to open a file from inside an already initiated ROOT session, you can do the folloiwng:

[jjesus@cmslpc37 ~]$ root -l
root [0]
.
Some ROOT directives
.
root [10] TFile *theFile = TFile::Open("root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root");
root [11] .ls();
TNetXNGFile**		root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root
 TNetXNGFile*		root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root

Hint: If you ever need a quick reminder about the proper way to open a ROOT file after you have initiated an interactive ROOT session, a quick way is just to open another file (on a different terminal or your local machine, for example). Once inside ROOT, hit the upwards arrow on your keyboard, and you will be presented with the line that ROOT used to open the file you indicated from the command line, ROOT appends that by default to the history of the current session (Show me how - [β]).

Inside a script

Existing EOS files that need to be open as readable inside scripts, should be open with the constructor shown in the previous section and using their URL's. For writing EOS Files from batch jobs, see below. On a C++-like script this would look just like accessing the file interactively:

#include <iostream>
.
.
.
#include "TFile.h"
.
.
.

int main(int argc, const char * argv[]) {
    Some code here

    //  Proper way to open an EOS ROOT file:
    TFile *theFile = TFile::Open("root://cmsxrootd.fnal.gov//store/user/jjesus/rootFile.root");

    Rest of code

    return 0;
}

Writing EOS files for batch jobs

Don't write directly to distributed storage (EOS) within a batch job. Instead, write directly to your job's local storage, and once completed, xrdcp the output.

The preferred methods to write to EOS from a batch job are:

  • Through crab3 jobs, which stage the output automatically to EOS at the end of the job when configured to output to T3_US_FNALLPC
  • Within condor batch jobs, transferring to EOS from the local worker machine at the end of the job, through xrdcp transfers, described earlier on this page.

Merge files with histograms and trees

Multiple ROOT files stored on EOS can be merged from the command line on any interactive LPC node (from any location on the local filesystem, usually somewhere inside /uscms_data, never from /eos/uscms). After setting the proper environment, hadd can be used by referring to the list of files to be merged by their corresponding URLs. The list can be easily obtained by using the -u option of the xrdfs tool for listing (as explained above). For example, from a CMSSW relase area located inside /uscms_data/d3/jjesus/, after cmsenv one can do:

[jjesus@cmslpc37 src]$ hadd myTarget.root `xrdfsls -u /store/user/jjesus/rootFiles | grep '\.root'`
hadd Target file: myTarget.root
hadd Source file 1: root://131.225.207.127:1094//store/user/jjesus/rootFiles/rootFile0.root
hadd Source file 2: root://131.225.207.127:1094//store/user/jjesus/rootFiles/rootFile1.root
hadd Source file 3: root://131.225.207.127:1094//store/user/jjesus/rootFiles/rootFile2.root
hadd Target path: myTarget.root:/
hadd Target path: myTarget.root:/2h_cone_eta_totalpT_nPU
hadd Target path: myTarget.root:/2h_cone_eta_totalpT_nPU/all
hadd Target path: myTarget.root:/2h_cone_eta_totalpT_nPU/h
hadd Target path: myTarget.root:/2h_cone_eta_totalpT_nPU/e
.
.
.
hadd Target path: myTarget.root:/2p_cone_eta_nPU_totalpT_nPV/h_HF
hadd Target path: myTarget.root:/2p_cone_eta_nPU_totalpT_nPV/egamma_HF
hadd Target path: myTarget.root:/2p_cone_eta_nPU_totalpT_nPV/chs

which merges the three files into a single one named myTarget.root.

There are multiple parts to the line responsible for merging the files. First xrdfsls is used to obtain the list of files on the LFN given as argument, the output will be formatted as URLs instead of just LFNs by adding the -u option. Then, the result of this command is filtered by piping to a grep command which will look just for the .root file extension. This is an important step, files to be merged need to be ROOT files and have the same structure. If the directory where the the files are stored contain other files, trying to use hadd without filtering will fail when trying to initialize a non-ROOT file, exiting due to the error. Note: if the ROOT files don't have the same structure, hadd will still create a target file, it will most likely show warning and error messages and the final product will probably be not precisely what it is supposed to be. Finally, as this command is enclosed inside grave accent marks (`), this will be executed first and the output will be put in place at the end of the hadd command as its last argument.

Things you shouldn't do

When listing

Do not use regular ls to list EOS areas over the FUSE mount:

                     - - - DON'T DO THIS - - -
[jjesus@cmslpc37 ~]$ ls /eos/uscms/store/.../
                     - - - DON'T DO THIS - - -

This is particularly bad for areas with large number of files. This is particularly bad when using wildcards: *, ?, etc. Do not do it on any interactive node and NEVER do that in a job. See Interacting with files on EOS on this page to find the correct way to do this.

When copying

Do not use regular cp to copy EOS files or directories over the FUSE mount:

                     - - - - - - - - - - - DON'T DO ANY OF THIS - - - - - - - - - - -
[jjesus@cmslpc37 ~]$ cp /eos/uscms/store/.../sourceFile /some/local/area/destFile
[jjesus@cmslpc37 ~]$ cp /some/local/area/sourceFile /eos/uscms/store/.../destFile
[jjesus@cmslpc37 ~]$ cp /eos/uscms/store/.../sourceFile /eos/uscms/store/.../destFile
                     - - - - - - - - - - - DON'T DO ANY OF THIS - - - - - - - - - - -

This is particularly bad when using wildcards, trying to copy large number of files or recursively. Do not do it on any interactive node and NEVER do that in a job. See Interacting with files on EOS on this page to find the correct way to do this.

When removing

Do not use regular rm to remove files or directories over the FUSE mount:

                     - - - - - - - - - - - - - DON'T DO THIS - - - - - - - - - - - - -
[jjesus@cmslpc37 ~]$ rm /eos/uscms/store/.../fileToRemove
                     - - - - - - - - - - - - - DON'T DO THIS - - - - - - - - - - - - -

When reading files

Do not run a script interactively from eos over the FUSE mount (copy it to NFS disk and run it from there):

                     - - - - - - - - - - - - - DON'T DO THIS - - - - - - - - - - - - -
[username@cmslpc37 ~]$ python /eos/uscms/store/username/myCoolScript.py
                     - - - - - - - - - - - - - DON'T DO THIS - - - - - - - - - - - - -
Webmaster | Last modified: Tuesday, 04-Apr-2017 15:32:05 CDT