uscms.org  www 

Using EOS at the LPC: Additional Commands

This is a secondary set of possibly useful EOS commands. Go to Using EOS at the LPC to find instructions for basic EOS use and operations.

Where do I find more information about EOS commands available?

What are some useful scripts for EOS?

  • Kevin Pedro's github utilities, see eosdu in particular
    • chmod 755 eosdu
    • ./eosdu -h /store/user/username
    • ./eosdu.sh -rh /store/user/lpcgroupname

  • There are some useful scripts in the github.com/FNALLPC/lpc-scripts area

  • From the interactive node command line type alias to find out what commands are aliased for your login
    [username@cmslpc39 WorkDir]$  alias
    eosgrpquota	eos root://cmseos.fnal.gov quota | grep -A4
    eosls	eos root://cmseos.fnal.gov ls
    eosmkdir	eos root://cmseos.fnal.gov mkdir
    eosquota	eos root://cmseos.fnal.gov quota | grep -B4 `whoami`
    eosrm	eos root://cmseos.fnal.gov rm
    xrdfsls	xrdfs root://cmseos.fnal.gov ls    

What are some errors and fixes for EOS?

  • Certain versions of CMSSW environment (CMSSW_9_2_4) may cause problems with the commands, giving an error such as:
    [username@cmslpc25 ~]$ eosls /store/user/CERNusername
    eos: symbol lookup error: eos: undefined symbol: _ZN5XrdCl4File4OpenERKSsNS_9OpenFlags5FlagsENS_6Access4ModeEt
    • Solve this error by doing the commands in a new terminal without a CMSSW environment
    • This should not affect xrdcp commands

  • If you changed shell by hand, the eos aliases (eosls, eosquota, etc.) are not loaded

How do I use EOS commands at the LPC on the interactive node command line?

Login to cmslpc-sl6.fnal.gov, use eos commands with this structure (where command is the eos command you put in yourself. These eos root://cmseos.fnal.gov commands only work on the interactive nodes and not the condor worker nodes.

[username@cmslpc37 ~]$ eos root://cmseos.fnal.gov command

Examples with the EOS find command:

du type command

This provides all files there with respective size. You can pipe this to a grep to do selection.

eos root://cmseos.fnal.gov find --size  

find with xurl

This runs find and gives the xurl path

eos root://cmseos.fnal.gov find --xurl

find with count

This maxes out at 100,000 files. All "user" find commands are limited to 100,000 lines of output. Note that sysadmins don't have this limit, so if more information is required, one can contact LPC Computing support to get the information from the sysadmins.

eos root://cmseos.fnal.gov find --count 

find with childcount

This counts number of a files in a directory, use with a directory -d

eos root://cmseos.fnal.gov find --childcount -d /store/user/username

Given that all "user" find commands are limited to 100,000 lines of output, we use the following example to not count too many subdirectories deep:

eos root://cmseos.fnal.gov find --childcount -d --maxdepth 1 /store/user/username

find by age

This finds files older than 365 days in a directory

eos root://cmseos.fnal.gov find -ctime +365 /store/user/lpcgroupname

More EOS copy options:

cp help:

eos root://cmseos.fnal.gov cp --help

cp example one file:

eos cp root://cmseos.fnal.gov///store/user/differentuser/miniaod.root root://cmseos.fnal.gov//store/user/tonjes/miniaodCopy.root

cp recursive:

This will work with the local path, can copy local to remote.

eos root://cmseos.fnal.gov cp --recursive

cp example directory recursive:

Note that the / at the end of the directory name is essential in formatting this command:

eos cp -r root://cmseos.fnal.gov//store/user/cmsdas/2014/ root://cmseos.fnal.gov//store/user/belt/cmsdas/

cp xrdcp many files:

To copy many files at the end of a condor batch job for instance, we have: for i in *.root; do xrdcp $i root://cmeos.fnal.gov//store/user/lpcgroupname/MyAnalysis/$i;. However, you can instead use xrdcp recursive, this is from the man page on xrootd.org:

xrdcp -r | --recursive

recursively copy all files starting at the given source directory. This option is only supported for local files.

EOS file renaming and moving:

EOS file rename

If you want to just rename a folder/file without copying inside eos, i.e., change metadata:

eos root://cmseos.fnal.gov file rename

EOS file move options

eos root://cmseos.fnal.gov mv --help
eos root://cmseos.fnal.gov mv /store/user/tonjes/step4_test.root /store/group/lpcci2dileptons/tonjes_test.root

EOS and xrootd:

Note that these are both xrootd access methods and fine at the cmslpc. Non-xrootd commands should never use /eos/uscms.


FNAL EOS and CERN EOS access at either site:

Here are some examples on how to access CERN EOS files at FNAL:
[username@cmslpc27 ~]$ eos root://eoscms.cern.ch/ ls /store/user 

How to access FNAL EOS files at CERN:
[username@lxplus095 ~]$ eos root://cmseos.fnal.gov ls /store/user 

Checksum files on EOS:

One can do checksum with xrdcp (which copies the file while calculating checksum), as well as xrootd (does not copy the file).

Two examples (note the long command - scroll right):

  1. xrdcp copies the file, here it copies it locally (.), which may not be ideal depending on the file size.
    [username@cmslpc29 CMSSW_9_2_4]$  xrdcp --cksum md5:print root://cmseos.fnal.gov//store/user/drberry/dynlo.lhe .
    md5: 87ee2aedc42926e66d0041645c2ceba4 .//dynlo.lhe 260873891
    Check your local copy like this:
    [tonjes@cmslpc27 ~]$ md5sum dynlo.lhe 
    87ee2aedc42926e66d0041645c2ceba4  dynlo.lhe
  2. xrootd calculates the adler32 checksum:
    [username@cmslpc29 CMSSW_9_2_4]$  xrdfs root://cmseos.fnal.gov//store/user/drberry/dynlo.lhe query checksum /store/user/drberry/dynlo.lhe
    eos 1e053b51
    Check your local copy like this:
    [tonjes@cmslpc27 ~]$  adler32 dynlo.lhe
Webmaster | Last modified: Friday, 03-Nov-2017 10:19:20 CDT