Computing Environment Setup: Batch Systems

The batch system available for users of the CMS LPC CAF is condor which allows the user to submit jobs into the lpc batch farm. On this page we will describe how to use this batch system.

The preferred way to access the most CPU (including the CMS LPC) is through CRAB

For any information not covered below, visit the condor user's manual. Find the version of condor running on lpc with condor_q -version

One important note: compilation of code is not supported on the remote worker node. This includes ROOT's ACLiC (i.e. the plus signs at the end of root -b -q foo.C+). You can compile on the LPC interactive nodes and transfer the executables and shared libraries for use on the worker. For root -b -q foo.C+, you will need to transfer the recently compiled foo* to the condor job.

How do I use CRAB to submit batch jobs?

Guides:

• CMS Workbook for CRAB
• Be sure to follow the CRAB pre-requisites for setup
• CRAB3 Hands On Tutorial at the LPC
• In addition to the SWGuideCrab above, the CRAB3FAQ is useful
• CRAB3 Troubleshooting
• CMS Connect (access Grid resources for bare-condor analysis jobs without using CRAB)

How do I use Condor to submit to the lpc batch farm?

What are the pre-requisites for all condor jobs on the cmslpc farm?

• Submitting and running jobs, as well as all condor queries now require the user to have a valid grid proxy in the CMS VO
• When you obtain your proxy (voms-proxy-init --valid 192:00 -voms cms), it will be saved in your home directory where it can be read by the condor job on the worker node
• The CMS LPC CAF system must know about the association of your grid certificate and FNAL username. This is usually done as part of the Enable EOS area ticket. You must do this at least once for your grid certificate to be associated with your account, which also lets you write to your EOS area from CRAB.
• Go to the LPC Service Portal: https://fermi.servicenowservices.com/lpc
  • Use Fermilab SSO, paying attention to instructions to configure your browser once for Kerberos login. Note that your Services credentials are different from Kerberos
  • Choose "CMS Storage Space Request", and select "Enable" under "Action Required"
  • It will prompt you for for your DN (Your DN is the result of voms-proxy-info --identity) and CERN username. Submit that to register your DN. The grid certificate will be known to the nodes within 1-3 hours during FNAL business hours.

A simple condor example: jdl file

The first step to using the condor system is writing the condor jdl (job description language) file. This file will tell the system what you want it to do and how. Below is an example, which will run a system program that will sleep for one minute, then quit. Use your favorite text editor to create the sleep-condor.jdl file with the following contents. Click on any of the green lines below for an explanation of what it does.

A simple condor example: sh file

The next step is to create the executable, in this case a shell script called sleep.sh. Create the file below in your favorite text editor in the same directory as your sleep-condor.jdl file.
#!/bin/bash
set -x
# Sleep
sleep $1
echo "##### HOST DETAILS #####"
echo "I slept for $1 seconds on:"
hostname
date


Submit, monitor, and remove your condor job

After you've created the jdl file sleep-condor.jdl, the shell script sleep.sh, and authenticated your grid certificate to the CMS VO, you can submit it to the condor system using the command condor_submit followed by the name of your submit description file (jdl), in this example's case "sleep-condor.jdl":

Your output should look something like this:

You can see the status of all jobs you have submitted (unless they have completed) to all the possible schedulers (lpcschedd1.fnal.gov for example) with the following command:

[username@cmslpc132 condor]$ condor_q

-- Schedd: lpcschedd1.fnal.gov : <131.225.188.55:9618?... @ 03/21/19 10:09:17
 ID        OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
914329.0   username        3/21 10:08   0+00:00:08 R  0    0.0 sleep.sh 60

Total for query: 1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended 
Total for username: 1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended 
Total for all users: 1642 jobs; 0 completed, 0 removed, 0 idle, 99 running, 1543 held, 0 suspended

-- Schedd: lpcschedd2.fnal.gov : <131.225.188.57:9618?... @ 03/21/19 10:09:17
 ID        OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD

Total for query: 0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended 
Total for username: 0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended 
Total for all users: 2669 jobs; 0 completed, 0 removed, 0 idle, 285 running, 2384 held, 0 suspended

-- Schedd: lpcschedd3.fnal.gov : <131.225.188.235:9618?... @ 03/21/19 10:09:17
 ID        OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD

Total for query: 0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended 
Total for username: 0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended 
Total for all users: 3723 jobs; 1 completed, 0 removed, 0 idle, 70 running, 3652 held, 0 suspended

Note that there is an -allusers option lets you see condor jobs submitted by all users on the schedulers (condor_q -allusers). To see just your own, omit that option.

• To understand job status (ST) from condor_q, you can refer to the condor user manual (8.8) - see the condor job troubleshooting page to understand why a job is in each status:
  • "Current status of the job, which varies somewhat according to the job universe and the timing of updates. H = on hold, R = running, I = idle (waiting for a machine to execute on), C = completed, X = removed, S = suspended (execution of a running job temporarily suspended on execute node), < = transferring input (or queued to do so), and > = transferring output (or queued to do so)."
• Be sure to know which scheduler your job was submitted to and is running on.

You can specifically get a list of all the jobs and their status for a specific user username from any machine using this command:

If you want to view the entire queue, you can use the following command:

This gives all the jobs from all users, including jobs coming in from CRAB and CMS connect into pilots, and jobs in the T1_US_FNAL workers. Note that cmslpc jobs submitted on local condor will not run on T1_US_FNAL workers. See the condor system status web page for some command line methods to find out only jobs for the local batch.

To cancel a job type condor_rm followed by the job number, for this example, 60000042.0 and 60000042.1

If you don't remember your scheduler name for that job, use the condor_q command from above, it will tell you what scheduler your job was submitted from.

condor_rm -name lpcschedd1.fnal.gov -all; condor_rm -name lpcschedd2.fnal.gov -all; condor_rm -name lpcschedd3.fnal.gov -all

If you wish to see the end of a current running job stdout, you can use the condor_tail command together with the job number. The results will give you ~20 lines of output, it is cut here for space in this example:

== CMSSW: 09-Mar2019 22:26:32 CST  Successfully opened file root://cmseos.fnal.gov//store/user/username/myfile.root
== CMSSW: Begin processing the 1st record. Run 1, Event 254, LumiSection 6 at 09-Mar-2019 22:30:12.001 CST
== CMSSW: Begin processing the 101st record. Run 1, Event 353, LumiSection 8 at 09-Mar-2019 22:30:28.673 CST
== CMSSW: Begin processing the 201st record. Run 1, Event 451, LumiSection 10 at 09-Mar-2019 22:30:32.670 CST

Details of the condor submit file (jdl)

universe = vanilla
The universe variable defines an execution environment for your job, in this example we use the vanilla universe which has the least amount of built in services, but also the least amount of restrictions. For a complete list of universes and what they do, so the condor user's manual. The cmslpc does NOT support the standard universe, and thus does not have condor_compile.
BACK

Executable = sleep.sh
This is the program you want to run. If the program is in the same directory as your batch file, just the name will work, example: yourscript.sh. If it is in a different directory than your batch file then you must give the pathname, example: myscripts/yourscript.sh runs the script yourscript.sh located in the directory myscripts.

• Note that we do not use /bin/sleep as our executable, as system level executables are read from the remote scheduler nodes. The lpcschedd*.fnal.gov nodes run SL7, and your jobs run on SL7 Docker containers (the wrapper in condor_submit will detect the OS of the submission node and use that to set the OS of the container).
• In general, either use a shell script (best because you can add useful comments about the worker node environment), or your own compiled executable
• The executable you specify in the Executable line in the condor jdl is automatically sent to the worker node, you do not have to specify it in transfer_input_files

should_transfer_files = YES
when_to_transfer_output = ON_EXIT

These options tell condor to take input files from your computer and send output files back. If these options are not activated then you must provide input through some other means and extract the output yourself. Users cannot do direct read/write from the NFS filesystems as they are not mounted on the condor worker nodes. The Trasfer_Input_Files option can not take a directory for input but can take files with an absolute path in case your input files are not in the submit directory.
Note that the ### means this line is commented to make the example work without any input files, remove the comment and replace file1, file2 with your file information.. Also note that in .jdl files, comments are not recognized in line. Some more expanded examples can be found below for good and bad NFS usage at this link.
BACK

Output = sleep_$(Cluster)_$(Process).stdout
This directs the standard output of the program to a file, in other words, everything that would normally be displayed on the screen, so that you can read it after it is finished running. Where you see $(Cluster) condor will substitute the job number, and $(Process) will become the process number, in this case, 0-4. Make sure you are NOT using a full path on this setting so the condor scratch area is utilized.
BACK

Error = sleep_$(Cluster)_$(Process).stderr
This is the same as the Output line, except it applies to standard error, this is extremely useful for debugging or figuring out what is going wrong (all most always something). Where you see $(Cluster) condor will substitute the job number, and $(Process) will become the process number, in this case, 0-4. Make sure you are NOT using a full path on this setting so the condor scratch area is utilized.
BACK

Log = sleep_$(Cluster)_$(Process).log
The log file contains information about the job in the condor system, the ip address of the computer that is processing the job, the time it starts and finishes, how many attempts were made to start the job and other such data. It is recommended to use a log file, where you see $(Cluster) condor will substitute the job number, and $(Process) will become the process number, in this case, 0-4. Make sure you are NOT using a full path on this setting so the condor scratch area is utilized.
BACK

notify_user = ${LOGNAME}@FNAL.GOV
Specifies to whom the system will automatically email when the job finishes (your email), in the example, the computer should have put your email address here. You will recieve a seperate email for every process in you job that completes.
The use in the sleep example is done in the context of the cat command, where it evaluates the content of the system environment variable ${LOGNAME}, and places it in the condor job description file sleep-condor. If you are making your own condor job description file with a text editor, using the system environment variable as above would not work, you will need to copy the line from the sleep-condor file instead.

BACK

x509userproxy = ${X509_USER_PROXY}
This line tells your job to bring along your grid proxy, which is needed for accessing files over xrootd (root://). You will have needed to authenticate your grid certificate for this line to work.
The use in the sleep example is done in the context of the cat command, where it evaluates the content of the system environment variable ${X509_USER_PROXY}, and places it in the condor job description file sleep-condor. If you are making your own condor job description file with a text editor, using the system environment variable as above would not work, you will need to use this $ENV(X509_USER_PROXY) instead for jdl to evaluate.
BACK

Arguments = 60
Here you put any command line arguments for your program, if you have none, exclude this line. In this example the program needs one argument for the number of seconds to wait. This argument tells the program to wait for one minute.
BACK

Queue 5
This is how many times you want to run the program, without this line the program will not run. The processes will be numbered starting at zero, so in this example they will be: 0, 1, 2, 3, and 4. In case you only want one job, use Queue 1.
BACK

3. How do I manage file input/output from within Condor?

Condor has options availble to automatically transfer input files to the worker where the job runs and then copy any output files back to the directory you submit from. Jobs should use these or read/write data from dcache instead of doing file reads or writes directlly from the NFS mounted file systems. Direct reads/writes to NFS can cause severe performance problems. If this happens the CMS T1 Facility team may end of up having to kill your jobs.

The options for telling condor to copy files into and out of your job are:
Should_Transfer_Files = YES
Transfer_Input_Files = file1, file2
WhenToTransferOutput = ON_EXIT

In addtion you need to make sure you use the correct pathname for files. Please take into account that the NFS disks are not mounted on the condor worker nodes.

BACK

4. How to I make sure my batch jobs are not accessing shared/NFS disk?

After October 1, 2017, the NFS shared filesystems is no longer be mounted on condor worker nodes. If you try to access files from /uscms/homes and /uscms_data/d* on the condor worker node, that will fail.

Guide to not using NFS on condor worker nodes, note that this page has more features documented (than here) in condor scripts and may be useful to you.

Example A: A good example

good1.jdl:

universe = vanilla
Executable = good1.sh
Output = good1.out
Error = good1.err
Log = good1.log
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = /uscms_data/d1/username/input_file.root, /uscms_data/d1/username/good1.C


good1.sh:

#!/bin/bash
# cms software setup not included here for brevity
cd ${_CONDOR_SCRATCH_DIR}
root -b -q good1.C

good1.C:

{
TFile f("input_file.root");
output_ntuples = process(f); // do some calculation
TFile g("output_file.root", "w");
g.write(output_ntuples);
}

Example B: A good example with tarring - tcsh script

Note: Please note that if your CMSSW tar is large (1 GB or larger), it may have two problems: First it will restrict the machines you can run upon due to the input transfer size; Secondly it may saturate the network if you have very many jobs all starting at once. Instructions are below to exclude caches and reduce the size of your .tar.gz. You will have your best networking results by transferring the file to EOS using xrdcp, as shown below, and xrdcp it to the local working area at the start of the condor batch job.

There are two ways to use this technique which will be referred to below:

1. Best for networking throughput: the small tarred CMSSW and large (root) input files are transferred to personal EOS storage. The tarred CMSSW is transferred to the local worker node disk, and large input files are opened in root following the EOS instructions for file reference inside a script. Note that the EOS transfer at the end of the job is best for large root files.
2. The tarred CMSSW and any input files are transferred to the local worker node disk from NFS - may slow down your job starting due to heavy NFS network usage for many jobs in parallel, as well as input file size.

tar -zcvf CMSSW_10_6_4.tgz CMSSW_10_6_4

• Note that you can exclude large files for instance from your tar with the following argument:
  --exclude="Filename*.root"
  
• Note that you can exclude CMSSW caches for instance with a command like this one:
  tar --exclude-caches-all --exclude-vcs -zcf CMSSW_10_6_4.tar.gz -C CMSSW_10_6_4/.. CMSSW_10_6_4 --exclude=src --exclude=tmp
• When using --exclude-caches-all, you should mark directories you want to exclude with a CACHEDIR.TAG file, for more, see these links for information and examples:
• http://www.brynosaurus.com/cachedir/spec.html
• https://github.com/TreeMaker/TreeMaker/blob/Run2/Production/test/condorSub/CACHEDIR.TAG
https://github.com/TreeMaker/TreeMaker/blob/Run2/Production/test/cache_all.sh

1. To transfer this to your personal EOS area do (for your CERN username):
xrdcp CMSSW_10_6_4.tgz root://cmseos.fnal.gov//store/user/username/CMSSW_10_6_4.tgz
2. In your condor.jdl file which you submit, refer to: Transfer_Input_Files = cmsRun.csh, ExampleConfig.py, CMSSW_10_6_4.tar.gz

xrdcp Filename1.root root://cmseos.fnal.gov//store/user/username/Filename1.root
xrdcp Filename2.root root://cmseos.fnal.gov//store/user/username/Filename2.root

Example cmsRun.csh:

#!/bin/tcsh
echo "Starting job on " `date` #Date/time of start of job
echo "Running on: `uname -a`" #Condor job is running on this node
echo "System software: `cat /etc/redhat-release`" #Operating System on that node
source /cvmfs/cms.cern.ch/cmsset_default.csh  ## if a bash script, use .sh instead of .csh
### for case 1. EOS have the following line, otherwise remove this line in case 2.
xrdcp -s root://cmseos.fnal.gov//store/user/username/CMSSW_10_6_4.tgz .
tar -xf CMSSW_10_6_4.tgz
rm CMSSW_10_6_4.tgz
setenv SCRAM_ARCH slc7_amd64_gcc820
cd CMSSW_10_6_4/src/
scramv1 b ProjectRename
eval `scramv1 runtime -csh` # cmsenv is an alias not on the workers
echo "Arguments passed to this script are: for 1: $1, and for 2: $2"
cmsRun ${1} ${2}
xrdcp nameOfOutputFile.root root://cmseos.fnal.gov//store/user/username/nameOfOutputFile.root
### remove the output file if you don't want it automatically transferred when the job ends
rm nameOfOutputFile.root
cd ${_CONDOR_SCRATCH_DIR}
rm -rf CMSSW_10_6_4

Be sure to make your cmsRun.csh executable: chmod +x cmsRun.csh

Your condor.jdl will look something like this (default for case 1. EOS):

universe = vanilla
Executable = cmsRun.csh
Should_Transfer_Files = YES
WhenToTransferOutput = ON_EXIT
Transfer_Input_Files = cmsRun.csh, ExampleConfig.py
Output = sleep_$(Cluster)_$(Process).stdout
Error = sleep_$(Cluster)_$(Process).stderr
Log = sleep_$(Cluster)_$(Process).log
x509userproxy = $ENV(X509_USER_PROXY)
Arguments = ExampleConfig.py 100
Queue 5

Case 2. Transfer needs this line modified:

Transfer_Input_Files = cmsRun.csh, ExampleConfig.py, CMSSW_10_6_4.tar.gz

Here, the username and X509_USER_PROXY location are read from the environment variables with a different method than in the cat command above.

BACK

Example C: A good example with making a new CMSSW during the batch job - bash script

Example cmsRun.sh:

#!/bin/bash
echo "Starting job on " `date` #Date/time of start of job
echo "Running on: `uname -a`" #Condor job is running on this node
echo "System software: `cat /etc/redhat-release`" #Operating System on that node
source /cvmfs/cms.cern.ch/cmsset_default.sh  ## if a tcsh script, use .csh instead of .sh
export SCRAM_ARCH=slc7_amd64_gcc820
eval `scramv1 project CMSSW CMSSW_10_6_4`
cd CMSSW_10_6_4/src/
eval `scramv1 runtime -sh` # cmsenv is an alias not on the workers
echo "CMSSW: "$CMSSW_BASE
echo "Arguments passed to this script are: for 1: $1, and for 2: $2"
cmsRun ${1} ${2}

Be sure to make your cmsRun.sh executable: chmod +x cmsRun.sh

Your condor.jdl will look something like this (default for case 1. EOS):

universe = vanilla
Executable = cmsRun.sh
Should_Transfer_Files = YES
WhenToTransferOutput = ON_EXIT
Transfer_Input_Files = cmsRun.sh, ExampleConfig.py
Output = sleep_$(Cluster)_$(Process).stdout
Error = sleep_$(Cluster)_$(Process).stderr
Log = sleep_$(Cluster)_$(Process).log
x509userproxy = $ENV(X509_USER_PROXY)
Arguments = ExampleConfig.py 100
Queue 5

Here, the username and X509_USER_PROXY location are read from the environment variables with a different method than in the cat command above.

BACK

Example D: A partial good example script to loop over all root files created during the condor batch job and transfer to EOS - bash script

Example cmsRun.sh:

Some notes: If you passed a root file to the job as input to the cmsRun, you want to delete it before it is transferred back, and only get the output, thus add in something like this: rm inputRootFileName.root. Then we loop over the rest of the root files and transfer them to EOS. We remove them from the local job node working area after transferring them so that the condor jdl doesn't pick up any of those already transferred root files with the option: Should_Transfer_Files = YES. Thanks to Kevin Pedro for the code.

#!/bin/bash
echo "Starting job on " `date` #Date/time of start of job
echo "Running on: `uname -a`" #Condor job is running on this node
echo "System software: `cat /etc/redhat-release`" #Operating System on that node
source /cvmfs/cms.cern.ch/cmsset_default.sh  ## if a tcsh script, use .csh instead of .sh
export SCRAM_ARCH=slc7_amd64_gcc820
eval `scramv1 project CMSSW CMSSW_10_6_4`
cd CMSSW_10_6_4/src/
eval `scramv1 runtime -sh` # cmsenv is an alias not on the workers
echo "CMSSW: "$CMSSW_BASE
echo "Arguments passed to this script are: for 1: $1, and for 2: $2"
cmsRun ${1} ${2}
### Now that the cmsRun is over, there is one or more root files created
echo "List all root files = "
ls *.root
echo "List all files"
ls 
echo "*******************************************"
OUTDIR=root://cmseos.fnal.gov//store/user/username/MyCondorOutputArea/
echo "xrdcp output for condor"
for FILE in *.root
do
  echo "xrdcp -f ${FILE} ${OUTDIR}/${FILE}"
  xrdcp -f ${FILE} ${OUTDIR}/${FILE} 2>&1
  XRDEXIT=$?
  if [[ $XRDEXIT -ne 0 ]]; then
    rm *.root
    echo "exit code $XRDEXIT, failure in xrdcp"
    exit $XRDEXIT
  fi
  rm ${FILE}
done

BACK

How do I troubleshoot my condor job problems?

Spun off to new page: Condor batch system troubleshooting