U.S. CMS
Search
uscms.org  www 

Computing Environment Setup: Batch Systems

The batch system available for users of the CMS LPC CAF is condor which allows the user to submit jobs into the lpc batch farm. On this page we will describe how to use this batch system.

NOTE: March 21, 2019

this page is being overhauled, for the most up to date submission details, check out Condor refactor updates

The preferred way to access the most CPU (including the CMS LPC) is through CRAB

For any information not covered below, visit the condor user's manual. Find the version of condor running on lpc with condor_q -version

How do I use CRAB to submit batch jobs?

Note that this is the preferred method to access the most CPU

Guides:

How do I use CMS Connect to submit batch jobs?

How do I use Condor to submit to the cmslpc batch farm?

What are the pre-requisites for all condor jobs on the cmslpc farm?

  • Submitting and running jobs, as well as all condor queries now require the user to have a valid grid proxy in the CMS VO
  • When you obtain your proxy (voms-proxy-init --valid 192:00 -voms cms), it will be saved in your home directory where it can be read by the condor job on the worker node
  • The CMS LPC CAF system must know about the association of your grid certificate and FNAL username. This is usually done as part of the Enable EOS area ticket. You must do this at least once for your grid certificate to be associated with your account, which also lets you write to your EOS area from CRAB.
    • Go to the LPC Service Portal: https://fermi.service-now.com/lpc
      • Use Fermilab SSO, paying attention to instructions to configure your browser once for Kerberos login. Note that your Services credentials are different from Kerberos
      • Choose "CMS Storage Space Request", and select "Enable" under "Action Required"
      • It will prompt you for for your DN (Your DN is the result of voms-proxy-info --identity) and CERN username. Submit that to register your DN. The grid certificate will be known to the nodes within 1-3 hours during FNAL business hours.

A simple condor example: jdl file

The first step to using the condor system is writing the condor jdl (job description language) file. This file will tell the system what you want it to do and how. Below is an example, which will run a system program that will sleep for one minute, then quit. Use your favorite text editor to create the sleep-condor.jdl file with the following contents. Click on any of the green lines below for an explanation of what it does.


universe = vanilla
Executable = sleep.sh
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
Output = sleep_\$(Cluster)_\$(Process).stdout
Error = sleep_\$(Cluster)_\$(Process).stderr
Log = sleep_\$(Cluster)_\$(Process).log
Arguments = 60
Queue 2

A simple condor example: sh file

The next step is to create the executable, in this case a shell script called sleep.sh. Create the file below in your favorite text editor in the same directory as your sleep-condor.jdl file.
#!/bin/bash
set -x
# Sleep
sleep $1
echo "##### HOST DETAILS #####"
echo "I slept for $1 seconds on:"
hostname
date

Submit, monitor, and remove your condor job


After you've created the jdl file sleep-condor.jdl, the shell script sleep.sh, and authenticated your grid certificate to the CMS VO, you can submit it to the condor system using the command condor_submit followed by the name of your submit description file (jdl), in this example's case "sleep-condor.jdl":

condor_submit sleep-condor.jdl

Your output should look something like this:

[username@cmslpc32 ~]$ condor_submit sleep-condor.jdl
Querying the CMS LPC pool and trying to find an available schedd...
Attempting to submit jobs to lpcschedd1.fnal.gov
Submitting job(s)..
2 job(s) submitted to cluster 914329.

You can see the status of all jobs you have submitted (unless they have completed) to all the possible schedulers (lpcschedd1.fnal.gov for example) with the following command:

condor_q Your queue ought to show the processes you just submitted, they may be idle for up to a minute or so, maybe longer if the system is very busy:

[username@cmslpc32 condor]$ condor_q

-- Schedd: lpcschedd1.fnal.gov : <131.225.188.55:9618?... @ 03/21/19 10:09:17
 ID        OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
914329.0   username        3/21 10:08   0+00:00:08 R  0    0.0 sleep.sh 60

Total for query: 1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended 
Total for username: 1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended 
Total for all users: 1642 jobs; 0 completed, 0 removed, 0 idle, 99 running, 1543 held, 0 suspended

-- Schedd: lpcschedd2.fnal.gov : <131.225.188.57:9618?... @ 03/21/19 10:09:17
 ID        OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD

Total for query: 0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended 
Total for username: 0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended 
Total for all users: 2669 jobs; 0 completed, 0 removed, 0 idle, 285 running, 2384 held, 0 suspended

-- Schedd: lpcschedd3.fnal.gov : <131.225.188.235:9618?... @ 03/21/19 10:09:17
 ID        OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD

Total for query: 0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended 
Total for username: 0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended 
Total for all users: 3723 jobs; 1 completed, 0 removed, 0 idle, 70 running, 3652 held, 0 suspended

Note that there is an -allusers option lets you see condor jobs submitted by all users on the schedulers (condor_q -allusers). To see just your own, omit that option.

  • To understand job status (ST) from condor_q, you can refer to the condor user manual (8.8) - see the condor job troubleshooting page to understand why a job is in each status:
    • "Current status of the job, which varies somewhat according to the job universe and the timing of updates. H = on hold, R = running, I = idle (waiting for a machine to execute on), C = completed, X = removed, S = suspended (execution of a running job temporarily suspended on execute node), < = transferring input (or queued to do so), and > = transferring output (or queued to do so)."
  • Be sure to know which scheduler your job was submitted to and is running on.

You can also monitor jobs with the landscape at FNAL LPC web monitor, keep in mind the results may be delayed up to 5 minutes.

You can specifically get a list of all the jobs and their status for a specific user username from any machine using this command:

condor_q -submitter username

If you want to view the entire queue, you can use the following command:

condor_status -submitters

This gives all the jobs from all users, including jobs coming in from CRAB and CMS connect into pilots, and jobs in the T1_US_FNAL workers. Note that cmslpc jobs submitted on local condor will not run on T1_US_FNAL workers. See the condor system status web page for some command line methods to find out only jobs for the local batch.

To cancel a job type condor_rm followed by the job number, for this example, 60000042.0 and 60000042.1

condor_rm -name lpcschedd3.fnal.gov 60000042

If you don't remember your scheduler name for that job, use the condor_q command from above, it will tell you what scheduler your job was submitted from.

If you want to remove all your jobs that went to all the schedulers, you would have do do the following:
condor_rm -name lpcschedd1.fnal.gov -all; condor_rm -name lpcschedd2.fnal.gov -all; condor_rm -name lpcschedd3.fnal.gov -all

If you wish to see the end of a current running job stdout, you can use the condor_tail command together with the job number. The results will give you ~20 lines of output, it is cut here for space in this example:

condor_tail 60000042.1

== CMSSW: 09-Mar2019 22:26:32 CST  Successfully opened file root://cmseos.fnal.gov//store/user/username/myfile.root
== CMSSW: Begin processing the 1st record. Run 1, Event 254, LumiSection 6 at 09-Mar-2019 22:30:12.001 CST
== CMSSW: Begin processing the 101st record. Run 1, Event 353, LumiSection 8 at 09-Mar-2019 22:30:28.673 CST
== CMSSW: Begin processing the 201st record. Run 1, Event 451, LumiSection 10 at 09-Mar-2019 22:30:32.670 CST

Consult the condor batch troubleshooting page for more on how to troubleshoot job problems.
Consult the condor system status web page for more on how to monitor the condor batch system.

Details of the condor submit file (jdl)


universe = vanilla
The universe variable defines an execution environment for your job, in this example we use the vanilla universe which has the least amount of built in services, but also the least amount of restrictions. For a complete list of universes and what they do, so the condor user's manual. The cmslpc does NOT support the standard universe, and thus does not have condor_compile.
BACK

Executable = sleep.sh
This is the program you want to run. If the program is in the same directory as your batch file, just the name will work, example: yourscript.sh. If it is in a different directory than your batch file then you must give the pathname, example: myscripts/yourscript.sh runs the script yourscript.sh located in the directory myscripts.

  • Note that we do not use /bin/sleep as our executable, as system level executables are read from the remote scheduler nodes. The lpcschedd*.fnal.gov nodes run SL7, and your jobs could run on SL6 or SL7 Docker containers (the wrapper in condor_submit will detect the OS of the submission node and use that to set the OS of the container).
  • In general, either use a shell script (best because you can add useful comments about the worker node environment), or your own compiled executable
  • The executable you specify in the Executable line in the condor jdl is automatically sent to the worker node, you do not have to specify it in transfer_input_files

BACK

should_transfer_files = YES
when_to_transfer_output = ON_EXIT

These options tell condor to transfer files to/from the batch job. If these options are not activated then you must provide input through some other means and extract the output yourself. Examples below show how to handle input/output to/from FNAL EOS within a shell script. Users cannot do direct read/write from the NFS filesystems within a condor job as NFS is not mounted on the condor worker nodes. Follow this link further down on the page to learn more about condor I/O and how to best manage it for the cmslpc systems.
BACK

Output = sleep_$(Cluster)_$(Process).stdout
This directs the standard output of the program to a file, in other words, everything that would normally be displayed on the screen, so that you can read it after it is finished running. Where you see $(Cluster) condor will substitute the job number, and $(Process) will become the process number, in this case, 0-1.
BACK

Error = sleep_$(Cluster)_$(Process).stderr
This is the same as the Output line, except it applies to standard error, this is extremely useful for debugging or figuring out what is going wrong (all most always something). Where you see $(Cluster) condor will substitute the job number, and $(Process) will become the process number, in this case, 0-1.
BACK

Log = sleep_$(Cluster)_$(Process).log
The log file contains information about the job in the condor system, the ip address of the computer that is processing the job, the time it starts and finishes, how many attempts were made to start the job and other such data. It is recommended to use a log file, where you see $(Cluster) condor will substitute the job number, and $(Process) will become the process number, in this case, 0-1.
BACK

Arguments = 60
Here you put any command line arguments for your program, if you have none, exclude this line. In this example the program needs one argument for the number of seconds to wait. This argument tells the program to wait for one minute, and is used within the shell script as $1.
BACK

Queue 2
This is how many times you want to run the program, without this line the program will not run. The processes will be numbered starting at zero, so in this example they will be: 0, and 2. In case you only want one job, use Queue 1.
BACK

Details of the condor shell script (sh)


Note that a shell script is used here because we are using a system executable. When condor creates the job on the worker node, it transfers the executable from the scheduler, lpcschedd*.fnal.gov (where * is a number). The scheduler nodes run SL7, so will transfer a SL7 system executable, even if you wish to run in a SL6 Docker container on the worker node. A shell script will use the proper operating system for the container.

#!/bin/bash
This line is required for the start of all shell scripts within the condor refactor, to tell the Docker container what type of shell to use. You cannot have anything before this line, not even comments (#). Errors when this is missing is described at the troubleshooting page. BACK

set -x
This line is optional. It has the benefit of reporting to you in your .stderr file the contents of the shell script that the condor worker node is running. BACK

# Sleep
The line which starts with # indicates a comment, and is ignored by the shell script. BACK

sleep $1
The system executable sleep command is used with the argument $1, which is passed from the Arguments 20 line from the sleep-condor.jdl file. BACK

echo "##### HOST DETAILS #####"
echo "I slept for $1 seconds on:"
hostname
date
BACK

How do I manage file input/output from within Condor?

Overall input/output guidelines

  • Your job is limited to 40GB of disk usage on the condor worker node.
  • Do not add a disk requirements in your condor.jdl file unless you are sure you need more as it affects your priority and books resources that aren't used.
  • Condor creates its working are on the condor worker node in the environment variable _CONDOR_SCRATCH_DIR

EOS file input and/or output

The FNAL EOS filesystem is accessible on the condor worker nodes through xrootd. It is not mounted as the EOS FUSE mount on the lpc schedulers (lpcschedd*.fnal.gov). Therefore, we cannot use condor file I/O, and must do our in outside of condor and within condor batch scripts on the condor worker node. Recall that EOS works best on individual files of 1-5GB in size. From here down not yet updated for the refactor.

1) use EOS within the job. 2) condor automatic NFS. Condor has options availble to automatically transfer input files to the worker where the job runs and then copy any output files back to the directory you submit from. Jobs should use these or read/write data from dcache instead of doing file reads or writes directlly from the NFS mounted file systems. Direct reads/writes to NFS can cause severe performance problems. If this happens the CMS T1 Facility team may end of up having to kill your jobs.

The options for telling condor to copy files into and out of your job are:
Should_Transfer_Files = YES
Transfer_Input_Files = file1, file2
WhenToTransferOutput = ON_EXIT

In addtion you need to make sure you use the correct pathname for files. Please take into account that the NFS disks are not mounted on the condor worker nodes.

BACK

How do I compress (tar) my custom CMSSW so that it's small and not overloading the network when transferred to many jobs at the same time?

Simple compression (tar)

Example (use your own CMSSW)
tar -zcvf CMSSW8025.tgz CMSSW_8_0_25

Exclude large files from your tar, like root files, caches, etc...

BACK

4. How to I make sure my batch jobs are not accessing shared/NFS disk?

After October 1, 2017, the NFS shared filesystems is no longer be mounted on condor worker nodes. If you try to access files from /uscms/homes and /uscms_data/d* on the condor worker node, that will fail.

Guide to not using NFS on condor worker nodes, note that this page has more features documented (than here) in condor scripts and may be useful to you.


Example A: A good example

In this case, we avoid the shared filesystem as much as possible. Note that there are no full path names anywhere except for the transfer_input_files. The inputs are transferred to a temporary area on the worker node's local disk, pointed to by the variable _CONDOR_SCRATCH_DIR. More complicated examples can be found in Example B, and Example C, below.
good1.jdl:

universe = vanilla
Executable = good1.sh
Output = good1.out
Error = good1.err
Log = good1.log
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = /uscms_data/d1/username/input_file.root, /uscms_data/d1/username/good1.C


good1.sh:

#!/bin/bash
# cms software setup not included here for brevity
cd ${_CONDOR_SCRATCH_DIR}
root -b -q good1.C

good1.C:

{
TFile f("input_file.root");
output_ntuples = process(f); // do some calculation
TFile g("output_file.root", "w");
g.write(output_ntuples);
}

Example B: A good example with tarring - tcsh script



In the following case, we avoid the shared filesystem as much as possible. Inputs are tarred and transferred to a temporary area on the worker node's local disk (40GB limit), pointed to by the variable _CONDOR_SCRATCH_DIR.

Note: Please note that if your CMSSW tar is large (1 GB or larger), it may have two problems: First it will restrict the machines you can run upon due to the input transfer size; Secondly it may saturate the network if you have very many jobs all starting at once. Instructions are below to exclude caches and reduce the size of your .tar.gz. You will have your best networking results by transferring the file to EOS using xrdcp, as shown below, and xrdcp it to the local working area at the start of the condor batch job.

There are two ways to use this technique which will be referred to below:

  1. Best for networking throughput: the small tarred CMSSW and large (root) input files are transferred to personal EOS storage. The tarred CMSSW is transferred to the local worker node disk, and large input files are opened in root following the EOS instructions for file reference inside a script. Note that the EOS transfer at the end of the job is best for large root files.
  2. The tarred CMSSW and any input files are transferred to the local worker node disk from NFS - may slow down your job starting due to heavy NFS network usage for many jobs in parallel, as well as input file size.
First tar your CMSSW working area. This presumes that in your CMSSW_8_0_25/src is the following files: cmsRun.csh, and ExampleConfig.py
tar -zcvf CMSSW8025.tgz CMSSW_8_0_25

  1. To transfer this to your personal EOS area do (for your CERN username):
  2. xrdcp CMSSW8025.tgz root://cmseos.fnal.gov//store/user/username/CMSSW8025.tgz
  3. In your condor.jdl file which you submit, refer to: Transfer_Input_Files = cmsRun.csh, ExampleConfig.py, CMSSW8025.tar.gz

Any large root files should be transferred to your personal EOS area, and referred to following the EOS instructions for file reference inside a script:
xrdcp Filename1.root root://cmseos.fnal.gov//store/user/username/Filename1.root
xrdcp Filename2.root root://cmseos.fnal.gov//store/user/username/Filename2.root

Your executable file cmsRun.csh will take arguments ${1} (the name of the python configuration file ExampleConfig.py), and ${2}, some other variable you are passing to the configuration file, like number of events. As always, be sure to test these scripts and python configuration files interactively for a single test before submitting many condor jobs.

Example cmsRun.csh:


#!/bin/tcsh
echo "Starting job on " `date` #Date/time of start of job
echo "Running on: `uname -a`" #Condor job is running on this node
echo "System software: `cat /etc/redhat-release`" #Operating System on that node
source /cvmfs/cms.cern.ch/cmsset_default.csh  ## if a bash script, use .sh instead of .csh
### for case 1. EOS have the following line, otherwise remove this line in case 2.
xrdcp -s root://cmseos.fnal.gov//store/user/username/CMSSW8025.tgz .
tar -xf CMSSW8025.tgz
rm CMSSW8025.tgz
setenv SCRAM_ARCH slc6_amd64_gcc530
cd CMSSW_8_0_25/src/
scramv1 b ProjectRename
eval `scramv1 runtime -csh` # cmsenv is an alias not on the workers
echo "Arguments passed to this script are: for 1: $1, and for 2: $2"
cmsRun ${1} ${2}
xrdcp nameOfOutputFile.root root://cmseos.fnal.gov//store/user/username/nameOfOutputFile.root
### remove the output file if you don't want it automatically transferred when the job ends
rm nameOfOutputFile.root
cd ${_CONDOR_SCRATCH_DIR}
rm -rf CMSSW_8_0_25

Be sure to make your cmsRun.csh executable: chmod +x cmsRun.csh

Your condor.jdl will look something like this (default for case 1. EOS):


universe = vanilla
Executable = cmsRun.csh
Should_Transfer_Files = YES
WhenToTransferOutput = ON_EXIT
Transfer_Input_Files = cmsRun.csh, ExampleConfig.py
Output = sleep_\$(Cluster)_\$(Process).stdout
Error = sleep_\$(Cluster)_\$(Process).stderr
Log = sleep_\$(Cluster)_\$(Process).log
x509userproxy = $ENV(X509_USER_PROXY)
Arguments = ExampleConfig.py 100
Queue 5

Case 2. Transfer needs this line modified:


Transfer_Input_Files = cmsRun.csh, ExampleConfig.py, CMSSW8025.tar.gz

Here, the username and X509_USER_PROXY location are read from the environment variables with a different method than in the cat command above.

BACK

Example C: A good example with making a new CMSSW during the batch job - bash script


Example cmsRun.sh:


#!/bin/bash
echo "Starting job on " `date` #Date/time of start of job
echo "Running on: `uname -a`" #Condor job is running on this node
echo "System software: `cat /etc/redhat-release`" #Operating System on that node
source /cvmfs/cms.cern.ch/cmsset_default.sh  ## if a tcsh script, use .csh instead of .sh
export SCRAM_ARCH=slc6_amd64_gcc530
eval `scramv1 project CMSSW CMSSW_8_0_25`
cd CMSSW_8_0_25/src/
eval `scramv1 runtime -sh` # cmsenv is an alias not on the workers
echo "CMSSW: "$CMSSW_BASE
echo "Arguments passed to this script are: for 1: $1, and for 2: $2"
cmsRun ${1} ${2}

Be sure to make your cmsRun.sh executable: chmod +x cmsRun.sh

Your condor.jdl will look something like this (default for case 1. EOS):


universe = vanilla
Executable = cmsRun.sh
Should_Transfer_Files = YES
WhenToTransferOutput = ON_EXIT
Transfer_Input_Files = cmsRun.sh, ExampleConfig.py
Output = sleep_\$(Cluster)_\$(Process).stdout
Error = sleep_\$(Cluster)_\$(Process).stderr
Log = sleep_\$(Cluster)_\$(Process).log
x509userproxy = $ENV(X509_USER_PROXY)
Arguments = ExampleConfig.py 100
Queue 5

Here, the username and X509_USER_PROXY location are read from the environment variables with a different method than in the cat command above.

BACK

Example D: A partial good example script to loop over all root files created during the condor batch job and transfer to EOS - bash script

Example cmsRun.sh:

Some notes: If you passed a root file to the job as input to the cmsRun, you want to delete it before it is transferred back, and only get the output, thus add in something like this: rm inputRootFileName.root. Then we loop over the rest of the root files and transfer them to EOS. We remove them from the local job node working area after transferring them so that the condor jdl doesn't pick up any of those already transferred root files with the option: Should_Transfer_Files = YES. Thanks to Kevin Pedro for the code.


#!/bin/bash
echo "Starting job on " `date` #Date/time of start of job
echo "Running on: `uname -a`" #Condor job is running on this node
echo "System software: `cat /etc/redhat-release`" #Operating System on that node
source /cvmfs/cms.cern.ch/cmsset_default.sh  ## if a tcsh script, use .csh instead of .sh
export SCRAM_ARCH=slc6_amd64_gcc530
eval `scramv1 project CMSSW CMSSW_8_0_25`
cd CMSSW_8_0_25/src/
eval `scramv1 runtime -sh` # cmsenv is an alias not on the workers
echo "CMSSW: "$CMSSW_BASE
echo "Arguments passed to this script are: for 1: $1, and for 2: $2"
cmsRun ${1} ${2}
### Now that the cmsRun is over, there is one or more root files created
echo "List all root files = "
ls *.root
echo "List all files"
ls 
echo "*******************************************"
OUTDIR=root://cmseos.fnal.gov//store/user/username/MyCondorOutputArea/
echo "xrdcp output for condor"
for FILE in *.root
do
  echo "xrdcp -f ${FILE} ${OUTDIR}/${FILE}"
  xrdcp -f ${FILE} ${OUTDIR}/${FILE} 2>&1
  XRDEXIT=$?
  if [[ $XRDEXIT -ne 0 ]]; then
    rm *.root
    echo "exit code $XRDEXIT, failure in xrdcp"
    exit $XRDEXIT
  fi
  rm ${FILE}
done

BACK

How do I troubleshoot my condor job problems?

Spun off to new page: Condor batch system troubleshooting


Advanced condor topics such as: higher memory, more cpu, more disk space, and partitionable slots can be found in the separate Batch System Advanced Topics web page.

BACK


Webmaster | Last modified: Monday, 25-Mar-2019 10:07:07 CDT