U.S. CMS
Search
uscms.org  www 

Troubleshooting the Condor Batch System at the CMS LPC CAF

NOTE: This page is newly created to be up to date for the Condor refactor.

There is a search bar in the top right of this page to search this whole site with google. In addition, you can search this page for a key phrase from your error.

Failures that indicate you need a valid voms-proxy in CMS VO

    All of these numbered errors indicate that you need to get a valid proxy in the CMS VO

    with: voms-proxy-init --valid 192:00 -voms cms

    Note that I have added obtaining a proxy as long as possible to the command

  1. Error with many different condor commands including submission (one example shown):
    [username@cmslpc34 condor]$ condor_submit condor_test.jdl
    Querying the CMS LPC pool and trying to find an available schedd...
    
    Attempting to submit jobs to lpcschedd2.fnal.gov
    
    Submitting job(s)
    ERROR: Failed to connect to local queue manager
    AUTHENTICATE:1003:Failed to authenticate with any method
    AUTHENTICATE:1004:Failed to authenticate using GSI
    GSI:5003:Failed to authenticate.  Globus is reporting error (851968:28).  
    There is probably a problem with your credentials.  (Did you run grid-proxy-init?)
    
    • Note that this may also fail if the FNAL CMS LPC CAF doesn't have your CMS grid certificate associated with your username. Follow the directions below to Enable your EOS area.

  2. Condor jobs held:
    [username@cmslpc34 condor]$ condor_q
    -- Schedd: lpcschedd2.fnal.gov : <131.225.188.57:9618?... @ 01/22/19 18:54:16
     ID          OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
    30000998.0   username          1/22 18:53   0+00:00:00 H  0    0.0 condor_test.sh 30000998 0
    30000998.1   username          1/22 18:53   0+00:00:00 H  0    0.0 condor_test.sh 30000998 1
    
    • First, find out why the job is held (it could be for other reasons!):
    • [username@cmslpc34 condor]$ condor_q -better-analyze 30000998.0
      
      The key line is:
      
      Hold reason: Error from slot1_1@cmswn1518.fnal.gov: SHADOW at 131.225.188.57 
      failed to send file(s) to <131.225.206.130:36653>: error reading from 
      /uscms/home/username/x509up_u55555: (errno 2) No such file or directory; 
      STARTER failed to receive file(s) from <131.225.188.57:9618>
      
    • Get your proxy: voms-proxy-init --valid 192:00 -voms cms, and release your jobs from the scheduler they are submitted to:
    • [username@cmslpc34 condor]$ condor_release -name lpcschedd2.fnal.gov -all
      All jobs have been released
      
    • It could instead be that due to being over quota on the home directory NFS disk you were unable to write a proxy file.
  3. If you get an error as follows, it shows that something is wrong with your authentication. Usually this is because you have not associated your CMS grid certificate to the Fermilab CMS LPC CAF account.
    • Submitting job(s)
      ERROR: Failed to connect to local queue manager
      SECMAN:2007:Failed to received post-auth ClassAd
      

      CMS Storage Space Request: Grid certificate at FNAL

    • The CMS LPC CAF system must know about the association of your grid certificate and FNAL username. This is usually done as part of Enable EOS area ticket. You must do this at least once for your grid certificate to be associated with your account, which also lets you write to your EOS area from CRAB.
    • Go to the LPC Service Portal: https://fermi.service-now.com/lpc
      • And do "CMS Storage Space Request", and select "Enable" under "Action Required"
      • It will prompt you for for your DN (Your DN is the result of voms-proxy-info --identity). and CERN username. Submit that to register your DN. It will take a few hours (during weekdays) for it to propagate everywhere

Condor executable (script) errors: "exec format error"

Symptom: standard_init_linux.go:190: exec user process caused "exec format error" in your condor stderr.
  • In general, this means that the Docker container on the worker node doesn't know enough to be able to execute your executable
  • The first line of your script should give the executable shell to use, like:
    • #!/bin/sh
    • #!/bin/bash
    • #!/bin/tcsh
  • Do not have any extra space or anything (not even a ###comment) before the first line of your script that condor executes

My jobs failed due to missing libraries on the worker node

  • Errors are like the following three examples:
    ./Analyzer: error while loading shared libraries: libCore.so: cannot open shared object file: No such file or directory                                              
    xrdcp: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by xrdcp)
    ./condor_exec.exe: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./condor_exec.exe)
    
    • Solution: Something is wrong in setting up your CMSSW or other environment on the condor worker node. If you have been using in your condor.jdl the line getenv = true to pass your interactive environment, an example is given on the condor refactor page
    • Solution: the ./condor_exec.exe error was an example of passing a system installed executable to the condor job from a SL6 system. The condor schedulers are SL7, so will pass a SL7 executable to a SL6 container and it will fail. Call your system executables (if needed) inside a shell script.

  • None of that worked? Fill out a LPC Service Portal ticket/incident (Click on "I'm having a problem!") and give the details of your jobs and the errors they had.

Unable to locate local daemon

  • Error in running script (gridpack generation) looks like:
  • root: could not import htcondor python API:
    Unable to locate local daemon
    
  • The gridpack scripts were written assuming that the condor schedulers (schedd) are on the local machine you are using. In the case of the condor refactor, they are not. condor_submit is actually a python wrapper around the real condor_submit command which first talks to negotiators and then the remote schedulers.
  • This is reported here: https://github.com/cms-sw/genproductions/issues/2101
  • Specifically the following line in the code is related: self.schedd = htcondor.Schedd()
  • Until that issue above is resolved, the workaround is to use CMS Connect

General troubleshooting methods

The following techniques may be useful to troubleshoot condor job problems. The condor user's manual guide to managing jobs goes into more depth for troubleshooting.

Web-based condor monitoring

  • The Landscape LPC and User batch summary web page is quite useful to understand the status of jobs at the cmslpc. The landscape web page requires a valid CMS grid certificate for authentication.
    • In the User Batch summary, choose or type in the username from the pulldown menu
    • Note that landscape updates every ~5 minutes so command line condor monitoring will give you quicker results, although neither are instantaneous
    • In the user batch summary, you can click on "Cluster" to find out all the jobs with the same Cluster (the first number in JobID before the period) running on a particular node. In this view you can find the condor_q -long information, including reason job is held, CPU efficiency, resident set size (memory), and disk usage.
    • In the first LPC landscape screen, the average job wait time is shown as well as some brief information about EOS health
    • The LPC batch summary screen gives more detailed information about the batch system, including condor user priority
    • In all landscape screens, the user can adjust the time range in the upper right corner
    • By logging into landscape (Fermilab services password) in the orange swirl menu in the upper left, a user can customize plots for their view, for instance making a log scale, or a white background

Command line job monitoring

  • From the command line: condor_status -submitters tells you the status of all condor jobs. If your jobs are Idle, they are not yet running, and waiting for an empty slot that matches your job requirements. If your jobs are Held they are stopped for some reason.

User priority

Job logs

  • You can find useful troubleshooting information in your job logs, the jobname_Cluster_Process.log file contains information about where the job is running/ran, the jobname_Cluster_Process.stdout, and jobname_Cluster_Process.stderr files contain the actual executable stdout and stderr. The stdout and stderr are transferred after the job is finished.

condor_q

  • To understand job status (ST) from condor_q, you can refer to the condor user manual (8.7) - see below for condor job troubleshooting to understand why a job is in each status:
    • "Current status of the job, which varies somewhat according to the job universe and the timing of updates. H = on hold, R = running, I = idle (waiting for a machine to execute on), C = completed, X = removed, S = suspended (execution of a running job temporarily suspended on execute node), < = transferring input (or queued to do so), and > = transferring output (or queued to do so)."
  • Be sure to know which scheduler your job was submitted to and is running on. You can query all the schedulers with condor_q. For instance if your condor jobs were scheduled from lpcschedd2.fnal.gov, then you may use the argument -name to query that one along, for instance: condor_q -name lpcschedd2.fnal.gov.
  • Find the jobID of the job you are concerned about with condor_q , you can see why it has that status it has with:

Analyze a condor job


The command to use it (example):
condor_q -name lpcschedd1.fnal.gov -analyze jobID

Note that -analyze and -better-analyze may give misleading results. This is because there are two matchmakers (negotiators) in the pool. One is for T1 pilots from MC production and one for cmslpc jobs. Therefore not only will more machines be reported than you may have access to, but you may occasionally get a message that I am not considering your job for matchmaking (when that's not actually the case for your job, you just landed on the wrong matchmaker).
    [username@cmslpc36 condor]$ condor_q -submitter username
    -- Submitter: lpcschedd1.fnal.gov : <131.225.188.55:9618?... : lpcschedd1.fnal.gov @ 01/22/19 19:40:11
     ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
     974.0   username          1/22 19:39   0+00:00:00 I  0    0.0 condor_test.sh 974 0
    [username@cmslpc36 condor]$ condor_q -analyze 974.0 -name lpcschedd1.fnal.gov
    -- Schedd: lpcschedd1.fnal.gov : <131.225.188.55:9618?...
    ---
    970.000:  Request is held.
    
    Hold reason: Error from slot1@cmswn1898.fnal.gov: STARTER at 131.225.191.192 failed 
    to send file(s) to <131.225.188.55:9618>; SHADOW at 131.225.188.55 failed to write to file 
    /uscms_data/d1/username/filename.root: (errno 122) Disk quota exceeded
    
    In the above case, the user's quota on /uscms_data, the NFS disk is exceeded.
  • In another case, the job requirements didn't allow for the job to run on any machines. It is important to note that due to the way the lpc cluster is partitioned, individual job slots can use more (or less) memory and disk than a standard amount. Therefore in a busy cmslpc farm, job requirements for memory and disk will needlessly restrict a job. It is best unless you need them for certain (and are willing to wait for resources to be available), to not add Requirements.
  • Here is an example output from condor_q -name lpcschedd1.fnal.gov -analyze where an old condor jdl was used:
    
    [username@cmslpc36 condor]$ condor_q -name lpcschedd1.fnal.gov -analyze 1033325
    -- Schedd: cmslpc28.fnal.gov : <131.225.188.55:9618?...
    	Last successful match: Tue Jan 22 14:25:07 2019
    	Last failed match: Wed Jan 23 14:32:46 2019
    
    	Reason for last match failure: no match found 
    
    The Requirements expression for your job is:
    
        ( ( OpSys == "LINUX" ) && ( Arch != "DUMMY" ) ) &&
        ( TARGET.Disk >= RequestDisk ) && ( TARGET.Memory >= RequestMemory ) &&
        ( TARGET.HasFileTransfer )
    
    
    Suggestions:
    
        Condition                         Machines Matched    Suggestion
        ---------                         ----------------    ----------
    1   ( OpSys == "LINUX" )              0                   REMOVE
    2   ( Arch != "DUMMY" )               0                   REMOVE
    3   ( TARGET.Memory >= 2500 )         1302                 
    4   ( TARGET.Disk >= 1000000 )        3736                 
    5   ( TARGET.HasFileTransfer )        6504 
    
    In the above case, condor_q -name lpcschedd1.fnal.gov -better-analyze is a better tool. However, condor_q -name lpcschedd1.fnal.gov -better-analyze wouldn't print the error about being over quota. It turned out the user above had extra commands in the condor job file to hold the job if it went over requested memory. Additionally there were some no longer supported requirements being implemented, which analyze suggested removing.

  • Here is an example with condor_q -name lpcschedd1.fnal.gov -better-analyze, where the user requested a machine with more memory available than any in the cluster.
  • 
    [username@cmslpc36 testJobRestrict]$ condor_q -name lpcschedd1.fnal.gov -better-analyze 1545569
    
    
    -- Schedd: cmslpc36.fnal.gov : <131.225.188.55:9618?...
    User priority for username@fnal.gov is not available, attempting to analyze without it.
    ---
    1545569.000:  Run analysis summary.  Of 6076 machines,
       6076 are rejected by your job's requirements 
          0 reject your job because of their own requirements 
          0 match and are already running your jobs 
          0 match but are serving other users 
          0 are available to run your job
    	No successful match recorded.
    	Last failed match: Tu Jan 22 17:02:40 2019
    
    	Reason for last match failure: no match found 
    
    WARNING:  Be advised:
       No resources matched request's constraints
    
    The Requirements expression for your job is:
    
        ( TARGET.Arch == "x86_64" ) && ( TARGET.OpSys == "LINUX" ) &&
        ( TARGET.Disk >= RequestDisk ) && ( TARGET.Memory >= RequestMemory ) &&
        ( TARGET.HasFileTransfer )
    
    Your job defines the following attributes:
    
        RequestDisk = 10000000
        RequestMemory = 210000
    
    The Requirements expression for your job reduces to these conditions:
    
             Slots
    Step    Matched  Condition
    -----  --------  ---------
    [0]        6076  TARGET.Arch == "x86_64"
    [1]        6076  TARGET.OpSys == "LINUX"
    [3]        1718  TARGET.Disk >= RequestDisk
    [5]           0  TARGET.Memory >= RequestMemory
    
    Suggestions:
    
        Condition                         Machines Matched    Suggestion
        ---------                         ----------------    ----------
    1   ( TARGET.Memory >= 210000 )       0                   MODIFY TO 24028
    2   ( TARGET.Disk >= 10000000 )       1718                 
    3   ( TARGET.Arch == "x86_64" )       6076                 
    4   ( TARGET.OpSys == "LINUX" )       6076                 
    5   ( TARGET.HasFileTransfer )        6076 
    

Querying condor ClassAds

  • Querying some or all of the job ClassAds is a useful technique to understand what's wrong with jobs. To find the most information, you can query on the interactive machine the jobs were submitted from with: condor_q -long JobID
  • Another way to find out the reason a job is held from the command line (use the appropriate -name lpcschedd1.fnal.gov for your job:
    • For all the jobs that are on a particular interactive machine: condor_q -af:jh User HoldReason -name lpcschedd1.fnal.gov.
    • For a single job, put in the jobID, for instance: condor_q 3988530.0 -af:jh User HoldReason -name lpcschedd1.fnal.gov

What are the system limits for jobs running on the cmslpc condor batch system?

  • Time: Jobs are removed for: Running for more than 2 days, held for more than 7 days, idle for more than 30 days
  • Disk: Jobs cannot use more than 40GB of local disk space on the condor worker node
  • Memory: The default memory (if not specified) is 2100 MB, if your job uses more than this, or more than you requested, then it is removed. Got to the Additional condor commands page to learn more about profiling and submitting high memory jobs
  • Technical details:
  • To find out the configuration of the complete system defaults, you can use the command: condor_config_val -dump -name lpcschedd1.fnal.gov, the most important line is:
    • SYSTEM_PERIODIC_REMOVE = (JobUniverse == 5 && JobStatus == 2 && ((time() - EnteredCurrentStatus) > 86400*2)) || (JobRunCount > 10) || (JobStatus == 5 && (time() - EnteredCurrentStatus) > 86400*7) || (DiskUsage > 40000000) || (ResidentSetSize > (RequestMemory * 1000) || (JobStatus == 1 && (time() - EnteredCurrentStatus) > 86400*30) )
    • If your job is removed, you get this which is more in English:
    • SYSTEM_PERIODIC_REMOVE_REASON = strcat("LPC job removed by SYSTEM_PERIODIC_REMOVE due to ", ifThenElse((JobUniverse == 5 && JobStatus == 2 && ((time() - EnteredCurrentStatus) > 86400*2)), "job running for more than 2 days.", ifThenElse((JobRunCount > 10), "job restarting more than 10 times.", ifThenElse((JobStatus == 5 && (time() - EnteredCurrentStatus) > 86400*7), "job held for more than 7 days.", ifThenElse((DiskUsage > 40000000), "job exceeding 40GB disk usage.", ifThenElse((ResidentSetSize > (RequestMemory * 1000)), "job exceeding requested memory.", ifThenElse((JobStatus == 1 && (time() - EnteredCurrentStatus) > 86400*30), "idle more than 30 days", "unknown reasons." ) ) ) ) ) ) )

Advanced condor topics

Advanced condor topics such as: higher memory, more cpu, more disk space, and partitionable slots can be found in the separate Batch System Advanced Topics web page.

Webmaster | Last modified: Tuesday, 02-Jul-2019 15:20:19 CDT