uscms.org  www 

Connect to the LPC CAF (Central Analysis Facility)

The LPC CAF (cmslpc cluster) is a group of interactive nodes running Scientific Linux Fermi (or SL) that LPC users are able to connect to, to develop and debug their code, submit jobs, do analysis, and so on. Users connect to cmslpc-sl6.fnal.gov, running SL6, arranged in the following way:

cmslpc23.fnal.gov, cmslpc24.fnal.gov, cmslpc25.fnal.gov, cmslpc26.fnal.gov, cmslpc27.fnal.gov, cmslpc28.fnal.gov, cmslpc29.fnal.gov, cmslpc30.fnal.gov, cmslpc31.fnal.gov, cmslpc31.fnal.gov, cmslpc32.fnal.gov, cmslpc33.fnal.gov, cmslpc34.fnal.gov, cmslpc35.fnal.gov, cmslpc36.fnal.gov, cmslpc37.fnal.gov, cmslpc38.fnal.gov, cmslpc39.fnal.gov, cmslpc41.fnal.gov, cmslpc42.fnal.gov all reachable from cmslpc-sl6.fnal.gov round robin, cmslpc40 is SL7

Most operating systems with OpenSSH configured with GSSAPI and Kerberos configured for the FNAL.GOV realm (see below for configuration) should be able to connect to the LPC cluster.

It is assumed that you already have a Fermilab Kerberos principal account. If not visit the "Get an Account" section.

Access to the LPC CAF is controlled by a "round robin" which redirects incoming connections to individual nodes and this page should serve as a guide on how to connect to the LPC CAF in most typical situations.

The status of the cmslpc interactive nodes can be found on ganglia.


Kerberos configuration file

To connect to the CMS LPC cluster you need to have Kerberos installed on your system. This is already included in Scientific Linux and OS X. In addtion you will need to get the krb5.conf file for Fermilab that corresponds to your specific OS. You can download this file from: Fermilab Kerberos Configuration Files.

The krb5.conf file was updated Feb 21, 2018, be sure you have v5.3 of the file on your computer.

/etc/krb5.conf is a text file that you can review and compare to the one found on Fermilab Kerberos Configuration Files to determine whether you already have the most recent one or you need to update it.

Save the appropriate file to your home directory as krb5.conf (you may need to rename it as some browsers will save it as krb5.conf.txt). Note: if you already have a krb5.conf file, it is a good idea to backup it up before placing the new file. Then run (depending on your system, you will be prompted for your root or admin password):

[localMachine:Desktop localUser]$ sudo mv -v krb5.conf /etc/krb5.conf
krb5.conf -> /etc/krb5.conf

Once done, the file is now in the appropriate location on your system.

SSH configuration file

In order to give SSH the proper configuration to login to the CMS LPC cluster, make sure the following lines are present in your ~/.ssh/config file (depending on your OS and). Note: you may not have such a file, and you may need to create the .ssh directory and the file

- - - - - - - - Create ~/.ssh/config file  - - - - - - - - - -
[localMachine:Desktop localUser]$ cd
[localMachine:Desktop localUser]$ mkdir .ssh
[localMachine:Desktop localUser]$ touch .ssh/config
- - - - - - - - Create ~/.ssh/config file  - - - - - - - - - -

Contents of ~/.ssh/config:

[localMachine:Desktop localUser]$ more ~/.ssh/config
# - - - - - - - - - - - - LINUX  - - - - - - - - - - - - - - - - - -
Host cmslpc*.fnal.gov
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
    GSSAPITrustDNS yes
# - - - - - - - - - - - - LINUX additional - - - - - - - - - - - - -
Host cmslpc*.fnal.gov
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
# - - - - - - - - - - - - LINUX additional - - - - - - - - - - - - -

# - - - - - - - - - - - - OS X  - - - - - - - - - - - - - - - - - 
Host cmslpc*.fnal.gov
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
# - - - - - - - - - - - - OS X additional - - - - - - - - - - - -
Host cmslpc*.fnal.gov
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
# - - - - - - - - - - - - OS X additional - - - - - - - - - - - -

Note about additional config: You may find some or all of these options useful to be added to your ~/.ssh/config for macOS 10.12-10.13, PC, and some versions of Linux. Only apply them to cmslpc*.fnal.gov, as they would be insecure for other hosts. See the note about RSA keys at the bottom of this page to understand more.

Connecting to the cmslpc-sl6 LPC CAF

Once your system has the appropriate prerequisites to connect to the LPC CAF do the following to login.

Get a Kerberos ticket

A valid Kerberos ticket for the FNAL.GOV Kerberos realm is needed to login to the LPC CAF, you can get one from your terminal by doing:

[localMachine:Desktop localUser]$ kinit username@FNAL.GOV
- - - - - - LINUX - - - - - -
Password for username@FNAL.GOV:
- - - - - - LINUX - - - - - -

- - - - - - OS X - - - - - -
username@FNAL.GOV's password:
- - - - - - OS X - - - - - -

if successful, there is no output to this command. You can always verify that a Kerberos ticket was created by issuing the following command:

[localMachine:Desktop localUser]$ klist
- - - - - - - - - - - - - - LINUX - - - - - - - - - - - - - -
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: username@FNAL.GOV

Valid starting     Expires            Service principal
04/25/16 16:46:53  04/26/16 18:45:29  krbtgt/FNAL.GOV@FNAL.GOV
        renew until 05/02/16 16:45:29
- - - - - - - - - - - - - - LINUX - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - OS X - - - - - - - - - - - - - - - -
Credentials cache: API:123DS3AA-1554-4A12-9A08-1982938328
        Principal: username@FNAL.GOV
  Issued                Expires               Principal
Apr 25 16:46:53 2016  Apr 26 18:45:29 2016  krbtgt/FNAL.GOV@FNAL.GOV
- - - - - - - - - - - - - - - - OS X - - - - - - - - - - - - - - - -

Show me these two commands (β).

SSH to the LPC CAF

With a valid Kerberos ticket, you can now login to a SL6 machine on the LPC CAF by doing:

- - - - - - - - - - - - - - LINUX  or OS X - - - - - - - - - - - - - - - - 
[localMachine:Desktop localUser]$ ssh -Y username@cmslpc-sl6.fnal.gov
- - - - - - - - - - - - - - LINUX  or OS X - - - - - - - - - - - - - - - - 

Warning: Permanently added 'cmslpc-sl6.fnal.gov,' (RSA) to the list of known hosts.
Last login: Tue Feb 20 13:26:23 2018 from
                              NOTICE TO USERS

       This  is a Federal computer (and/or it is directly connected to a
       Fermilab local network system) that is the property of the United
       States Government.  It is for authorized use only.  Users (autho-
       rized or unauthorized) have no explicit or  implicit  expectation
       of privacy.

       Any  or  all uses of this system and all files on this system may
       be intercepted, monitored, recorded,  copied, audited, inspected,
       and  disclosed  to authorized site, Department of Energy  and law
       enforcement personnel, as  well as authorized officials of  other
       agencies,  both  domestic and foreign.  By using this system, the
       user consents to such interception, monitoring, recording,  copy-
       ing,  auditing,  inspection,  and disclosure at the discretion of
       authorized site or Department of Energy personnel.

       Unauthorized or improper use of this system may result in  admin-
       istrative  disciplinary  action and civil and criminal penalties.
       By continuing to use this system you indicate your  awareness  of
       and  consent to these terms and conditions of use.  LOG OFF IMME-
       DIATELY if you do not agree to  the  conditions  stated  in  this

       Fermilab  policy  and  rules for computing, including appropriate
       use, may be found at http://www.fnal.gov/cd/main/cpolicy.html
                     ..::Powered by CMS-LPC::..                      

   Hostname: cmslpc23.fnal.gov           OS Release: SLF 6.9 (Ramsey)          
         IP:                 Subnet:             

     Kernel: 2.6.32-696.20.1                   Arch: x86_64                    
        RAM: 31.34 GiB                         Swap: 32.00 GiB                 
      Cores: 24                             Virtual: physical                  

 SSH Logins: 5                             Load Avg: 1.31 1.31 1.3             
   For information about computing at the LPC go to: lpc.fnal.gov/computing   
[username@cmslpc23 ~]$ 

Note: the lines just above the prompt each time you login have an important message, is a quick reminder on where to go if you need to look for some information about computing at the LPC.

Connecting to the cmslpc SL7 node (cmslpc40)

As of October, 2017, there is one SL7 node available for interactive use in the LPC CAF: cmslpc40.fnal.gov. After you have obtained your Kerberos ticket, connect with:

[localMachine:Desktop localUser]$ ssh -Y username@cmslpc40.fnal.gov
Caveats about this single interactive node:
  • No ability to submit condor jobs, as there are no SL7 condor batch machines available at the cmslpc at this time
  • Should be able to submit CMS Connect jobs to SL7 nodes
  • Should be able to run SL6 code on SL7 - let us know!
  • Contact LPC Computing Support (Marguerite) with problems you have found

Connecting to the LPC Cluster from Mac OSX

  • Mac OSX has the Terminal program installed in the Applications/Utilities folder which you can use to enter the kinit and ssh commands above after you have configured your /etc/krb5.conf and ~/.ssh/config as described above.
  • Mac OSX does not come with an X11 server installed. The XQuartz server can be installed from xquartz.org. Download the XQuartz package, double click on the .dmg, and follow the instructions to install it.

Connecting to the LPC Cluster from a Windows PC

Connecting to the LPC cluster through the round-robin queue from a Windows PC is known to work with Cygwin/X or Kerberized PuTTY. Directions for installing, configuring, and connecting to the LPC CAF are given below. Also included are directions for using WinSCP to access files in your account.


Cygwin/X is an port of the UNIX-like API and X-Window system to Windows.

The full Cygwin/X install, setup/configuration, and usage for the LPC CAF is shown with screenshots and extensive explanation on this ShowMeHow page.

A description of the install is listed here without images- note you may need Windows management privileges on your computer:
  • To install Cygwin/X follow the download and installation directions from the user's guide.
  • Be sure the local package directory is set to C:\cygwin54
  • Use the search function to find, and click the checkbox to install the "bin" of the following packages:
    • krb5-auth-dialog
    • krb5-workstation
    • openssh
    • xinit
  • You may find the following optional packages useful:
    • wget for getting programs to install under cygwin
    • emacs for local text editing on the cygwin command line
    • nano for local text editing on the cygwin command line
    • xclock for testing the local cygwin X11 server
  • Let Cygwin/X resolve dependencies, ignore the error about "libsub0-Win32 driver"
  • Start the "Cygwin64 Terminal" program
  • Edit (using emacs or nano) your ~/.bashrc to contain: export DISPLAY=:0.0
  • Then follow the instructions similar to above for configuration
    • Get the Linux krb5.conf file and save on the Cygwin/X terminal command line to /etc/krb5.conf. This is actually in Windows filesystem: C:\cygwin64\etc
    • The krb5.conf file was updated Feb 21, 2018, be sure you have v5.3 of the file on your computer.

    • Make and edit your ~/.ssh/config (Windows filesystem: C:\cygwin64\home\yourusername\.ssh\config) file to contain the following:
        # - - - - - - - - - - - -  cygwin - - - - - - - - - - - - -
        Host cmslpc*.fnal.gov
            GSSAPIAuthentication yes
            GSSAPIDelegateCredentials yes
            StrictHostKeyChecking no
            UserKnownHostsFile /dev/null
        # - - - - - - - - - - - -  cygwin - - - - - - - - - - - - -
  • Daily use, Start the "Cygwin64 Terminal" program (last two items will look the same in Linux/MacOSX as in your Cygwin64 terminal:
    • Start the Cygwin/XWindow server with the command: startxwin&
    • Get your Fermilab Kerberos ticket with: kinit username@FNAL.GOV
    • ssh to the cmslpc cluster with: ssh -Y username@cmslpc-sl6.fnal.gov
Note: Documentation for cygwin adapted from local FNAL instructions, and local cmslpc user instructions.

Kerberos and SSH with PuTTY and XMing:

The usage of PuTTY at Fermilab is described in this ServiceNow Knowledge Base article

To get your Kerberos ticket:
The Ticket Manager lives in the taskbar notification area (lower right side of the screen). Right-click on its icon (the one with a silhouette of a K), and select MIT Kerberos Window. In the Ticket Manager window, select Get Ticket. Enter your Kerberos principal name (username@FNAL.GOV), and password. Make sure the options for "Remember this Principal", "Forwardable and Proxiable", and "Renewable" are selected. Click "OK".

To connect to the LPC cluster for the first time with PuTTY:
Double click on the PuTTY icon in the directory where you unzipped the zip file.
In the PuTTY configuration window:

  • select Session and enter cmslpc-sl6.fnal.gov in the HostName field
  • select Connection -> Data and enter your username in the Auto-login username field
  • select Connection -> SSH and select "2 only" for "Preferred SSH protocol version"
  • select Connection -> SSH -> X11 and check "Enable X11 forwarding"
  • select Connection -> SSH -> Auth -> GSSAPI and check both boxes: "Attempt GSSAPI authentication" and "Allow GSSAPI credential delegation" settings
  • select Session and enter LPC in the Saved Sessions field and click Save
double click on LPC in the Saved Sessions list

Daily use: to connect to the LPC cluster with PuTTY (after you have obtained your Kerberos ticket):
Double click the PuTTY icon.
In the PuTTY configuration window:
select Session and double click on LPC in the Saved Sessions list

Kerberos and SFTP:

WinSCP supports Kerberos authentication for SFTP on Windows can be downloaded from http://winscp.net/eng/index.php. Follow the directions here check Advanced options and under SSH -> Authentication check both boxes for GSSAPI Authentication. Use this program to transfer files to and from the LPC cluster.

Troubleshooting your connection

Troubleshooting MacOSX connections

  • If XQuartz is not opening remote windows (occasionally after a software update), be sure that you did the following:
    • Use ssh -Y (the -Y enables trusted X11 forwarding)
    • In XQuartz Preferences, the "Security" tab, ensure that both "Authenticate connections" is checked. If you still have problems, ensure "Allow connections from network clients" is checked.
    • If you have changed your network identity (switching wireless access point, IP address), you may have to restart XQuartz and possibly Terminal to make new Xauthority access control keys if Xwindow forwarding no longer works
    • If you get this error: Warning: No xauth data; using fake authentication data for X11 forwarding.
      • First, be sure you did ssh -Y and you have XQuartz running with the settings listed above
      • Add to your ~/.ssh/config file:
        Host *
            XAuthLocation /opt/X11/bin/xauth
  • Additional troubleshooting note: if you get OpenGL errors, you can remedy them by entering this in Mac OSX Terminal program of your laptop: defaults write org.macosforge.xquartz.X11 enable_iglx -bool true

Rare Linux OpenSSH bug:

If you are running a Linux machine and have setup your ~/.ssh/config as shown above, you have a valid Kerberos ticket (klist to check), and the appropriate /etc/krb5.conf, but using ssh -vv gives you the following errors:

 debug1: An invalid name was supplied
Permission denied (gssapi-keyex,gssapi-with-mic).
This occurred with the following versions and we understand is related to this bug as reported in archlinux: https://bugs.archlinux.org/task/50054. You should be able to log directly into a cmslpc node (see the image above for the list, for instance: ssh -Y username@cmslpc24.fnal.gov
  • OpenSSH_7.5p1, OpenSSL 1.1.0f, May 25, 2017
  • OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g, Nov 16, 2017

Troubleshooting gssapi failures:

If you get the following error:

Warning: Permanently added 'cmslpc-sl6.fnal.gov' (RSA) to the list of known hosts.
Permission denied (gssapi-keyex,gssapi-with-mic).
Note: The warning about adding the RSA key is normal and you can accept the key.
For errors such as above, you need to ensure the following:
  • Did you get an account on the cmslpc? You would have gotten an email with the following: "You should have received a separate email with information about getting started with CMS. More information on using the USCMS LPC can be found at: http://www.uscms.org/uscms_at_work/computing/getstarted/index.shtml"
    • Alternately, a colleague logged into the cmslpc cluster can check with finger username to check if you have an account
    • Be sure you spelled your username correctly
  • Did you get the latest /etc/krb5.conf as described above?
  • Did you configure your ~/.ssh/config properly following the procedure above? (note cygwin users follow the Linux configuration)
    • You can also check for any other settings that may have been changed (not usually, but could be the case) in your /etc/ssh/ssh-config
  • Did you get a valid Kerberos ticket? Follow this to kinit
    • Note: Do not do kinit -af as you may have done in the past, that was for older versions of kerberos and will give you the error above if you use it for the current version. You will need to kdestroy, and then kinit user@FNAL.GOV to get a correct kerberos ticket
    • Be sure to check what your alias is defined as in case you have an old alias configured for kinit or ssh
    • Check your kerberos ticket with klist
    • If you did NOT get a valid kerberos ticket, a ssh -vvv username@cmslpc-sl6.fnal.gov will report this error: No credentials cache file found
  • Otherwise, you can test logging into one of the nodes listed at the top of the page, for example: ssh -vvY username@cmslpc31.fnal.gov and see if that works for you

Troubleshooting credential failures:

  • [MyComputer:~] username% kinit username@fnal.gov
    kinit: krb5_cc_get_principal: No credentials cache file found
    In this case, the kinit was done to the wrong domain, it should be capitalized FNAL.GOV
  • Error: kinit: krb5_get_init_creds: Error from KDC: CLIENT EXPIRED
    In this case the user's Kerberos password expired and/or the account credentials need renewal.


  • As of Spring, 2018, the LPC CAF (cmslpc cluster) no longer mounts /afs directories. You may still access the /afs filesystem from CERN lxplus, but be aware that it is being phased out.

RSA keys and the cmslpc cluster

As described above, a login to the cmslpc-sl6.fnal.gov goes to one of multiple nodes. This will redirect you to one of the nodes and as of July 2017 is not a true load balancer, just a "round robin". Certain clients (Mac specifically) will often get "stuck" on one node. A true load balancer will be installed in early 2018 for the SL7 machines.

Since each of the individual node (see image at the top of this page) is a different machine, they will have different RSA keys even though you can reach them all with ssh -Y username@cmslpc-sl6.fnal.gov. When you modify your ~/.ssh/config, be sure to only apply these settings to cmslpc*.fnal.gov machines as they may be insecure for other clusters. You will need to accept a new RSA key fingerprint for each login, and in the current configurations above, they are sent to /dev/null and thus not retained for cmslpc*.fnal.gov machines.
Webmaster | Last modified: Tuesday, 13-Mar-2018 09:44:42 CDT