U.S. CMS
Search
uscms.org  www 

Connect to the LPC CAF (Central Analysis Facility)

The LPC CAF (cmslpc cluster) is a group of interactive nodes running Scientific Linux Fermi (or SL) that LPC users are able to connect to, to develop and debug their code, submit jobs, do analysis, and so on. Users connect to cmslpc-sl6.fnal.gov, running SL6, arranged in the following way:

cmslpc23.fnal.gov, cmslpc24.fnal.gov, cmslpc25.fnal.gov, cmslpc26.fnal.gov, cmslpc27.fnal.gov, cmslpc28.fnal.gov, cmslpc29.fnal.gov, cmslpc30.fnal.gov, cmslpc31.fnal.gov, cmslpc31.fnal.gov, cmslpc32.fnal.gov, cmslpc33.fnal.gov, cmslpc34.fnal.gov, cmslpc35.fnal.gov, cmslpc36.fnal.gov, cmslpc37.fnal.gov, cmslpc38.fnal.gov, cmslpc39.fnal.gov, all reachable from cmslpc-sl6.fnal.gov round robin, cmslpc40, cmslpc41, and cmlspc42 are SL7



Most operating systems with OpenSSH configured with GSSAPI and Kerberos configured for the FNAL.GOV realm (see below for configuration) should be able to connect to the LPC cluster.

It is assumed that you already have a Fermilab Kerberos principal account. If not visit the "Get an Account" section.

Access to the LPC CAF is controlled by a "round robin" which redirects incoming connections to individual nodes and this page should serve as a guide on how to connect to the LPC CAF in most typical situations.

The status of the cmslpc interactive nodes can be found on ganglia.

Prerequisites

Kerberos configuration file

To connect to the CMS LPC cluster you need to have Kerberos installed on your system. This is already included in Scientific Linux and OS X. In addtion you will need to get the krb5.conf file for Fermilab that corresponds to your specific OS. You can download this file from: Fermilab Kerberos Configuration Files.

The krb5.conf file was updated Feb 21, 2018, be sure you have v5.3 of the file on your computer.

/etc/krb5.conf is a text file that you can review and compare to the one found on Fermilab Kerberos Configuration Files to determine whether you already have the most recent one or you need to update it.

Save the appropriate file to your home directory as krb5.conf (you may need to rename it as some browsers will save it as krb5.conf.txt). Note: if you already have a krb5.conf file, it is a good idea to backup it up before placing the new file. Then run (depending on your system, you will be prompted for your root or admin password):

[localMachine:Desktop localUser]$ sudo mv -v krb5.conf /etc/krb5.conf
Password:
krb5.conf -> /etc/krb5.conf

Once done, the file is now in the appropriate location on your system.

SSH configuration file

In order to give SSH the proper configuration to login to the CMS LPC cluster, make sure the following lines are present in your ~/.ssh/config file (depending on your OS and). Note: you may not have such a file, and you may need to create the .ssh directory and the file

- - - - - - - - Create ~/.ssh/config file  - - - - - - - - - -
[localMachine:Desktop localUser]$ cd
[localMachine:Desktop localUser]$ mkdir .ssh
[localMachine:Desktop localUser]$ touch .ssh/config
- - - - - - - - Create ~/.ssh/config file  - - - - - - - - - -


Contents of ~/.ssh/config:

[localMachine:Desktop localUser]$ more ~/.ssh/config
# - - - - - - - - - - - - LINUX  - - - - - - - - - - - - - - - - - -
Host cmslpc*.fnal.gov
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
    GSSAPITrustDNS yes
# - - - - - - - - - - - - LINUX additional - - - - - - - - - - - - -
Host cmslpc*.fnal.gov
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
# - - - - - - - - - - - - LINUX additional - - - - - - - - - - - - -

# - - - - - - - - - - - - OS X  - - - - - - - - - - - - - - - - - 
Host cmslpc*.fnal.gov
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
# - - - - - - - - - - - - OS X additional - - - - - - - - - - - -
Host cmslpc*.fnal.gov
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
# - - - - - - - - - - - - OS X additional - - - - - - - - - - - -

Note about additional config: You may find some or all of these options useful to be added to your ~/.ssh/config for macOS 10.12-10.13, PC, and some versions of Linux. Only apply them to cmslpc*.fnal.gov, as they would be insecure for other hosts. See the note about RSA keys at the bottom of this page to understand more.

Connecting to the cmslpc-sl6 LPC CAF

Once your system has the appropriate prerequisites to connect to the LPC CAF do the following to login.

Get a Kerberos ticket

A valid Kerberos ticket for the FNAL.GOV Kerberos realm is needed to login to the LPC CAF, you can get one from your terminal by doing:

[localMachine:Desktop localUser]$ kinit username@FNAL.GOV
- - - - - - LINUX - - - - - -
Password for username@FNAL.GOV:
- - - - - - LINUX - - - - - -

- - - - - - OS X - - - - - -
username@FNAL.GOV's password:
- - - - - - OS X - - - - - -

if successful, there is no output to this command. You can always verify that a Kerberos ticket was created by issuing the following command:

[localMachine:Desktop localUser]$ klist
- - - - - - - - - - - - - - LINUX - - - - - - - - - - - - - -
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: username@FNAL.GOV

Valid starting     Expires            Service principal
04/25/16 16:46:53  04/26/16 18:45:29  krbtgt/FNAL.GOV@FNAL.GOV
        renew until 05/02/16 16:45:29
- - - - - - - - - - - - - - LINUX - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - OS X - - - - - - - - - - - - - - - -
Credentials cache: API:123DS3AA-1554-4A12-9A08-1982938328
        Principal: username@FNAL.GOV
    
  Issued                Expires               Principal
Apr 25 16:46:53 2016  Apr 26 18:45:29 2016  krbtgt/FNAL.GOV@FNAL.GOV
- - - - - - - - - - - - - - - - OS X - - - - - - - - - - - - - - - -

Show me these two commands (β).

SSH to the LPC CAF

With a valid Kerberos ticket, you can now login to a SL6 machine on the LPC CAF by doing:

- - - - - - - - - - - - - - LINUX  or OS X - - - - - - - - - - - - - - - - 
[localMachine:Desktop localUser]$ ssh -Y username@cmslpc-sl6.fnal.gov
- - - - - - - - - - - - - - LINUX  or OS X - - - - - - - - - - - - - - - - 


Warning: Permanently added 'cmslpc-sl6.fnal.gov,131.225.190.121' (RSA) to the list of known hosts.
Last login: Tue Feb 20 13:26:23 2018 from 131.225.100.153
                              NOTICE TO USERS

       This  is a Federal computer (and/or it is directly connected to a
       Fermilab local network system) that is the property of the United
       States Government.  It is for authorized use only.  Users (autho-
       rized or unauthorized) have no explicit or  implicit  expectation
       of privacy.

       Any  or  all uses of this system and all files on this system may
       be intercepted, monitored, recorded,  copied, audited, inspected,
       and  disclosed  to authorized site, Department of Energy  and law
       enforcement personnel, as  well as authorized officials of  other
       agencies,  both  domestic and foreign.  By using this system, the
       user consents to such interception, monitoring, recording,  copy-
       ing,  auditing,  inspection,  and disclosure at the discretion of
       authorized site or Department of Energy personnel.

       Unauthorized or improper use of this system may result in  admin-
       istrative  disciplinary  action and civil and criminal penalties.
       By continuing to use this system you indicate your  awareness  of
       and  consent to these terms and conditions of use.  LOG OFF IMME-
       DIATELY if you do not agree to  the  conditions  stated  in  this
       warning.

       Fermilab  policy  and  rules for computing, including appropriate
       use, may be found at http://www.fnal.gov/cd/main/cpolicy.html
------------------------------------------------------------------------------
                     ..::Powered by CMS-LPC::..                      

   Hostname: cmslpc23.fnal.gov           OS Release: SLF 6.9 (Ramsey)          
         IP: 131.225.190.121                 Subnet: 255.255.252.0             

     Kernel: 2.6.32-696.20.1                   Arch: x86_64                    
        RAM: 31.34 GiB                         Swap: 32.00 GiB                 
      Cores: 24                             Virtual: physical                  

 SSH Logins: 5                             Load Avg: 1.31 1.31 1.3             
------------------------------------------------------------------------------
   For information about computing at the LPC go to: lpc.fnal.gov/computing   
------------------------------------------------------------------------------
[username@cmslpc23 ~]$ 

Note: the lines just above the prompt each time you login have an important message, is a quick reminder on where to go if you need to look for some information about computing at the LPC.

Connecting to the cmslpc SL7 nodes (cmslpc-sl7.fnal.gov)

As of October, 2018, there are three SL7 node available for interactive use in the LPC CAF: cmslpc40.fnal.gov, cmslpc41.fnal.gov, and cmslpc42.fnal.gov. After you have obtained your Kerberos ticket, connect with:

[localMachine:Desktop localUser]$ ssh -Y username@cmslpc-sl7.fnal.gov
Caveats about these SL7 interactive nodes:
  • No ability to submit condor jobs, as there are no SL7 condor batch machines available at the cmslpc at this time
  • Should be able to submit CMS Connect or CRAB jobs to SL7 nodes
  • Be careful if your executable links to slc6 architecture, you may find you need to build it with a new slc7 CMSSW version.
  • Contact LPC Computing Support (Marguerite) with problems you have found
  • Note that a SL7 node will indicate the operating system in the message of the day, for instance:

    
    ------------------------------------------------------------------------------
                         ..::Powered by CMS-LPC::..                      
    
       Hostname: cmslpc42.fnal.gov           OS Release: SLF 7.5 (Nitrogen) 
       

Connecting to the LPC Cluster from Mac OSX

  • Mac OSX has the Terminal program installed in the Applications/Utilities folder which you can use to enter the kinit and ssh commands above after you have configured your /etc/krb5.conf and ~/.ssh/config as described above.
  • Mac OSX does not come with an X11 server installed. The XQuartz server can be installed from xquartz.org. Download the XQuartz package, double click on the .dmg, and follow the instructions to install it.

Connecting to the LPC Cluster from a Windows PC

Connecting to the LPC cluster through the round-robin queue from a Windows PC is known to work with Cygwin/X or Kerberized PuTTY. Directions for installing, configuring, and connecting to the LPC CAF are given below. Also included are directions for using WinSCP to access files in your account.

Cygwin/X:

Cygwin/X is an port of the UNIX-like API and X-Window system to Windows.

The full Cygwin/X install, setup/configuration, and usage for the LPC CAF is shown with screenshots and extensive explanation on this ShowMeHow page.

A description of the install is listed here without images- note you may need Windows management privileges on your computer:
  • To install Cygwin/X follow the download and installation directions from the user's guide.
  • Be sure the local package directory is set to C:\cygwin64
  • Use the search function to find, click the "+" to expand, and click the "Skip" to change it to check the checkbox for the "bin" of the following packages:
    • krb5-auth-dialog
    • krb5-workstation
    • openssh
    • xinit
  • You may find the following optional packages useful:
    • wget for getting programs to install under cygwin
    • emacs for local text editing on the cygwin command line
    • nano for local text editing on the cygwin command line
    • xclock for testing the local cygwin X11 server
  • Let Cygwin/X resolve dependencies, ignore the error about "libsub0-Win32 driver". If packages fail to download, be sure to "Retry" instead of continue to get those missing packages. In most cases they will work. You can always install more packages using the Cygwin-X "setup-x86_64" installer that you have downloaded (choosing a different mirror site if need be).
  • Start the "Cygwin64 Terminal" program
  • Edit (using emacs or nano) your ~/.bashrc to contain: export DISPLAY=:0.0
  • Then follow the instructions similar to above for configuration
    • Get the Linux krb5.conf file and save on the Cygwin/X terminal command line to /etc/krb5.conf. This is actually in Windows filesystem: C:\cygwin64\etc
    • The krb5.conf file was updated Feb 21, 2018, be sure you have v5.3 of the file on your computer.

    • Make and edit your ~/.ssh/config (Windows filesystem: C:\cygwin64\home\yourusername\.ssh\config) file to contain the following:
      • 
        # - - - - - - - - - - - -  cygwin - - - - - - - - - - - - -
        Host cmslpc*.fnal.gov
            GSSAPIAuthentication yes
            GSSAPIDelegateCredentials yes
            StrictHostKeyChecking no
            UserKnownHostsFile /dev/null
        # - - - - - - - - - - - -  cygwin - - - - - - - - - - - - -
        
  • Daily use, Start the "Cygwin64 Terminal" program (last two items will look the same in Linux/MacOSX as in your Cygwin64 terminal:
    • Start the Cygwin/XWindow server with the command: startxwin&
    • Get your Fermilab Kerberos ticket with: kinit username@FNAL.GOV
    • ssh to the cmslpc cluster with: ssh -Y username@cmslpc-sl6.fnal.gov
Note: Documentation for cygwin adapted from local FNAL instructions, and local cmslpc user instructions.

Kerberos and SSH with PuTTY and XMing:

The usage of PuTTY at Fermilab is described in this ServiceNow Knowledge Base article

  • Version 0.63 of PuTTY supports Kerberos authentication for SSH on Windows and can be found at https://www.chiark.greenend.org.uk/~sgtatham/putty/releases/0.63.html. You would want to use the .EXE installer created with Inno Setup putty-0.63-installer link. Putty will be installed in c:/Program Files/Putty.
  • You will also need to download and install MIT Kerberos for Windows (version 4.1 is recommended for Windows 10). The installer can be found at the MIT Kerberos download page. Scroll down to download the "MIT Kerberos for Windows" program (.msi), either 64 bit for newer computers or 32 bit for older versions of Windows. Here is a link to the Fermilab Knowledge Base Article about using Kerberos on Windows.
  • In addition you will need get the krb5.conf file for Fermilab and save it to C:\ProgramData\MIT\Kerberos5\krb5.ini replacing this file if it exists.
  • The krb5.conf file was updated Feb 21, 2018, be sure you have v5.3 of the file on your computer.

  • For XWindows with PuTTY, you will need Xming, an implementations of X11 on the Windows platform. With X11 forwarding enabled in PuTTY and an X server(XMing) running, programs like Root and emacs can be displayed on the Windows desktop.

To get your Kerberos ticket:
The Ticket Manager lives in the taskbar notification area (lower right side of the screen). Right-click on its icon (the one with a silhouette of a K), and select MIT Kerberos Window. In the Ticket Manager window, select Get Ticket. Enter your Kerberos principal name (username@FNAL.GOV), and password. Make sure the options for "Remember this Principal", "Forwardable and Proxiable", and "Renewable" are selected. Click "OK".

To connect to the LPC cluster for the first time with PuTTY:
Double click on the PuTTY icon in the directory where you unzipped the zip file.
In the PuTTY configuration window:

  • select Session and enter cmslpc-sl6.fnal.gov in the HostName field
  • select Connection -> Data and enter your username in the Auto-login username field
  • select Connection -> SSH and select "2 only" for "Preferred SSH protocol version"
  • select Connection -> SSH -> X11 and check "Enable X11 forwarding"
  • select Connection -> SSH -> Auth -> GSSAPI and check both boxes: "Attempt GSSAPI authentication" and "Allow GSSAPI credential delegation" settings
  • select Session and enter LPC in the Saved Sessions field and click Save
double click on LPC in the Saved Sessions list

Daily use: to connect to the LPC cluster with PuTTY (after you have obtained your Kerberos ticket):
Double click the PuTTY icon.
In the PuTTY configuration window:
select Session and double click on LPC in the Saved Sessions list

Kerberos and SFTP:

WinSCP supports Kerberos authentication for SFTP on Windows can be downloaded from http://winscp.net/eng/index.php. Follow the directions here check Advanced options and under SSH -> Authentication check both boxes for GSSAPI Authentication. Use this program to transfer files to and from the LPC cluster.

Troubleshooting your connection

Troubleshoot repeating X11 connection rejected error

See also Mac troubleshooting below for other X11 errors.
  • On cmslpc41, the following error has been observed on Mac and PC, repeating on an intermittent basis while connected to that interactive node:
    • X11 connection rejected because of wrong authentication.
  • Solution is to do the following:
    • On your laptop, add the following to your local ~/.ssh/config:
      
      Host cmslpc*fnal.gov
         ForwardX11 yes 
         ForwardX11Trusted yes 
         ForwardAgent yes 
      
    • On the cmslpc cluster, remove your existing .Xauthority file: rm ~/.Xauthority (this will break existing X11 sessions)
    • Connect a new terminal to the cmslpc cluster and you should not get the error anymore

Troubleshooting MacOSX connections

  • If XQuartz is not opening remote windows (occasionally after a software update), be sure that you did the following:
    • Use ssh -Y (the -Y enables trusted X11 forwarding)
    • In XQuartz Preferences, the "Security" tab, ensure that both "Authenticate connections" is checked. If you still have problems, ensure "Allow connections from network clients" is checked.
    • If you have changed your network identity (switching wireless access point, IP address), you may have to restart XQuartz and possibly Terminal to make new Xauthority access control keys if Xwindow forwarding no longer works
    • If you get this error: Warning: No xauth data; using fake authentication data for X11 forwarding.
      • First, be sure you did ssh -Y and you have XQuartz running with the settings listed above
      • Add to your ~/.ssh/config file:
        Host *
            XAuthLocation /opt/X11/bin/xauth
        
  • Additional troubleshooting note: if you get OpenGL errors, you can remedy them by entering this in Mac OSX Terminal program of your laptop: defaults write org.macosforge.xquartz.X11 enable_iglx -bool true

Windows troubleshooting

  • CygwinX:
    • Problem: Unable to open remote XWindows:
      • Be sure to have started startxwin& in your session. If need be, close all windows and start a new CygwinX session and run startxwin&
    • Problem: gssapi and other connection errors:
      • CygwinX runs internally a lot like Linux, check the Linux debugging below, be sure to use the Linux local ~/.ssh/config and the Linux krb5.conf
      • Try to see if you can connect to an individual node instead of cmslpc-sl6.fnal.gov see the full list at the top of this page
  • PuTTy:
    • Problem:: Configured PuTTy apparently correctly, but still unable to connect.
    • possible solutions:
      • If you have PuTTy 0.7 installed, you have to remove it completely, and only have PuTTy version 0.63 installed
      • Be sure you got a valid FNAL.GOV Kerberos ticket with MIT Kerberos, see above
      • Be sure you have the latest krb5.conf

OpenSSH bug (Linux/Cygwin (Windows):

If you are running a Linux machine (have also observed in Cygwin on Windows and once on a Mac) and have setup your ~/.ssh/config as shown above, you have a valid Kerberos ticket (klist to check), and the appropriate /etc/krb5.conf, but using ssh -vv gives you the following errors:

 debug1: An invalid name was supplied
Permission denied (gssapi-keyex,gssapi-with-mic).
This occurred with the following versions and we understand is related to this bug as reported in archlinux: https://bugs.archlinux.org/task/50054. You should be able to log directly into a cmslpc node (see the image above for the list, for example: ssh -Y username@cmslpc24.fnal.gov
  • OpenSSH_7.5p1, OpenSSL 1.1.0f, May 25, 2017
  • OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g, Nov 16, 2017

Troubleshooting gssapi failures (Mac/Windows/Linux):

If you get the following error:

Warning: Permanently added 'cmslpc-sl6.fnal.gov' (RSA) to the list of known hosts.
Permission denied (gssapi-keyex,gssapi-with-mic).
Note: The warning about adding the RSA key is normal and you can accept the key.
For errors such as above, you need to ensure the following:
  • Did you get an account on the cmslpc? You would have gotten an email with this subject line when your account was created: "Welcome to the USCMS UAF (Central Analysis Facility)".
    • Alternately, a colleague logged into the cmslpc cluster can check with finger username to check if you have an account
    • Be sure you spelled your username correctly
  • Did you get the latest /etc/krb5.conf as described above?
  • Did you configure your ~/.ssh/config properly following the procedure above? (note cygwin users follow the Linux configuration)
    • You can also check for any other settings that may have been changed (not usually, but could be the case) in your /etc/ssh/ssh-config
  • Did you get a valid Kerberos ticket? Follow this to kinit
    • Note: Do not do kinit -af as you may have done in the past, that was for older versions of kerberos and will give you the error above if you use it for the current version. You will need to kdestroy, and then kinit user@FNAL.GOV to get a correct kerberos ticket
    • Be sure to check what your alias is defined as in case you have an old alias configured for kinit or ssh
    • Check your kerberos ticket with klist
    • If you did NOT get a valid kerberos ticket, a ssh -vvv username@cmslpc-sl6.fnal.gov will report this error: No credentials cache file found
  • Otherwise, you can test logging into one of the nodes listed at the top of the page, for example: ssh -vvY username@cmslpc31.fnal.gov and see if that works for you

Troubleshooting credential/kinit failures (Mac/Windows/Linux):

  • [MyComputer:~] username% kinit username@fnal.gov
    kinit: krb5_cc_get_principal: No credentials cache file found
    In this case, the kinit was done to the wrong domain, it should be capitalized FNAL.GOV
  • Error: [MyComputer:~] username% kinit username@FNAL.GOV
    kinit: krb5_get_init_creds: Error from KDC: CLIENT EXPIRED
    In this case the user's Kerberos password expired and/or the account credentials need renewal.
  • Error: [MyComputer:~] username% kinit username@FNAL.GOV
    username@FNAL.GOV's password:
    kinit: krb5_get_init_creds: unable to reach any KDC in realm FNAL.GOV, tried 0 KDCs
    In this case, you need to update your /etc/krb5.conf

AFS:

  • As of Spring, 2018, the LPC CAF (cmslpc cluster) no longer mounts /afs directories. You may still access the /afs filesystem from CERN lxplus, but be aware that it is being phased out.

RSA keys and the cmslpc cluster

As described above, a login to the cmslpc-sl6.fnal.gov goes to one of multiple nodes. This will redirect you to one of the nodes and as of July 2017 is not a true load balancer, just a "round robin". Certain clients (Mac specifically) will often get "stuck" on one node. A true load balancer will be installed in early 2018 for the SL7 machines.

Since each of the individual node (see image at the top of this page) is a different machine, they will have different RSA keys even though you can reach them all with ssh -Y username@cmslpc-sl6.fnal.gov. When you modify your ~/.ssh/config, be sure to only apply these settings to cmslpc*.fnal.gov machines as they may be insecure for other clusters. You will need to accept a new RSA key fingerprint for each login, and in the current configurations above, they are sent to /dev/null and thus not retained for cmslpc*.fnal.gov machines.

Notifications of downtimes; getting help

As a new user, you have been automatically subscribed to the cms_uaf_users at fnal.gov email list. This list will be used by the CMS T1 Facilities Support Team to announce downtimes and provide updates on the CMSLPC and other facilities integral to its use.
System status information, including the cms_uaf_users archives are listed here.

To get help for computing, please consult the LPC Computing Get Help page at the lpc.fnal.gov pages.
Webmaster | Last modified: Friday, 12-Oct-2018 15:13:00 CDT