U.S. CMS
Search
uscms.org  www 

Connect to the LPC CAF (Central Analysis Facility)

The LPC CAF (cmslpc cluster) is a group of interactive nodes running Linux that LPC users are able to connect to, to develop and debug their code, submit jobs, do analysis, and so on. The main interactive nodes are behind a load balancer. Multiple parallel analysis should be handled by the batch system.

Most operating systems with OpenSSH configured with GSSAPI and Kerberos configured for the FNAL.GOV realm (see below for required configuration prerequisites) should be able to connect to the LPC cluster.

It is assumed that you already have a Fermilab Kerberos principal account (you got the email "Welcome to the CMS LPC CAF (Central Analysis Facility)"). If not visit the "Get an Account" section.

The system status of the cmslpc interactive nodes can be found on the SSI metrics page.

Guidelines for use of the interactive nodes.

Prerequisites

Kerberos configuration file

To connect to the CMS LPC cluster you need to have Kerberos installed on your system. This is already included in Scientific Linux and macOS, but not Windows. In addtion you will need to get the up to date krb5.conf file for Fermilab that corresponds to your specific OS. You can download this file from: Fermilab Kerberos Configuration Files.

The krb5.conf file was updated on May 16, 2023, be sure you have v5.7-3 of the file on your computer.

/etc/krb5.conf is a text file that you can review and compare to the one found on Fermilab Kerberos Configuration Files to determine whether you already have the most recent one or you need to update it.

Save the appropriate file to your home directory as krb5.conf (you may need to rename it as some browsers will save it as krb5.conf.txt). Note: if you already have a krb5.conf file, it is a good idea to backup it up before placing the new file. Then run (depending on your system, you will be prompted for your root or administrator password):

[localMachine:Desktop localUser]$ sudo mv -v krb5.conf /etc/krb5.conf
Password:
krb5.conf -> /etc/krb5.conf

Once done, the file is now in the appropriate location on your system.

SSH configuration file

In order to give SSH the proper configuration to login to the CMS LPC cluster, make sure the following lines are present in your ~/.ssh/config file (depending on your OS and). Note: you may not have such a file, and you may need to create the .ssh directory and the file

- - - - - - - - Create ~/.ssh/config file  - - - - - - - - - -
[localMachine:Desktop localUser]$ cd
[localMachine:Desktop localUser]$ mkdir .ssh
[localMachine:Desktop localUser]$ touch .ssh/config
- - - - - - - - Create ~/.ssh/config file  - - - - - - - - - -


Use your favorite text editor to edit this file on your local machine with the following contents for ~/.ssh/config

[localMachine:Desktop localUser]$ more ~/.ssh/config
# - - - - - - - - - - - - LINUX  - - - - - - - - - - - - - - - - - -
Host cmslpc*.fnal.gov
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
#  - - - - - - - - - - - - LINUX  - - - - - - - - - - - - - - - - - -

Note: GSSAPITrustDNS yes may be needed in addition for some older versions of Linux if your computer complains that the WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

# - - - - - - - - - - - - macOS  - - - - - - - - - - - - - - - - - 
Host cmslpc*.fnal.gov
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
# - - - - - - - - - - - - macOS  - - - - - - - - - - - - - - - - - 

Note: The settings for StrictHostKeyChecking and UserKnownHostsFile should only be applied to cmslpc*.fnal.gov, as they would be insecure for other hosts. See the note about RSA keys at the bottom of this page to understand more.
Note: In case your ssh connection keeps timing out, you may find this useful in your local computer's ~/.ssh/config file:

Host cmslpc*.fnal.gov
    ServerAliveInterval 60
    TCPKeepAlive yes
    KeepAlive yes

Connecting to the cmslpc nodes LPC CAF

Once your system has the appropriate prerequisites to connect to the LPC CAF, you must have a valid Kerberos ticket before logging in. Note that in the examples below we use cmslpc-sl7.fnal.gov for the interactive node, for newer Operating Systems and development nodes, see the full list below.

Get a Kerberos ticket

A valid Kerberos ticket for the FNAL.GOV Kerberos realm is needed to login to the LPC CAF, you can get one from your terminal by doing:

[localMachine:Desktop localUser]$ kinit username@FNAL.GOV
- - - - - - LINUX - - - - - -
Password for username@FNAL.GOV:
- - - - - - LINUX - - - - - -

- - - - - - macOS - - - - - -
username@FNAL.GOV's password:
- - - - - - macOS - - - - - -

if successful, there is no output to this command. You can always verify that a Kerberos ticket was created by issuing the following command (you do not need to do this on a normal login):

[localMachine:Desktop localUser]$ klist
- - - - - - - - - - - - - - LINUX - - - - - - - - - - - - - -
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: username@FNAL.GOV

Valid starting     Expires            Service principal
09/30/19 15:06:33  10/01/19 17:06:30  krbtgt/FNAL.GOV@FNAL.GOV
- - - - - - - - - - - - - - LINUX - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - macOS - - - - - - - - - - - - - - - -
Credentials cache: API:123DS3AA-1554-4A12-9A08-1982138328
        Principal: username@FNAL.GOV
    
 Issued                Expires               Principal
Sep 30 15:06:33 2019  Oct  1 17:06:30 2019  krbtgt/FNAL.GOV@FNAL.GOV/font>
- - - - - - - - - - - - - - - - macOS - - - - - - - - - - - - - - - -

SSH to the LPC CAF

With a valid Kerberos ticket, you can now login to a SL7 machine on the LPC CAF by doing:

- - - - - - - - - - - - - - LINUX  or macOS - - - - - - - - - - - - - - - - 
[localMachine:Desktop localUser]$ ssh -Y username@cmslpc-sl7.fnal.gov
- - - - - - - - - - - - - - LINUX  or macOS - - - - - - - - - - - - - - - - 
Warning: Permanently added 'cmslpc-sl7.fnal.gov,131.225.204.176' (ECDSA) to the list of known hosts.
Last login: Thu Sep 19 21:17:10 2019 from 131.225.189.108
                              NOTICE TO USERS

       This  is a Federal computer (and/or it is directly connected to a
       Fermilab local network system) that is the property of the United
       States Government.  It is for authorized use only.  Users (autho-
       rized or unauthorized) have no explicit or  implicit  expectation
       of privacy.

       Any  or  all uses of this system and all files on this system may
       be intercepted, monitored, recorded,  copied, audited, inspected,
       and  disclosed  to authorized site, Department of Energy  and law
       enforcement personnel, as  well as authorized officials of  other
       agencies,  both  domestic and foreign.  By using this system, the
       user consents to such interception, monitoring, recording,  copy-
       ing,  auditing,  inspection,  and disclosure at the discretion of
       authorized site or Department of Energy personnel.

       Unauthorized or improper use of this system may result in  admin-
       istrative  disciplinary  action and civil and criminal penalties.
       By continuing to use this system you indicate your  awareness  of
       and  consent to these terms and conditions of use.  LOG OFF IMME-
       DIATELY if you do not agree to  the  conditions  stated  in  this
       warning.

       Fermilab  policy  and  rules for computing, including appropriate
       use, may be found at http://www.fnal.gov/cd/main/cpolicy.html
------------------------------------------------------------------------------
                     ..::Powered by CMS-LPC::..                      

   Hostname: cmslpc155.fnal.gov          OS Release: SLF 7.9 (Nitrogen)        
         IP: 131.225.189.191                 Subnet: 255.255.252.0             

     Kernel: 3.10.0-1160.99.1                  Arch: x86_64                    
        RAM: 11.57 GiB                         Swap: 10.00 GiB                 
      Cores: 8                              Virtual: rhev                      

 SSH Logins: 1                             Load Avg: 0.02 0.04 0.05            
------------------------------------------------------------------------------
   For information about computing at the LPC go to: lpc.fnal.gov/computing   
------------------------------------------------------------------------------
[username@cmslpc130 ~]$ 

Notes about the cmslpc interactive nodes:
  • The lines just above the prompt each time you login have an important message as well as the software version, is a quick reminder on where to go if you need to look for some information about computing at the LPC.
  • If your bash shell login doesn't look right, see this link about setting up your ~/.bash_profile
If you are unable to login, see the troubleshooting section below.

Interactive nodes to connect to:

Be sure to get your Kerberos ticket, and ssh per the directions above. In this section is a list of possible node names which you can use instead of cmslpc-sl7.fnal.gov, depending on your analysis or development needs. NOTE: SL7 nodes will be removed May 31, 2024.

For regular interactive connection, code development & debugging, batch job submission, and interactive analysis, the following nodes are available:

  • cmslpc-sl7.fnal.gov, running Scientific Linux Fermi 7
    • The load balancer corresponds to nodes cmslpc101.fnal.gov to cmslpc149.fnal.gov
  • cmslpc-el8.fnal.gov, running Alma Linux 8
    • The load balancer corresponds to nodes cmslpc201.fnal.gov to cmslpc230.fnal.gov
  • cmslpc-el9.fnal.gov, running Alma Linux 9
    • The load balancer corresponds to nodes cmslpc301.fnal.gov to cmslpc330.fnal.gov
  • GPU interactive nodes are described on a separate page, connection to the cmslpc interactive GPUs requires the same pre-requisites above

Note, the following are "heavy" nodes for software and singularity/apptainer development. These are NOT intended for regular analysis, use the interactive nodes above. Each "heavy" node has a local large filesystem: /scratch/containers/$USER, in which files are removed after 3 days.

  • cmslpc-el8-heavy01.fnal.gov, running Alma Linux 8, for development only
  • cmslpc-el8-heavy02.fnal.gov, running Alma Linux 8, for development only
  • cmslpc-el9-heavy01.fnal.gov, running Alma Linux 9, for development only

If a different OS (i.e., SL6, SL7) is needed, one can use Singularity/apptainer.

Connecting to the LPC Cluster from macOS

  • macOS has the Terminal program installed in the Applications/Utilities folder which you can use to enter the kinit and ssh commands above after you have configured your /etc/krb5.conf and ~/.ssh/config as described above.
  • macOS does not come with an X11 server installed. The XQuartz server can be installed from xquartz.org. Download the XQuartz package, double click on the .dmg, and follow the instructions to install it. Be sure to log out and back in after installation for it to work!

Connecting to the LPC Cluster from a Windows PC

Connecting to the LPC cluster through the round-robin queue from a Windows PC is known to work with Cygwin/X or Kerberized PuTTY. Directions for installing, configuring, and connecting to the LPC CAF are given below. Also included are directions for using WinSCP to access files in your account.

Cygwin/X:

Cygwin/X is an port of the UNIX-like API and X-Window system to Windows.

Cygwin/X installation

The full Cygwin/X install, setup/configuration, and usage for the LPC CAF is shown with screenshots and extensive explanation on this ShowMeHow page.

A description of the install is listed here without images- note you may need Windows management privileges on your computer:
  • To install Cygwin/X follow the download and installation directions from the user's guide.
  • Be sure the local package directory is set to C:\cygwin64
  • To Select Packages, click the "+" to expand next to the all, and put in a search term for each of these packages listed below.
  • (Newer version of Cygwin) After searching, click the down arrow to the left of the "Bin" column for the package you want and choose the version, usually the most recent. Then you will see the "Bin" column for that package checked.
  • (older version Cygwin) Use the search function to find, click the "+" to expand, and click the "Skip" to change it to check the checkbox for the "bin" of the following packages:
    • krb5-auth-dialog
    • krb5-workstation
    • openssh
    • xinit
  • You may find the following optional packages useful:
    • wget for getting programs to install under cygwin
    • emacs for local text editing on the cygwin command line
    • nano for local text editing on the cygwin command line
    • xclock for testing the local cygwin X11 server
  • Let Cygwin/X resolve dependencies, ignore the error about "libsub0-Win32 driver". If packages fail to download, be sure to "Retry" instead of continue to get those missing packages. In most cases they will work. You can always install more packages using the Cygwin-X "setup-x86_64" installer that you have downloaded (choosing a different mirror site if need be).
  • Start the "Cygwin64 Terminal" program
  • Edit (using emacs or nano) your ~/.bashrc to contain: export DISPLAY=:0.0. Note that in some cases you may need to remove everything else in that file.
  • Then follow the instructions similar to above for configuration
    • Get the Linux krb5.conf file and save on the Cygwin/X terminal command line to /etc/krb5.conf. This is actually in Windows filesystem: C:\cygwin64\etc
    • The krb5.conf file was updated Feb 21, 2018, be sure you have v5.3 of the file on your computer.

    • Make and edit your ~/.ssh/config (Windows filesystem: C:\cygwin64\home\yourusername\.ssh\config) file to contain the following:
      • 
        # - - - - - - - - - - - -  cygwin - - - - - - - - - - - - -
        Host cmslpc*.fnal.gov
            GSSAPIAuthentication yes
            GSSAPIDelegateCredentials yes
            StrictHostKeyChecking no
            UserKnownHostsFile /dev/null
            ForwardX11 yes
            ForwardX11Trusted yes
        # - - - - - - - - - - - -  cygwin - - - - - - - - - - - - -
        

Connect to cmslpc with Cygwin/X

  • Daily use, Start the "Cygwin64 Terminal" program (last two items will look the same in Linux/macOS as in your Cygwin64 terminal:
    • Start the Cygwin/XWindow server with the command: startxwin&
    • Get your Fermilab Kerberos ticket with: kinit username@FNAL.GOV
    • ssh to the cmslpc cluster with: ssh -Y username@cmslpc-sl7.fnal.gov
    Note: Documentation for cygwin adapted from local FNAL instructions, and local cmslpc user instructions.

Kerberos and SSH with PuTTY and XMing:

The usage of PuTTY at Fermilab is described in this ServiceNow Knowledge Base article

  • Version 0.63 of PuTTY supports Kerberos authentication for SSH on Windows and can be found at https://www.chiark.greenend.org.uk/~sgtatham/putty/releases/0.63.html. You would want to use the .EXE installer created with Inno Setup putty-0.63-installer link. Putty will be installed in c:/Program Files/Putty.
  • You will also need to download and install MIT Kerberos for Windows (version 4.1 is recommended for Windows 10). The installer can be found at the MIT Kerberos download page. Scroll down to download the "MIT Kerberos for Windows" program (.msi), either 64 bit for newer computers or 32 bit for older versions of Windows. Here is a link to the Fermilab Knowledge Base Article about using Kerberos on Windows.
  • In addition you will need get the krb5.conf file for Fermilab and save it to C:\ProgramData\MIT\Kerberos5\krb5.ini replacing this file if it exists.
  • The krb5.conf file was updated Feb 21, 2018, be sure you have v5.3 of the file on your computer.

  • For XWindows with PuTTY, you will need Xming, an implementations of X11 on the Windows platform. With X11 forwarding enabled in PuTTY and an X server(XMing) running, programs like Root and emacs can be displayed on the Windows desktop.

To get your Kerberos ticket:
The Ticket Manager lives in the taskbar notification area (lower right side of the screen). Right-click on its icon (the one with a silhouette of a K), and select MIT Kerberos Window. In the Ticket Manager window, select Get Ticket. Enter your Kerberos principal name (username@FNAL.GOV), and password. Make sure the options for "Remember this Principal", "Forwardable and Proxiable", and "Renewable" are selected. Click "OK".

To connect to the LPC cluster for the first time with PuTTY:
Double click on the PuTTY icon in the directory where you unzipped the zip file.
In the PuTTY configuration window:

  • select Session and enter cmslpc-sl7.fnal.gov in the HostName field
  • select Connection -> Data and enter your username in the Auto-login username field
  • select Connection -> SSH and select "2 only" for "Preferred SSH protocol version"
  • select Connection -> SSH -> X11 and check "Enable X11 forwarding"
  • select Connection -> SSH -> Auth -> GSSAPI and check both boxes: "Attempt GSSAPI authentication" and "Allow GSSAPI credential delegation" settings
  • select Session and enter LPC in the Saved Sessions field and click Save
double click on LPC in the Saved Sessions list

Daily use: to connect to the LPC cluster with PuTTY (after you have obtained your Kerberos ticket):
Double click the PuTTY icon.
In the PuTTY configuration window:
select Session and double click on LPC in the Saved Sessions list

Kerberos and SFTP:

WinSCP supports Kerberos authentication for SFTP on Windows can be downloaded from http://winscp.net/eng/index.php. Follow the directions here check Advanced options and under SSH -> Authentication check both boxes for GSSAPI Authentication. Use this program to transfer files to and from the LPC cluster.

Troubleshooting your connection

Host Key errors

If you get an error like this one:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:y41zjGTp4uBcp+CwV2OVG/EqEHLMzU9qSwS8VPYHO4A.
Please contact your system administrator.
Add correct host key in /Users/username/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/username/.ssh/known_hosts:3
ECDSA host key for cmslpc-sl7.fnal.gov has changed and you have requested strict checking.
Host key verification failed.
This means that you need to properly configure your ~/.ssh/config file on the computer you are trying to ssh from (Cygwin/X .ssh/config).
Click here to learn more about Host RSA keys and the cmslpc cluster, if you want to understand why the "Host key has changed".

Troubleshoot repeating X11 connection rejected error

See also Mac troubleshooting below for other X11 errors.
  • On cmslpc124, the following error has been observed on Mac and PC, repeating on an intermittent basis while connected to that interactive node:
    • X11 connection rejected because of wrong authentication.
  • Solution is to do the following:
    • On your laptop, add the following to your local ~/.ssh/config:
      
      Host cmslpc*fnal.gov
         ForwardX11 yes 
         ForwardX11Trusted yes 
         ForwardAgent yes 
      
    • On the cmslpc cluster, remove your existing .Xauthority file: rm ~/.Xauthority (this will break existing X11 sessions)
    • Connect a new terminal to the cmslpc cluster and you should not get the error anymore.
    • You may find you also need to restart your X11 server and terminal program.
    • This error is harmless, and you can usually keep opening X11 windows, however, the repeated message is annoying. If none of the above fixes it, log in to a different node (see list at the top of the page) and should be ok

Troubleshooting macOS connections

  • If XQuartz is not opening remote windows (occasionally after a software update), be sure that you did the following:
    • Use ssh -Y (the -Y enables trusted X11 forwarding)
    • In XQuartz Preferences, the "Security" tab, ensure that both "Authenticate connections" is checked. If you still have problems, ensure "Allow connections from network clients" is checked.
    • If you have changed your network identity (switching wireless access point, IP address), you may have to restart XQuartz and possibly Terminal to make new Xauthority access control keys if Xwindow forwarding no longer works
    • If you get this error: Warning: No xauth data; using fake authentication data for X11 forwarding.
      • First, be sure you did ssh -Y and you have XQuartz running with the settings listed above
      • Add to your ~/.ssh/config file:
        Host *
            XAuthLocation /opt/X11/bin/xauth
        
  • You can troubleshoot Permission denied (gssapi-keyex,gssapi-with-mic) errors with the section below -usually due to existing conda/anaconda/pip/MacPorts installation on your computer, Macs in particular are affected but not only them.
  • Additional troubleshooting note: if you get OpenGL errors, you can remedy them by entering this in Mac OSX Terminal program of your laptop: defaults write org.macosforge.xquartz.X11 enable_iglx -bool true

Windows troubleshooting

  • CygwinX:
    • Problem: Unable to open remote XWindows:
      • Be sure to have started startxwin& in your session. If need be, close all windows and start a new CygwinX session and run startxwin&
    • Problem: gssapi and other connection errors:
      • CygwinX runs internally a lot like Linux, check the Linux debugging below, be sure to use the Linux local ~/.ssh/config and the Linux krb5.conf
      • Try to see if you can connect to an individual node instead of cmslpc-sl7.fnal.gov see the full list at the top of this page
  • PuTTy:
    • Problem:: Configured PuTTy apparently correctly, but still unable to connect.
    • possible solutions:
      • If you have PuTTy 0.7 installed, you have to remove it completely, and only have PuTTy version 0.63 installed
      • Be sure you got a valid FNAL.GOV Kerberos ticket with MIT Kerberos, see above
      • Be sure you have the latest krb5.conf

OpenSSH bug (Linux/Cygwin (Windows):

If you are running a Linux machine (have also observed in Cygwin on Windows and once on a Mac) and have setup your ~/.ssh/config as shown above, you have a valid Kerberos ticket (klist to check), and the appropriate /etc/krb5.conf, but using ssh -vv gives you the following errors:

 debug1: An invalid name was supplied
Permission denied (gssapi-keyex,gssapi-with-mic).
This occurred with the following versions and we understand is related to this bug as reported in archlinux: https://bugs.archlinux.org/task/50054. You should be able to log directly into a cmslpc node (see the image above for the list, for example: ssh -Y username@cmslpc120.fnal.gov
  • OpenSSH_7.5p1, OpenSSL 1.1.0f, May 25, 2017
  • OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g, Nov 16, 2017

Troubleshooting gssapi failures (Mac/Windows/Linux):

If you get the following error:

Warning: Permanently added 'cmslpc-sl7.fnal.gov' (RSA) to the list of known hosts.
Permission denied (gssapi-keyex,gssapi-with-mic).
Note: The warning about adding the RSA key is normal and you can accept the key.
For errors such as above, you need to ensure the following:
  • Did you get an account on the cmslpc? You would have gotten an email with this subject line when your account was created: "Welcome to the CMS LPC CAF (Central Analysis Facility)" or "Welcome to the USCMS UAF (Central Analysis Facility)".
    • Alternately, a colleague logged into the cmslpc cluster can check with finger username to check if you have an account
    • Be sure you spelled your username correctly
  • Did you get the latest /etc/krb5.conf as described above?
  • Did you configure your ~/.ssh/config properly following the procedure above? (note cygwin users follow the Linux configuration)
    • You can also check for any other settings that may have been changed (not usually, but could be the case) in your /etc/ssh/ssh-config
  • Did you get a valid Kerberos ticket? Follow this to kinit
    • Note: Do not do kinit -af as you may have done in the past, that was for older versions of kerberos and will give you the error above if you use it for the current version. You will need to kdestroy, and then kinit user@FNAL.GOV to get a correct kerberos ticket
    • Be sure to check what your alias is defined as in case you have an old alias configured for kinit or ssh
    • Check your kerberos ticket with klist
    • If you did NOT get a valid kerberos ticket, a ssh -vvv username@cmslpc-sl7.fnal.gov will report this error: No credentials cache file found
  • Otherwise, you can test logging into one of the nodes listed at the top of the page, for example: ssh -vvvY username@cmslpc121.fnal.gov and see if that works for you
  • Anaconda/conda/pip/MacPorts errors:

  • These are different different user reported solutions
    1. The conda environment is automatically starting for your login. You can test this with which kinit, if you are NOT in a conda/Anaconda environment it points to /usr/bin/kinit, if it points to another place follow these guidelines to work outside of conda:
      • Favored solution:, you can turn off conda base environment as default with conda config --set auto_activate_base false and then logging out and back in (or rebooting your computer). The conda environment will still be available but not on by default. Be sure to kinit outside of a conda environment. Source of solution at stack overflow weblink.
      • Alternately: Make sure you are outside a conda environment before doing your kinit: use the following command:
        • conda deactivate
        • If you need to then login, after the above command destroy your previous Kerberos ticket: kdestroy, then kinit as usual
        • Note that Anaconda has installed some software in your ~/.zshrc or your ~/.bashrc to automatically be in a conda environment
        • To work in an Anaconda environment again in your Terminal you may use conda activate
        • If you still have trouble, make sure to open a new Terminal after making changes above (one user reported reboot was necessary)
        • Note that if the command line says (base) then you are operating in a conda environment and likely have the above issue
        • Note that in a Mac if klist reports Ticket cache: KCM:501, you likely have the above issue, and can confirm with which kinit to see that the command line is using the anaconda/conda/pip kinit
    2. If you don't want to disable anaconda by default (or some other software changed your kerberos, like pip), then you can do the following:
      • When you use kinit, and similar commands, always call the macOS system version (if you have used the anaconda or pip version, you must log out and back in to make this work):
        • /usr/bin/kinit username@FNAL.GOV
        • You can make an alias in your local Mac .zshrc and use it, for instance: alias kfnal="/usr/bin/kinit username@FNAL.GOV"
        • Any other commands you may need to reference /usr/bin include /usr/bin/klist, /usr/bin/kdestroy
    3. Also note that some Mac updates may modify /etc/ssh/ssh_config or ~/.ssh/config, please follow one of these methods to configure it:
    4. MacPorts:
      • MacPorts ssh update may have the gssapi connection failure above together with:
        
        /Users/laptopuser/.ssh/config line 198: Unsupported option "gssapiauthentication"
        /Users/laptopuser/.ssh/config line 199: Unsupported option "gssapidelegatecredentials"
        
      • The solution is to either roll back the MacPorts ssh upgrade or use Mac OS ssh tools with /usr/bin/ssh
      • Troubleshooting: which ssh, and which kinit point to code in /opt/local/bin, which means some custom install of software that's not the Mac OS defaults

Troubleshooting credential/kinit failures (Mac/Windows/Linux):

  • Error: [MyComputer:~] username% debug1: Unspecified GSS failure. Minor code may provide more information
    No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

    In this case the Kerberos ticket expired or wasn't gotten, and user must kinit username@FNAL.GOV again to have access
  • [MyComputer:~] username% kinit username@fnal.gov
    kinit: krb5_cc_get_principal: No credentials cache file found
    or: kinit: krb5_get_init_creds: Error from KDC: CLIENT_NOT_FOUND
    In this case, the kinit was done to the wrong domain, it should be capitalized FNAL.GOV
  • Error: [MyComputer:~] username% kinit username@FNAL.GOV
    kinit: krb5_get_init_creds: Error from KDC: CLIENT EXPIRED or
    kinit: Client's entry in database has expired while getting initial credentials
    In this case the user's Kerberos password expired and/or the account credentials need renewal.
  • [MyComputer:~] username% kinit
    Error: kinit: krb5_get_init_creds: CLIENT EXPIRED
    In this case the user is using the wrong username (kinit without any arguments uses the username of the local computer which may not be the same as remote), be sure to specify with your Fermilab username: kinit username@FNAL.GOV.
  • Error: [MyComputer:~] username% kinit username@FNAL.GOV
    username@FNAL.GOV's password:
    kinit: krb5_get_init_creds: unable to reach any KDC in realm FNAL.GOV, tried 0 KDCs
    In this case, you need to update your /etc/krb5.conf
  • Error: kinit: krb5_get_init_creds: Error from KDC: CLIENT_NOT_FOUND
    In this case, the username was either wrongly entered, @fnal.gov (should be uppercase), or the user didn't get a Fermilab kerberos account. Click here to get an E-892/919 (CMS) account
  • kinit: krb5_get_init_creds: Error from KDC: CLIENT LOCKED OUT, in this case you have let some required Fermilab security training lapse, like cybersecurity training or something similar. Please complete the training and contact the Fermilab Service Desk to get your access restored

Troubleshooting scp/sftp or singularity failures

Troubleshooting getting "stuck" on an overloaded interactive VM

The cmslpc-sl7.fnal.gov load balancer is tuned to place you on a machine with low load, but as people work during the day that can of course change. These are shared resources between over 100 active users each day. If you find you are on a machine that is loaded, you can directly log into another instead, or you can log out and have the load balancer place you on another machine with low load. Because the load balancer will try to keep you associated with the same node, you will need to log off entirely and let it re-set first however. To do this:
  1. Log out all ssh/sftp connections (Xwindows/emacs/etc.) from that node
  2. Wait 5 minutes for the host affinity to re-set (what "glues" you to a particular node)
  3. Log back into cmslpc-sl7.fnal.gov. It should connect you to an LPC node with lower load.
Checking what may be the cause of the overload:
  • Interactive node monitoring can be found at this link
  • You can also use top to see if a particular user is overloading the resources, take note of what's available on that node when you login. Use q to exit top
  • LPC Computing discussion March 16, 2018 slides 5-11, discussion of how users overload nodes - this was when there were more CPU available per node in sl6 (and fewer nodes), and how to read the output of top
  • Note that the shared NFS filesystems can also be overloaded and is difficult to monitor, but it will affect all interactive nodes when this occurs


AFS:

  • As of Spring, 2018, the LPC CAF (cmslpc cluster) no longer mounts /afs directories. You may still access the /afs filesystem from CERN lxplus, but be aware that it is being phased out.

RSA keys and the cmslpc cluster

As described above, a login to the cmslpc-sl7.fnal.gov goes to one of multiple nodes. This will redirect you to one of the nodes using a load balancer.

Since each of the individual node (see list of nodes) is a different virtual machine, they will have different RSA keys even though you can reach them all with ssh -Y username@cmslpc-sl7.fnal.gov. When you modify your ~/.ssh/config, be sure to only apply these settings to cmslpc*.fnal.gov machines as they may be insecure for other clusters. You will need to accept a new RSA key fingerprint for each login, and in the current configurations above, they are sent to /dev/null and thus not retained for cmslpc*.fnal.gov machines.

Login shell

The default login shell at the cmslpc cluster is bash (June 4, 2019), the default was tcsh before then. There is more information and instructions on how to change your default login shell (using the LPC Service Portal) on the Software Setup page, including how to fix your prompt.

Notifications of downtimes; getting help

As a new user, you have been automatically subscribed to the cms_uaf_users at fnal.gov email list. This list will be used by the CMS at Fermilab Facilities Support Team to announce downtimes and provide updates on the CMSLPC and other facilities integral to its use.
System status information, including the cms_uaf_users archives are listed here.

To get help for computing, please consult the LPC Computing Get Help page at the lpc.fnal.gov pages.
Webmaster | Last modified: Wednesday, 13-Mar-2024 11:11:01 CDT