Connect to the LPC CAF (Central Analysis Facility)

The LPC CAF (cmslpc cluster) is a group of interactive nodes running Linux that LPC users are able to connect to, to develop and debug their code, submit jobs, do analysis, and so on. The main interactive nodes are behind a load balancer. Multiple parallel analysis should be handled by the batch system.

Most operating systems with OpenSSH configured with GSSAPI and Kerberos configured for the FNAL.GOV realm (see below for required configuration prerequisites) should be able to connect to the LPC cluster.

It is assumed that you already have a Fermilab Kerberos principal account (you got the email "Welcome to the CMS LPC CAF (Central Analysis Facility)"). If not visit the "Get an Account" section.

The system status of the cmslpc interactive nodes can be found on the SSI metrics page.

Guidelines for use of the interactive nodes.

Prerequisites

Kerberos configuration file

To connect to the CMS LPC cluster you need to have Kerberos installed on your system. This is already included in Scientific Linux and macOS, but not Windows. In addtion you will need to get the up to date krb5.conf file for Fermilab that corresponds to your specific OS. You can download this file from: Fermilab Kerberos Configuration Files.

The krb5.conf file was updated on May 16, 2023, be sure you have v5.7-3 of the file on your computer.

/etc/krb5.conf is a text file that you can review and compare to the one found on Fermilab Kerberos Configuration Files to determine whether you already have the most recent one or you need to update it.

Save the appropriate file to your home directory as krb5.conf (you may need to rename it as some browsers will save it as krb5.conf.txt). Note: if you already have a krb5.conf file, it is a good idea to backup it up before placing the new file. Then run (depending on your system, you will be prompted for your root or administrator password):

[localMachine:Desktop localUser]$ sudo mv -v krb5.conf /etc/krb5.conf
Password:
krb5.conf -> /etc/krb5.conf

Once done, the file is now in the appropriate location on your system.

SSH configuration file

In order to give SSH the proper configuration to login to the CMS LPC cluster, make sure the following lines are present in your ~/.ssh/config file (depending on your OS and). Note: you may not have such a file, and you may need to create the .ssh directory and the file

- - - - - - - - Create ~/.ssh/config file  - - - - - - - - - -
[localMachine:Desktop localUser]$ cd
[localMachine:Desktop localUser]$ mkdir .ssh
[localMachine:Desktop localUser]$ touch .ssh/config
- - - - - - - - Create ~/.ssh/config file  - - - - - - - - - -

Use your favorite text editor to edit this file on your local machine with the following contents for ~/.ssh/config

[localMachine:Desktop localUser]$ more ~/.ssh/config
# - - - - - - - - - - - - LINUX  - - - - - - - - - - - - - - - - - -
Host cmslpc*.fnal.gov
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
#  - - - - - - - - - - - - LINUX  - - - - - - - - - - - - - - - - - -

Note: GSSAPITrustDNS yes may be needed in addition for some older versions of Linux if your computer complains that the WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

# - - - - - - - - - - - - macOS  - - - - - - - - - - - - - - - - - 
Host cmslpc*.fnal.gov
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
    GlobalKnownHostsFile /dev/null
    LogLevel ERROR
# - - - - - - - - - - - - macOS  - - - - - - - - - - - - - - - - -

Note: The settings for StrictHostKeyChecking and UserKnownHostsFile should only be applied to cmslpc*.fnal.gov, as they would be insecure for other hosts. See the note about RSA keys at the bottom of this page to understand more.
Note: In case your ssh connection keeps timing out, you may find this useful in your local computer's ~/.ssh/config file:

Host cmslpc*.fnal.gov
    ServerAliveInterval 60
    TCPKeepAlive yes
    KeepAlive yes

Connecting to the cmslpc nodes LPC CAF

Once your system has the appropriate prerequisites to connect to the LPC CAF, you must have a valid Kerberos ticket before logging in. Note that in the examples below we use cmslpc-el9.fnal.gov for the interactive node, for newer Operating Systems and development nodes, see the full list below.

Get a Kerberos ticket

A valid Kerberos ticket for the FNAL.GOV Kerberos realm is needed to login to the LPC CAF, you can get one from your terminal by doing:

[localMachine:Desktop localUser]$ kinit username@FNAL.GOV
- - - - - - LINUX - - - - - -
Password for username@FNAL.GOV:
- - - - - - LINUX - - - - - -

- - - - - - macOS - - - - - -
username@FNAL.GOV's password:
- - - - - - macOS - - - - - -

if successful, there is no output to this command. You can always verify that a Kerberos ticket was created by issuing the following command (you do not need to do this on a normal login):

[localMachine:Desktop localUser]$ klist
- - - - - - - - - - - - - - LINUX - - - - - - - - - - - - - -
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: username@FNAL.GOV

Valid starting     Expires            Service principal
May  6 10:02:15 2024  May  7 12:02:12 2024  krbtgt/FNAL.GOV@FNAL.GOV
- - - - - - - - - - - - - - LINUX - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - macOS - - - - - - - - - - - - - - - -
Credentials cache: API:42509457-B3FA-4905-BAB8-51A42173D5CD
        Principal: username@FNAL.GOV
    
 Issued                Expires               Principal
May  6 10:02:15 2024  May  7 12:02:12 2024  krbtgt/FNAL.GOV@FNAL.GOV/font>
- - - - - - - - - - - - - - - - macOS - - - - - - - - - - - - - - - -

Note that if the klist for Mac OS says: KCM:501, this is anaconda/miniconda kerberos and does not work with Fermilab and needs to be not used.

SSH to the LPC CAF

With a valid Kerberos ticket, you can now login to a Alma9 machine on the LPC CAF by doing:

- - - - - - - - - - - - - - LINUX  or macOS - - - - - - - - - - - - - - - - 
[localMachine:Desktop localUser]$ ssh -Y username@cmslpc-el9.fnal.gov
- - - - - - - - - - - - - - LINUX  or macOS - - - - - - - - - - - - - - - - 
Warning: Permanently added 'cmslpc-el9.fnal.gov' (ED25519) to the list of known hosts.
                              NOTICE TO USERS

       This  is a Federal computer (and/or it is directly connected to a
       Fermilab local network system) that is the property of the United
       States Government.  It is for authorized use only.  Users (autho-
       rized or unauthorized) have no explicit or  implicit  expectation
       of privacy.

       Any  or  all uses of this system and all files on this system may
       be intercepted, monitored, recorded,  copied, audited, inspected,
       and  disclosed  to authorized site, Department of Energy  and law
       enforcement personnel, as  well as authorized officials of  other
       agencies,  both  domestic and foreign.  By using this system, the
       user consents to such interception, monitoring, recording,  copy-
       ing,  auditing,  inspection,  and disclosure at the discretion of
       authorized site or Department of Energy personnel.

       Unauthorized or improper use of this system may result in  admin-
       istrative  disciplinary  action and civil and criminal penalties.
       By continuing to use this system you indicate your  awareness  of
       and  consent to these terms and conditions of use.  LOG OFF IMME-
       DIATELY if you do not agree to  the  conditions  stated  in  this
       warning.

       Fermilab  policy  and  rules for computing, including appropriate
       use, may be found at http://www.fnal.gov/cd/main/cpolicy.html
Last login: Wed Apr 10 09:37:24 2024 from 131.225.172.74
------------------------------------------------------------------------------
                     ..::Powered by CMS-LPC::..                      

   Hostname: cmslpc333.fnal.gov          OS Release: Alma Linux 9.3            
         IP: 131.225.191.79                  Subnet: 255.255.252.0             

     Kernel: 5.14.0-362.18.1                   Arch: x86_64                    
        RAM: 11.42 GiB                         Swap: 4.00 GiB                  
      Cores: 8                              Virtual: kvm                       

 SSH Logins: 1                             Load Avg: 0.02 0.03 0.0             
------------------------------------------------------------------------------
  For information about computing at the LPC: http://lpc.fnal.gov/computing   
------------------------------------------------------------------------------
                 Welcome to the CMS EL9 interactive cluster!
[username@cmslpc333 ~]$

Notes about the cmslpc interactive nodes:

The lines just above the prompt each time you login have an important message as well as the software version, is a quick reminder on where to go if you need to look for some information about computing at the LPC.
If your bash shell login doesn't look right, see this link about setting up your ~/.bash_profile

If you are unable to login, see the troubleshooting section below.

Interactive nodes to connect to:

Be sure to get your Kerberos ticket, and ssh per the directions above. In this section is a list of possible node names which you can use instead of cmslpc-el9.fnal.gov, depending on your analysis or development needs.

For regular interactive connection, code development & debugging, batch job submission, and interactive analysis, the following nodes are available:

cmslpc-el8.fnal.gov, running Alma Linux 8

The load balancer corresponds to nodes cmslpc201.fnal.gov to cmslpc250.fnal.gov

cmslpc-el9.fnal.gov, running Alma Linux 9

The load balancer corresponds to nodes cmslpc301.fnal.gov to cmslpc350.fnal.gov

GPU interactive nodes are described on a separate page, connection to the cmslpc interactive GPUs requires the same pre-requisites above

Note, the following are "heavy" nodes for software and singularity/apptainer development. These are NOT intended for regular analysis, use the interactive nodes above. Each "heavy" node has a local large filesystem: /scratch/containers/$USER, in which files are removed after 3 days.

cmslpc-el8-heavy01.fnal.gov, running Alma Linux 8, for development only
cmslpc-el8-heavy02.fnal.gov, running Alma Linux 8, for development only
cmslpc-el9-heavy01.fnal.gov, running Alma Linux 9, for development only

If a different OS (i.e., SL6, SL7) is needed, one can use Singularity/apptainer.

Connecting to the LPC Cluster from macOS

macOS has the Terminal program installed in the Applications/Utilities folder which you can use to enter the kinit and ssh commands above after you have configured your /etc/krb5.conf and ~/.ssh/config as described above.
macOS does not come with an X11 server installed. The XQuartz server can be installed from xquartz.org. Download the XQuartz package, double click on the .dmg, and follow the instructions to install it. Be sure to log out and back in after installation for it to work!

If for some reason the XQuartz program will not open after you have installed it, you can follow these instructions from Apple to open installed programs from a non-identified developer.

Connecting to the LPC Cluster from a Windows PC

Connecting to the LPC cluster through the round-robin queue from a Windows PC is known to work with Cygwin/X or Kerberized PuTTY. Directions for installing, configuring, and connecting to the LPC CAF are given below. Also included are directions for using WinSCP to access files in your account.

Cygwin/X:

Cygwin/X is an port of the UNIX-like API and X-Window system to Windows.

Cygwin/X installation

The full Cygwin/X install, setup/configuration, and usage for the LPC CAF is shown with screenshots and extensive explanation on this ShowMeHow page.

A description of the install is listed here without images- note you may need Windows management privileges on your computer:

To install Cygwin/X follow the download and installation directions from the user's guide.
Be sure the local package directory is set to C:\cygwin64
To Select Packages, click the "+" to expand next to the all, and put in a search term for each of these packages listed below.
(Newer version of Cygwin) After searching, click the down arrow to the left of the "Bin" column for the package you want and choose the version, usually the most recent. Then you will see the "Bin" column for that package checked.
(older version Cygwin) Use the search function to find, click the "+" to expand, and click the "Skip" to change it to check the checkbox for the "bin" of the following packages:

krb5-auth-dialog
krb5-workstation
openssh
xinit

You may find the following optional packages useful:

wget for getting programs to install under cygwin
emacs for local text editing on the cygwin command line
nano for local text editing on the cygwin command line
xclock for testing the local cygwin X11 server

Let Cygwin/X resolve dependencies, ignore the error about "libsub0-Win32 driver". If packages fail to download, be sure to "Retry" instead of continue to get those missing packages. In most cases they will work. You can always install more packages using the Cygwin-X "setup-x86_64" installer that you have downloaded (choosing a different mirror site if need be).
Start the "Cygwin64 Terminal" program
Edit (using emacs or nano) your ~/.bashrc to contain: export DISPLAY=:0.0. Note that in some cases you may need to remove everything else in that file.
Then follow the instructions similar to above for configuration

Get the Linux krb5.conf file and save on the Cygwin/X terminal command line to /etc/krb5.conf. This is actually in Windows filesystem: C:\cygwin64\etc

The krb5.conf file was updated Feb 21, 2018, be sure you have v5.3 of the file on your computer.

Make and edit your ~/.ssh/config (Windows filesystem: C:\cygwin64\home\yourusername\.ssh\config) file to contain the following:


# - - - - - - - - - - - -  cygwin - - - - - - - - - - - - -
Host cmslpc*.fnal.gov
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
    ForwardX11 yes
    ForwardX11Trusted yes
# - - - - - - - - - - - -  cygwin - - - - - - - - - - - - -

Connect to cmslpc with Cygwin/X

Daily use, Start the "Cygwin64 Terminal" program (last two items will look the same in Linux/macOS as in your Cygwin64 terminal:

Start the Cygwin/XWindow server with the command: startxwin&
Get your Fermilab Kerberos ticket with: kinit username@FNAL.GOV
ssh to the cmslpc cluster with: ssh -Y username@cmslpc-el9.fnal.gov

Note

local FNAL instructions

local cmslpc user instructions

Kerberos and SSH with PuTTY and XMing:

The usage of PuTTY at Fermilab is described in this ServiceNow Knowledge Base article

Version 0.63 of PuTTY supports Kerberos authentication for SSH on Windows and can be found at https://www.chiark.greenend.org.uk/~sgtatham/putty/releases/0.63.html. You would want to use the .EXE installer created with Inno Setup putty-0.63-installer link. Putty will be installed in c:/Program Files/Putty.

As of October 2017, later versions of PuTTY are available by may not work at FNAL
More about PuTTY and Kerberos are detailed at this Fermilab Knowledge Base Article about using Kerberos on Windows.
FNAL computing has links to installers, including a local copy of PuTTY 0.66 64-bit installer.

You will also need to download and install MIT Kerberos for Windows (version 4.1 is recommended for Windows 10). The installer can be found at the MIT Kerberos download page. Scroll down to download the "MIT Kerberos for Windows" program (.msi), either 64 bit for newer computers or 32 bit for older versions of Windows. Here is a link to the Fermilab Knowledge Base Article about using Kerberos on Windows.
In addition you will need get the krb5.conf file for Fermilab and save it to C:\ProgramData\MIT\Kerberos5\krb5.ini replacing this file if it exists.

The krb5.conf file was updated Feb 21, 2018, be sure you have v5.3 of the file on your computer.

For XWindows with PuTTY, you will need Xming, an implementations of X11 on the Windows platform. With X11 forwarding enabled in PuTTY and an X server(XMing) running, programs like Root and emacs can be displayed on the Windows desktop.

To get your Kerberos ticket:
The Ticket Manager lives in the taskbar notification area (lower right side of the screen). Right-click on its icon (the one with a silhouette of a K), and select MIT Kerberos Window. In the Ticket Manager window, select Get Ticket. Enter your Kerberos principal name (username@FNAL.GOV), and password. Make sure the options for "Remember this Principal", "Forwardable and Proxiable", and "Renewable" are selected. Click "OK".

To connect to the LPC cluster for the first time with PuTTY:
Double click on the PuTTY icon in the directory where you unzipped the zip file.
In the PuTTY configuration window:

select Session and enter cmslpc-el9.fnal.gov in the HostName field
select Connection -> Data and enter your username in the Auto-login username field
select Connection -> SSH and select "2 only" for "Preferred SSH protocol version"
select Connection -> SSH -> X11 and check "Enable X11 forwarding"
select Connection -> SSH -> Auth -> GSSAPI and check both boxes: "Attempt GSSAPI authentication" and "Allow GSSAPI credential delegation" settings
select Session and enter LPC in the Saved Sessions field and click Save

double click on LPC in the Saved Sessions list

Daily use: to connect to the LPC cluster with PuTTY (after you have obtained your Kerberos ticket):
Double click the PuTTY icon.
In the PuTTY configuration window:
select Session and double click on LPC in the Saved Sessions list

Kerberos and SFTP:

WinSCP supports Kerberos authentication for SFTP on Windows can be downloaded from http://winscp.net/eng/index.php. Follow the directions here check Advanced options and under SSH -> Authentication check both boxes for GSSAPI Authentication. Use this program to transfer files to and from the LPC cluster.

Troubleshooting your connection

Host Key errors

If you get an error like this one:


@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:y41zjGTp4uBcp+CwV2OVG/EqEHLMzU9qSwS8VPYHO4A.
Please contact your system administrator.
Add correct host key in /Users/username/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/username/.ssh/known_hosts:3
ECDSA host key for cmslpc-el9.fnal.gov has changed and you have requested strict checking.
Host key verification failed.

This means that you need to properly configure your ~/.ssh/config file on the computer you are trying to ssh from (Cygwin/X .ssh/config).
Click here to learn more about Host RSA keys and the cmslpc cluster, if you want to understand why the "Host key has changed".

Troubleshoot repeating X11 connection rejected error

See also Mac troubleshooting below for other X11 errors.

On cmslpc124, the following error has been observed on Mac and PC, repeating on an intermittent basis while connected to that interactive node:

X11 connection rejected because of wrong authentication.

Solution is to do the following:

On your laptop, add the following to your local ~/.ssh/config:


Host cmslpc*fnal.gov
   ForwardX11 yes 
   ForwardX11Trusted yes 
   ForwardAgent yes

On the cmslpc cluster, remove your existing .Xauthority file: rm ~/.Xauthority (this will break existing X11 sessions)
Connect a new terminal to the cmslpc cluster and you should not get the error anymore.
You may find you also need to restart your X11 server and terminal program.
This error is harmless, and you can usually keep opening X11 windows, however, the repeated message is annoying. If none of the above fixes it, log in to a different node (see list at the top of the page) and should be ok

Troubleshooting macOS connections

If XQuartz is not opening remote windows (occasionally after a software update), be sure that you did the following:

Use ssh -Y (the -Y enables trusted X11 forwarding)
In XQuartz Preferences, the "Security" tab, ensure that both "Authenticate connections" is checked. If you still have problems, ensure "Allow connections from network clients" is checked.
If you have changed your network identity (switching wireless access point, IP address), you may have to restart XQuartz and possibly Terminal to make new Xauthority access control keys if Xwindow forwarding no longer works
If you get this error: Warning: No xauth data; using fake authentication data for X11 forwarding.
- First, be sure you did ssh -Y and you have XQuartz running with the settings listed above
- Add to your ~/.ssh/config file:
```
Host *
    XAuthLocation /opt/X11/bin/xauth
```

You can troubleshoot Permission denied (gssapi-keyex,gssapi-with-mic) errors with the section below -usually due to existing conda/anaconda/pip/MacPorts installation on your computer, Macs in particular are affected but not only them.
Additional troubleshooting note: if you get OpenGL errors, you can remedy them by entering this in Mac OSX Terminal program of your laptop: defaults write org.macosforge.xquartz.X11 enable_iglx -bool true

Windows troubleshooting

CygwinX:

Problem: Unable to open remote XWindows:

Be sure to have started startxwin& in your session. If need be, close all windows and start a new CygwinX session and run startxwin&

Problem: gssapi and other connection errors:

CygwinX runs internally a lot like Linux, check the Linux debugging below, be sure to use the Linux local ~/.ssh/config and the Linux krb5.conf
Try to see if you can connect to an individual node instead of cmslpc-el9.fnal.gov see the full list earlier on this page

PuTTy:

Problem:: Configured PuTTy apparently correctly, but still unable to connect.
possible solutions:

If you have PuTTy 0.7 installed, you have to remove it completely, and only have PuTTy version 0.63 installed
Be sure you got a valid FNAL.GOV Kerberos ticket with MIT Kerberos, see above
Be sure you have the latest krb5.conf

OpenSSH bug (Linux/Cygwin (Windows):

If you are running a Linux machine (have also observed in Cygwin on Windows and once on a Mac) and have setup your ~/.ssh/config as shown above, you have a valid Kerberos ticket (klist to check), and the appropriate /etc/krb5.conf, but using ssh -vv gives you the following errors:


 debug1: An invalid name was supplied
Permission denied (gssapi-keyex,gssapi-with-mic).

This occurred with the following versions and we understand is related to this bug as reported in archlinux: https://bugs.archlinux.org/task/50054. You should be able to log directly into a cmslpc node (see the image above for the list, for example: ssh -Y username@cmslpc333.fnal.gov

OpenSSH_7.5p1, OpenSSL 1.1.0f, May 25, 2017
OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g, Nov 16, 2017

Troubleshooting gssapi failures (Mac/Windows/Linux):

If you get the following error:


Warning: Permanently added 'cmslpc-el9.fnal.gov' (RSA) to the list of known hosts.
Permission denied (gssapi-keyex,gssapi-with-mic).

Note: The warning about adding the RSA key is normal and you can accept the key.
For errors such as above, you need to ensure the following:

Did you get an account on the cmslpc? You would have gotten an email with this subject line when your account was created: "Welcome to the CMS LPC CAF (Central Analysis Facility)" or "Welcome to the USCMS UAF (Central Analysis Facility)".

Alternately, a colleague logged into the cmslpc cluster can check with finger username to check if you have an account
Be sure you spelled your username correctly

Did you get the latest /etc/krb5.conf as described above?
Did you configure your ~/.ssh/config properly following the procedure above? (note cygwin users follow the Linux configuration)

You can also check for any other settings that may have been changed (not usually, but could be the case) in your /etc/ssh/ssh-config

Did you get a valid Kerberos ticket? Follow this to kinit

Note: Do not do kinit -af as you may have done in the past, that was for older versions of kerberos and will give you the error above if you use it for the current version. You will need to kdestroy, and then kinit user@FNAL.GOV to get a correct kerberos ticket
Be sure to check what your alias is defined as in case you have an old alias configured for kinit or ssh
Check your kerberos ticket with klist
If you did NOT get a valid kerberos ticket, a ssh -vvv username@cmslpc-el9.fnal.gov will report this error: No credentials cache file found

Otherwise, you can test logging into one of the nodes listed at the top of the page, for example: ssh -vvvY username@cmslpc333.fnal.gov and see if that works for you

Anaconda/conda/pip/MacPorts errors:

These are different different user reported solutions
1. The conda environment is automatically starting for your login. You can test this with which kinit, if you are NOT in a conda/Anaconda environment it points to /usr/bin/kinit, if it points to another place follow these guidelines to work outside of conda:
  - Favored solution:, you can turn off conda base environment as default with conda config --set auto_activate_base false and then logging out and back in (or rebooting your computer). The conda environment will still be available but not on by default. Be sure to kinit outside of a conda environment. Source of solution at stack overflow weblink.
  - Alternately: Make sure you are outside a conda environment before doing your kinit: use the following command:
    - conda deactivate
    - If you need to then login, after the above command destroy your previous Kerberos ticket: kdestroy, then kinit as usual
    - Note that Anaconda has installed some software in your ~/.zshrc or your ~/.bashrc to automatically be in a conda environment
    - To work in an Anaconda environment again in your Terminal you may use conda activate
    - If you still have trouble, make sure to open a new Terminal after making changes above (one user reported reboot was necessary)
    - Note that if the command line says (base) then you are operating in a conda environment and likely have the above issue
    - Note that in a Mac if klist reports Ticket cache: KCM:501, you likely have the above issue, and can confirm with which kinit to see that the command line is using the anaconda/conda/pip kinit
2. If you don't want to disable anaconda by default (or some other software changed your kerberos, like pip), then you can do the following:
  - When you use kinit, and similar commands, always call the macOS system version (if you have used the anaconda or pip version, you must log out and back in to make this work):
3. Also note that some Mac updates may modify /etc/ssh/ssh_config or ~/.ssh/config, please follow one of these methods to configure it:
  - CMS documentation of config: Configure as described above
  - FNAL documentation: Fermilab knowledgebase for kerberos for Mac to configure
4. MacPorts:

Troubleshooting credential/kinit failures (Mac/Windows/Linux):

Error: [MyComputer:~] username% debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)
In this case the Kerberos ticket expired or wasn't gotten, and user must kinit username@FNAL.GOV again to have access
[MyComputer:~] username% kinit username@fnal.gov
kinit: krb5_cc_get_principal: No credentials cache file found
or: kinit: krb5_get_init_creds: Error from KDC: CLIENT_NOT_FOUND
In this case, the kinit was done to the wrong domain, it should be capitalized FNAL.GOV
Error: [MyComputer:~] username% kinit username@FNAL.GOV
kinit: krb5_get_init_creds: Error from KDC: CLIENT EXPIRED or
kinit: Client's entry in database has expired while getting initial credentials
In this case the user's Kerberos password expired and/or the account credentials need renewal.
- Password reset: follow directions at the Fermilab pages for password reset, or the Fermilab Service Desk may also be contacted for assistance
- Follow directions here for account renewal
[MyComputer:~] username% kinit
Error: kinit: krb5_get_init_creds: CLIENT EXPIRED
In this case the user is using the wrong username (kinit without any arguments uses the username of the local computer which may not be the same as remote), be sure to specify with your Fermilab username: kinit username@FNAL.GOV.
Error: [MyComputer:~] username% kinit username@FNAL.GOV
username@FNAL.GOV's password:
kinit: krb5_get_init_creds: unable to reach any KDC in realm FNAL.GOV, tried 0 KDCs
In this case, you need to update your /etc/krb5.conf
Error: kinit: krb5_get_init_creds: Error from KDC: CLIENT_NOT_FOUND
In this case, the username was either wrongly entered, @fnal.gov (should be uppercase), or the user didn't get a Fermilab kerberos account. Click here to get an E-892/919 (CMS) account
kinit: krb5_get_init_creds: Error from KDC: CLIENT LOCKED OUT, in this case you have let some required Fermilab security training lapse, like cybersecurity training or something similar. Please complete the training and contact the Fermilab Service Desk to get your access restored

Troubleshooting scp/sftp or singularity failures

Did you put something interactive like in your ~/.bashrc? Put it in your ~/.bash_profile instead. More about your login shell at this link.
Are you over quota?

AFS:

As of Spring, 2018, the LPC CAF (cmslpc cluster) no longer mounts /afs directories. You may still access the /afs filesystem from CERN lxplus, but be aware that it is being phased out.

RSA keys and the cmslpc cluster

As described above, a login to the cmslpc-el9.fnal.gov goes to one of multiple nodes. This will redirect you to one of the nodes using a load balancer.

Since each of the individual node (see list of nodes) is a different virtual machine, they will have different RSA keys even though you can reach them all with ssh -Y username@cmslpc-el9.fnal.gov. When you modify your ~/.ssh/config, be sure to only apply these settings to cmslpc*.fnal.gov machines as they may be insecure for other clusters. You will need to accept a new RSA key fingerprint for each login, and in the current configurations above, they are sent to /dev/null and thus not retained for cmslpc*.fnal.gov machines.

Login shell

The default login shell at the cmslpc cluster is bash (June 4, 2019), the default was tcsh before then. There is more information and instructions on how to change your default login shell (using the LPC Service Portal) on the Software Setup page, including how to fix your prompt.

Notifications of downtimes; getting help

As a new user, you have been automatically subscribed to the cms_uaf_users at fnal.gov email list. This list will be used by the CMS at Fermilab Facilities Support Team to announce downtimes and provide updates on the CMSLPC and other facilities integral to its use.
System status information, including the cms_uaf_users archives are listed here.

To get help for computing, please consult the LPC Computing Get Help page at the lpc.fnal.gov pages.

US CMS News

In This Section: