Disk Quota and Network mounted FileSystems (NFS)

The FNAL LPC Analysis Cluster provides several file sytems for its users. These filesytems are configured in different ways for different purposes. Information is provided below on purpose and configuration differences, quota and usage statistics, instructions on how to check your quota and options available to you if you should run out of quota.

What is the maximum personal quota?

The LPC Coordinators have decided to set an individual user quota of 2 TB per user in EOS /store/user, 200GB in NFS /uscms_data, and 2GB in /uscms home.

Requests for increase in space of EOS must be approved by LPC Coordinators and can be submitted as a ServiceNow Request.

• This is the link with instructions for EOS quota increase.
• NFS quota increases are not given.

NFS Disk Space

/uscms 

Backup snapshots for /uscms are daily starting at 18:00 and kept for 4 days.
  Purpose: Home areas, area small quota (2GB), backed up to hidden snapshot directory.
Note that as of May 4, 2017, the home directory path is /uscms/home/u/username, where /u/ is the first letter of your username. There is a soft link to the previous path in place.

To recover accidentally removed files, Submit a "I'm having a problem" trouble ticket with Fermilab LPC Service Portal, using your Fermilab single sign on, report what file(s), and what date they were removed for recovery. Note that files in ~username/nobackup are on a different disk and not backed up, as discussed below. Files in EOS are not backed up either.

Nobackup Data Areas: /uscms_data/d1

There are several data areas located on the NFS disk. The /uscms_data/d1 area is simply a collection of symbolic links that point to a user's actual data area (currently /uscms_data/d2 or /uscms_data/d3)
Its best if users use the /uscms_data/d1 (or ~username/nobackup) path to access their data just in case the actual data area needs to be moved to a different file system for space reasons.
Individual user quotas are set to a default of 200GB in the data areas. There are NO BACKUPS for this area, and no way to recover accidentally deleted files.

/uscms_data/d2

No backups
No snapshots

  Purpose Data area with quotas, not backed up to snapshot

/uscms_data/d3

No tape backups
No snapshots

  Purpose: Data area with quotas, not backed up to snapshot

/uscmst1b_scratch/lpc1/3DayLifetime

No tape backups
No snapshots

  Purpose: Data area WITHOUT quota; not backed up in any fashion. Do not store unreproducible work in this area - there are NO BACKUPS. The 3DayLifetime area (/uscmst1b_scratch/lpc1/3DayLifetime) is accessible by ALL users. If you do not have a directory, create one with mkdir /uscmst1b_scratch/lpc1/3DayLifetime/username (with your username). As the name suggests, files stored here are automatically removed after 3 days. Since there are no quotas on this file system we expect users to clean up their directories on a regular basis. If the file system begins to get full we will send out email to the users asking them to clean up their areas.

Checking your quota usage on NFS mounted areas:

  Users can use the UNIX "quota" command to check their disk/quota usage. The output includes information on all NFS mounted file systems even those that may not have quotas enforced.
An example - over quota on nobackup:  

[username@cmslpc333 ]$ quota -s
Disk quotas for user username (uid 55555): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
cms-nas-0.fnal.gov:/uscms
                  1158M    0  2048M          25746k       0       0        
cmsnfs-scratch1.fnal.gov:/uscms/data3/
                   212G*   200G    220G   6days    538k       0       0      

• The -s option tells the quota command to try to use units for showing the usage/limit output.
• The most pertinent number fields in this output are the first and third. The first shows how much disk is being used and the third shows what your limit is set to.
• In this example, the * is indicating that this user is over quota(second item) on ~/nobackup, however they will still be able to write files until they reach the limit within the grace period (fourth field).
• The grace period listed in this example is 6 days as the user just went over quota: once that time has passed, the user will not be able to write more than 200GB (their quota).
• If, instead, grace lists none, the user will be unable to write over the 200GB quota.
• Note that the /uscms area for your home directory has only a hard limit, so there will be no warning or grace, and once you are over limit you will not be able to write to that area.
• The first mounted disk listed is your home area, the second is your (soft linked) nobackup area, which is centrally linked as /uscms_data/d1, which is mounted as /uscms/data2, or /uscms/data3, and soft linked as /uscms_data/d2 or /uscms_data/d3

Here is an example of a user who is over quota on /uscms/homes:  

[username@cmslpc333 ]$ quota -s
Disk quotas for user username (uid 55555): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
cms-nas-0.fnal.gov:/uscms
                  2048M       0   2048M          25960k       0       0     
cmsnfs-scratch1.fnal.gov:/uscms/data3/
                 47720M    200G    220G            424k       0       0 
                   

Note that the "blocks" used is the same size as the "limit", and there is no *. This is a hard cutoff, and if you exceed it, you will see an error message is like this:

[username@cmslpc333 ~/temp]$ cp -pr testFile.root test.root
cp: closing `test.root': Disk quota exceeded

Going over quota on your home directory will have unintended consequences like not being able to write an ~/.Xauthority file upon login and use X-window forwarding, so you will need to cleanup and/or move files to other filesystems.

What to do if you exceed your quota limit on NFS:

1. Remove uneeded files
2. Utilize the 3DayLifetime area in /uscmst1b_scratch/lpc1/3DayLifetime/username. As the name implies any data stored here is automatically removed after 3 days.
3. Copy data to your personalEOS area (recursive examples)
4. If you are a member of one of the LPC collaborative groups, you can utilize the storage areas allocated to that LPC Collaborative group on EOS.
5. If all of the above are not sufficient, you can request more disk space in EOS with the following ServiceNow ticket below which will be approved or rejected by LPC Coordinators. Ensure you have exhausted all the possibilities above including the 3Daylifetime, LPC Collaborative groups, and EOS areas.
   • This is the link with instructions for EOS quota increase, as well as how to find out your current quota.
   • There is no longer a ServiceNow ticket for NFS disk quota increases, and LPC coordination does not approve NFS quota increase requests.

AFS mounts

As of Spring, 2018, the LPC CAF (cmslpc cluster) no longer mounts /afs directories on interactive or worker nodes. You may still access the /afs filesystem from CERN lxplus, but be aware that it is being phased out.