The National Institute for Computational Sciences

File Systems


Introduction


At the time of this writing, two filesystems are available to ACF users: the Network File System (NFS) and Lustre Haven. NFS contains home and project directories. Lustre space contains project and scratch directories. Table 1.1 summarizes the available filesystems.

Table 1.1 - File System Summary
File SystemPathQuotaPurged
NFS Home Directory /nics/[a,b,c,d]/home/<username> 10GB Not Purged
Lustre Scratch Directory /lustre/haven/user/<username> No Quota Purged
Lustre Project Directory /lustre/haven/proj/<project> By Request Not Purged

Please note that while both NFS and Lustre are reliable filesystems, errors and corruptions can still occur. It is your responsibility to back up your data. To learn about this process, please review the Data Transfer document.

Home Directories


Home directories use the NFS (Network File System). NFS enables a local system to access remote storage resources as though they were part of that local system. On the ACF, fifteen terabytes of storage space is available to NFS. Home directories on NFS are regularly backed up.

When you create a new account on the ACF, you receive your own home directory on NFS. This is your personal storage location. Only you have access to your home directory; no other users can access it. Here you can store job scripts, virtual environments, and other types of files and data as you require. For convenience, you can refer to your home directory with the environment variable $HOME or with the tilde (~) character.

By default, your home directory is limited to 10GB of storage space. It is not intended to store large amounts of job-related data, nor is it a location for data backups. For job-related data, please use your scratch directory. If you require data backups, please transfer your data from the ACF to your own storage resources. Please refer to the Data Transfer document for more information.

To determine how much storage space you have consumed in your home directory, execute the quota -s command. Figure 2.1 shows the possible output of this command. Of interest are the first “space,” “quota,” and “limit” fields. The “space” field shows how much storage space is currently in use. The “quota” field displays the soft quota placed on your home directory. The “limit” field defines the hard quota, which is the absolute maximum storage space you can consume. When you exceed the quota, you will start a grace period that gives you time to reduce your storage space usage. If you do not reduce your storage space usage during this period, the soft limit defined by the “quota” field will be enforced.

Disk quotas for user user-x (uid 00001): 
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
nfs.nics.utk.edu:/nfs/b
                  3144M  10240M  10752M           58432       0       0
Figure 2.1 - Output of quota -s

Project Directories


To share data with your research group, you will need a project directory. Both NFS and Lustre project space can be provided for your project. Please submit a project directory request to receive storage space for your group. Specify the project, directory location, size of the storage space, storage duration, and an explanation of how you intend to use the storage. For NFS project directories, they will be accessible at the /nics/a/proj/<proj-directory> absolute path. For Lustre project directories, they will be accessible at the /lustre/haven/proj/<proj-name> absolute path.

Note that NFS project directories are regularly backed up. Lustre project directories can be backed up for a fee.

NFS project directories are usually limited to 10GB of storage space. Quotas on Lustre project directories can be requested by the PI of the project. Neither NFS nor Lustre project directories are purged.

Scratch Directories


Scratch directories use the high-performance Lustre filesystem. On the ACF, 1.7 petabytes of Lustre storage space is available to all resources. No Lustre scratch directories are backed up. Additionally, Lustre scratch space is purged every thirty days. Please transfer your data from Lustre scratch space to your own storage resources before purges take place. For more information on how to transfer data, please review the Data Transfer document.

Each user has access to a scratch directory in Lustre. The name of the directory is the same as your ACF username. To access your scratch directory, execute cd /lustre/haven/user/<user>. For convenience, use the $SCRATCHDIR environment variable to refer to your Lustre space.

Lustre scratch space is intended to be used for job execution. All your jobs on the ACF should be executed from within Lustre space. After the job has successfully completed its tasks, any data created by the job should be archived and transferred to your own storage resources. If job data is purged, it cannot be recovered.

If you wish to determine which files are eligible to be purged from Lustre space, execute the lfs find $SCRATCHDIR -mtime +30 -type f command. If you wish to view your total usage of Lustre space, execute the lfs quota -u <user> /lustre/haven command. Please note that any attempts to circumvent the ACF’s purge policies will not be tolerated. If such attempts are discovered, your access to Lustre space may be revoked.


Last Updated: 01 / 07 / 2020