Overview
The Campus Cluster file system provides projects with a high performant storage space that is attached to a large compute service and the high-speed CARNE network. Use models include operational data storage and scratch space for analysis, mid-term storage for research data, a central storage space for compute and data sharing workflows for cross-campus collaboration, and more. The Campus Cluster file system offers scalable storage from 1 terabyte up to many petabytes available in dedicated project file sets available across a wide variety of protocols including Direct HPC compute access, Globus, SFTP, rsync, NFS, and Samba. The storage provides high-availability and high-reliability through the use of RAID technologies and multi-host presentations to all core servers from redundant controllers, to protect against controller and server failures.
Hardware
The Campus Cluster file system hardware is comprised of 6 Dell R740 servers each with 32 cores/384GB memory, dual EDR,FDR, and 40GbE interfaces, and 2 Samsung PM1725 3.2TB NVME SSDs for metadata, these make up the core I/O servers. The system deploys a DDN SFA 14KX which is currently half populated with 420 8TB drives providing 2.5PB of usable space (capacity can grow to 5PB usable) and is connected via 12 x 100Gb EDR Infiniband for the bulk storage capacity. Also the system includes 3 Dell R6515 servers each with 64 cores/256GB memory and a 1 x 100GbE NIC dedicated to data access via Globus, NFS, SMB, rsync, scp, bbcp, and more.
Quotas
Home: Users have a 5GB soft quota and 7GB hard quota on their home directory. A user’s home directory is intended to be used for things such as applications installation, scripts, and job log storage. There is no inode quota on home space.
Projects: Each project has a quota based upon their level of investment and will vary from project to project. The default per user, per project directory inode quota is 20 million inodes, if this becomes and issue the user and/or their tech rep should contact the project office to discuss options.
Scratch: Users have a 10TB quota in their scratch space, this quota spans usage between their /scratch/users/ directory and any data they may have generated in /scratch/batch. There is no inode quota on scratch space.
Purge
Purge is not run on home or projects space, but is run daily on scratch space removing files that are older than 30 days. If data needs to be retained on the cluster for longer than 30 days, that data needs to be moved into home or projects locations.
Snapshots
Snapshots are intended to provide users and projects the ability to recover accidentally deleted files in the case of a scripting, command line, or overwrite error. Snapshots reside on the same physical hardware as the active data.
Home directories are snapshotted once per day, with snapshots being retained for 30 days. Data residing only in the snapshots is not counted towards a user’s quota.
Project spaces are snapshotted once per day, with snapshots being retained by default for 30 days. If a project wishes to shorten the time snapshots are retained, please file a request with the project and the storage team will accommodate it.
Backups
The campus cluster program offers a DR (Disaster recovery) service that allows users to pay an additional rate (lower than the primary storage rate) for their data to be backed up daily to an off-site/off-campus location. If this option is of interest to you or your team, please check out the Buy Storage page for more information. While Campus Cluster Operators will do everything to make the service as reliable as possible, having only one copy of data is a risk. So no data on the Campus Cluster system should be considered safe with a single copy. Campus Cluster customers are encouraged to back up their important data themselves or by enrolling in the DR program.