Compute resources on our HPCs (Stokes and Newton) are shared, and we rely allocation accounting and a system of queues to ensure that such resources are managed fairly. Limits to compute resources include queue limits and account limits.
Queue Limits
Both Stokes and Newton provide two main queues: a main queue (called normal) and a secondary one (called preemptable). The normal queue is used in almost all cases; the preemptable queue can be used when a user is out of compute hours in a given month (jobs in this queue will be paused, or preempted, when a job arrives in the normal queue).
Submitted jobs are scheduled onto nodes using a fairshare approach. This approach automatically computes a priority for each submitted job based upon the usage of the cluster by that user over the past 14 days. A user that has used the cluster a lot in recent days will have new jobs placed lower in priority; a user who has had used the cluster recently will have new jobs placed at a higher priority. This allows the fair use of the cluster and doesn't allow any one user to dominate use of the cluster.
Account Limits
In addition to the limits imposed on users in terms of queue limitations, Stokes and Newton have several limitations at the level of the faculty group account. First of all, each faculty group account has a specific number of dedicated processor hours (DPH) allocated to it per month. A DPH is an hour of computation on any single core of the system. So a 10 hour job fully occupying 3 16-core nodes consumes 10*3*16=480 DPH. By default, users receive 80,000 DPH to use per month, unless the faculty member has financially contributed to Stokes (see Faculty Buy-In below). Once that allocation has been depleted, users can no longer charge DPH to that account until it is refilled. Accounts are refilled on the 1st of each month. All users on a given PI account share this limitation.
If a user is unable to submit more work because the account that they are using has used all of its monthly allocation, the user should use the preemptable queue.
More Information
To learn more about the details of using our batch system to submit to our queues, please see our tutorial page on Job Submission within the ARCC.
Faculty Buy-In
Faculty may contribute financially to help the ARCC extend available computational capacity and are awarded additional monthly computational hours for doing so. See Faculty Buy-In for all the information.