Changes due to ARCC Fall 2019 Maintenance Cycle

Details

Written by R. Paul Wiegand R. Paul Wiegand

Published: 19 December 2019 19 December 2019

Stokes and Newton have returned to operation four days early! Please remember that we have two such scheduled maintenance downtimes per year, one after Fall term (the one we just completed) and one after Spring term.

Please take a moment to read over the changes:

1. We upgraded our resource manager, SLURM, to version 19.05.4. The new version has much better GPU support. We will be revising our documentation online soon to show how to take advantage of these features. Until then, here is more information: https://slurm.schedmd.com/SLUG19/GPU_Scheduling_and_Cons_Tres.pdf
2. Singularity support was deployed on both clusters. Singularity is a container service like Docker (and has the ability to run Docker images). To run singularity, just load the singularity/singularity-3.5.2-go-1.13.5 module! Again, we will be rolling out improved documentation in the coming weeks to explain how to use this. Until then, here is more information: https://sylabs.io/guides/3.5/user-guide/
3. The module system (lmod) was upgraded, and its configuration was changed so that the module hierarchy will be cached. This should improve latencies when using the 'ml' and 'module' commands, including during login.
4. We upgraded our NVIDIA GPU drivers to the latest versions and made the CUDA 10.2 module available to users.
5. We completed internal tool configuration changes, as well as our regular benchmarking of the compute, file system, and network performance. All show the clusters operating as expected.

We appreciate your patience and wish you the best with your research!

Fall Maintenance cycle downtime, Dec 15 - Dec 22

Details

Written by R. Paul Wiegand R. Paul Wiegand

Published: 22 November 2019 22 November 2019

• downtime
• maintenance

Stokes and Newton will be taken down per our bi-annual routine maintenance cycle during mid-December. Specifically, the clusters will be unavailable from Sunday, December 15 through Sunday, December 22.

The primary objectives during this downtime is to upgrade SLURM and our module system.

Recall that we now routinely bring the system down twice a year, once in late Fall and once in late Spring. We will keep the users notified in advance of such downtimes, but we recommend you build such expectations into your workflow. Though we anticipate no data loss during this time, it's never a bad idea to backup your materials. So we suggest you use this opportunity to copy salient data and code off of Stokes and Newton prior to the downtime.

Power loss in research park, ARCC outage

Details

Written by R. Paul Wiegand R. Paul Wiegand

Published: 27 September 2019 27 September 2019

• outage

Last night (Thursday, September 26) at 11:30p, a commercial power loss in research park caused an outage in the Partnership buildings, where our clusters reside. The outage lasted 2.5 hours. The good news is that our system functioned precisely as designed: Our UPS held the load of the cluster for the designed length of time, and the core infrastructure elements (e.g., file system, infiniband backbone) were protected the entire time via generator.

But the bad news is that the length of the outage exceeded our UPS's designed limits, which means that almost all compute nodes lost power. All jobs on Stokes, and many jobs on Newton were lost. We apologize for the inconvenience.

The clusters are back up and accessible now. Queued jobs have deployed and are running.