The AC unit compressor failed yesterday afternoon, and was turned off around 4pm. All servers were shutdown a few minutes later, since temperature spiked from 32 to 35 degrees in a few minutes, and damage to servers would result from operation at such temperatures for any extended period of time.

Core network and Email services were turned on at approximately 9:30am, since a fan was being used to circulate modest amounts of ambient air into the server room. No additional equipment could be sustained with this cooling.

A new compressor was flow in from Montreal, delivered around 12:30, and was installation was completed just before 4pm. The unit appears to be running smoothly, however

  • Given the consequence of any surprises with the unit operating under heavy load overnight...
  • It has been recommended we run the AC at a low-to-moderate thermal load until tomorrow AM
  • As a result, most servers (anchor, jade, kt-4800, etc etc) will remain OFFLINE until tomorrow.
  • once we get a "clean bill of health" following inspection of the AC tomorrow morning, it will be considered safe to power on all equipment.

I anticipate that equipment should be turned on starting at ~10am tomorrow, unless we have any problems. Please do NOT consider this a rock-solid timeline though.


THURSDAY OCT.4th - UPDATE:

The AC unit in the server room seems to have had a good night; temperature was solid at 19 degrees under moderate load. The unit has checked out OK and is getting a small top-up of coolant. Otherwise, everything is operating perfectly.

I've just powered on the main UPS / power supply in the server room, which is the first step in the boot up process for bringing all servers back online.

Anchor, Jade, and KT-4800 will likely take 20-30 minutes before they are online and operating normally.

Most other servers will come up a bit faster than this. In theory, all equipment should be online and operating normally by ~10am at the latest.

If any major problems are observed after this time, please contact Tim Chipman.