Update; Campus Power Outage July 4, 2008

Power was lost to the northwest section of the Vancouver campus at 8pm on Friday July 4th. Approximately 60 buildings were affected, including the L.S. Klinck Building. At 2:30 am Saturday, a faulty fuel sensor in the UBC IT generator caused it to shut down unexpectedly and without warning. Consequently all systems in the L.S. Klinck IT data centre failed due to lack of power.
Most systems/services were up by mid-day Saturday. Some sporadic outages continued on into Monday July 7th, notably the Finance system and non-critical aspects of the SIS. Most of these issues were resolved the same day. The remaining issues, associated with Interchange email vacation messages and forwarding, were resolved on Tuesday evening.
UBC IT is continuing to work with Plant Operations to determine why the generator’s fuel sensor failed. The UBC IT generator is tested monthly to ensure proper operation when needed. These tests were fully up to date. The testing protocol will be reviewed to ensure the risk of future incidents of this type are mitigated. A formal IT Critical Incident Review will be conducted.