Excitement
Yesterday afternoon, an employee at the Pasadena power plant did something to shut power down to 1/3 of the city for about 10 seconds.
It came right back up, but not before doing some collateral damage. In the process of restarting, Caltech’s chilled water plant blew a pipe.
As in, no more chilled water coming out of the plant.
Guess what keeps our computer rooms cool?
Last night the temperature in the server room shot up past 80, and all “non-essential” machines were shut down. This includes all the development and testing servers, a handful of operational (but apparently “non-essential”) web servers, and parts of our mail system. As of this morning, only 2 of the IPAC operational servers are still up, and I have no idea how long that will last. If it keeps getting hotter in there, they’ll shut them down to keep them from getting damaged.
So, here I am at work, playing on my new MacBook Pro, waiting for news. I’m hoping to at least be able to redirect all our currently-down services to an error message page somewhere, but everyone who can edit DNS is currently running around and not easily grab-able. No other IRSA folks have bothered to show up (my boss would be here, but he’s in Seattle for a big astronomy conference). Most of the lights in the building are off, and the doors to the computer room and to outside are wide open, with fans blowing outside air in.
And the sun just came out, so now it will get warmer. Keep your fingers crossed for our poor servers!
In other news, the horses that drive the campus wagon bus are sick, so students will need to need to ride their own horses to class.
Seriously, I thought Caltech and NASA were, like, high-tech. Do you take turns shoveling coal for steam power to run the server room as well?
Comment by Jeff — 1/10/2007 @ 11:03 am
There are tanks of liquid nitrogen at various places on campus. Bring in a couple of tanks, crack the valves, and just remember you really don’t need the full 21% of O2 to survive ;).
Comment by Carol — 1/10/2007 @ 3:07 pm
I love Jeff’s comment!
Comment by Jen Yu — 1/11/2007 @ 9:59 am