Saturday December 20th at 10:09am eloise.cementhorizon.com and kalxstaff.org went down for scheduled maintenance while we moved the server from it's existing facility which is closing down at the end of the year to its new facility in Sacramento. The server came back up as planned but there were settings I failed to setup correctly. I've only just now gotten back to a computer to fix them. Those sites came back up at 2:26pm today Sunday December 21st.

Starting around 1:30am November 29th sites hosted on the odessa server (e.g. photo galleries, confluence, jira) started experiencing 5% packet loss. This has caused the sites to be intermittently unavailable. Nevin is going to go down to the datacenter tomorrow and swap out what he suspects is a bad cable.

On 10/29 at 10:14am odessa (the new server) restarted for some reason. I'm looking into the cause of this. Additionally I moved DNS services over from gloria to odessa a month ago. I forgot to configure DNS to automatically startup when the server restarted and as a result DNS services were down at that point.

Our backup DNS provider has been providing DNS resolution since 10/29 at 10:14am but finally gave up realizing that we weren't fixing our DNS server this morning 11/5 at 8:45am.

Between 11/5 at 8:45am and 2:02pm all cementhorizon DNS services were down. The result of this is that though our web services were up, they couldn't be reached by name (for example, typing in http://www.cementhorizon.com/ into your browser would not pull up the web page). The second result is that email sent to addresses @cementhorizon.com during this 5 hour window would be delayed potentially up to 10 or 12 hours from when it was sent.

I've set DNS to autostart in case the server reboots again and I've also setup monitoring explicitly on DNS so that I'm alerted if this problem occurs again.

From Sunday October 5th at 9:30am until Tuesday October 7th at 1:19pm cementhorizon and associated sites were down. This was caused by blog spam triggering oom-killer. There was nobody at the site on Monday to powercycle the server.

eloise.cementhorizon.com went down yesterday. I've moved kalxstaff.org back to gloria and it should be available shortly. I'll try to send eloise.cementhorizon.com to a maintenance page.

Yesterday morning, Saturday Aug 23 at 3:05am the server became unavailable. I'll have it powercycled tomorrow morning.

At 12:10pm today Thursday August 14 the site became unavailable. I've requested our provider restart the server but I expect it won't happen until Friday morning.

Cementhorizon became unavailable at around 9pm tonight, Sunday night. I'll keep trying to contact the server but if I'm unable to, I'll have to wait to have it powercycled tomorrow morning.
-Gene