On 10/29 at 10:14am odessa (the new server) restarted for some reason. I'm looking into the cause of this. Additionally I moved DNS services over from gloria to odessa a month ago. I forgot to configure DNS to automatically startup when the server restarted and as a result DNS services were down at that point.

Our backup DNS provider has been providing DNS resolution since 10/29 at 10:14am but finally gave up realizing that we weren't fixing our DNS server this morning 11/5 at 8:45am.

Between 11/5 at 8:45am and 2:02pm all cementhorizon DNS services were down. The result of this is that though our web services were up, they couldn't be reached by name (for example, typing in http://www.cementhorizon.com/ into your browser would not pull up the web page). The second result is that email sent to addresses @cementhorizon.com during this 5 hour window would be delayed potentially up to 10 or 12 hours from when it was sent.

I've set DNS to autostart in case the server reboots again and I've also setup monitoring explicitly on DNS so that I'm alerted if this problem occurs again.