Re: !! SERVICE OUTAGE !!
Posted: April 29th, 2009, 12:18 pm
I told my unix admin and my windows admin (Lee) that our old servers needed to be cancelled by Monday night. Which meant they had until Sunday night to migrate data, and then all of Monday to get services setup.
Unix services were done on time, including the virtual infrastructure.
Windows services required 3 "servers" to be installed, patched, configured, files migrated, and services setup. Since file transfers were slow and Lee didn't prioritize things as much as I would have liked, the process continued until today (an extra day and a half).
Due to the fact that all of this hardware is extremely expensive and our customers (and tsugaming users) contribute to an extremely small percentage of our costs, the priority was not uptime; the priority was maximum data retention while canceling the boxes on time. Which they were.
So long story short:
[1] Migration started
[2] Data was moved over to new box
[3] Old boxes were cancelled
[4] New box was setup
[5] New virtual servers were setup
Had step 2 taken a couple hours rather than overnight ... and had step 5 taken 4-6 hours rather than a full day, we'd have been set up on time. But it was impossible to completely set up the new infrastructure and test it before shutting down old services. Put simply: it wasn't financially worth it. There was over $1000/month worth of hardware involved in this process.
Unix services were done on time, including the virtual infrastructure.
Windows services required 3 "servers" to be installed, patched, configured, files migrated, and services setup. Since file transfers were slow and Lee didn't prioritize things as much as I would have liked, the process continued until today (an extra day and a half).
Due to the fact that all of this hardware is extremely expensive and our customers (and tsugaming users) contribute to an extremely small percentage of our costs, the priority was not uptime; the priority was maximum data retention while canceling the boxes on time. Which they were.
So long story short:
[1] Migration started
[2] Data was moved over to new box
[3] Old boxes were cancelled
[4] New box was setup
[5] New virtual servers were setup
Had step 2 taken a couple hours rather than overnight ... and had step 5 taken 4-6 hours rather than a full day, we'd have been set up on time. But it was impossible to completely set up the new infrastructure and test it before shutting down old services. Put simply: it wasn't financially worth it. There was over $1000/month worth of hardware involved in this process.
