[TCH-Ryan]

We will be conducting maintenance on tarvis this evening at 12:00am (10/07/07) to swap out hardware on the server in an effort to correct stability issues. The expected down time will be less than 30 minutes but can be up to 1 hour if there are any unpexected issues while the server is returning online with new hardware.

Our apologies in advance for any inconvenience this may cause and we will keep you updated in this thread as the maintenance progresses.

[TCH-James]

We are bringing this server down now. We will be replacing the drive and the raid card. When it comes back online, the raid array will be rebuilt. Once this completes, we will take it down momentarily to replace the second drive.

[TCH-James]

Tarvis is currently running a filesystem check. I will post back when the check is completed.

[TCH-James]

The server found some issues during the filesystem check which required a manual run of the filesystem check. We are currently working our way through this process and will post updates as we go.

[TCH-James]

Tarvis is back online and we are currently rebuilding the raid array with the new disk. I will post updates as we continue.

[TCH-James]

During the rebuild of the raid array, we lost the filesystem journal causing the server to crash. After rebooting the server, the raid array came up in a bad state. Due to these issues, we have placed a new machine online and will begin working on a restore from backups. We will keep updating this post as the restore continues.

[TCH-James]

We were able to bring the old machine back online so that all users will be back online. We are currently prepping a new machine and will then restore from backups. Once this is done, we will put the new machine online in place of the old one.

Updates will be posted here as we progress.

Thank you for your patience and understanding during this time.

[TCH-Vivek]

The old box is online as Jesse said. We have prepared the new box for production and accounts are being restored from Saturday's backup run. Once this is over we will be syncing databases and mails with the old box and bring the new box in place of the old one. 12% done so far.

[TCH-Vivek]

23%

[TCH-Vivek]

37%

[TCH-Vivek]

47%

[TCH-Vivek]

63%

[TCH-Vivek]

72%

[TCH-Vivek]

80% done

[TCH-Vivek]

90% done.

[TCH-Vivek]

Restore is over. We have also synced the mysql databases and mails with the old box to make sure things are up to date. The new box is now up as tarvis and everything looks fine. We are still monitoring the server. Should you have any issues please raise a helpdesk ticket. Thank you all for your co operation with us while we were working on it.