Well My Website and Blog along with all my sub domains are finally back up once again after my hosting company Hosting Unlimited (hosting-unlimited.net) had server difficulties as detailed in my previous posts, which I managed to write in between them remapping the server and then taking all the sites offline to restore the most recent backups from the 4th September 2005 at 06:00 hrs BST.

Dougie Brown the Administrative Director at Hosting Unlimited has fully apologized for the initial lack of updates when the Cassandra server was first down and since that has been updating us all the way as you can see from the quote below that I will update as more information becomes available:

We have a technical issue with Cassandra which has resulted in that server being off-line since yesterday.

I would firstly apologise for no post being entered here on the forums, which unfortunately was due to an oversight. We fully appreciate the need for information which usually goes a long way to allay fears and reduce frustration, and this post will now provide the updates which should have been here from the start.

We have as always been working flat-out on the issue since it ocurred; please be assured we are doing everything possible to restore service on Cassandra.

Thank you for your patience.

** UPDATE 13:10 hrs **
The server is now back on-line but is still not behaving properly. We continue to work on it.

** UPDATE 14:10 hrs **
We are installing a new kernel at present after working with the software, which is the issue (i.e. software). We should know in about 15 minutes if this resolves the problem. If it does not, we will update here again shortly afterwards with what we propose to do.

** UPDATE 14:45 hrs **
Unfortunately, this has failed. Here is the course of action which is now being taken, which despite taking some time, is the quickest and safest course available

1. The primary drive is being re-mapped
2. The server software will then be re-installed
3. The most recent data backups will be restored (as at 06:00 hrs BST 4th September 2005)

Min & Max timescales are:-
1. 2 – 12 hours
2. 3 hours
3. Immediate – 5 hours, depending on how your account username falls alphabetically.

Work has already begun as above. We are confident it will not run to anything like the maximum timescales, but are telling you them just in case.

** UPDATE 16:45 hrs **
The main problems regarding Cassandra appear to stem from the server having a hard reboot during the time it was FSCKing; this has caused some OS corruption. The server is being reimaged now as we speak.

** UPDATE 17:45 hrs **
The disk reimaging is progressing well, and should complete near the minimum period above.

** UPDATE 18:35 hrs **
The reimaging is complete, and cPanel is now installing.

** UPDATE 20:15 hrs **
Software install is nearing completion (point 2 above). Data restoration will begin immediately thereafter.

** UPDATE 21:00 hrs **
We had hoped to offer more recent data than 06:00 hrs 04/09/2005, but it is clearly unreliable and has suffered damage. This means that ALL data will be restored from 06:00 hrs 04/09/2005. Please therefore do not raise tickets asking for the more recent data.

** UPDATE 22:40 hrs **
The data restoration is well underway – as I write, 130 accounts have been restored (up to usernames beginning with “dj”).

** UPDATE 23.00 hrs **
Accounts now restored up to usernames beginning with “fh”. There may well be minor issues such as a few Fantastico icons not being available straightaway, but these will be taken care of once we are certain the server is settled.

** UPDATE 23.25 hrs **
This update is now closing. Restoration will continue – monitored – until completed throughout the night. A big thanks from me to all our staff here and in the DC who worked so hard to resolve this unwelcome issue, and to the affected clients who have been so patient and in many cases, encouraging.

To reiterate – accounts are continuing to be restored until completed, in alphabetical username order. This does mean that the “z”s will be last.

** UPDATE 09:00 hrs **
There are approximately 50 accounts which are still not restored, due to an undersized drive being fitted by the DC. These are accounts with usernames beginning with “sul” to “z”. We have been actively resolving this final hurdle since it became apparent, and these accounts will be restored shortly.

** UPDATE 09:55 hrs **
Cassandra will be off-line for 15-30 minutes shortly, in order to have a new drive fitted. When it is back on-line, all existing accounts will continue to function normally, and restoration of the remaining accounts will commence.

** UPDATE 10:45 hrs **
Downtime was restricted to 3 minutes. The new drive is installed and being worked on now.

** UPDATE 12:15 hrs **
All accounts have now been restored, and this update is now closed.

Thank you again for your patience.

Dougie Brown
Administrative Director.

So I would now like to say thank you to everybody at Hosting Unlimited (hosting-unlimited.net) and everybody else from the data centre you have done a great job, carry on the excellent work, I hope you will be my Hosting Company for years to come, I will surely recommend you too all my friend and family.

Hosting-Unlimited.net
Emergency Downtime Thread
Discussion of the problem at HU