Server Issue: s332.au.crucialx.net

[Ijan]

Hi,

We've had one of the drives fail in the primary (OS) array in s332. Currently this is not affecting any Virtual Machines as they are hosted on another array in the server however we will have to replace the failed drive. During this time the server will be off-line. I will update this thread as well as email all customers who will be effected once we confirm when the maintenance window will be.

[Ijan]

Hi,

We've scheduled maintenance for 8:30pm AEST tonight. An email has been sent to all customers who will be affected by the maintenance.

I will be on site performing the drive swap while Divya and Sherin will be online assisting me.

We will update this ticket prior to to starting the maintenance.

[timmygee]

Just a pointer, it's not evident in that email that you sent that only people affected were emailed.

I had to check this forum to confirm that

[Sherin]

Hello,

We are proceeding with maintenance.

We shall keep you updated.

[divya]

Hello,

The latest update on the maintenance is as follows. The drive has been swapped and the raid array is currently rebuilding. We shall update you as soon as the VMs are started.

Thanks,
Divya.

[Ijan]

Hi timmygee,

Good pointer, i'll make sure my announcements include that information in the future. I apologise to everyone who may have been consfused by the post.

[Ijan]

Hi Everyone,

A quick update, the server is currently undergoing a RAID rebuild of the primary RAID Array, once it has completed we will restart the Xend service and the VM's should start to come online.

We'll update you with it's progress, thanks for your patience.

[lplatypus]

Thanks so much for fixing this for us.

Out of curiosity, why does a disk failure require downtime? Do these servers not have hot swappable drives?

[Ijan]

Hi lplatypus,

Thanks for your question. In each server we have 10 drives and two RAID arrays. One Array holds the customer data and this consists of 8 x hard drives in RAID 10 configuration, these drives are hot swappable and when they fail and we replace them and the array usually rebuilds itself. The other array has the OS installed and consists of 2 x drives in RAID 1 and its one of these drives that failed, in this case once the drive was replaced the array started to rebuild however Xend service could not be restarted due to the array being in 'read only mode' so until the rebuild is complete we unfortunately cant restart the virtual machines.

We do have try to implement fail safes as you can see although unfortunately there are some issues that we cant prevent but can only mitigate the damage they can cause.

I might now make a note of saying that we are looking at a VMware solution that will incorporate fail over to spare nodes as well as live migrations and as long as all the testing goes well we'll have it out mid next year

[tdashadow]

Hi guys, is there an eta on this as i have alot of people waiting and it might just put their minds at peace