Hazard Notification - Maintenance: Impacts Cloud VPS / HA Hosting - 7/8th Feb 2012

[ross]

Hi Everyone,

We have scheduled the following maintenance. All affected Customers will be emailed with the relevant details.

Please note hazard notifications are intended for no or minimal impact maintenance windows.

This is a hazard notification regarding some scheduled emergency maintenance that potentially impacts our HA Cloud VPS Services (Cloud Linux VPS and Cloud Windows VPS) / HA Shared Hosting Services (HA Business Hosting / HA Reseller Hosting).

You are receiving this notice as you are impacted by this event. We have split this maintenance into two windows. Further details below,

MAINTENANCE START: Tuesday 7th and Wednesday 8th February 2012, 10:00PM - 06:00AM AEDT
EXPECTED TIME TO COMPLETION: 8 Hours
REASON FOR MAINTENANCE: We are performing maintenance on our SAN Infrastructure, further several patches are being installed on the Citrix XenServer software platform.
SERVICES IMPACTED: No impact is expected during this maintenance window due to the nature of live migrating services as the patching is completed on each affected blade / node. Further the SAN storage is clustered, allowing for maintenance to be completed with no impact.
EXTRA NOTES: This does not impact our legacy Linux / legacy Windows VPS solutions. This does not impact our legacy Business / Student / Reseller Hosting.

[ross]

Hi Everyone,

Please note there is currently a fault affecting some Customer Cloud VPS's and some of our internal HA hosting services.

During the routine maintenance a fault occurred with the virtualisation software, we are liaising with our vendor now to address the fault.

Currently there is no e.t.a.

The following HA Shared / Business and Reseller Hosting Nodes are impacted,

HA Reseller Nodes:

rs1-eqx-syd.ha-node.on.net
rs2-eqx-syd.ha-node.on.net
rs4-eqx-syd.ha-node.on.net
rs5-eqx-syd.ha-node.on.net

HA Business Hosting Nodes:

sh2-eqx-syd.ha-node.on.net
sh6-eqx-syd.ha-node.on.net

We apologize for the inconvenience, and are working with our vendors to resolve this issue as promptly as possible.

[ross]

Hi Everyone,

All affected HA Reseller and HA Shared hosting nodes have been restored.

We are continuing to work on restoring Customer Cloud VPS services.

Thankyou everyone for your patience.

[ross]

Hi Everyone,

95% of all Customer Cloud VPS's are now restored.

There are a couple with issues, that we are liaising with Citrix to resolve.

An incident report will be prepared once all logs / debriefs have occurred. An e.t.a on the incident report will be provided in 24 hours.

Thankyou for your patience and understanding so far.

[ross]

Hi Everyone,

Please note, the maintenance work to patch several critical bugs on the citrix xenserver platform is continuing now that all services have been restored.

As indicated to us by senior Citrix engineers, we are continuing with the patching, as they advised that leaving some blades unpatched and others patched can lead to stability issues.

The estimated time to completion of the work is mid day.

We apologize for the issues and the extended maintenance window pushing into business hours.

Please note, Citrix are investigating what caused the fault that impacted some services.

[ross]

Hi Everyone,

The E.T.A has now been re-calculated.

We believe the work will be completed at around 2:00PM.

Thankyou for your patience during this time.

Please note, we have a Citrix support officer on standby should any further issues arise.

[ross]

Hi Everyone,

Just an update, the maintenance is ongoing, and has been delayed further due to an Ubuntu 10 LTS bug that affected some Customers.

We are continuing with the maintenance work now.

E.T.A to completion is 8PM. (Live migrating services around is not a quick process, which is where all the major delays are coming from).

[ross]

Hi Everyone,

Apologies for the delayed update.

All maintenance has been completed.

Investigations are continuing, once complete an incident report will be prepared and emailed to all affected Customers.

[ross]

Hi Everyone,

We are just finalizing an incident report now.

Once the draft has been confirmed, we will send out the details to all impacted Customers.

[ross]

Hi Everyone,

All affected Customers have been notified via email. This includes our Cloud VPS Customers and any HA Business (Shared) / HA Reseller Customers.