Vocus Event Notification - bdr02.sjc01 reload

[Aaron]

From Vocus:

Start Date: Thursday 13th October 2011, 22:42 AEDT (UTC +11); 00:42 NZDT (UTC +13) End Date: Thursday 13th October 2011, 22:51 AEDT (UTC +11); 00:51 NZDT (UTC +13)

A border router in San Jose, California went into a low memory condition and ceased responding to commands. Vocus engineers collected the required logging data and upon conclusion of this exercise the device reloaded itself.

A high severity case has been logged with our vendor so that this issue can be properly investigated. Enough information was obtained to point to the process that was causing the issue so we are confident the incident can be recreated in the lab.

We will issue further details in coming days but do expect this to require maintenance at a later date in order to patch the code on this device.

[Aaron]

Update from Vocus:

Customers please note that the same Vocus border router in Jan Jose California that reloaded on Thursday evening (details below) is once again in a low memory condition. We have been monitoring the device closely and have both Vocus engineers and vendor engineers working on the high sev case.

Emergency maintenance may be required on this device during the course of the day. We will keep you informed.

[Aaron]

Update from Vocus:

Start Date: Saturday 15th October 2011, 11:00 AEDT (UTC +11); 13:00 NZDT (UTC +13) End Date: Saturday 15th October 2011, 11:30 AEDT (UTC +11); 13:30 NZDT (UTC +13)

During the above window, Vocus Engineers will be performing an emergency reload of a device at the Market Post Tower datacentre in San Jose. This is in response to the notification sent earlier today (listed below).

[Aaron]

Update from Vocus:

Update: bdr02.sjc01 Memory Leak

We are continuing to investigate the cause of the memory leak problem on bdr02.sjc01. No recent changes have been made on this device and we are working with our vendor to investigate the issue.

We are losing memory to a process at a rate of approximately 1% per hour.

Our calculations with our vendor support lead us to the conclusion that to leave this device we would have been facing an unscheduled reload at approximately 7PM AEDT (9PM NZDT) this evening so the decision was made to free up the memory now to avoid peak impact to the network. Unfortunately as this is an IOS-XE device, the only way to free that memory up was to reload the device.

After the reload we have observed with our vendor that the leak is still occurring but we will need to take action within the next 48 hours in order to avoid a crash as the free memory pool reduces. We are continuing to work with them on the analysis of this problem and will provide updates as we form an action plan.

[Aaron]

Update from Vocus:

Resolved: bdr02.sjc01 Memory Leak

We have tracked down the trigger for the leaking memory on bdr02.sjc01 and disabled it. The device is now sitting at 68% free memory and has been steady at this level for the past 30 minutes.

A defect is being generated by our vendor and a software update will be required at a later date to permanently resolve the cause of this problem but no further unplanned service impact is anticipated.