Network Issue - 4/Dec/09 9:00AM

[Aaron]

We have a few customers reporting issues after last nights upgrade. If you are having issues please post a ping, traceroute, download test, and any other relevant information here. You should provide these results between the source and destination of the issue.

[Aaron]

We believe this issue has been fixed now, if anybody is still having issues please let us know!

The issue appeared to be a slight misconfiguration on the port to one of our upstream providers.

[Aaron]

The issues seem to have reappeared, and we are working on it currently.

[Aaron]

We have Equinix working on the cross connect on their end to try and fix this issue, while this is happening you may experience some periods of packetloss.

Unfortunately we are not able to disable Vocus BGP session while we do this work, as our route has been dampened.

[Aaron]

We disabled our Vocus link until we have resolved the issue with the cross connect.

[Aaron]

The cross connect issue has been resolved, and Vocus has been turned back on. All issues should now be resolved.

If you do experience any additional issues, please let us know!

[jbngar]

Hi Aaron

I noticed this morning that I'm pinging my own sites at around 50ms - when normally a Sydney site would be 30-35ms. Tracerouting crucial's servers resulted in this:

Target Name: N/A
IP: 203.98.88.34
Date/Time: 7/12/2009 12:45:17 PM to 7/12/2009 12:47:16 PM

Hop Sent Err PL% Min Max Avg Host Name / [IP]
1 120 0 0.0 1 4 1 SpeedTouch.lan [10.0.0.138]
2 120 0 0.0 27 37 28 [172.18.219.3]
3 120 0 0.0 27 43 28 [172.18.76.14]
4 120 0 0.0 27 49 28 vl20.core2.civ.bigpond.net.au [61.9.207.136]
5 120 0 0.0 27 149 30 TenGigabitEthernet7-3.civ17.Canberra.telstra.net [165.228.176.45]
6 120 0 0.0 28 53 29 TenGigabitEthernet0-8-0-2.civ-core2.Canberra.telstra.net [203.50.10.129]
7 120 0 0.0 28 41 29 TenGigE0-8-0-0-1.dkn-core1.Canberra.telstra.net [203.50.6.126]
8 120 0 0.0 34 57 35 POS0-5-0-0.ken-core4.Sydney.telstra.net [203.50.6.130]
9 120 0 0.0 34 42 34 TenGigabitEthernet8-1.pit1.Sydney.telstra.net [203.50.20.39]
10 120 0 0.0 33 51 34 pacnet.lnk.telstra.net [139.130.55.26]
11 120 0 0.0 34 71 35 gi1-0-1.gw1.syd2.asianetcom.net [202.147.40.194]
12 120 0 0.0 34 617 95 PTR-0001.gw2.syd1.asianetcom.net [203.192.136.206]
13 120 0 0.0 48 181 55 [203.98.88.34]


Target Name: N/A
IP: 203.98.88.34
Date/Time: 7/12/2009 12:48:11 PM to 7/12/2009 12:50:10 PM

Hop Sent Err PL% Min Max Avg Host Name / [IP]
1 120 0 0.0 1 5 1 SpeedTouch.lan [10.0.0.138]
2 120 0 0.0 28 34 28 [172.18.219.3]
3 120 0 0.0 27 37 28 [172.18.76.14]
4 120 0 0.0 27 39 28 vl20.core2.civ.bigpond.net.au [61.9.207.136]
5 120 0 0.0 27 202 29 TenGigabitEthernet7-3.civ17.Canberra.telstra.net [165.228.176.45]
6 120 0 0.0 28 35 28 TenGigabitEthernet0-8-0-2.civ-core2.Canberra.telstra.net [203.50.10.129]
7 120 0 0.0 28 39 29 TenGigE0-8-0-0-1.dkn-core1.Canberra.telstra.net [203.50.6.126]
8 120 0 0.0 34 44 34 POS0-5-0-0.ken-core4.Sydney.telstra.net [203.50.6.130]
9 120 0 0.0 34 43 34 TenGigabitEthernet8-1.pit1.Sydney.telstra.net [203.50.20.39]
10 120 0 0.0 33 40 34 pacnet.lnk.telstra.net [139.130.55.26]
11 120 0 0.0 34 42 35 gi1-0-1.gw1.syd2.asianetcom.net [202.147.40.194]
12 120 4 3.3 34 538 99 PTR-0001.gw2.syd1.asianetcom.net [203.192.136.206]
13 120 2 1.7 48 218 57 [203.98.88.34]


Note the comparatively high average pingtime on the server PTR-0001.gw2.syd1.asianetcom.net - is this supposed to be happening?

[Aaron]

Originally Posted by jbngar

Hi Aaron

I noticed this morning that I'm pinging my own sites at around 50ms - when normally a Sydney site would be 30-35ms. Tracerouting crucial's servers resulted in this:

Target Name: N/A
IP: 203.98.88.34
Date/Time: 7/12/2009 12:45:17 PM to 7/12/2009 12:47:16 PM

Hop Sent Err PL% Min Max Avg Host Name / [IP]
1 120 0 0.0 1 4 1 SpeedTouch.lan [10.0.0.138]
2 120 0 0.0 27 37 28 [172.18.219.3]
3 120 0 0.0 27 43 28 [172.18.76.14]
4 120 0 0.0 27 49 28 vl20.core2.civ.bigpond.net.au [61.9.207.136]
5 120 0 0.0 27 149 30 TenGigabitEthernet7-3.civ17.Canberra.telstra.net [165.228.176.45]
6 120 0 0.0 28 53 29 TenGigabitEthernet0-8-0-2.civ-core2.Canberra.telstra.net [203.50.10.129]
7 120 0 0.0 28 41 29 TenGigE0-8-0-0-1.dkn-core1.Canberra.telstra.net [203.50.6.126]
8 120 0 0.0 34 57 35 POS0-5-0-0.ken-core4.Sydney.telstra.net [203.50.6.130]
9 120 0 0.0 34 42 34 TenGigabitEthernet8-1.pit1.Sydney.telstra.net [203.50.20.39]
10 120 0 0.0 33 51 34 pacnet.lnk.telstra.net [139.130.55.26]
11 120 0 0.0 34 71 35 gi1-0-1.gw1.syd2.asianetcom.net [202.147.40.194]
12 120 0 0.0 34 617 95 PTR-0001.gw2.syd1.asianetcom.net [203.192.136.206]
13 120 0 0.0 48 181 55 [203.98.88.34]


Target Name: N/A
IP: 203.98.88.34
Date/Time: 7/12/2009 12:48:11 PM to 7/12/2009 12:50:10 PM

Hop Sent Err PL% Min Max Avg Host Name / [IP]
1 120 0 0.0 1 5 1 SpeedTouch.lan [10.0.0.138]
2 120 0 0.0 28 34 28 [172.18.219.3]
3 120 0 0.0 27 37 28 [172.18.76.14]
4 120 0 0.0 27 39 28 vl20.core2.civ.bigpond.net.au [61.9.207.136]
5 120 0 0.0 27 202 29 TenGigabitEthernet7-3.civ17.Canberra.telstra.net [165.228.176.45]
6 120 0 0.0 28 35 28 TenGigabitEthernet0-8-0-2.civ-core2.Canberra.telstra.net [203.50.10.129]
7 120 0 0.0 28 39 29 TenGigE0-8-0-0-1.dkn-core1.Canberra.telstra.net [203.50.6.126]
8 120 0 0.0 34 44 34 POS0-5-0-0.ken-core4.Sydney.telstra.net [203.50.6.130]
9 120 0 0.0 34 43 34 TenGigabitEthernet8-1.pit1.Sydney.telstra.net [203.50.20.39]
10 120 0 0.0 33 40 34 pacnet.lnk.telstra.net [139.130.55.26]
11 120 0 0.0 34 42 35 gi1-0-1.gw1.syd2.asianetcom.net [202.147.40.194]
12 120 4 3.3 34 538 99 PTR-0001.gw2.syd1.asianetcom.net [203.192.136.206]
13 120 2 1.7 48 218 57 [203.98.88.34]


Note the comparatively high average pingtime on the server PTR-0001.gw2.syd1.asianetcom.net - is this supposed to be happening?

Having a high ping at one of the points along the way does not mean that there is a slow down in the network, this can happen time to time as many routers don't put a high priority on ICMP (ping/traceroute) packets.

Are you experiencing any slow down other than the higher than normal pings?

[jbngar]

Hi Aaron - thanks for the reply

I'd like to be able to tell the real life 15ms difference between 50ms and 35ms, but my eyes just aren't that good ;-)

Sites etc seem to be loading fine, and there doesn't seem to be the packet loss like I had previously.

BUT it is definitely pinging slower than it was last week, and the preceding servers on the route are all pinging around the 30-35ms mark, when it gets to the asianet.com server it goes up to anywhere between 50 and 100ms, and after that the crucial servers ping around 50ms.

If you've intentionally lowered the priority of ping requests on our servers then that would explain things, if you haven't - it may be an indicator of something screwy somewhere.

It's not a massive deal, I'm sure no-one will notice, but a healthy ping value comparison is what I sometimes use to sell hosting to clients...

[Aaron]

Originally Posted by jbngar

Hi Aaron - thanks for the reply

I'd like to be able to tell the real life 15ms difference between 50ms and 35ms, but my eyes just aren't that good ;-)

Sites etc seem to be loading fine, and there doesn't seem to be the packet loss like I had previously.

BUT it is definitely pinging slower than it was last week, and the preceding servers on the route are all pinging around the 30-35ms mark, when it gets to the asianet.com server it goes up to anywhere between 50 and 100ms, and after that the crucial servers ping around 50ms.

If you've intentionally lowered the priority of ping requests on our servers then that would explain things, if you haven't - it may be an indicator of something screwy somewhere.

It's not a massive deal, I'm sure no-one will notice, but a healthy ping value comparison is what I sometimes use to sell hosting to clients...

You shouldn't get any lower performance, as we have actually increased our capacity by a significant amount, and added an additional ISP. Please keep on eye on the ping time, and if it persists reply back to this post and I'll have a closer look into this.