August 29, 2014

Unfortunately we and other customers in Cork Internet Exchange suffered an outage this afternoon, which caused a complete loss of network connectivity to and from our servers; with the exception of DNS and backup services, and customers on our US, UK and DE machines. There should be no long-term effects, but if you do notice anything untoward, please let us know.  Sincere apologies for any inconvenience caused. An explanation of the cause of the problem from CIX is below.

Today, Thursday 28th August, at approximately 4:30pm, CIX NOC began to see networking problems that affected a large number of customers. The problem cleared at 5:10pm. All customers connected to two Cisco 1Gbps core switches lost public connectivity to the Internet during all or part of this window. Customers connected to our Juniper 10Gbps and 40Gbps core switches were not affected.

We believe that, at 4:30pm, a customer working in the data centre inadvertently created a link between the two Cisco switches by patching a connection between them. This link started a broadcast storm in that network. By 4:45pm all services on those two switches stopped working. At 5:05 the customer removed the link at the network resumed normal operation shortly afterwards.

If your services were affected by this event then we apologise. We will investigate preventative measures to avoid a recurrence.

Yes, that person is a complete dipstick.