Here is the explanation for what happened to our USA access numbers today:
Downtime: 45 Minutes
Area Affected: USA
Impact: Primary access numbers only allowed intermittent connections.
Compensation: 90 free minutes for each USA account, 3 million minutes total.
As you may know, our access number provider added additional capacity to our access numbers on Friday. This was necessary to handle the growing amount of traffic on our system. When they added this capacity, a technician accidentally changed the routing scheme that is used to send SIP connections from their trunks to our colo facility. The change in the routing rules resulted in our router getting flooded with connection requests and not rolling those requests over to our other servers.
In a nutshell, our router was getting sent inbound calls in a less efficient manner than it is used to. When we hit our Saturday peak, the router was not able to process all of those calls. Furthermore, the fail over method we use could not kick in because it too did not understand what the incoming requests were saying. We are working closely with our provider to prevent this from happening again.
I was in the park with my daughter when I got the first Facebook notification we were down. 45 minutes later our techs had the issue resolved. One of them was getting ready for his girlfriend’s college graduation (Sorry Hollee and congrats!) and the other had just landed at LAX. Thanks to them and our access number provider’s NOC, we had the problem fixed quickly.
To make up for being down during this peak period, I have added 90 minutes of free calling to the accounts of all active USA customers. If you are on a fixed minute plan, your packaged balance has 90 more minutes. If you are on the 1.5 cent calling card, your balance has an extra $1.35 on it. If you are on Quarter Calls, your balance has an extra $1.25 minutes.
In total, I just gave away over 3 million minutes of free calling. My partners will probably kill me for this or dock my pay severely, but its important for IndiaLD to let all of you know how much we value your business. This is especially true for all of our new customers who haven’t enjoyed hundreds of hours of calling to India with us yet.
If you were one of the callers who was affected this morning, please accept these minutes as our apology.
If you were not affected this morning and are just finding out about this now, enjoy the free minutes.