CIO update: Post-mortem on the Skype outage
As a follow-up to last week’s outage, here is a detailed explanation of what transpired, the root cause, and plans to mitigate this from happening again in the future. For starters, it helps to understand that Skype is based on a peer-to-peer (P2P) network, which is explained here. Last week, the P2P network became unstable and suffered a critical failure. The failure lasted approximately 24 hours from December 22, 0800 PST/1600 GMT to December 23, 0800 PST/1600 GMT.
What was the cause for the failure?
On Wednesday, December 22, a cluster of support servers responsible for offline instant messaging became overloaded. As a result of this overload, some Skype clients received delayed responses from the overloaded servers. In a version of the Skype for Windows client (version 5.0.0152), the delayed responses from the overloaded servers were not properly processed, causing Windows clients running the affected version to crash.
Users running either the latest Skype for Windows (version 5.0.0.156), older versions of Skype for Windows (4.0 versions), Skype for Mac, Skype for iPhone, Skype on your TV, and Skype Connect or Skype Manager for enterprises were not affected by this initial problem.
However, around 50% of all Skype users globally were running the 5.0.0.152 version of Skype for Windows, and the crashes caused approximately 40% of those clients to fail. These clients included 25–30% of the publicly available supernodes, also failed as a result of this problem.
If approximately 20% of total Skype clients failed, why was there a much bigger disruption to Skype functionality?
Although Skype staff responded quickly to disable the overloaded servers and to eliminate client requests to them, a significant number of supernodes had already failed. A supernode is important to the P2P network because it takes on additional responsibilities compared to regular nodes, acting like a directory, supporting other Skype clients, helping to establish connections between them and creating local clusters typically of several hundred peer nodes per each supernode.
Once a supernode has failed, even when restarted, it takes some time to become available as a resource to the P2P network again. As a result, the P2P network was left with 25–30% fewer supernodes than normal. This caused a disproportionate load on the remaining available supernodes.
Why weren’t the other supernodes available to help?
The failure of 25–30% of supernodes in the P2P network resulted in an increased load on the remaining supernodes. While we expect this kind of increase in the instance of a failure, a significant proportion of users were also restarting crashed Windows clients at this time. This massively increased the load as they reconnected to the peer-to-peer cloud. The initial crashes happened just before our usual daily peak-hour (1000 PST/1800 GMT), and very shortly after the initial crash, which resulted in traffic to the supernodes that was about 100 times what would normally be expected at that time of day.
Supernodes have a built in mechanism to protect themselves and to avoid adverse impact on the systems hosting them when operational parameters do not fall into expected ranges. We believe that increased load in supernode traffic led to some of these parameters exceeding normal limits, and as a result, more supernodes started to shut down. This further increased the load on remaining supernodes and caused a positive feedback loop, which led to the near complete failures that occurred a few hours after the triggering event.
Regrettably, as a result of the confluence of events – server overload, a bug in Skype for Windows clients (version 5.0.0.152), and the decline in available supernodes – Skype’s functionality became unavailable to many of our users for approximately 24 hours.
How did Skype help support supernode recovery?
In order to restore Skype functionality, the Skype engineering and operations team introduced hundreds of instances of the Skype software into the P2P network to act as dedicated supernodes, which we nick-named “mega-supernodes,” to provide enough temporary supernode capacity to accelerate the recovery of the peer-to-peer cloud.
By late Wednesday night (PST) it was evident that only a proportion (about 15-20%) of Skype users connections were ‘healing’ and the volume of load on the supernodes continued to be unusually high. In response, our team introduced several thousand more mega-supernodes through the night. During Wednesday night, full recovery of the P2P network was underway and the majority of users were able to connect to the P2P network normally by early morning (California-PST) on December 23rd.
As we reported during the incident, in order to recover the core Skype functionality as quickly as possible, we utilized resources normally used to support Group Video Calling, to deploy supernodes, and over the course of Thursday night and Friday morning we returned these to their normal use and restored Group Video Calling functionality in time for Christmas.
The supernodes stabilized overnight on Thursday and by Friday, several tens of thousands of supernodes were supporting the P2P network. During Friday, we withdrew a significant proportion of the mega-supernodes from service, leaving some in operation to ensure stability of the P2P network over Christmas and New Year.
What is Skype doing to prevent this from happening again?
We understand how important the reliability, security and quality of our software is to Skype users around the world, and we work hard to maintain high standards, as well as develop new features and products.
First, we will continue to examine our software for potential issues, and provide ‘hotfixes’ where appropriate, for download or automatic delivery to our users. Since a bug was identified in Skype for Windows (version 5.0.0.152), we had provided a fix to v5.0 of our Windows software prior to the incident, and we will provide further updates for download this week. We will also be reviewing our processes for providing ‘automatic’ updates to our users so that we can help keep everyone on the latest Skype software. We believe these measures will reduce the possibility of this type of failure occurring again.
Second, we are learning the lessons we can from this incident and reviewing our processes and procedures, looking in particular for ways in which we can detect problems more quickly to potentially avoid such outages altogether, and ways to recover the system more rapidly after a failure.
Third, while our Windows v5 software release was subject to extensive internal testing and months of Beta testing with hundreds of thousands of users, we will be reviewing our testing processes to determine better ways of detecting and avoiding bugs which could affect the system.
Finally, as we continue to grow, we will keep under constant review the capacity of our core systems that support the Skype user base, and continue to invest in both capacity and resilience of these systems. An investment program we initiated a year ago has significantly increased our capacity already and more investment is planned for 2011 both to support the ongoing roll out of our paid and enterprise products, and to continue to support the growth of our core Skype software that we know millions of users rely on every day.
We are truly grateful to all of our users and humbled by your continued support. We know how much you rely on Skype, and we know that we fell short in both fulfilling your expectations and communicating with you during this incident. Lessons will be learned and we will use this as an opportunity to identify and introduce areas of improvement to our software, further assess and invest in capacity and stability, and develop better processes for outage recovery and communications to our user base. Thank you to everyone.
Thanks for the info. Skype is a great product. Keep up the great work!
Wow! What an excellent description.
My personal opinion of Skype has gone up following this technical issue, simply on the way you handled it! I think most others would agree.
Funny how a disaster can actually be mitigated by good customer service and an open honest approach. If only other companies acted similarly.
Awesome thanks for the detailed explanation and compensation of 7 day credit.
Skype is and will always be my main mode for business communication remotely or office as well as for family connections.
Will continue to support such a excellent company..
Happy New Year Skype, see you next year & the next & the next…..
Regards
Shazz
The humorous thing in all of this, is that if no one would of updated to 5.0 the problem would never of happened, stupid buggy 5.0 update, GO DIE.
I am very glad that I HATE AND DESPISE the new skype 5.0 so therefore I wasn’t apart of the problem of the network going to down, due to their stupid useless and buggy 5.0 update.
OH that thing about automatic update, I choose to NOT update my skype to 5.0 and I won’t until the make it work exactly like 4.0 does.
They sacrificed the mic settings somehow that makes them 10x more sensitive than necessary and it causes the audio to bounce around and raise and lower on it’s own.
I do not have this problem on skype ver 4.2.0.187, it work as it should and how I’ve grown to love it.
I do not want to be forced to update to any client in the future.
Great attitude and approach from an Executive of a Service Provision organization. Straight talk – an alien trend in the marketplace!
I remain your customer!
Thank you to take the time to explain the situation…. I use a Mac OS X and I’m a Skype user for at least 3 years now and I’ve to say that this is the first time I had problem with.
Thank you also for the voucher, I can understand some people frustration but although is only 1$ voucher (1$ for me but how many for you?) I think is a gesture of appreciation for your customer. Keep up to good work!
to all the Skype family Happy New Year!
isabella
Thank you for explaining what happened and for your honesty. It is extremely rare that businesses explain to paying customers what has happened and then offers credit, and as of today even an extension of 1-week’s service. Usually there is no explanation and “what is lost is lost — too bad for you (the paying customer)”. This attitude absolutely annoys me. So thank you for your openness. You (Skype) are definitely above everyone else in my books.
Dear CIO,
Thanks for the explanation. A very nice gesture from skype to reach every customer. Most importantly considering everyone – even if they use free PC-2-PC – as customer is a great gesture!
regards,
Jagmohan
Hey folks — I appreciate your explanation. I for one am surely NOT going to ask for any voucher from you folks. You provide an incredible service for which I am very grateful! Thanks so much!
First of all thanks to Skype for the services it offers!
… second Is better to send a mil with the word “sorry” that a mail with an empty voucher!!!
I wish you a better 2011!!!
Thank you for your efforts to compensate and appologise. Greatly appreciated.
Please don’t autoupdate without permission from the user. Some of us have a data plan with limited downloads, or we just need our full bandwith at a particular time. Instead, show a message prompting us to update, and informing whether it’s a critical update or a minor fix.
Your failure of December 22 is a sign of incompetence, your reply is pure bla bla and that the rate increase you have put up is absolutel outrageous. You do not help to keep clients with increases of 150% in my case from 7.85 for three months to 6.99 for one month. I won’t renew and will surepy try and find alternatives. Not to mention your inability to serve well your clients using samsung galaxy under android .
Regards
Yvan Gauthier
I agree with the comment above by markskessler:
I really appreciate the long and thorough explanation, and I accept all the apologies, but a voucher for 1 dollar is just ridiculous. Just the apologies would have left a better impression.
To the great people at Skype,
Everyone has an off-day, but not everyone is so forthcoming and honest. Thank you for your gracious emails, straightforwardness with the press, and technical detail on how Skype works. I am very happy to call myself a Skype subscriber and continue to recommend the service to my friends and acquaintances all over the world.
Again, thank you, and you freakin’ rock!
Most Sincerely,
Luke Landis
Vancouver, Canada
I am so very impressed with Skype and how you stepped up and acknowledged the problem, explained it to us (even though very few could actually understand it, I still appreciated the work you went through to give us answers), and offered us compensation beyond the amount of time some of us were without Skype. Your customer service and genuine concern for the quality of service we get harkens back to an era when people actually expected to be treated with respect.
The more I use Skype, the better I like it. I’m now considering having incoming calls as well as outgoing (land line) calls.
Happy New Year to all of you.
thank you, Lars, for the profound explanation. i run an online-based project and i’m constantly amazed at how you guys have been able to pull off such a great service as Skype. we cannot imagine our lives without Skype anymore! i wouldn’t be able to work between two continents without it..
one day with interruption is not a problem – and i personally didn’t experience any issues on that day.
happy New Year to you and all the team behind Skype, Liza
I think your problem is Quality Management or in other words plain QA.
Nor overload and not version distribution.
Dani.Tal@cio-1.com call me for feedback.
Thanks for the explanation, and I appreciate the offer for some skype credit, however, several users, including myself have received the sorry for your trouble email with a blank code. I dont know if gmail stripped the code out, but nevertheless, it looks bad. Oh, and way to post a video about the new iphone video calling feature that you cant watch on an iPhone.
I find it remarkable that your system is so large and successful that it’s not possible to fully test and simulate it before release. We never covered that one in software engineering class. The life of the pioneer is always full of surprises.
If you wouldn’t force updating to Skype 5 without even asking the user, the percentage of clients running the buggy version would probably have been low enough to not cause any harm.
Instead you’re annoying all the people who like Skype 4 way better than Skype 5 in a rather rude way. (And as I don’t use group video conferences, I won’t update, because I would lose some essential features.)
If you should decide to completely block the Skype 4 clients one day, I will completely drop Skype, if the new version can’t fulfill my personal needs, which only Skype 4 can do so far.
Also, your Email is pretty much indistinguishable from a Phishing attempt. It contains:
- External graphics
- Several links which don’t point to the URL their link text is suggesting, but something cryptic instead, which seems to be used for email click tracking.
- Instructions to enter the Skype account name and password immediately following one of those seemingly fake links.
Anybody can have problems and have an outage. Its ok. The thing is that I received a mail explaining all this, and also saying that I had a credit to talk around half an hour. That I only had to put my user name, passw and the credit code which I never received. Im really surprised….
Great explanation, but you forgot to make one recommendastion for users; always have a backup plan. It’s surprising that many of the complaints posted on the web are from users that relied soley on Skype. The frameworks of modern technology (Internet, Distributed Computing, Open or Semi-open source software) are great but they all have the potential for failure, even Skype. It’s a fact of life, organizations need to plan backups for everything they do. Bravo that you stepped up and acknowledged your mistake and are planning around it. However, you can’t control the enitre web; outages will occur, users should plan accordingly.
Thanks for the candid explanation. I appreciate the technical detail because it helps restore my confidence in Skype. I recognize that no software is perfect or bug free. As long as the crashes are extremely rare, I can accept them.
While it is good that you are reviewing your vulnerabilities, I would encourage you to also put emphasis on improving version 5. Group video calling is a valuable feature to add, but it has come at price. I used to be able to run 720×480 video from my DV camcorder connected my PC firewire input and/or S-Video capture card. But not anymore. These inputs no longer work with Skype! I now have to use a third party product to capture video, and then it will only do 320×240. I had abandoned alternatives to Skype because Skype was the best. However, if version 5 is not tweaked soon to be as good as version 4 was, I may find myself using one of the others again.
I will say that your audio quality still remains heads and shoulders above the competition, and is worth the $3/mo. cost over your ‘free’ competitor (which sounds like a cellphone call).
Give us back those full installations and it’ll be easier to be up to date. It is kind of boring to steal a real setup file from an online installation every time or to download it on every machine.
Also, you repel people from updating by adding features/making changes people don’t want. I caught myself few times unwilling to update it even knowing how crucial it is to have up to date software. Just because previous version was faster logging in and had less animated stuff.
At some level of development a program like this would have separated it’s transport layer from its GUI. But this will never happen for Skype because of it is closed.
Kley Moment,
Ukraine.
Thank You very much Skype and Mr. Bates CEO of Skype for the very detailed explaination and also for extending my online number and giving the credit as you did.
The way and the timely manor in which you and your company handled the problem was “Top Notch Professionalism” at it’s best !!! P2P, Nodes, Super/Mega Nodes or whatever, I’m not certain that any cell phone provider or landline company would come close to taking care of it’s customers as you have done, especially with the amount of people who use your service and the timely way all of this was handled.
Thank You Again!!!
Best Wishes for 2011
david
I would also like to thank all of the Skype team for explaining what happened. I was not affected personally by it. It take a great company to see a problem and explain why it happened. Not like my ISP who every-time something is going wrong they say that it was a fiber cut. I swear they have cut the same fiber line 100 times by now. I would think they would know where it is.
Thanks for your honesty,
Fielious
THANK YOU for the transparency.
The plan going forward looks good.
Other mission critical software providers should take note of your prudent measures and emulate them too.
Thank you for your explanation (most of which I will admit I didn’t understand) and thank you for the extended subscription gift. But most of all, thank you for the WONDERFUL service you have provided me since 2005. Skype is absolutely priceless to me, I cannot believe how little I pay to stay in contact with my friends and family worldwide. Considering the vast number of customers you have (and how much FREE service you provide) it did not surprise me that the system was down for a day — although I assumed that it just got overloaded for the holidays. Anyway, you guys are great. Keep up the good work! And don’t fret over this little failure.
You clearly have an architecture problem. Better client testing is a whitewash (but maybe it is all you can do at the moment?) If you had a resilient architecture, you could easily tolerate lower software quality. Fix it now, or suffer again. Mr Bates is familiar with routing protocols, and will be able to help you in this regard.
I wish all companies took after Skype. Now THIS is what I call customer service and it’s exactly why I am a loyal customer.
Hi.
Happy New Year.
Thanks for the report. It is good that Skype took the right approach and provided customer with a detailed honest explanation.
Keep up the good work and hope such situation or similar will not happen again.
Thanks.
Matthew
Nearly every s/w co. has to deal with updates, bugs, different client versions etc. It is a massive task to cater for all these factors not to mention the millions of users. I can imagine the pressure upon the engineers to resolve a complex issue as fast as possible.
Due to the nature of P2P technology, it’s not so easy to pinpoint the cause, so all you ‘users’ out there, don’t be quick to point the finger!
Of course, not every person will be pleased by an outage but I believe that Skype has performed well both in it’s technical rectification, news updates, this final debrief and compensation.
Keep up the good work. Your honesty and excellence are the signs of what makes a business successful.
Like ‘santeld’ and ‘basiclife’, I have worked in development and technical support for both private and govt sectors. It is rare to see such technical transparency and honesty, especially from the CEO.
‘melaperson’: Please read the fineprint. Skype says ’30 minutes to a landline in some of our most popular countries’.
‘ultra-fine’: You are on the right track. P2P relies on decentralized servers, however, Skype will have their own servers for routing the calls and other . In a nutshell, the problem exists with the nature of a ‘super-node’, which is a universal directory of what users are online. If the ‘phonebook’ isn’t working, then you can’t call anyone. Research ‘supernode’ and P2P at Wikipedia. Drawing a chart etc of Skype technology could lead to divulging confidential information.
Thanks for this detailed explanation… This is what i expect from a reputable company, and strengthens my confidence in you.
Hi,
After couple of days of frustration of not being able to use skype i was glad to see the email and receive that small token of appreciation….only that the coupon code is NOT included in my email. I have been trying to find out where to send an email to notify about this issue and after browsing the website i have not found an email address or anyother point of contact. It would be great if i have received the coupon code.
And lastly, it is true that I am currently in Poland and have tried using skype from Poland, but my primary language is English and not sure why did i receive email in Polish and had to go above and beyond to be able to read it.
As much as i enjoy using skype the last few days and this customer care experience have been very frustrating.
Thank you
Thumbs up for Skype on this one!! They could as well have left us in the dark explaining nothing, referring to the “no guarantees”-clause in their licence agreement and go further with their business as usual, but they didn’t. Congratulations Lars and everybody else at Skype.
@sangdelan, you can avoid auto updating by changing “automatic download and install” to “notification” in you Skype advanced settings. I never had any attempt to auto install a new version.
Well done for putting your hands up an being honest, Skype is a very good and very useful piece of software, more people need to subscribe to give skype the income they need to have spare capacity for these instances, we have become a nation that loves everything for free, that is of course unless its the latest gadget and it appears we will pay any price to be “cool”
A good explanation – thanks. As noted by oldmrbill and others, it’s refreshing to get truth and it increases my confidence in the product. On the subject of supernodes, are these normal users with extra-large machines, and do they know they are servicing others? My normal Skype use is via a Dualphone, which I suspect has no spare capacity for such services. As other (e.g. mobile phones) use Skype will the load on such supernodes increase?
As a note to melaperson, when I’m paying around 3.5 euros per month for the service a payback of 1 euro for a 24-hour hiatus seems around 9 times more generous than necessary. I wish I got the same service from, say, the airlines or my ISP.
Some suggestions based on personal experience with the Corp of Engineers’ email system in 1991:
A) Do not have supernodes auto update, let them ask the user
If a supernodes version is too old, it stops being a supernode
C) Warn Supernode users when their current version is likely to age out
D) When a user’s software requests permission to do an update, it displays the estimated time the update is likely to take based in part on how long it took a nearby user
E) Users do not have to upgrade to the latest version, some recent versions that are considered rock solid can be chosen
The trick is to ensure the system can not do itself in – really hard^2 to model.
TTFN Art “Boots” Coleman
I wasn’t hugely inconvenienced by this glitch, and have for the most part been pleased with Sykpe’s services. But I find their response to this problem preposterous. As with others above, I received an email of “apology” from Skype a few days ago, with a offer of a voucher redeemable for “30 minutes” of landline calls. The amount? $1.00. I would have had more respect, and been more understanding, had the company simply sent an apology and left it at that. Better nothing that this piddly, insulting sum, which – as with users above – won’t cover more than a few minutes of the calls I usually use Skype to make. There are a number of ways in which Skype could have sought to achieve redress for the inconvenience caused to their customers – this was the poorest. As soon as Google Chat offers VoIP, I’m done with this company.
Thank you for the thoughtful gesture. The way Skype handled this provides a benchmark for other services companies to aspire to.
Mark.
Excellent Mr Rabbe and Mr Bates. And thanks for the voucher, very much appreciated. Excellent service. Ben Smith-Haddon
Well articulated explanation of the outage…
Despite best efforts, outages happen to every company regardless of it’s size. Honesty and transparency are the best tools to keep a happy and loyal customer base – check! You nailed it.
Happy New Year and may 2011 be outage free!
All the best,
Matt
Great job at getting everyone back online, can’t always expect things to be perfect. Although I do have to question…if you offer a “gratitude with a credit voucher worth a call of more than 30 minutes to a landline in some of our most popular countries, such as USA, UK, Germany, China, Japan” why did I see a charge on my account for $1 for redeeming this. You mentioned it as a CREDIT VOUCHER, right? A little confused here.
Curious, is there any mechanism in Skype’s protocol whereby clients talking to each other know the other client’s OS and version number (many protocols seem to have a stage where the software identifies itself to the other party)? In addition to checking for updates every x days, the software could also possibly know it needs to check for updates if it sees that you are actively having a call with someone else using a higher version of Skype on the same OS, for example.
As others, I also appreciate how Skype handled the incident. As a result of this handling, in my view Skype’s image actually improves (provided it does not happen again very soon of course).
That said, as others, I also think that forcing all users to upgrade to the latest version is indeed one of the root causes of the problem. What if the failure had been on version 5.0.0.156 after it hit the majority of the users?
Good job Lars in explaining this. Skype – a new telco – has already differentiated itself from old-telco with technology and business-model… now you’re doing it in PR & communications. Congrats.
As a business, we understand that things sometimes break. That being said, we depend on this service heavily for communications. The voucher you sent out is useless to us as there seems to be no way to apply it to a business account. If you are going to send something out, make sure its useful or it just makes your customers unhappy.
Only 18 comments out of the 18 million or so users? Must not have been that big of a deal. Slightly inconvenient for me, but made me appreciate you that much more really. Thank you for your excellent services and for your good communications afterward. And thank you for the gift of free minutes, which you certainly didn’t have to do, and which most of us do appreciate, (except for ingrates like markskessler.) Loved reading that you’re facilitating communications for Iranians like MRK. Keep up the great work.
Lars, you deserve praise for your candor in explaining what happened. It is appreciated.
I wish that Skype’s non-existant customer service, and Skype’s non-existant customer communication, could be just 1% as effective as this single blog post.
In general, on any other day, Skype seems to not care about its users AT ALL.
Not only do the users keep Skype afloat financially, but Skype USERS ARE the so-called “supernodes”, providing the infrastructure and bandwidth that makes Skype run, LITERALLY.
Very much like Microsoft Windows, every newer version of Skype software has been FAR WORSE than the previous version. Every Skype user knows that.
There hasn’t been a good solid stable reliable version of Skype since version 3.8 (and yes, everyone knows, you can Google skype 3.8 and still get it — which I highly recommend to everyone). The Linux version works much better than any of the Windows 4.x versions. And the 5.x versions don’t run AT ALL! I don’t understand how you can even offer the 5.x versions to the public with a straight face.
Anyway, Skype is 99% a peer-to-peer network — just like Bittorrent — and we’re all waiting for the day when FOSS P2P software catches up to Skype in audio quality.
When that day comes, PROPRIETARY Skype will be history.