CIO update: Post-mortem on the Skype outage
As a follow-up to last week’s outage, here is a detailed explanation of what transpired, the root cause, and plans to mitigate this from happening again in the future. For starters, it helps to understand that Skype is based on a peer-to-peer (P2P) network, which is explained here. Last week, the P2P network became unstable and suffered a critical failure. The failure lasted approximately 24 hours from December 22, 0800 PST/1600 GMT to December 23, 0800 PST/1600 GMT.
What was the cause for the failure?
On Wednesday, December 22, a cluster of support servers responsible for offline instant messaging became overloaded. As a result of this overload, some Skype clients received delayed responses from the overloaded servers. In a version of the Skype for Windows client (version 5.0.0152), the delayed responses from the overloaded servers were not properly processed, causing Windows clients running the affected version to crash.
Users running either the latest Skype for Windows (version 5.0.0.156), older versions of Skype for Windows (4.0 versions), Skype for Mac, Skype for iPhone, Skype on your TV, and Skype Connect or Skype Manager for enterprises were not affected by this initial problem.
However, around 50% of all Skype users globally were running the 5.0.0.152 version of Skype for Windows, and the crashes caused approximately 40% of those clients to fail. These clients included 25–30% of the publicly available supernodes, also failed as a result of this problem.
If approximately 20% of total Skype clients failed, why was there a much bigger disruption to Skype functionality?
Although Skype staff responded quickly to disable the overloaded servers and to eliminate client requests to them, a significant number of supernodes had already failed. A supernode is important to the P2P network because it takes on additional responsibilities compared to regular nodes, acting like a directory, supporting other Skype clients, helping to establish connections between them and creating local clusters typically of several hundred peer nodes per each supernode.
Once a supernode has failed, even when restarted, it takes some time to become available as a resource to the P2P network again. As a result, the P2P network was left with 25–30% fewer supernodes than normal. This caused a disproportionate load on the remaining available supernodes.
Why weren’t the other supernodes available to help?
The failure of 25–30% of supernodes in the P2P network resulted in an increased load on the remaining supernodes. While we expect this kind of increase in the instance of a failure, a significant proportion of users were also restarting crashed Windows clients at this time. This massively increased the load as they reconnected to the peer-to-peer cloud. The initial crashes happened just before our usual daily peak-hour (1000 PST/1800 GMT), and very shortly after the initial crash, which resulted in traffic to the supernodes that was about 100 times what would normally be expected at that time of day.
Supernodes have a built in mechanism to protect themselves and to avoid adverse impact on the systems hosting them when operational parameters do not fall into expected ranges. We believe that increased load in supernode traffic led to some of these parameters exceeding normal limits, and as a result, more supernodes started to shut down. This further increased the load on remaining supernodes and caused a positive feedback loop, which led to the near complete failures that occurred a few hours after the triggering event.
Regrettably, as a result of the confluence of events – server overload, a bug in Skype for Windows clients (version 5.0.0.152), and the decline in available supernodes – Skype’s functionality became unavailable to many of our users for approximately 24 hours.
How did Skype help support supernode recovery?
In order to restore Skype functionality, the Skype engineering and operations team introduced hundreds of instances of the Skype software into the P2P network to act as dedicated supernodes, which we nick-named “mega-supernodes,” to provide enough temporary supernode capacity to accelerate the recovery of the peer-to-peer cloud.
By late Wednesday night (PST) it was evident that only a proportion (about 15-20%) of Skype users connections were ‘healing’ and the volume of load on the supernodes continued to be unusually high. In response, our team introduced several thousand more mega-supernodes through the night. During Wednesday night, full recovery of the P2P network was underway and the majority of users were able to connect to the P2P network normally by early morning (California-PST) on December 23rd.
As we reported during the incident, in order to recover the core Skype functionality as quickly as possible, we utilized resources normally used to support Group Video Calling, to deploy supernodes, and over the course of Thursday night and Friday morning we returned these to their normal use and restored Group Video Calling functionality in time for Christmas.
The supernodes stabilized overnight on Thursday and by Friday, several tens of thousands of supernodes were supporting the P2P network. During Friday, we withdrew a significant proportion of the mega-supernodes from service, leaving some in operation to ensure stability of the P2P network over Christmas and New Year.
What is Skype doing to prevent this from happening again?
We understand how important the reliability, security and quality of our software is to Skype users around the world, and we work hard to maintain high standards, as well as develop new features and products.
First, we will continue to examine our software for potential issues, and provide ‘hotfixes’ where appropriate, for download or automatic delivery to our users. Since a bug was identified in Skype for Windows (version 5.0.0.152), we had provided a fix to v5.0 of our Windows software prior to the incident, and we will provide further updates for download this week. We will also be reviewing our processes for providing ‘automatic’ updates to our users so that we can help keep everyone on the latest Skype software. We believe these measures will reduce the possibility of this type of failure occurring again.
Second, we are learning the lessons we can from this incident and reviewing our processes and procedures, looking in particular for ways in which we can detect problems more quickly to potentially avoid such outages altogether, and ways to recover the system more rapidly after a failure.
Third, while our Windows v5 software release was subject to extensive internal testing and months of Beta testing with hundreds of thousands of users, we will be reviewing our testing processes to determine better ways of detecting and avoiding bugs which could affect the system.
Finally, as we continue to grow, we will keep under constant review the capacity of our core systems that support the Skype user base, and continue to invest in both capacity and resilience of these systems. An investment program we initiated a year ago has significantly increased our capacity already and more investment is planned for 2011 both to support the ongoing roll out of our paid and enterprise products, and to continue to support the growth of our core Skype software that we know millions of users rely on every day.
We are truly grateful to all of our users and humbled by your continued support. We know how much you rely on Skype, and we know that we fell short in both fulfilling your expectations and communicating with you during this incident. Lessons will be learned and we will use this as an opportunity to identify and introduce areas of improvement to our software, further assess and invest in capacity and stability, and develop better processes for outage recovery and communications to our user base. Thank you to everyone.
So all of this could have been prevented if Skype’s auto update actually worked. Just a quick note here: every update I’ve ever gotten for my Skype client was through a manual upgrade check.
First of all I would like to congratulate with you Mr. Rabbe, because of your detailed explanation.
Skype is a very great service and every day I rely on it, for both work and personal use.
Every person can do an error, but there aren’t many people that are willing to admit theirs guilties.
I wasn’t impacted too much by the outage. However, Leo at twit.tv was for a couple of his shows. It proved how much Skype is needed as their other options were limited and not that great. Everything they do with Skype is fantastic and when they had to switch to other methods, it just wasn’t the same. The quality was way down.
On one hand, the outage was bad for a lot of people who couldn’t use it.. On the other hand, this will help Skype identify any weak points and patch those up.
Thanks for your explanation of all troubles since the dsipruption as well the way you tackled the disaease.
Fully understandable for me, because although I’m retired (almost 65) and formerly worked in an hospital (Hospital Information Systems) I “enjoyed” your explantion how jou tackled this major disease.
As the french say “CHAPEAU”.
Personally my skype activities never were down in my location.
With deep respect I wish you lots of succes in the future and a happy new year.
king regards
Gerard Roelands
groelands@zonnet.nl
Lars,
Having worked for the likes of AT&T, it is fantastic that you’ve taken time to explain the issue as you have, along with the plan to insure this will not happen again. It’s not often candor such as yours is expressed by a CIO. It demostrates a maturity and confidence in your team and yourself. Please keep up the great work!
BTW, I’ve just signed up for your service for this very reason.
I appreciate how you have handled this incident. You’ve shown a Skypish kind of customer care we all appreciate in your actions and communications.
Hi
Good to hear that everything go to normal again! I’m not testing new version (156) yet but I do not like any of v5 of skype! (Due to some old features leak) I prefer to use v4.2.0.187 and any time I’m reinstalling v5 to test it on different ISPs at Iran, I had to get back to old v4 due to faster connecting to server and more stability.
Another thing, I hope you guys can find a way for us at Iran to bypass some filtering on skype client so we can connect to skype without using any VPN. most ISPs block skype at here!!!!
I love Skype more than others messengers for 3 thing: 1) easy to use for my family 2) really unique and great sound Quality 3) Desktop Sharing which help me to show my families to do somethings at their systems via this great feature.
Love you and I hope you have great holidays beside your families and more greater and successful business days at new year!
Best Regards
MRK
The automatic software update contributed, in part, to the Skype
outage last week.
When I used Skype last week, the program was automatically upgraded
to the “buggy” version while I sat and was forced to wait.
If I could have chosen to perform the upgrade on my own time
schedule, I would have done the upgrade in the future so I did
not have to sit and wait for the update
when I wanted to make a phone call; thus, my computer would have been
available as a functioning super-node (running the bug-free older code)
during last week’s outage.
Kudos to CIO Lars Rabbe for a straight-forward explanation of the outage. No smoke and mirrors, just the facts. How refreshing!
Thank you Skype customer services for acknowledging the problem, taking responsibility for it (although technology failure is impossible to avoid completely), and giving customers a credit voucher. It shows you remember who keeps you in business; a lesson some big companies have yet to learn!
And after that lengthy explanation which probably 99.99% of people will not understand (super-nodes, p2p clouds, mega-nodes – really???) you are giving your long time, loyal customers a whole $1 credit to compensate us for the outage. Wow – that’s incredible!!!! Merry Chrsitmas indeed. Reminds me of the Mortimer brothers from Trading Places – one whole dollar.
Thanks for the information – very informative and with hindsight, an obvious problem (isn’t that always the way). Congratulations to your engineers for identifying and solving this problem as quickly as they did. I work as a developer and understand how much of a pain this must’ve been.
I appreciate that Skype is willing to acknowledge and explain its’ problems rather than try to hide them – You’ve won some good will from me for owning up and explaining.
What if someone doesn’t want to update the release of Skype? I am perfectly happy with my old one and I don’t see any reason why I should change it. The few times that I tried, I always went back to the old one, as the new releases I did not like and gave trobles to my PC.
Last week I saw that often there were attempt to “auto-instal” a new version of the programme. I was quick enough to avoid it…. Will I be able to avoid it also in the future?
I hope so.
Regards.
Having read the entire explanation I still cannot get how the Skype network infrastructure works. Are supernodes – just usual PCs running Skype? Does that mean that you offload some of your core networking services onto users?
Can you draw a little chart/scheme how everything works?
After i read that message i was very pleased. Hope all goes well with the plans for further stability.
Skype is super Awesome! good to know, thanks for sharing this.
I don’t understand why your email to me suggests that I will get 30 minutes worldwide but when I put in the code that you sent me, it says “Your voucher or prepaid card was succesfully redeemed, and we are now activating the following services:
$1.00 Skype Telephony Voucher for $1.00″ Which is true? I call Ecuador. $1.00 won’t cover ten minutes.
Dear Mr. Rabbe,
We are proud to use Skype and I am using Skype for more than 1.5 years for my software business. And this is the first time I got problem with Skype. But the one who works hard has to give exams. So it was your exam time, and you passed it with flying colors. Dont be sorry or think that you fell short to users’ expectations. You are the best.
Thanks for your services and great support.
Regards,
Nirav Vora
Well done! What a refreshing approach to service. that’s why your company is highly respected and will continue to be so. Without skype my business would still be paying rip off teleco bills. Thanks to skype I have emplyees on every continent and daily conference calls and for sure my conventional voip (non peer to peer) is always out performed by Skype.
Hey! Just a couple of things…
(1) Where my dollar be at? I read a few people complaining they got a dollar for their troubles and my Skype was as inaccessible as everybody elses in that period so, ahem, cough up
(2) I see the updates are still coming thick and fast for Skype (Beta) 2.1.0.81 for Linux huh?
I’m using sarcasm for #2 there, I really do want the dollar in #1
Perhaps it’s time for Skype to host some supernodes, for paying customers? Piggybacking on other clients might be “scale for free” but it’s unreliable (as we’ve seen) and I wonder what an evil person could do if they pretend to be a supernode, or subvert a supernode?
We can all run into unexpected problems from time to time. Understandable. HOWEVER, do not say you are sorry, if you do not mean it. Sending a $1 (yes, I do mean ONE pathetic dollar) is adding insult to injury as for those that are regular Skype users paid over the odds to use alternatives during the outage. I only found out about the outage after hours trying to connect. Perhaps an email from Skype as soon as they found out would have gone a long way to reduce anxieties. Skype chose not to inform users until problem was under control (too late, we already knew it was sorted!). Perhaps apart from reviewing their testing procedures (which proved to be insufficient) they should MAINLY review their emergency communications strategy (if they have one!).
From what I understood is that they were to credit everyone 30 minutes. So what turned out to be giving people 30 minutes of free calling to landlines anywhere in the world was actually just a $1 credit!! WTF was that all about?? Why not just give the people 30 minutes credit of international calling then to where they have 30 minutes of credit towards all their international calls instead of this stupid $1 credit thing. If one was calling the middle-east that $1 would be gone in a few minutes time. Bottom line is Skype pulled a fast one by lying about the 30 minutes credit for international calls.
The voucher can be used to give you approximately 30 minutes of free calling to landlines anywhere in the world.
Lars, thanks for the detailed explanation; it is clear that the trigger, a buggy client, should be shot.
But your claim that the software is robustly tested does not wash. The 152 version also crashes on log-in for user names containing more than two consecutive periods (“…”). This is the kind of bug that should be spotted in routine test scripts, checking the user name base or at least a subset of it. I’ve had to go back to 4.2.0.187 AND turn off automatic updates, because a subsequent version I unwillingly installed did not fix this bug.
I’m afraid this is pathetic Lars, and no amount of explanation can correct shoddy processes. Hire someone who knows about testing and sort this out. In the meantime, I’m sticking to 4.2.
Let’s just say I wanna help out and dedicate resources to running a supernode, which will be online and on high bandwidth. How can I help?
Hi, thanks for the update. It was horrible because I had a conf call planned at the time.
However Skype remains one of my favorite apps.
Also one remark, I have so many people in my contact list and I don’t need to see their photos, would be better is there would be an option to change the views.
thanks, marko
I was impacted by this outage, but have not seen any offer from Skype to compensate. I do pay Skype monthly. Let’s see —
Thank you Mr Rabbe for explaining. Really appreciate the honesty, and good luck with China!
I’m not going to comment on the problem itself — I hardly noticed it — but it’s nice to see a fairly detailed description of what happened. And what’s even nicer is how polite and encouraging the user comments are.
Maybe the Christmas spirit hasn’t worn off yet.
yes, it is kind of ridiculous to offer 1 dollar as compensation for lost services?? I appreciate excuse, but it cost us all money and we relied on Skype, so this compensation is rather insutling
Yes, I agree. It is sort of incredible to see such a detailed, thorough, and honest explanation of a company’s technical issues. Thank you for the forthrightness.
Skype 4.x does an auto upgrade whether I like it or not. Skype 5.x audio is so choppy (both receive and sending) I cannot use it (Vista 64-bit), I have tried both regular and business. So I downgrade to version 4 which works great, but auto upgrade kills me every time. What to do?
I am extremely impressed by this post. While the issues were unfortunate, I think Skype’s response to them has been very constructive and transparent.
In the future will you more rapidly be able to roll out mega-supernodes? Have you looked at deploying instances on to a cloud service?
Have you considered a beta branch of Skype for the more technical crowd (e.g. Firefox nightly builds)? While you do offer betas it doesn’t update to the latest test version (and you also introduce bugs between “beta” and “live”).
I completely agree with the other posters that while the down time was unfortunate, I think your efforts to fix it and your detailed explination were a very refreshing view of what is going on and what happened. Sure the outage was unfortunate and caused issues for people. I know I had difficulty moving information around during it, but without Skype I wouldn’t be able to do what I wanted anyway and nobody else comes close to the transparency at the same level of service as Skype does. Thank you very much for your efforts.
AJ
I echo previous comments of appreciation regarding how you handled this incident, your detailed explanation, and the vouchers you sent to your customers. Your attitude and customer sense is dead on. I will tell my friends all about how pleased I am with Skype. Happy New Year!
There is something I do not understand about the super-node and client interdependence.
Can a user’s client be a super-node? Can my PC e a super-node without my knowledge?
If yes, then what does it mean in degradation of performance on my PC?
Is this the cause of the irritating delay of about half a minute between calling a partner and the ringing tone?
I’m curious to know if your development team reads any of the power systems research? They have been dealing with cascading failures for quite some time, and there may be some research done there that is useful to Skype’s operations.
Mr. Rabbe:
This is a model example of how to respond to such a situation. By being sufficiently transparent about the details of the event, speaking plainly about what went wrong, and enumerating how you’re addressing the problem, you’ve repaired and engendered trust.
Not only have I learned that Skype has a robust service (and is improving); I’ve learned that their CIO has integrity… in copious amounts.
Well done.
Regards,
John S. Ryan
Everyone,
I just wanted to say that i was offered (like everyone else i’m sure) a credit for the outage. I just wanted to say that i will decline to take a credit. I have been very happy with the service that i have been given by Skype over the years, and proudly pay the 50 dollars or so a year for the unlimited land line calling. The service is excellent, and with anything in life there will always be issues that arise from time to time. I find it troubling that i see as many complaints as about this as i do. For what most people get for free, i gladly will continue to pay for. Excellent product.
Thanks!
Thanks for the explanation of the outage.
Isn’t this the same fundamental cause of the outage a few years ago, in which Windows Update restarted a significant number of computers simultaneously, causing the Skype super-nodes to overload in the same way as described above? In that case, the original cause wasn’t Skype clients crashing, but being shut down due to OS patching, but the rest of the explanation still fits.
In the future, you may want to have a super-node, when shutting down due to traffic, to signal the network to start up a cloud server at Amazon (or another provider) to replace itself– that way you could save the cost of running your own super-nodes all the time, while still ensuring that enough super-nodes stay running during events like this.
Guidelines used while developing the Internet Protocol stack, as conveyed by Dave Mills (a.k.a., Mr. NTP himself, creator of fuzzballs, original IP routers) to his university students:
Amendment:
* No system works correctly.
Acid test:
Would you trust your paycheck to the system?
Implementation Hint:
Use unique timeouts such that even in combination they become signatures of where problems may exist. Use a different prime number for each to aid in discovering the combination of compounded timeouts.
Thank you for the transparency. As I didn’t suffer during the outage, I won’t redeem the outage voucher. Your high standards for service, functionality, vision and performance have kept me, for one, on board as a satisfied customer. Happy 2011 to you, and to all.
I have to say. I’ve dealt with many of the other phone companies both personally and professionally. Things happen in a large network. What separates the good from the great is honesty and communication. We have a term we call “phone magic” we use whenever we report a problem to a telco and they claim nothing is wrong but the problem mysteriously goes away later. Skype has not only been honest about the situation but has even taken measures to compensate users for the problem. Something other companies usually refuse to do unless you complain.
As for the Auto Update I’m not sure what the problem is there as I’m always notified that an update is available and install updates as soon as I’m aware of them. I think the problem is configuration and Skype should come with Auto-Updates fully turned on by default as a suggestion.
I remember Lars when he was at Fidelity Investments, he is one of the straightest shooters in the technology biz. I use Skype regularly and will continue to do so with confidence knowing that honest, hard-working and dedicated technology professionals such as Lars Rabbe are behind it!
Regards,
Jason Matthews
I too am just stunned at the generous $1 Skype credit – a 30 second call is about what it will cover -
I also received what looked like a phishing message when I logged on during the outage period; I blocked and reported it, and hope that no one on Skype was harmed ….
Skype – I really appreciate you taking ownership of the problem and trying to explain the situation. But a $1 voucher …? Originally I read this and thought “This is a good company, they’ve owned up and they’re making it right with a voucher for 30 minutes.” Then I saw the voucher amount after I logged in and redeemed and … well, honestly I think it’s a pretty poor showing. I think it might have been better if you didn’t offer the $1 and just left it at this blog and apology.
Dear Lars
Read about the problems and well, this kind of thing happens, read also about some users may receive compensation, well your/the skype service in my view so just beyond fantastic. Therefore, if some form of compensation is to come my way PLEASE keep it just keep on providing a fantastic service and l am happy to put up with the odd disruption. Happy New Year in advance to all the Skype team, from top to the cleaner.
Best regards, fleetwooda
I like Skype and I like the openness about the problem. But I am afraid that it is not the whole explanation. I use the PC-PC and the Skype-Out facility. Unfortunately the later versions seem to be less stable. At the moment I use v. 5.0.0.156 and regularly encounter the infamous message “This program encountered a problem and had to close …” It seems that most of the effort goes into adding more bells and whistles which most people don’t use but decrease the stability of the program. Further I hate the loss of the possibility to “ring” the computer’s internal speaker at incoming calls.
Yes, it is good that the CIO can be so frank, it’s a shame the support staff don’t share his candour and zest for problem solving. After the recent issues I had an idea that I literately had to spell out to the support staff and ask them to pass it on to the product team. She promised that they would reply to give me some confidence that my idea was being considered…. no reply yet.
Why can’t users (businesses) set up their own private super nodes? You run Skype, perhaps in super-node only mode on your DMZ with some kind of filter to allow it to serve only your employees (perhaps by username or just a specific port). Then allow clients to add their own preferred super-nodes to their list. Thus ensuring your employees communication is not reliant on random peers whilst not having the burden of being a public super-node .
What do you think?
Thanks for the detailed explanation.
I have been in the software development business – so I am sympathetic to your frustration. More pathological testing seems to be in order for your upcoming releases.
Dear Mr. Rabbe,
This response should serve as perfect example to all service providers on how to deal with public relations and outages. Open, honest and detailed. As an IT manager I can only imagine how intensely your operations staff worked to remediate this. Few people, so very few people have any idea of the extreme complexity of the Skype IT product mix, the scope of the virtual and real infrastructure and how much of it is outside of your direct control.
Everyone has 20/20 hindsight. The value for money ratio provided by Skype is world class. If, in the years of perfect service I have experienced, I must endure one single day of outage, in order that your company can unearth, understand and kill such a corner case threat, then so be it. You are pushing forward a wave on the boundaries of technology and we surf that wave with you.
Thanks for the wake up call to just how valuable your service is and good luck in the future.
Graham Mitchell