The Great GoogleLapse

Web sites go down. Circuits fail. Network engineers goof router configs. And few of these outages ever make the nightly news…

But if you happen to be Google and your content constitutes up to 5% of all Internet traffic, people notice.  Network engineers around the world frantically email traceroutes to mailing lists. IRC channels fill with speculation (“definitely was a DDoS attack”, “no, a worm”, “it was ISP xxx’s  fault!”). And end users Twitter (a lot).

So what does it look like when 5% of the Internet disappears on an otherwise uneventful Thursday morning? The below graph shows average traffic across 10 tier1/2 ISPs in North America from Google’s network (ASN 15169). Outage began roughly at 10:15am and lasted through 12:15pm EDT.

Looking at the data, most large transit providers appear to have been impacted (e.g., Level3, AT&T, etc.). Other providers (e.g. large consumer DSL / Cable) showed no drop in traffic from/to Google.

Looking at BGP (below snapshot is from Arbor’s Routeviews Servers) we see a lot of churn in Google’s BGP routes around the outage timeframe — one prefix I choose at random flaps across half a dozen providers before getting withdrawn.

In a recent official company blog post, Google blamed some combination of airplanes and BGP for the outage.

Reblog this post [with Zemanta]

50 Responses to “The Great GoogleLapse”

May 14, 2009 at 6:29 pm, Fenixnordic Group » Blog Archive » When Google Goes Down, It Falls Hard said:

[…] via » The Great GoogleLapse · Security to the Core | Arbor Networks Security. […]

May 14, 2009 at 6:56 pm, Whither Google, so goes the Web | AERS - Advanced Ecommerce Research Systems said:

[…] even when it’s just a simple routing error, as it was this morning.  This diagram from Arbor Networks, whose servers are positioned at key NAPs and ISPs globally, portrays the […]

May 14, 2009 at 5:28 pm, Stat Shot: Google’s Circumnavigation Edition said:

[…] Networks, which provides telecommunications gear and network security information, noted on its blog that Google comprises up to 5 percent of the web’s traffic, and when that traffic disappeared […]

May 14, 2009 at 8:46 pm, When Google fails, the Internet fails » VentureBeat said:

[…] Deal. Above, you can see how much data was transferred by 10 different Internet service providers monitored by security company Arbor Networks. Below, you can see traffic to sites run by MerchantCircle, which manages the Internet presence for […]

May 14, 2009 at 9:11 pm, Matt’s Blog » Blog Archive » Google Routing Fail said:

[…] all over the place, which means that they have some pretty good insight into things. As they posted about today, Google apparently fudged a route and it propagated out, routing most of their traffic through some […]

May 14, 2009 at 11:15 pm, Searching for Google « Rich’s Random Walks said:

[…] SANS links seem to be working now.   There is also a very interesting, somewhat tongue-in-cheek blog post from Arbor Networks, showing the impact of the Google problem on Internet traffic. Possibly related […]

May 14, 2009 at 11:34 pm, Google we have a problem said:

[…] Una fotografia del calo del traffico Your web sites go down. Circuits fail. Network engineers goof router configs. And mostly you don’t worry about making the nightly news. But if you happen to be Google and your content constitutes up to 5% of all Internet traffic, people notice.  Network engineers around the world email traceroutes to each other. IRC channels fill with speculation (”definitely was a DDoS attack”, “no, a worm”, “it was ISP xxx’s  fault!”). And end users Twitter (a lot). […]

May 15, 2009 at 5:00 am, 评论:Google 打个喷嚏 整个 Internet 都感冒 « 每日IT新闻,最新IT资讯,聚合多站点消息,保证你与世ç said:

[…] 新闻来源:VentureBeat Google 的全球访问故障已经告一段落,经查是 Google 错误地将部分流量转到了亚洲地区导致阻塞,虽然尘埃落定,然而从这件事却可以发现 Google 对全球互联网的影响,Arbor Networks 的流量统计分析了故障期间10个不同 ISP 的流量数据,下面这张流量图很明显地说明了问题。 […]

May 15, 2009 at 5:41 am, ODWGOOG said:

Whether we like it or not,Google is almost synonymous with Internet. And this behavior is bad, bad for all of us. For more information check our blog, too.

May 15, 2009 at 5:57 am, Upstream Connections - SEO » What Happens When Google Stops Working? said:

[…] /blog/asert/2009/05/the-great-googlelapse/ Share this post: […]

May 15, 2009 at 9:44 am, Cuando Google se cae, se cae bien. Impresionante | Pateando Piedras said:

[…] gráfica muestra el descenso de tráfico en la red cuando los servicios de Google se caen. Esto demuestra lo enorme que es Google, le daría […]

May 15, 2009 at 10:01 am, Bill Hartzer said:

Interesting data and certainly that’s an interesting chart. As people were reporting that Google was down, luckily I was on one of those providers that didn’t see a Google outage. I didn’t have any issues at all.

May 15, 2009 at 11:31 am, Repubblica.it - Blog - Scene Digitali » Blog Archive » Google-botto. E due in pochi mesi... e la rete ne subisce le conseguenze said:

[…] la più accreditata è quella che indica in due ore circa la fase di arresto: come documentato da un centro per la sicurezza della rete.  Del resto l’azienda si è assunta la responsabilità del disservizio,  spiegando che circa […]

May 15, 2009 at 12:21 pm, Andrew P. said:

It’s the best argument against Web 2.0 that we’ve seen so far. I’ve always believed and continue to believe, that individuals and enterprises need to own the software that is critical to their activities and have it installed on local machines that can continue to function without the Internet.

May 15, 2009 at 12:25 pm, The great Google collapse | Digital Prank said:

[…] at 12:48am Australian time and the slowdown peaked around morning in US, mid-afternoon in Europe. Arbor Networks which monitors the web usage said that traffic from Google servers to major North American Internet […]

May 15, 2009 at 12:54 pm, Terry Slattery said:

It would be interesting to know how Google determined that they had goofed and how they determined that it was a routing mistake. I think a lot of us could learn from it.

May 15, 2009 at 2:02 pm, jeremy c said:

Ha! AT&T has had an outage here for the past 2 days so I never noticed google being down!

May 15, 2009 at 2:25 pm, Google Cloud - Jak Łatwo Zawisnąć W Chmurach | SEO Blog said:

[…] ale jak pokazuje wykres jednej z amerykaÅ„kich firm zajmujÄ…cych siÄ™ bezpieczeÅ„stwem w sieci (Arbor Networks), niewiele brakowaÅ‚o do zastoju w ruchu poÅ›ród najwiÄ™kszych firm przesyÅ‚owych w […]

May 15, 2009 at 2:35 pm, Charles Bronson said:

Can you please clarify the description of the traffic represented by your graph? I want to make sure I understand what it is relating.

Is the graph representing *all* of the average traffic across 10 tier 1 and 2 ISP’s or is it *only* the average of traffic originating from ASN 15169 as seen across 10 tier 1 and 2 ISP’s?

Thanks

May 15, 2009 at 4:52 pm, The Great Google Depression of 2009 Happened Thursday, May 14, at 7:48am PST | Rev2.org said:

[…] graph from Arbor Networks shows the down time.  See that big post-hole?  That’s Google falling off the Internet for a […]

May 15, 2009 at 6:54 pm, Craig Labovitz said:

The latter — average only includes traffic originating or transiting AS15169.

May 15, 2009 at 10:48 pm, Ike’s World – Weblog Edition said:

[…] By ichristo Google outage takes out 5% of the internet traffic. /blog/asert/2009/05/the-great-googlelapse/ […]

May 16, 2009 at 12:39 am, When Google Goes Down, So Does 5 Percent Of The Internet? « Multan POST said:

[…] the statistic offered by Arbor Networks’ Craig Labovitz, who in a Thursday blog post and Web traffic graph noted that between 10:15 a.m. and 12:15 p.m. Eastern Daylight Time, Thursday, […]

May 16, 2009 at 6:15 am, Finance Geek » Google’s Outage Caused 5% Internet Traffic Decline said:

[…] Networks chief scientist Craig Labovitz observed in a blog post that failures happen. “[B]ut if you happen to be Google (NSDQ: GOOG) and your content […]

May 16, 2009 at 9:04 am, 鱼儿在线 » 评论:Google 打个喷嚏 整个 Internet 都感冒 said:

[…] Google 的全球访问故障已经告一段落,经查是 Google 错误地将部分流量转到了亚洲地区导致阻塞,虽然尘埃落定,然而从这件事却可以发现 Google 对全球互联网的影响,Arbor Networks 的流量统计分析了故障期间10个不同 ISP 的流量数据,下面这张流量图很明显地说明了问题。 […]

May 16, 2009 at 9:26 am, TheTradingReport » Blog Archive » links for 2009-05-16 said:

[…] Craig Labovitz: The Great GoogleLapse […]

May 16, 2009 at 9:27 am, TheTradingReport » Blog Archive » Google’s Outage Caused 5% Internet Traffic Decline said:

[…] Networks chief scientist Craig Labovitz observed in a blog post that failures happen. “[B]ut if you happen to be Google (NSDQ: GOOG) and your content […]

May 16, 2009 at 9:27 am, 墨尔本 said:

I’m glad it comes back to normal!

May 16, 2009 at 10:40 am, La Gran Caída de Google -- Reikuaa.com | Tu blog said:

[…] Craig Labovitz Traducción: Raúl Batista Fuente: Arbor Networks « Cómo salvar un móvil que ha caído al […]

May 16, 2009 at 7:00 pm, Every Cloud has a silver lining? | New Beltane said:

[…] was quite significant, with @ 5% of traffic disappearing between 10.15 and 12.15 EDT according to Arbor Networks Security. According to their investigation, the big boys were effected significantly. If wasn’t […]

May 17, 2009 at 2:45 am, Wenn Google in Nordamerika 2 Stunden nicht verfügbar ist « Der Jupp said:

[…] from Google’s network (ASN 15169). Outage began roughly at 10:15am and lasted through 12:15pm EDT.Arbor Networks: Security to the Core, May […]

May 17, 2009 at 2:00 pm, ZitZot » Google’s outage: internet traffic went down by 5% said:

[…] the assertion of Arbor Networks’ Craig Labovitz, who in a Thursday blog post and Web traffic graph noted that when Google has an hour and a half outage this week, Internet […]

May 18, 2009 at 3:12 am, A.H.Banen said:

According to the Google Apps Status Dashboard – http://www.google.com/appsstatus# – there was no disruption of service…

May 18, 2009 at 4:13 am, Google jumitti verkon : Viimeisellä Rannalla said:

[…] voi olla valtava vaikutus koko infrastruktuurin toimintaan, sillä Googlen järjestelmien ongelmat 14.5. hidastivat verkkoliikennettä merkittävästi kautta maailman. Useat verkkopalvelut […]

May 18, 2009 at 6:32 am, Google’s Downtime Affected 5% of the Internet said:

[…] doubt about that.  And one of the most interesting observations/reactions was perhaps made by the Arbor Networks Security Blog which stated that  5% of overall Internet traffic was affected by the Google […]

May 18, 2009 at 11:49 am, Google’s Downtime Affected 5% of the Internet | Mujtaba-K.com - Daily Latest Technology News Buzz said:

[…] doubt about that.  And one of the most interesting observations/reactions was perhaps made by the Arbor Networks Security Blog which stated that  5% of overall Internet traffic was affected by the Google […]

May 18, 2009 at 11:57 pm, More on the Great Google Crash of 2009! — Brian Hancock said:

[…] The Great GoogleLapse […]

May 19, 2009 at 6:01 pm, Rational Survivability » Google Gaffe - The Cloud Needs a Snuggie…Or a Wedgie said:

[…] By now you’ve undoubtedly heard that Google had a little operational hiccup.  I particularly enjoyed Craig Labovitz’s (arbor) account of “The Great GoogleLapse” […]

May 21, 2009 at 4:55 am, Lennie said:

@Andrew P.

It’s not an argument against Web 2.0 (which is mostly just a certain kind of web-application/-page), it is however an argument against cloud-computing, utility-computing, SAAS, etc.

You could do Web 2.0 on your intranet as well.

An even better argument against cloud-computing, etc. is Microsoft Azure, which was down for 23 hours if I remember correctly.

May 21, 2009 at 5:16 pm, El día que Google falló « ESNOG said:

[…] manera directa en el funcionamiento de Internet tal y como se refleja en el análisis realizado por ArborNetworks donde se aprecia una caída de hasta 15Gbps de tráfico en el tráfico acumulado de unos 10 […]

May 22, 2009 at 10:52 am, Online Outages, Outrage and Ordeals | A Second Opinion said:

[…] But to people who rely on Google for their e-mail, their calendars and their documents, seeing them suddenly drop offline must have felt as if somebody had carved a chunk out of the Internet. […]

May 24, 2009 at 10:50 am, Kai said:

Its good also to compare to any other web properties when it occurs.
“we see a lot of churn in Google’s BGP routes around the outage timeframe”

May 24, 2009 at 11:00 pm, Joe Wilcox » Antitrust Primer: Google and Microsoft said:

[…] over Internet commerce extends much farther. Last week’s four-hour Google outage affected as much as 5 percent of Internet traffic. Google standards influence search and advertising, and much more, not just on the desktop but […]

May 25, 2009 at 7:20 am, Bruno Kerouanton » Le SSTIC, et autres sujets de recherche said:

[…] Je suis certain qu’il y a un énorme besoin de faire des travaux de recherche sur le domaine. Certes, c’est un tantinet moins technique que le piratage des signaux TNT, mais c’est intéressant car les décideurs (bref, ceux qui ont les sous…) n’en sont pas encore conscients (des soucis) et on pourrait bien aller droit à la catastrophe si par malheur tout passe par Google et que ce dernier plante, par exemple… […]

October 09, 2009 at 4:46 pm, Antitrust Primer: Google and Microsoft said:

[…] over Internet commerce extends much farther. Last week’s four-hour Google outage affected as much as 5 percent of Internet traffic. Google standards influence search and advertising, and much more, not just on the desktop but […]

January 01, 2010 at 6:32 pm, aimClear’s 2009 Daily Training Link Library » aimClear Search Marketing Blog said:

[…] The Great GoogleLapse, asert.arbornetworks.com […]

August 11, 2010 at 10:06 pm, Google Downtime,First In Over 3 years said:

[…] 8 min of downtime. And one of the most interesting observations/reactions was perhaps made by the Arbor Networks Security Blog which stated that  5% of overall Internet traffic was affected by the Google downtime, this is […]

November 13, 2010 at 9:59 pm, Antitrust Primer: Google and Microsoft « 5 minutes with Joe said:

[…] over Internet commerce extends much farther. Last week’s four-hour Google outage affected as much as 5 percent of Internet traffic. Google standards influence search and advertising, and much more, not just on the desktop but […]

September 05, 2011 at 11:27 pm, Phantom Link Cloaker said:

affiliate link cloaker…

The Great GoogleLapse | Security to the Core | Arbor Networks Security…

Comments are closed.