The Internet After Dark (Part 1)
by Craig LabovitzAfter dark when the dinner dishes are put away and the kids are safely tucked into bed, the Internet subtly changes. Starting in the twilight of early evening, business traffic slows to a crawl, previously dormant applications flicker on home computer screens, and like clockwork, Internet activity begins its nightly climb towards a regular after hours bandwidth peak.
But before we get too carried away with metaphor and innuendo, some background.
In our last post blog post, we found (somewhat unexpectedly) that the pattern of North American daily Internet traffic differs from Europe and Asia. Unlike European Internet traffic which peaks around 7pm GMT and then quickly drops off until morning business hours, US Internet traffic reaches its peak at 11pm EDT and then stays relatively high until 3am in the morning (i.e. stays above 60% of peak or more).
This uniquely American traffic pattern holds true across dozens of individual ISPs, tens of millions of subscribers, and petabytes of daily Internet traffic.
The question is what are Americans doing at night?
To begin answering this question, we first recap Internet Observatory data from our earlier post. The below graph shows the daily average traffic fluctuations of 40 North American consumer / regional providers (taking the average of 10 weekdays in July). To make the graph more readable, we show traffic as a percentage of peak traffic levels. All times are EDT.
The way to interpret the graph above is that at 6am EDT North American traffic volumes are at 50% of their daily peaks. Traffic then climbs to a local maxima at 4pm and then a daily peak around 11pm EDT before again dropping during the early morning hours.
To understand the two North American traffic peaks at 4pm and 11pm graphed above, it helps to look only at consumer Internet traffic (i.e. as opposed to enterprise and tier1 transit). Below we overlay in yellow the average daily traffic from only US and Canadian consumer providers (i.e. only showing cable / DSL and excluding tier2, research, content, tier1, etc.).
From the graph its pretty clear consumer traffic plays a large role in the midnight North American traffic peak. We also see consumer traffic tends to climb later in the day (i.e. consumer traffic crosses 50% threshold after 9am as opposed to the broader Internet average of 6am) and consumer traffic trends towards filling the bulk of after-hours traffic. Perhaps most telling is the change in slope of graphed average consumer traffic around 6pm and then again around 8-9pm — all likely related to Americans turning to the Internet after dinner and during evening leisure hours.
But though we now know it is consumers driving the late night Internet traffic peak, we still have not answered what they are doing.
In response to our last post on the somewhat mysterious differences between American and European traffic patterns, readers offered a range of theories including:
- More so than Europe, American traffic grows with web surfing and at night
- Larges surges of P2P would explain North American traffic spikes
- Americans watch more video and related adult entertainment late at night
- In general, Europeans use the Internet less at night, have better social interactions, eat better food and generally live better lives
We finish this blog post by exploring which of the above theories do not account for the large midnight spike in North American traffic. I have no way to evaluate the last bullet point around higher European quality of life (although I’m pretty sure my high school French teacher is still insisting this is true).
Web
During both the day and night the single largest Internet application is the web (52% of all Internet traffic on average).
But while web surfing plays a large role in North American traffic trends, the graph below shows web does not provide the complete explanation behind the American bandwidth peaks.
The “Daily Web Traffic” graph shows web as an average daily percentage of all North American Internet traffic. For purposes of this blog post, we define “web” as traffic on port 80, 8080 and 443. We note that web traffic includes both html page downloads as well as video and other applications running over HTTP.
From a daily low of 42%, web traffic grows by 10% at night to account for 52% of all Internet traffic. So web accounts for slightly over half of the late night traffic, but what is consuming the other half of American traffic?
P2P
Given all the press and provider angst over P2P traffic, many commenters suggested (incorrectly) that P2P is the source of the post midnight bulge in American Internet traffic. As a category, P2P is the second largest source of American Internet traffic coming in at roughly 15-20% of all North American traffic.
[Note: We'll devote an article on the evolution of P2P traffic in an upcoming post. And, of course, the upcoming "2009 Internet Observatory" report goes into far more detail on the statistics and methodology than our more casual blog postings.]
Since most P2P does not use standard ports and/or includes encryption, we extrapolate the data below using a combination of Observatory port data with statistics from application payload characterization across several large US and Canadian cable operators. We again graph P2P as an average daily percentage of all North American Internet traffic.
Unlike the web and almost all other applications, the daily average P2P cycle does not coincide with broader traffic trends. In fact, the P2P daily trend is pretty much completely inverted from daily traffic. In other words, P2P reaches it low at 4pm when web and overall Internet traffic approaches its peak. P2P traffic only bursts from a low of 8% to a high of 17% of Internet traffic after midnight and then drops off at 6am.
As a side note, the cyclical inverted traffic pattern of P2P is interesting in its own accord. The inversion is highly suggestive of either persistent congestion or, more likely, evidence of widespread provider manipulation of P2P traffic rates.
So P2P also does not explain the midnight spike of American Internet traffic. What does? Next week we’ll complete this blog post in Part II and explore the applications behind North American traffic after dark.
September 2, 2009 Update:
Given some of the comments / questions around P2P as a percentage of Internet traffic versus a percentage of peak, a new graph below:
Looking at P2P as a percentage of P2P peak traffic shows even more clearly the inverted pattern (i.e. since Internet traffic begins to climb at 6am, the earlier graph previously obscured the even more pronounced 6am peak in relative P2P traffic levels).
Editor’s Note: This blog is the second in a series of weekly (or possibly semimonthly) posts leading up to the publication of the joint University of Michigan, Merit Network and Arbor Networks “2009 Internet Observatory Report”. The full technical reports goes into detail on the evolving Internet topology, commercial ecosystem and traffic patterns — available this October. Next week: “The Internet After Dark (Part II)”
For me, 3AM in the eastern US is “chat with friends in Australia on IRC” time.