The trending/popular domains list is interesting [0]. Since Cloudflare effectively serves as part of the backbone for so many sites and services, I'm assuming their ranking is more able to count usage of sites that are mostly accessed by smartphone app, e.g. Tiktok #3, Facebook #4. Unlike Comscore [1], which seems to overly represent media sites.
Too bad Radar omits pornography sites, I was curious whether they'd actually be among the top sites. FWIW, SimilarWeb [2] (which has Tiktok at #46), does include porn sites (xvideos and pornhub are #8 and #9, respectively).
Though I don't quite understand how mozilla.org could rank as #10 on Radar's sites, just below Instagram and above Youtube, Whatsapp, and Twitter.
edit: It's also interesting to compare its Top Browser rankings with analytics.usa.gov [3]. It appears that Radar's measured users are significantly more on Chrome – 55% (desktop+mobile) vs 47.6% on usa.gov sites; whereas usa.gov users are significantly more on Safari (iOS+desktop): 35.2% vs 13% on Radar. I suppose this reflects the assumption that iOS has a bigger proportion of U.S. users than it does worldwide users.
Some domains are up there because they're being used by applications, for exemple no one goes directly to microsoft.com but Windows does it for many things like update, monitoring, telemetry ect ...
*.mozilla.org is probably resolved when you use Firefox.
Cloudflare can't know if the resolve was made by a user or by internal application logic.
I imagine microsoft.com received quite a bit of direct traffic last Tuesday when preorders opened for the next Xbox consoles. Also, .NET Framework, .NET Core, Entity Framework, Entity Framework Core, and Azure documentation is all hosted at microsoft.com (to name just a few).
I don't mean to discredit your main point, that a lot of domain use can be credited to application or OS logic, but to say no one goes to microsoft.com is a bit weird.
Edge and IE are still a decent chunk of desktop traffic as well. And opening them with home page will probably kick off traffic to microsoft.com as well. Not sure how much that will be, but probably factors in.
Edit: Although their metrics have the browser share lower than others. They have Edge at <4% and IE doesn't register in top 10. Whereas something like https://netmarketshare.com/ has Edge at 7% and IE 5%. So there is seemingly a bias towards other browsers in their methods. Which makes sense, seems unlikely someone uses 1.1.1.1 and IE.
Btw, one piece of feedback: the interface for selecting a certain country is rather confusing. A (non-tiny) dropdown selector at the top would be much more intuitive.
Wouldn't this include all traffic? As long as it's enabled Windows and Firefox are constantly sending statistics in the background. If this is measured by # of requests, then this would make sense.
Does this mean that Firefox is sending statistics in far greater frequency than Windows? Either these stats are wrong or Firefox is sending telemetry far too frequently to rival these sites (as Firefox usage is simply not high enough otherwise).
The popular domains metric is probably measured through the 1.1.1.1 dns service and firefox recently defaulted to resolving domains through cloudflares DoH service, so popular domains may be biased toward firefox users. Browser popularity is probably less biased as it would have to be measure by HTTP requests to sites using cloudflare.
Microsoft being high up wouldn't surprise me if it included Microsoft 365 and Outlook traffic. But I just can't imagine that surpassing Google search+Drive+GMail+Maps
Around ten years ago I built the analytics system for an Alexa 100 company. The company had a policy of supporting IE6 until its share of traffic was below 1% and it was sitting at around 4% at the time IIRC. This was the figure that was reported industry-wide and this was also what we were seeing internally in our stats. However, partway through the project we decided to switch from an image pixel to javascript tracking. At that point we saw the share of IE6 traffic drop immediately to below 1% and it dawned on me what was happening: A lot of bot creators had been using an IE6 user agent to disguise their activity. This was in the days before headless browsing so the vast majority of them weren't downloading / processing javascript. I thought 4% was high then for bot activity but it wouldn't surprise me if the share has exploded since then.
Oh, I don't doubt that bot activity is rampant. The question is what Cloudflare's false-positive and false-negative rates are; I know from personal experience that they sometimes are convinced that humans are bots, and I assume sometimes bots get by them and therefore get counted as humans. But between those, my personal conclusion is that CF's claims about bot activity should be read with error bars.
But keep in mind CF has a lot of options for filtering traffic. Just because you got a CAPTCHA on a CF site doesn't necessarily mean considered you a bot.
Were these crawlers for big well-known things like google/bing/archive.org? I'm wondering if there are other types of crawlers that I don't even know about.
A site for one of my clients was, at one point, getting absolutely hammered by a bot that crawls for plagiarised content. It was ignoring our robots.txt so I ended up blocking it via CloudFlare. There are bots for pretty much anything you can think of out there.
edit: replaced "scrapes" with "crawls" for accuracy.
The Earth has a (mostly) free society. Therefore, many programmers find a web crawler to be an advantageous endeavor. From search engines, to social media companies, to ad networks, to government agencies, to financial institutions, to students learning about programming, to nefarious actors, and even shadow versions of everything above. Bots are neat. One can do a lot with a freely accessible public database of information, only limit is one's creativity.
On the consumer side. There is a hard limit on bandwidth, though, on the provider side that can be exhausted by an overwhelming amount of bot traffic. In effect, that's what a DDoS is, even if unintentional.
That's actually lower that I would have guessed. And lower than what I see on my servers. It's more than 1/2 for sure, and it's hard to guess how much of that other (less than half) part is real people, and not just bots pretending to be people. My 1/2 educated guess is about 40% of traffic to the sites I run is real people on any given day. These are all smallish websites for non-profits, so maybe not an "average" website, whatever that might mean.
Hadn't noticed that but that's a good point. "Way higher": I would have guessed the same as you but, upon further thought, I guess I'm not surprised: creating a crawler is 100 lines of Python so I would guess that there are actually tens of thousands of bots running around grabbing web pages. Humans are an ever diminishing portion of the "web", especially given the (old) focus on "semantic web" and machine-friendly-ish formats...
Really interesting numbers. I just wish it was possible to see all the other statistics in relation to letgit vs bot traffic.
eg if mobile traffic is 35% of total, what percentage of legit vs bot traffic is mobile? TLS 1.2 vs 1.3 is a 38%/61% split, but how much bot traffic is encrypted?
This is just the beginning. We are going to be building out more and more functionality. My hope is that we deploy jupyter and let people crunch numbers in the ways they choose.
A single bot can generate thousands of times more traffic than a person. Even a small number of bots could easily generate a huge percentage of traffic.
Wow, tiktok.com steadily ranking at #3, more popular than facebook.com, amazon.com, apple.com and netflix.com, only behind google.com and microsoft.com. Really curious to see detailed stats.
I can see tiktok being more popular than apple, people watch videos much more often than they buy phones. But microsoft being more popular than tiktok? What's so popular on that site?
Cloudflare says they exclude "content servers"[1], I assume that means CDNs. https://www.verizonmedia.com/ says it is an ad network, including video ads, with brands like Yahoo, HuffPost and TechCrunch.
The leading UK food delivery company, for popularity I'm not sure if a higher or lower number means more popular - but to me it looks like the site has more traffic on a weekday than a weekend. And also most traffic comes from Australia, where this site wouldn't even work.
Really weird results, I have a popular site on Cloudflare and the domain popularity by visitor country shown on this public dashboard doesn't match traffic statistics I see in my account at all.
I'm guessing that it depends on the kind of traffic taken into account? If we count all requests to and from computers and not only web requests, you end up with a whole lot of computers calling home all day
How is this data collected? I can see how they would be able to get statistics from clients hitting CloudFlare servers, but how can they report on the traffic hitting google.com or microsoft.com?
I'm guessing the different metrics are collected through different means. Their blog mentions HTTP request to sites that use cloudflare as well as DNS [0]. It would make sense that domain popularity is meaused through DNS and bots and DOS measured by the systems built for handling those things.
Any idea why the "worldwide change in internet traffic" graph is not flatter and has daily dips ? Is this graph normalized on the actual time, disregarding time-zones?
I would say likely due to the uneven population distribution around the world, combined with day-of-week trends due to people being off work/school.
The more interesting datapoint to me is the hour-by-hour breakdowns of HTTP vs HTTPS, likely due to particular countries that discourage (and have ISPs that block) encrypted traffic.
It would be great if the site rankings had a longer time period than 30 days, such as 1 year. Though totally understand if they only started tracking such a ranking recently.
They don't really say how they are ranking things, it appears to be multiple metrics. They also removed porn, and I would not be surprised if they removed something like amazonaws.com since that just sits in front of s3 bucket.
Developers: The page must be fast! Google found an extra .5 seconds in search page generation time dropped traffic by 20%.
Also developers: Let's use Cloudflare - 'Checking your browser before accessing content - please allow up to 5 seconds'.
There is not-insignificant, if not huge group of site admins that take a "aha! more security therefore better!" approach and will just enable under attack mode (the 5s JS challenge) for everyone all the time.
Some are a bit more savvy and will take a still rather ham-fisted approach of doing so for most countries. That's not uncommon with retailer websites that realistically only serve a single country, and therefore have good enough reason to challenge all others to deter automated traffic from compromised machines outside their market.
I see it ALL of the time. I am not sure if it is due to Safari and ITP blocking whatever protection tracking stuff Cloudflare does, or because my DNS is routed through AWS vs some other provider (as I am on a split tunnel VPN, and need to resolve internal DNS entries).
It's frustrating, especially when I open a couple of tabs on the same site after Googling a particular problem.
It could be the ITP blocking or your DNS setup. If I had to guess it'd be the AWS setup you have as that is not seen as a client IP for a home network.
The interstitial page you're seeing attempts to validate IP location, HTTP headers, and the query string at the very least. So the fact that your requests look like they are coming directly from an AWS IP would, based on my gut feeling having troubleshot tons of these issues at CF, cause this issue.
If you're ever curious you might want to try and hit a site you know serves you these pages regularly without the VPN setup just to isolate the problem.
Well, yes and no.
They have their own user agent, so they can get their own classification. e.g.
Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 [FBAN/FBIOS;FBDV/iPhone11,8;FBMD/iPhone;FBSN/iOS;FBSV/13.3.1;FBSS/2;FBID/phone;FBLC/en_US;FBOP/5;FBCR/]
Too bad Radar omits pornography sites, I was curious whether they'd actually be among the top sites. FWIW, SimilarWeb [2] (which has Tiktok at #46), does include porn sites (xvideos and pornhub are #8 and #9, respectively).
Though I don't quite understand how mozilla.org could rank as #10 on Radar's sites, just below Instagram and above Youtube, Whatsapp, and Twitter.
edit: It's also interesting to compare its Top Browser rankings with analytics.usa.gov [3]. It appears that Radar's measured users are significantly more on Chrome – 55% (desktop+mobile) vs 47.6% on usa.gov sites; whereas usa.gov users are significantly more on Safari (iOS+desktop): 35.2% vs 13% on Radar. I suppose this reflects the assumption that iOS has a bigger proportion of U.S. users than it does worldwide users.
[0] https://radar.cloudflare.com/#trending-domains
[1] https://www.comscore.com/Insights/Rankings
[2] https://www.similarweb.com/top-websites/
[3] https://analytics.usa.gov/