Whois records just got better at DomainTools today because we have integrated Compete.com
data into our records.
The metrics industry in general got a lot more exciting when Compete.com showed up last fall. Competition is what keeps innovation happening, and it was good to see a fresh face on the scene because things were stale. We have seen very little happen in the free global metrics space for years – comScore was thought to be more accurate but was a closed paid system that not many companies choose to afford. SEO marketing companies tend to go for the free service and until Compete.com showed up there was nothing to challenge the lone free Internet metrics service, Alexa. We have had Alexa data inside our records for a over a year and we thought it would be fair to add the new kid to the metrics industry as well. Some people, like Google’s insider Matt Cutts, question Compete’s data and say it is inaccurate. I think the verdict is still out, but I know more data can’t be a bad thing.
Alexa and Compete have huge difference when you look at their graphs. Last year DomainTools was ranked 190th in the world by Alexa. We certainly don’t feel that big, we have a few million visitors a month but should we be in the top 200? My friend Alex Algard founded another Seattle startup called Whitepages.com and they serve between 8-10 million pages a day. While DomainTools serves about 5% of that amount, both his traffic and ours are growing.
Geoffrey Mack the Product Manager of Alexa noted in his blog:
…[A] colleague who had noticed that Alexa had shown a steady decline of his site’s reach over the course of the year, which stood in stark contrast to his own internal stats which showed a steady increase in users. The answer to the apparent discrepancy is that Alexa’s Reach number is not the same thing as visitors. The reach is counting the percent of Internet users worldwide who visit a site. In the case of my colleague’s site, his global reach was declining, not because he was losing visitors, but because the rest of the Web was growing so quickly. As people are coming online in China, Korea, Vietnam, Brazil and countless other countries, they are counted as users in Alexa’s panel, but they aren’t visiting his site. So, in essence, his percent of global traffic is declining, even as his traffic goes up.
We have seen Alexa stats get even more World focused over the last two months and suspect that this is affecting their US-focused website stats. Alexa currently ranks DomainTools as 228th in the world and Whitepages.com at 650th in the world. DomainTools averaged 35.19% of our visitors from inside the US last week, while I am sure Whitepages.com had a far more concentrated US base. Companies can’t rely on Alexa graphs unless they have a truly global website.
This lack of non-US based customers explain why Whitepages.com fell from 400th in the world last year to 650th in the world while at the same time increased in visitors. Alexa seems to be increasing their global client install base faster than traffic increases on most U.S. geared audience websites. I think Alexa’s statistics are awesome at representing the world’s traffic, but their was an adjustment or some event that happened to the entire Alexa system 2 months ago which shows everyone dipping down a little. Geoffrey hints on his blog, “… we have as many U.S. users as ever, on a strict percentage basis [but] they are declining as the Alexa Toolbar grows in popularity across the globe“. The most interesting thing he points out is that Alexa has only 18% of its users from North America but he estimates North America actually has 21.2% of the Internet users. Which means Alexa data should be taken with a huge grain of salt and only considered a world metrics system. Even as a global metrics system on their own accord they discount the North America by 17.7%! Look to the new companies to eat Alexa’s lunch because they continue to not want to normalize. If they isolated U.S. based traffic and just showed those statistics they would have entirely new graphs and I suspect look closer to comScore numbers. All of the things I have read lead me to believe the world is growing faster than US targeted websites – even if those U.S. based sites are having phenomenal growth.
Many start-ups use free metrics, but if they are showing these numbers to investors, they put an asterisk on them. VCs should also be aware Alexa graphs are only good for global reach numbers as well. Sites like Skype would be accurate but then again, they are more an application then a website.
Conclusion: Alexa discounts North America by 17.7%
The youngest metric company is the start-up called Quantcast – a horrible name, but very interesting data. First, let me rail on this name. Quant is short for Quantitative, Quant is a slang word for an expert in the use of mathematics and related subjects, particularly in investment management and stock trading. I suspect a mathematician co-founder invented this name, and I think they should hold a few focus groups on names for this company before they decide to stick with it. After all, they are in beta and could rebrand when they come out of beta. But enough about their bad name, as I said I was impressed with their data. They are still young company and don’t have an API yet but their data is very impressive and their chart of DomainTools shows traffic on the rise. It appears they have isolated US traffic and show just that chart. I wish Alexa would learn a lesson from this and show traffic based on different areas of the world rather then trying to show the entire world as the default view. Check out Quantcast’s service, it offers a fresh perspective to both Compete and Alexa. We look forward to adding their data in once an API is available.
Robots.txt footnote
Alexa, Compete, and Quantcast are all guilty of firewalling unknown friendly search engine agents at the front gate. These sites that monitor the Internet should be the most in the know that unfriendly agents cloak as humans and will come in no matter what. So the general rule of thumb is that robots.txt directives are only for the good agents anyway. Having just one rule that applies towards all agents is considered a best practice. New search engines like Powerset are left with a huge firewall and no information about certain websites because webmasters don’t understand the power of robots.txt. One of these three companies sort of got the right idea by blocking certain locations but if you are a CEO of one of these companies, send your webmasters back to boot camp. If there are sensitive areas of the site, most search engines can use complex pattern matching rules and agents can be instructed to stay out of those areas. I am not naming names but they are all guilty. Why does one company allow 4 agents, the other 6 agents, and last 7 agents? I got started on the Internet as an SEO expert so I have to rail on large companies that are unaware or clueless.
Alexa’s Robots.txt
Compete’s Robots.txt
Quantcast’s Robots.txt
Subscribe
June 5th, 2007 by 
Jay Westerdal 
I am at SMX Seattle this week and talking with a lot of Industry leading SEO webmasters about what is going on in the Search Engine space right now. There is increased talk about how ranking first is even more important now then ever before now that Google has launched Universal Search. I attended a few after parties held by Google, Yaho0, and Microsoft, all of them were great, but personally I think Google’s party was the best. The Google Colors were every where and they had full candy bars (see picture below). The glow in the dark dring glasses were a nice touch as well. If you would like to see notes from the session they had good coverage by some bloggers in the audience which got posted at SE Roundtable. No relation to the Domain Roundtable. I think Domainers and SEO Experts need to work closers together so I have invited some of the best experts to the Domain Roundtable for our August conference. I will talk about these experts later.
7 Comments »
I was checking Green Lake Massage’s website. They are a small Seattle based company. Great domain name for a firm that gives messages to name itself after the city it is in and the thing it does. [City] [Occupation]. However when looking at there website and their whois record I am shocked at how disorganized everything is. First off their 
Appearing on the front page of Search Engine’s result page is actually fairly easy if a person is trained in the science and art of SEO; provided however that the term they targetting is below the 75th percentile on the keyword difficulty index score. Competitive terms (AKA Money Terms) are fought over in the SEO Wars everyday. If a webmaster with the right product can reach the first page of results with the right term it can mean thousands of dollars of additional income a month.
The 
I was talking with Frank Schilling of 
Often thought leaders define new words but forget to register the domain name. Seth Godin in his 
MattCutts.com
Above the whois, we also have this summary display of what the search engine robot would have seen. As we can see in the example, Matt Cutts does not have a meta description for his page. For those of you that do not know, Matt is a Google Engineer, so it is a bit ironic that he does not populate this. Overall Matt’s front page lacks a lot of things and his score was a 68% for on-page content.