How does Hubspot get Google statistics?

Put your website into Website Grader. The free report they come back with has:

Google pagerank
Google indexed pages
Last Google crawl date
Traffic rank (from Alexa)
Inbound links (matches neither Alexa nor Google links: operator)

How does Website Grader get this info? Are they screen scraping? Is it possible to get this information under a commercial license?

Do people have experiences to share about using screen-scraped data for their businesses?

Data

asked Jan 16 '10 at 05:43

Dan
303 points

Dharmesh should know that one ;) – Olivier Lalonde 15 years ago
Dharmesh, any comment? – Dan 15 years ago

2 Answers

Most of this data is easily retrievable using various APIs, ex. Google Ajax Search API.

Furthermore, the links: method is not very accurate, rather the better way is to fetch the indexed pages, number; hint: the one found in Google webmaster tools.

Furthermore, I dont think a user initiated report request would anyways get classified as screen-scraping. though I am in no way qualified to comment on the legal aspects.

Also, perhaps you should take the technical/programming aspect of the question over to Stackoverflow.

Cheers!!

answered Jan 16 '10 at 07:09

Dheer Gupta
191 points

"The implementation of the Service on your Property must be generally accessible to users without charge and must not require a fee-based subscription or other fee-based restricted access." http://code.google.com/apis/ajaxsearch/terms.html. Hmm, so all the Google features they use are free to their end-users. – Dan 15 years ago

It's interesting to see how they still get Page Rank. Page rank is no longer available publicly on the API and Google blocked SeoMoz from using Page rank data for their tool.

The rest is available from the various API's publicly. The links on Hubspot's report is from Yahoo Site Explorer.

answered Jan 16 '10 at 10:05

Romy Misra
71 points

Yahoo Site Explorer a) only allows you to get info on your own verified site, b) is limited to noncommercial use. Are they really using YSE? – Dan 15 years ago
I should say, what makes you think they are using YSE? – Dan 15 years ago
Because the data matches perfectly :) Try it for your self. – Romy Misra 15 years ago
So do you think they are illegal, or got a commercial license from Yahoo somehow? – Dan 15 years ago
The Yahoo site explorer api is free, anyone can use it! – Romy Misra 14 years ago
Romy, I believe the terms of service say limited to noncommercial use. Do you have a ref saying otherwise? – Dan 14 years ago