Our Blog

Expert Internet Marketing Advice for Small Businesses

How to use Simple SERP Scraper

Posted on Friday 10 January 2016 in Tools

scraper-header

Heres a quick walkthrough on how to configure Simple SERP Scraper, which can be downloaded for free here.

Prospect Tab

Settings Tab

  1. Country:В the Google TLD youd like to query
  2. Search Index: the index to scrape: the web, blog posts or blog homepages
  3. Scrape Delay: the amount of time to wait between each scrape
  4. Number of Results: how many results to return for each page
  5. Number of Pages: how many pages do you want to scrape
  6. Process Threads: currently set to 1 (may change later)
  7. Search Queries: search queries to scrape (Google search operators only)

Scrape Delay

This is the time the tool will wait between each scrape.В This setting is very important if you are not using proxies.

These are the recommended settings, to make sure Google doesnt block or ban your IP address:

Note: The random setting makes the searches look as natural as possible, delaying each scrape anywhere between 30 to 90 seconds. This does mean the tool will take a little longer to run, but at least you wont trigger any captchas. That being said, if you intend to do aВ large number of search queries using the site: and info: operators, captchas will beВ inevitable!

Proxies Tab

For most standard prospecting jobs, proxies areВ not necessary, if the scrape delay is set to Random we designed the tool to run without them.В But if you intend to do bulk prospecting (more than 100 search queries),В wed recommended using them.

TIP: Only ever use anonymous proxies and ideally you should use private ones.В If you want to use public proxies, bear in mind that they are slow. And you'll need a lot of them, as Google is very quick at blocking them.

scraper-proxies

Settings Tab

This is where you set up the link metrics you want to retrieve, and add the necessary keys to fetch the data.

scraper-settings-tab

1. PageRank

Fetches toolbar PageRank for each URL and the domain. The tool uses a third party service called PageRank.My-Addr.com toВ retrieve this data. We could have got the data without this service, but it means using proxies. The service is really cheap: $1 for 100,000 checks, which works out to beВ 0.001В cents per check!

To fetch this data, first sign up for an account and then add your API key into the field. Your API key is the long number found on this page.

pagerank-key

Dont use this key, its just an example! >В http://pagerank.my-addr.com/external_api/A5621F56440D928546234EB196609A50/[url]

2. Majestic SEO

To fetch Majestic data youll need a paidВ accountВ  silver or above. Click on theВ В Get Access Token to authorise the app and retrieve your OpenApps unique access token. Enter the access token into the field.

3. Mozscape

Mozscape data is available to anyone with a Moz account: free, pro or paid. Although, there are some major speed differences between them:

Click on the Get Credentials link to retrieve your Access ID and Secret Key.

NOTE:В Using a free API account willВ add considerable time to large prospecting jobs.

Scrape Away

Once youre all set up and youre ready to go click the Scrape button. The speed depends on various factors: your connection, proxies (if used) and the speed of link metric data providers.

By Sarah Sawyer

We help businesses succeed online - follow me on twitter or via Google+.

Share this post

Comments Please!

  • vseo

    Quite simple process, and quite useful.
    You can also build that query using google docs and majestic SEO.
    The good point in that, is that you can set triggers if any new player appears

  • Jonathan Jones

    Will be reviewing this tool on my blog, and I may possibly add this to my list of tools to use for 2016 article! :)

  • Seb

    Awesome tool, but could it be that it is only working for .com and .co.uk? If I try it for .DE (Germany), I only get an empty list with the headline but not values

    • Sarah Sawyer

      Hi Seb, Glad to hear you like the tool. Ive updated the code, and its all working again. Youll need to download the latest version.

    • anonymous dude

      I have the same problem. But it works when I use it as its shown in example:
      phrase intitle:phrase

      • http://www.wonderlandpromotions.co.uk/ Sarah Sawyer

        Windows or Mac? Are you using the the latest version? The latest version is 1.42. If you are, please drop me an email info@wonderlandpromotions.co.uk and Ill have a look at whats going on.

        • Seb

          Hi, I updated to 1.42 and it works now (Windows). Thank you!

  • Spook SEO

    Great tool and such an easy process to set up this. I would consider this one as my personal tool to use and Ill see what it can offer me. Thanks for introducing this me. Ill come back on this for my feedback.

  • josephnew87

    Chkout this breakthrough method to drop upto 3 Dress Sizes in a week http://www.girlyhacks.com/venusfactor

  • UpGiant

    Any plans to add a custom date range function?

  • Jonathan Jones

    Has this tool been removed?

© Wonderland Internet Promotions | 2018 | 50 Brown St, Manchester, Manchester M2 2JT