The AdWords Keyword Tool can be extremely useful for both SEO and AdWords Managers when planning and building campaigns to check metrics like advertiser competition, average monthly search volume, and approximate CPCs. Keyword Tool can also be used to perform analysis of a particular industry vertical, or perform a seasonal analysis for a set of keywords (using the “Local search trends” column).
The data can be so good, in fact, that you might find yourself trying to get your hands on lots of it, which can be a little difficult, especially since the Keyword Tool only returns a maximum of 800 keyword ideas per search, even if you enter multiple keywords for your initial query.
I wanted to examine a bulk keyword data set for some analysis this week, so I put together a simple tool using iMacros (the same tool we used to gather social media data for the Warmest 100 and predict the outcome of the world’s biggest music poll) that could *theoretically* be used to scrape a bulk data set from the AdWords Keyword Tool.
What is iMacros?
iMacros is a browser extension for Firefox, Chrome and Internet Explorer (although I will use Firefox running on Windows for this example) that allows users to record and automate repetitive browser tasks with relatively little coding knowledge. It can also be used to extract data from a website, or automatically fill out forms. Best of all for advanced users, iMacros can be fully integrated into almost any coding language, making it an extremely powerful tool.
Scraping The Keyword Tool With iMacros
The macro explained here performs a simple series of actions that allows you to extract a bulk set of data from the Keyword Tool. The macro uses a CSV list of initial keywords, and runs them through the Keyword Tool one by one, and downloads the keyword suggestions for each query, up to 800 keywords each time.
Step 1: Installing iMacros for Firefox
Visit the iMacros for Firefox extension page and click Add to Firefox to install the extension. Once installed you can use the iMacros toolbar icon to toggle the iMacros pane on/off.
Step 2: Setting up the macro
To create a new macro, click the Rec tab and click Record and then Stop without doing any actions in between to create a new #Current.iim macro file. Right click on the new #Current.iim file in the list of macros, and click Rename, and give your macro a name like Keyword Tool Extractor.
Now click the Edit tab and click Edit Macro to open the iMacros Editor window. Delete all of the existing line of code from the editor window, and paste in the code from this Pastebin. It should look like the code below. Click Save & Close to close the editor window.
Step 3: Explaining the code
To understand the macro, let’s run through the code and examine exactly what is happening on each line.
Line 4 opens up our CSV of initial keywords. In the example, this is saved to c:keywords.csv, but you can edit this to any file path you like.
Line 5 tells iMacros that we will only be using 1 column of the csv file.
Line 6 tells iMacros to move to the next line of the csv file every time the script loops.
Line 7 enters the next keyword from the csv into the Keyword Tool search box.
Line 8 makes sure the Keyword ideas tab is selected in the Keyword Tool.
Line 9 clicks the Search button.
Line 10 tells iMacros to wait for 3 seconds while the Keyword Tool results load. IMPORTANT: Depending on your internet connection speed, you may need to increase the number of seconds the script waits. 3 seconds was enough for me, but slower connections might take longer.
Line 11 clicks the Download button.
Line 12 clicks All search results from the download menu.
Line 13 clicks the next Download button.
Line 14 clicks Dismiss in the dialog that pops up following a CSV download.
Line 15 waits for 1 second before looping back to the beginning and starting again.
Step 4: Create a keyword CSV
This is fairly straightforward. Create a new spreadsheet in Excel with your list of initial keywords in column A. Make sure you save the file with a .CSV (Comma delimited) file type. I saved my CSV to c:keywords.csv, as per the example code, so if you saved yours somewhere else, you will need to edit line 4 of the macro code.
Step 5: Running the Macro
We’re almost ready to roll. First we need to setup the Keyword Tool to extract the right data. Login to AdWords and click Tools and Analysis > Keyword Tool to open the Keyword Tool. Edit the Locations, Languages, Devices and Columns settings to extract the data you want. You can also select or deselect Only show ideas closely related to my search terms to expand or refine your search results.
In the iMacros pane, select the Keyword Tool Extractor.iim macro from the list, and set the Repeat Macro Max to the number of keywords in your CSV file. My example file had 7 keywords, so I’ve entered 7 as the Max. This value determines the number of times iMacros needs to loop through the extraction code.
Click Play (Loop) to start running the code and watch the macro run!
The first time you run the macro, Firefox may display a download dialog similar to below. Click Save File and Do this automatically for files like this form now on to automatically save the CSV files to your downloads folder.
Step 6: Merging the data
After the script has finished running, you will have one CSV file in your Downloads folder for each keyword in your initial keyword CSV list. We need a quick way to merge them all together. I used a simple command line trick to merge all of my CSVs together. You should now have a GIANT spreadsheet of Keyword Tool data.
The only problem I encountered was when the Keyword Tool threw a captcha at me (as you would expect) after around 300 consecutive searches. This caused the macro to stop and display an error. You will just need to modify your CSV file and delete any of the keywords that have already been searched, and edit the Max loops in iMacros before running the macro again to continue working through the list of keywords.