If you’re not interested in my preamble, just jump to the tool.
Search Console, love it or hate it, has its shortcomings. If you are a technical SEO focusing on index bloat, crawl waste, or anything related to indexation, you know that Search Console only gives you cursory information. The Index Status report is about as high level as a report could possibly be. Exporting provides no URL detail. If you want to know what URLs are actually indexed, you won’t be getting it from Google Search Console.
And that’s a longstanding problem technical SEOs have faced. We’ve tried to scrape Google’s search result pages (maybe with a Scrapebox or similar tool), but we’ve mostly been relegated to checking indexation manually using Google.com. We might lean on search operators until Google starts throwing Captchas, but even that isn’t always consistent. See, Google doesn’t index everything it knows about. Google also doesn’t show everything it indexes. Sometimes a query that should logically pull a particular URL doesn’t, but if you query it a slightly different way… well, that might work for some reason – with (or without) using search operators. Cats and dogs, living together, mass hysteria.
Check out this example. We have an old employee who previously had a bio page on our website. (We miss you Megan, but you’re doing some really important stuff.) We redirected her page to our default “Meet The Team” page. A keyword search for Megan’s bio doesn’t pull it up as expected. But, we improperly (and accidentally) 302’d her page, which is keeping the URL in Google’s index.
Basically, Google is remembering the URL, but displaying the destination page. And Google says 302s are the same as 301s, huh? But that’s another story. So how did we discover that? Dumb luck and a lot of poking around in Google search results? Nah, we used the tool I’m about to introduce.
Legend has it, Anthony Moore, while facing a problem, with face in palm, once said:
Well, Sean Malseed was listening. He rolled it around his noodle, and went to work on the Greenlane Indexation Tester. When he rose from the cellar (or emerged from the cellar, triumphantly, as he puts it), a new tool was built.
Meet The Google Indexation Tester
If you know Greenlane, we like to build tools when we come across client issues. Then, we like to share them for the world to use – check out the Garage for all our free tools. This newest tool is one of my personal favorites to date. Our Director of Technology, Sean Malseed (who works in our Garage and has his own tools suite at RankTank.org) found a hidden gem – an unpublished API that can be tapped to check indexation.
If you were at my State of Search session in November 2016, I gave you the link to version 1. Now we’re sharing the new and improved version for everyone. It’s in Google Sheets, so you’ll simply need to make a copy. No additional plugins needed. Simply paste in a list of URLs (as values), or upload your XML sitemap, and let it do its thing.
- Note: This tool gets data without scraping Google search results, and without using your IP address!
Check out the screenshot or video tutorial below. Any questions? Email us at firstname.lastname@example.org.
Watch The Video Tutorial