Technical SEO

The Crucial SEO Importance of HTTP Headers

Published: 06.17.16

If you’re not checking your client’s HTTP headers, you’re not giving them good service.

I’m not talking about the stuff in between the <head> and </head> tags, either. I’m talking about the server response that you get before you get all that nice HTML, or that fancy PDF, or whatever else your client’s website is slinging.

That’s because, well, your client’s website isn’t slinging anything. It’s being slung by a server, and the server’s HTTP response is the first thing a web browser – or a web robot like Google’s crawler – will see. There’s information there you may not be properly reviewing.

Implementing important SEO items in the HTTP headers is more common than you may think, and most of the major SEO tools don’t check them. It’s the number one feature we’d love to see in Screaming Frog, apart from scheduled crawls which we also built a utility to do. HTTP headers are so critical to our SEO efforts that we built a tool to use internally called Head Exam. It checks headers for SEO factors and lets us bulk check a list of URLs so we can audit a whole site.

Head Exam is so important to our SEO efforts that we’re releasing it as a free tool.

head-exam-screenshotTool updated 6/22/16, current version 1.15

Open Greenlane Head Exam

The canonicals in the above screenshot aren’t being pulled from the web page source. Those canonicals are coming from the HTTP header response. Use this tool to check your clients’ headers right now. You might be surprised at what you’ve missed.

Watch the video walkthrough

Here’s a scenario – let’s pretend you’re working on a technical audit for a client, and you call out an absence of a canonical tag (which is traditionally found in the source code). You compile your findings in your report and send to your client. The client responds to your recommendation and sets forth with implementing a canonical tag to live in the source code. (Keep in mind, for some clients there might be a cost involved in this implementation.) But what if the HTTP header was already properly displaying a canonical URL request? Google reads this request and may have already honored it. You could be wasting your client’s time and money, and displaying a degree of your own negligence.

Why HTTP headers are absolutely critical for every SEO

Did you know that you can declare canonical URLs in the HTTP header? Did you know that you can declare canonical tags in the HTTP header and not have to declare them in the HTML code?

If not, then I’m glad you’re reading this now before you audit a client’s website and tell them to implement canonicals, and they look at you blankly and say that they have. In the HTTP headers. That you didn’t even check. The bigger the client, the more likely they’ve done this. Like who? Well, whitehouse.gov for example. Just take a look at their HTTP header:

whitehouse-header-canonical

In this snippet of header (pulled from our Head Exam tool), we can see that the Whitehouse is declaring canonicals, shortlinks and even share images in the header!

This is because the Whitehouse.gov site is built with Drupal, which is used by a full 6% of the Quantcast top 10k sites. In fact, Builtwith is tracking 617,090 live top websites using Drupal right now. Drupal puts canonicals in the HTTP header by default.

But don’t take my word for it, because I’m not the only one talking about this. Mike King just recently covered this in his incredible deck, The Technical SEO Renaissance.

OK, but is anyone really putting this stuff in their HTTP headers?

Um, yes. I plopped in a random sampling from Alexa top sites… prepare to have your socks blown off:

sites-using-http-canonicals

Not only do all of these huge websites use canonicals, shortlinks and other SEO elements in the HTTP headers, but two of them have huge glaring mistakes:

  • washingtonexaminer.com declares a canonical that’s a 404. Further, the domain newsok.com in the canonical is for The Oklahoman, not the Washinton Examiner. Whaaa… ?
  • radaronline.com has a shortlink declared that redirects to WordPress. Not their WordPress, but straight up the WordPress.com homepage. Whaaa… ?

The fact is this: WordPress and Drupal both include a lot of important SEO information in the HTTP headers by default, and together those platforms make up something like 70% of the top sites on the internet according to Quantcast. That makes looking at HTTP headers pretty damn important for you as an SEO.

What else can be hiding in the HTTP header?

Tons of stuff. In addition to canonical tags, some of the most important things for SEOs are:

  • HREFLANG tags
  • Shortlinks
  • Share image
  • Encoding
  • Language
  • Content length
  • File type
    … and more

Is your client using one or more of these? Have you ever checked their HTTP headers? If and when your client calls you out on this, “I didn’t know to check there” isn’t going to cut it.

5-minute HTTP header audit checklist

Use this checklist, along with our Head Exam tool above, to do a quick 5-minute health check of  your clients’ HTTP headers. Click the items to expand and see what you should check if an item applies to your site.

Are there canonical tags in the HTTP header?

☑ Are they correct?

☑ Do they contradict any on-page canonicals?

Are there hreflang tags in the HTTP header?

☑ Is the hreflang tag chain unbroken?

☑ Do they contradict with any on-page hreflang tags?

Is there a shortlink in the HTTP header?

☑ Is it right?

Does the HTTP header declare a share image?

☑ Is it formatted correctly?

☑ Does it resolve?

Is a language declared?

☑ Is it right?

☑ Does it match the language declared in page source?

☑ Does it match hreflang tags (if applicable)?

How to use Head Exam

Using head exam is really easy. You have two tabs:

http-tabs

Use the Bulk SEO HTTP header check tab to check HTTP headers for a bulk list of URLs. Use the Single HTTP header examiner tab to take an in-depth look at the full HTTP headers for a URL.

Bulk SEO HTTP header check tab

On this tab, simply paste a list of URLs into Column B, starting at row 6. The tool will pull each URL, split up the HTTP headers, and parse out relevant SEO metrics from the headers into the cells to the right. You can see canonicals, shortlinks, share images, and more.

head-exam-paste-here

Be careful not to change any of the other cells, as they will populate automatically.

no-touch-automated-http-header-seo

Single HTTP Header Examiner tab

On this tab, you can see the full header response for a single URL, parsed out with important stuff (like link rel tags) highlighted. Simply enter your URL in cell C4, hit enter, and wait for the magic to happen.

single-http-header-examiner

That’s it! Hope you start checking HTTP headers every time now. Any questions on how to better use Head Exam, let me know!

Sean Malseed
Sean Malseed
Sean is a developer, technical SEO, and entrepreneur with over 10 years of industry experience. Prior to Greenlane, he founded CircleRank, a successful technical SEO agency in Philadelphia. He has spoken at major digital marketing conferences around the world, including PubCon, SMX, SES/ClickZ, and OMCap. He also co-organizes Philly’s monthly digital marketing event, SEO Grail. …
Read Bio
  • Bill Rowland

    Sean,

    Great post. This really opened my eyes to an area that needs greater attention.

    Based on your explanation, the primary benefits of reviewing the headers are:

    – Identification of conflicts between elements in the http headers versus those on page
    – Page speed.

    Is that correct?

    Regarding page speed, is there any indication of how much time can actually be saved?

    Cheers,

    • Sean Malseed

      Hey Bill!

      The most important part of checking headers is looking for conflicts, not so much page speed. The first things I look for are canonicals, shortlinks, hreflang, and robots directives. I make sure they’re not broken or wrong, and that they don’t contradict with anything declared in page source or robots.txt.

  • Steve

    Great post, Sean. As you’ve mentioned in a comment, it’s important to check HTTP headers for robots directives – for example one of my clients noindexes all PDFs site-wide using the X-Robots-Tag, so a traditional check in robots.txt wouldn’t flag that up, and for a normal HTML page, the page would be indexed despite no noindex code in the page’s section. It’s very easy to get caught out by it if you don’t think to check it.

    • Sean Malseed

      So right, Steve!

  • Umesh Shejole

    Helpful post like to follow for my SEO work.

  • Juan

    Hello,

    Great post and great tool!

    I’m trying to check the meta robots no index tag on a website, in which section am I supposed to do that?

    Regards

    • Sean Malseed

      Hey Juan! There’s two places to check for that directive – in the page source, and in the HTTP header (using Head Exam).

  • Franck

    Hi Sean,
    i’ve tested your tool for my website, on which i have specified canonical urls and href lang too. But nothing shows up in column K.
    The instruction is :

    I notice that fox.com writes this instruction with /> at the end. Is the “slash” missing in my HTML code ?
    Many thanks,
    Franck

    • Sean Malseed

      Hi Franck!

      Thanks for trying out the tool. Head Exam checks for canonicals and other declarations in the HTTP header, not in page source. A lot of platforms include declarations there (like Drupal), and issues there aren’t picked up by tools like Screaming Frog.

  • Paul

    I’m not declaring canonicals in my header. What I can’t distill from the article is: ‘Is it better to declare a canonical than not doing that or is it just important that it doesn’t conflict?

    • Sean Malseed

      Hey Paul,

      I think the two main takeaways are this:

      1. It’s important to know if your platform is declaring canonicals or other important info in the HTTP headers. I think a lot of people are running something like Drupal for example, and have no idea that their headers are doing that.

      2. If you are declaring canonicals in the HTTP header, it’s important that they are right and don’t conflict with anything you’ve declared in the page source itself.

  • Sean Work

    What’s with the expiration date in the Whitehouse response?

© 2017 Greenlane. All rights reserved.

2550 Eisenhower Avenue, A203, Eagleville, PA 19403

A Philadelphia SEO and Digital Marketing Agency    Privacy Policy    RSS

Subscribe to our Newsletter