I have never heard of a 404 penalty. 404’s happen. Google has them sometimes too. On large sites, no matter how diligent you are, there will be abandoned or moved pages where an internal link wasn’t updated. A product may go out of stock. A news story may get pulled. A site might get a refresh and a URL may be left behind.
Search engines obviously don’t want to serve 404 pages. It makes them look bad, so they give you tools through their webmaster accounts to identify and clean up your own 404’s – I wrote about using them for backlink wins). Trying to minimize your 404’s is an excellent idea for users and to keep Google focused on the URLs they should be caring about.
To handle 404’s, I contain them in this order:
- 301 redirect any 404 error I find to the most relevant page on my site. If there is no relevant page, and the page will not be coming back, move to option 3. If the page is due to return soon, also move to option 2.
- Leave the 404 in place. Google has a big memory – if the page is due to return, Google will check back periodically and hopefully the trust and history of the page will continue. If after a certain amount of attempts there still is no page, Google will knock it out of their index.
- If the page is officially gone and will never return, use the “Remove URLs” feature in Google Webmaster Tools. Unfortunately for large sites with many 404’s, this is not a very scalable tool. But if you want to get Google to forget about this page and start focusing on your other content or products, this is a smart optimization step.