Hiding download pages and other pages from search engines
As you develop your website you will no doubt have various pages that are “backend” pages. These pages might include download pages, thank you pages, policy pages, or whatever. For web marketing we optimize our sites for search engines and we do not want these types of pages to show up in search engines.
Sometimes the reason we do not want search engines to show our page is for an obvious reason: we don’t want the search engines to show our product or free download page until the visitor has completed the proper steps of our funnel. But there is another reason we don’t want to show these various pages – dilution of our search results.
As the search spiders crawl our pages for information they are trying to digest the information on our site and determine what our site is about. If our site is actually about dog training, we want the spiders to only digest dog training information and not information about our support policy, payment policy, or a thank you message for subscribing. If the spiders digest that additional spurious information they can get confused as to what the site is about.
On method to discouraging search spiders from foraging on pages is the use of the “content” tag in our page. There are two relevant statements that we are going to use. The first is the “nofollow” statement which instructs the spiders not to follow any of the links on that page, and the second statement is “noindex” which instructs the spiders to ignore the page and don’t make any entries for it in the search engine’s index. The full statement will look something like this:
<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>
This statement goes into the <head> section of your page. It doesn’t really matter where in your <head> section it goes, but often it is put right after the <TITLE> tag and might look something like:
<head>
<title>My best download< /title>
<meta name=”robots” content=”noindex, nofollow”>
</head>
Of course, many of us are not html experts or don’t have access to the <head> section of the web page. This is especially true for WordPress users, of which I’m a heavy proponent. Thankfully, for the WordPress users there are a number of plugins available for adding this information on each of the pages we want the spiders to ignore. A couple of these WordPress plugins are Headspace and All in one SEO.