Cookies disclaimer

I agree Our site saves small pieces of text information (cookies) on your device in order to deliver better content and for statistical purposes. You can disable the usage of cookies by changing the settings of your browser. By browsing our website without changing the browser settings you grant us permission to store that information on your device.

Meta Robots Tags

Meta Robots

Contents:

It is possible to add ‘robots’ Meta tags to the source code of individual pages. This sets / specifies crawler access at page level, allowing you to stipulate how the page should be crawled and indexed.

This can be used in addition to the robots.txt file, which you can read more about by following the link.

Like the robots.txt file, the Meta tag version is an indicator and not a locked door, meaning that crawlers, especially malware, can ignore it completely. This article is part of a series on Meta tags; you can see some of the other articles by clicking a link below:

Benefits

The robots Meta tag helps to control access to a page by crawlers and is most commonly used to prevent search engines from crawling pages that you do not want to be indexed. This prevents wasted time crawling your site and thus increasing crawl efficiency.

This also enables you to prevent pages from being shown in the SERPS and can assist in limiting duplicate content issues.

Often websites will have custom landing pages for different campaigns such as SEM / PPC (Paid Search), EDM (Email Direct Marketing), partner traffic / campaigns, content marketing, etc.

It is often the case that you do not want these pages indexed and thus receiving organic traffic or appearing in the SERPS. Adding a robots Meta tag to the page can resolve this issue.

How to Implement

Like all Meta tags, they are located in the <head> section of the source code and use the Meta name ‘robots’, see examples below:

<meta name=”robots” content=”index, follow”>
<meta name=”robots” content=”noindex, nofollow”>
<meta name=”robots” content=”index, nofollow”>

The Meta name ‘robots’ simply defines this Meta tag as a robots tag and thus enables the use of robots content… We discuss in more detail below what all of these parameters mean and what impact they have on a page containing the code.

Content=””

This component of the Meta tag is where we specify what action the crawler should take, we have listed the all of the potential options below:

Index

Specifies that the page should be indexed, this is the default choice for crawlers anyway and hence is not a massively useful option. You can put these onto every page that you want indexed, but it won’t speed up or assist in indexing, rather it acts as a preventative measure.

Example: <meta name=”robots” content=”index”>

Noindex

Specifies that the page should not be indexed and prevents the page from appearing in the SERPs. Ensure that is used only where required.

Example: <meta name=”robots” content=”noindex”>

Follow

This informs crawlers / search engines that the links on this page should all be followed; this is regardless of whether the page itself can be indexed. Thus you can use this in conjunction with other statement such as ‘noindex’.

Example: <meta name=”robots” content=”follow”>

Nofollow

This informs crawlers / search engines that the links on this page should not all be followed; this is regardless of whether the page itself can be indexed. Thus you can use this in conjunction with other statement such as ‘index’.

Example: <meta name=”robots” content=”nofollow”>

Noodp

Google may use descriptions from the open directory project within the SERPs in place of the page title and Meta descriptions that have been specially created for that page.

Since Dmoz listings are factual and not very attractive for CTRs, it is recommended that we prohibit the DMOZ data from appearing in the SERPs. This is done with the use of the NOODP robots tag.

Example: <meta name=”robots” content=”noodp”>

Noydir

Google may use descriptions from Yahoo within the SERPs in place of the page title and Meta descriptions that have been specially created for that page. Since Yahoo listings are factual and not very attractive for CTRs, it is recommended that we prohibit the Yahoo data from appearing in the SERPs. This is done with the use of the NOYDIR robots tag.

Example: <meta name=”robots” content=”nodir”>

None

This is essentially shorthand for ‘noindex, nofollow’. Combining both tags into one shorter version. This will prevent a page from appearing in the SERPs, being indexed and will prevent all links from being followed.

Example: <meta name=”robots” content=”none”>

Noimageindex

This prevents images from being indexed on a page, the default is that they will be indexed, so you only need to specify this if you do not want images to be indexed and appear in Google Image Searches.

Example: <meta name=”robots” content=”noimageindex”>

Noarchive

This prevents search engines from displaying a cached copy of the page (not used by MSN/Live)

Example: <meta name=”robots” content=”noarchive”>

Nocache

This prevents search engines from displaying a cached copy of the page, but is only looked at by MSN/Live.

Example: <meta name=”robots” content=” nocache”>

Nosnippet

Specifies that Google cannot show snippets of content on this page from being shown in the SERPS and it also stops page caching.

Example: <meta name=”robots” content=”nosnippet”>

Common Mistakes

When adding robot Meta tags to pages through a template, it can often be an oversight that they are not customised or removed as required on each page. This can result in, depending on the contents of the tag, site-wide issues of pages not being indexed.

Issues can also occur when pages are copied and the tags not edited or removed, causing pages to either be indexed or not indexed, either of which can be good or bad!

Sign Up For Early Access
& Earn a Chance to Win 1 Years Free Subscription!

What You Get...

There's no obligation to become a full member after your trial, but we think that once you've seen what's available, you'll want to join us.

We are in the process of building our software and are ramping up to launch the Technical Auditing component in early 2018, soon to be followed by a suite of other components such as keyword ranking and backlink analysis.

Sign up today for 1 months free access and get a further 10% off of any package price when we launch for the first year as a reward for being an early subscriber.

Also, you will be entered into a lottery, where we will be giving away five 1-year subscriptions for free!

Sign up for early access today!