Cookies disclaimer

I agree Our site saves small pieces of text information (cookies) on your device in order to deliver better content and for statistical purposes. You can disable the usage of cookies by changing the settings of your browser. By browsing our website without changing the browser settings you grant us permission to store that information on your device.

Indexation Data

Indexation Data

Simple Definition: Indexation
- Crawlable
- Indexable
- Non-Indexable
- Has meta robots follow
- Has meta robots index
- Has meta robots nofollow
- Has meta robots noindex
Related Content
Benefits of this Data

 

Simple Definition: Indexation

Indexation is one of the most fundamental parts of SEO, if a page is not indexable then it will not appear in Google. There are many components to this which we look at on-page to determine if a page is indexable or not.

There are also a range of tags and attributes that our Web Crawler scrapes from pages that are related to indexation. All of these are covered here in this section.

 

Crawlable

Pages and resources that are not disallowed by the robots.txt.

Detail: 

This is a basic check, where every resource, whether is a HTML/Text page with a product for sale, or a JavaScript file, can be crawled by your site’s robots.txt.

The robots.txt can be the simplest of website files and the most influential, our algorithm checks and informs you of whether a resource is crawlable by Google, based on the instructions of the robots.txt file.

 

Indexable

HTML/Text pages that are not restricted by a robots.txt or meta robots tag from being indexed and have a status code of 200.

Detail: 

This is a valuable check and a great time saver! We check every page for these five things where present:

  • Is the URL allowed to be crawled by the robots.txt?
  • The URL does not contain the NOINDEX tag
  • The URL returns a status code of ‘200’
  • If the URL has a canonical tag, is self-referential?
  • URL contains at least 1 internal link from and indexable page

If the answer to all the questions is ‘yes’, then we determine the page to be indexable for the purpose of this check. There may be other reasons preventing a page from being indexed such as a penalty from Google, which we do not check for here.

This performs a technical assessment of whether a page is technically indexable by Google, not whether it is in fact indexed.

 

Non-Indexable

Pages that are not indexable due to robots.txt or meta robots' tags, or a status code other than 200.

Detail: 

This is a valuable check that can save you time. We check every page for these five things where present:

  • Is a URL disallowed to be crawled by the robots.txt?
  • The URL contains the NOINDEX tag
  • The URL returns a status code other than ‘200’
  • If the URL has a canonical tag that is not self-referential?
  • URL contains 0 internal links from indexable pages

If the answer to all the questions is ‘yes’, then we determine the page to be nn-indexable for the purpose of this check.

 

Has meta robots follow

If the page has a the meta robots tag with a 'FOLLOW' attribute specified.

Detail: 

We analyse meta robots tags (see example highlighted below) and look specifically for the ‘follow’ attribute.

  • <meta name=”robots” content=”index, follow”>

If the page contains this tag regardless of whether the tag is implemented correctly, we will mark the field as ‘true’. If the tag is set to anything else or is not used, the field will be empty.

 

Has meta robots index

If the page has a the meta robots tag with an 'Index' attribute specified.

Detail: 

We analyse meta robots tags (see example highlighted below) and look specifically for the ‘index’ attribute.

  • <meta name=”robots” content=”index, follow”>

If the page contains this tag regardless of whether the tag is implemented correctly, we will mark the field as ‘true’. If the tag is set to anything else or is not used, the field will be empty.

 

Has meta robots nofollow

If the page has a the meta robots tag with a 'NOFOLLOW' attribute specified.

Detail: 

We analyse meta robots tags (see example highlighted below) and look specifically for the ‘nofollow’ attribute.

  • <meta name=”robots” content=”index, nofollow”>

If the page contains this tag regardless of whether the tag is implemented correctly, we will mark the field as ‘true’. If the tag is set to anything else or is not used, the field will be empty.

 

Has meta robots noindex

If the page has a the meta robots tag with a 'Noindex' attribute specified.

Detail: 

We analyse meta robots tags (see example highlighted below) and look specifically for the ‘noindex’ attribute.

  • <meta name=”robots” content=”noindex, nofollow”>

If the page contains this tag regardless of whether the tag is implemented correctly, we will mark the field as ‘true’. If the tag is set to anything else or is not used, the field will be empty.

 

Related Content

The list of guides below might be useful if you are analysing this data and want to know more about it:

Related column headers in Raptor website crawler reports:

 

Benefits of this Data

There are several main benefits to analysing indexation data, some of the more popular ones we’ve listed below:

  • Identify Indexation Issues for SEO
  • Identify all indexable Resources
  • Identify all non-indexable Resources
  • Resolve indexation issues & conflicts
  • Audit a site’s indexation profile
  • Scrape competitor indexation data

 

Sign up for a Free 30-Day Trial today! Identify indexation Issues on your site