Examples of How Indexation Data is Used
- Example 1: Quickly Find Resources That Won’t Be Indexed by Google
- Example 2: Identify Follow Internal Links to Non-Indexable Resources
Benefits of Our Data
HTML/Text pages that are either restricted by a robots.txt or meta robots tag from being indexed or have a status code other than 200. Non-indexable, means a resource that Google will be unlikely to index and hence show in the SERPs (Search Engine Results Page).
Within the column for ‘non-Indexable’, the two options or the values we would return in the fields / cells are as follows:
A resource is deemed to be non-indexable if any of the following are true:
- the URL disallowed by the robots.txt
- The URL contains the NOINDEX tag
- The URL returns a status code of anything other than ‘200’
- URL has at no follow internal links from and indexable page
This does not mean that a resource is not indexed by Google, as it may be indexed for any number of reasons. Rather, this is an indication that there may be a problem either now or in the future with the indexation of a specified URL.
If you want these pages or resources to be indexed, it is worth looking at the reason why they might not be.
There may be other reasons preventing a page from being indexed such as a penalty from Google, which we do not check for.
There are various reasons why you would want to look at Indexation Data, we have set out some examples below.
Filtering for ‘False’ within the ‘non-indexable’ column will show you are resources that are not indexable. This can often immediately highlight errors where they exist, for example; the home page or product pages should always be indexable.
Non-Indexable resource (URLs with the value of ‘False’ in the ‘non-indexable’ column) should not have ‘follow’ internal links pointing to them. You can see how many follow links a URL has in the column header ‘follow in link’, a non-indexable resource should have none.
This leaks authority to pages that do not require it, adding the ‘Nofollow’ tag to all internal links pointing to a resource will resolve this issue.
The list of guides below might be useful if you are analysing this data and want to know more about it:
Related column headers in Raptor website crawler reports:
- Has Meta Robots Follow
- Has Meta Robots Index
- Has Meta Robots Nofollow
- Has Meta Robots Noindex
There are several benefits to analysing indexation data, such as those listed below:
- Identify Indexation Issues for SEO
- Identify all indexable Resources
- Identify all non-indexable Resources
- Resolve indexation issues & conflicts
- Audit a site’s indexation profile
- Scrape competitor indexation data
Sign up Today for a Free 30-Day Trial. Identify indexation Issues on your site!