HTML/Text pages that are not restricted by a robots.txt or meta robots tag from being indexed and have a status code of 200. Indexable content has at least the potential to appear in Google’s SERPs (Search Engine Results Page). How well it ranks within the SERPs is another matter, but a page has o be indexed to appear in them.
Within the column for ‘Indexable’, the two options or the values we would return in the fields / cells are as follows:
This is a valuable check and a great time saver! We check every page for these five things where present, if they are, then the resource is deemed indexable:
- Is the URL allowed to be crawled by the robots.txt?
- The URL does not contain the NOINDEX tag
- The URL returns a status code of ‘200’
- If the URL has a canonical tag, is self-referential?
- URL contains at least 1 internal link from and indexable page
This performs a technical assessment of whether a page is technically indexable by Google, not whether it is in fact indexed. There may be external factors such as a penalty that are preventing a page from being indexed that this assessment does not include.
There are various reasons why you would want to look at Meta Robots Data, we have set out some examples below.
Filtering for ‘true’ within the ‘indexable’ column will show you are resources that are indexable. When used with other data such as HTML/Text pages, Canonical Pages, etc, you can easily identify the main pages of the website you likely to want to optimise.
The list of guides below might be useful if you are analysing this data and want to know more about it:
Related column headers in Raptor website crawler reports:
- Has Meta Robots Follow
- Has Meta Robots Index
- Has Meta Robots Nofollow
- Has Meta Robots Noindex
There are several benefits to analysing indexation data, such as those listed below:
- Identify Indexation Issues for SEO
- Identify all indexable Resources
- Identify all non-indexable Resources
- Resolve indexation issues & conflicts
- Audit a site’s indexation profile
- Scrape competitor indexation data
Sign up Today for a Free 30-Day Trial. Identify indexation Issues on your site!