Disallowed URLs - Raptor SEO Data
Disallowed URLs are URLs that have been disallowed from the robots.txt file. This tag will prevent the page form being crawled by Google and thus will prevent them from appearing in the organic search results.
How to Disallow a resource from the Robots.txt File
There are various ways in which you can disallow a page or resource from being crawled by either Google, search engines or web crawlers from the robots.txt file. You can specify the whole site be disallowed, which is often done on staging servers in development. To do this you add this tag to the robots.txt file:
You can also specify specific search engines or crawlers by stating a user agent such as Googlebot:
You can also disallow any resource or type of resource from being crawled such as images, entire directories or PDFs, you can also use wild cards to save time specifying each page.
What Does Disallow Do?
Unlike the noindex tag, the disallow prevents crawlers from crawling a page or resource. The noindex tag requires Google to crawl the page to find the tag and respond accordingly. This can save crawl budget and also prevents resources from being indexed.
You can see our robots.txt file by clicking the link of you want to see an example. The disallow will be adhered to by search engines but most web crawlers will simply ignore them.
Why We Show You This Data
Often this is done intentionally but if it is not done intentionally, this can cause massive indexation problems. Often after a website migration this can be left in place and could prevent the whole site from being indexed. Identifying resources that can’t be crawled, can help to resolve problems with a site or specific pages appearing in the SERPs (Search Engine Result Pages).
Sometimes data and SEO can be confusing, with so many tools, the ever-changing environment and new technology it’s hard to keep up with the acronyms. We describe all the data we use and try where possible to use to use the most descriptive and common terminology. At Raptor our SEO tools and digital marketing software is designed for SEOs by SEOs, we give you everything you need to make informed strategic decisions and drive organic visibility. If you want to know more, check out the video below:
This guide is part of an extensive series of guides covering the data that we show in the summary tab of our SEO reporting feature. The following list of links shows all of the categories of data guides, videos and tutorials that we have. If you have any feedback on this or anything else, please fee free to get in touch:
Page Speed Data
Google Analytics Data