Cookies disclaimer

I agree Our site saves small pieces of text information (cookies) on your device in order to deliver better content and for statistical purposes. You can disable the usage of cookies by changing the settings of your browser. By browsing our website without changing the browser settings you grant us permission to store that information on your device.

Disallowed URLs

Disallowed URLs - Raptor SEO Data

Disallowed URLs are URLs that have been disallowed from the robots.txt file. This tag will prevent the page form being crawled by Google and thus will prevent them from appearing in the organic search results.

 

 

How to Disallow a resource from the Robots.txt File

There are various ways in which you can disallow a page or resource from being crawled by either Google, search engines or web crawlers from the robots.txt file. You can specify the whole site be disallowed, which is often done on staging servers in development. To do this you add this tag to the robots.txt file:

User-agent: *
Disallow: /

You can also specify specific search engines or crawlers by stating a user agent such as Googlebot:

User-agent: Googlebot
Disallow: /

You can also disallow any resource or type of resource from being crawled such as images, entire directories or PDFs, you can also use wild cards to save time specifying each page.

 

What Does Disallow Do?

Unlike the noindex tag, the disallow prevents crawlers from crawling a page or resource. The noindex tag requires Google to crawl the page to find the tag and respond accordingly. This can save crawl budget and also prevents resources from being indexed.

You can see our robots.txt file by clicking the link of you want to see an example. The disallow will be adhered to by search engines but most web crawlers will simply ignore them.

 

Why We Show You This Data

Often this is done intentionally but if it is not done intentionally, this can cause massive indexation problems. Often after a website migration this can be left in place and could prevent the whole site from being indexed. Identifying resources that can’t be crawled, can help to resolve problems with a site or specific pages appearing in the SERPs (Search Engine Result Pages).

 

Sometimes data and SEO can be confusing, with so many tools, the ever-changing environment and new technology it’s hard to keep up with the acronyms. We describe all the data we use and try where possible to use to use the most descriptive and common terminology. At Raptor our SEO tools and digital marketing software is designed for SEOs by SEOs, we give you everything you need to make informed strategic decisions and drive organic visibility. If you want to know more, check out the video below:

 

This guide is part of an extensive series of guides covering the data that we show in the summary tab of our SEO reporting feature. The following list of links shows all of the categories of data guides, videos and tutorials that we have. If you have any feedback on this or anything else, please fee free to get in touch:
Summary
Indexation
Canonicals
Canonical Content
Content Data
Linking Data
Page Speed Data
Meta Data
Google Analytics Data