Cookies disclaimer

I agree Our site saves small pieces of text information (cookies) on your device in order to deliver better content and for statistical purposes. You can disable the usage of cookies by changing the settings of your browser. By browsing our website without changing the browser settings you grant us permission to store that information on your device.

Web Crawler Guides

Web Crawler Guides

The guides in this part of the support documentation for our web crawler describe in illustrated step by step instructions, how to use our software. Our software is easy to use even if you are completely new to web crawling.

The ‘how to’ support documentation for the web crawler is segmented into five sections:


Project Management

These guides include the first steps that are required before you can crawl or report on anything. These guides include, how to add, edit and delete entities such as projects and websites. We also describe the hierarchical structure of the projects within our software.

Learn more about web crawler project management.



How to guides describing the different options for crawling websites, such as crawling a whole project or just one site. You will need to have added a project before you can follow any of these guides.

Learn more about crawling websites.



Reporting guides detail how to generate reports, as well as how to read reports. There are various options for exporting data depending on what you want. We are also constantly adding more data and reports for you to download.

You will need to have crawled at least one site to generate a report.

Learn more about reporting.


Usage & Analytics

These how to guides show you how to do the following two types of activity:

Learn more about usage & analytics data.


Understand Usage

Learn how to read your usage data, what it means and how it is generated. This is always worth looking at as we describe in here what we crawl and what counts as a used ‘URL’ from your usage. For example, 301 redirects are still counted as are 404 pages, we also count images and a range of file types.

You will need to have added and crawled a site to see usage data.

Learn more about usage data.


Project Analytics

These guides provide context around how to read and understand the project data that we collect and display within the software. You will need to have crawled a site to see project analytics data.

Learn more about project analytics.