Cookies disclaimer

I agree Our site saves small pieces of text information (cookies) on your device in order to deliver better content and for statistical purposes. You can disable the usage of cookies by changing the settings of your browser. By browsing our website without changing the browser settings you grant us permission to store that information on your device.

Add a Project

How to Add a Project

In this guide we explain how to add a project to Raptor’s web crawler software. A project is a container for sites that are related. Typically, projects have a main site and some competitors contained in them.

The number of URLs we must crawl, and the load times of those URLs are the two biggest factors in how long it will take to crawl a project. With that in mind, we typically crawl at around 10 URLs per second per site being crawled.


Step 1: Login

Open a web browser and navigate to:

Once there, login with your Raptor Username and Password:


Sign in to Raptor Web Crawler

You can click on the ‘Remember Me’ tick box to save your details for future access on that device.

Then click on ‘Sign In’.


Step 2: Click 'Add Project'

As illustrated in the screenshot below, click on the ‘Add Project’ link in the left-hand side menu:

Click Add a Project


Step 3: Add Site Details

This will take you to the ‘add project’ page, from here you can add details:

Add Project Details

For the main site, you must add the following details:

Project Name: Create a name for your project.

Site Name: Create a name for the site.

URL: Add the complete TLD (Top Level Domain) of the site you wish to add. Include the full URL, for example: = Correct = Wrong

Once these have been added, you do not need to complete any other field in order to save and create a project. You can also come back and add any of these details in at a late stage should you wish.

Sitemap URLs: If the site has an XML sitemap with an unusual filename or URL and is not detailed in the robots.txt file, you can add the URL of it here. If you have an index sitemap that lists all other sitemaps, you can add that URL here.

You can also add competitors into the web crawler, you can use the crawl data from this to benchmark against your own site/s. Currently you can add up to four competitors when creating a project, but you can add more after the project is created.

Competitor Name: Typically use the brand name of the competitor, but feel free to be creative!

URL: Same as the ‘Main URL’ described above, add the competitor’s URL into this field.


Step 4: Save Project

Once you have added the site details, click on the ‘Save Project’ button at the bottom of the page, as shown in the screenshot below:

Save a Project

The charts and data are updated every second.

The process for our crawler is the following:

  1. Find xml sitemap/s
  2. Crawl xml sitemap/s
  3. Build list of URLs from sitemaps
  4. Crawl list of URLs
  5. Add new URLs found during crawl to the list
  6. Continue until all URLs that we can find have been crawled
  7. Analyse data in preparation for reports

If usage is very high you may see that some sites have been queued before crawling.


Step 5: See Project In Manage Projects

Once you click the save button, you will be taken back to the ‘Manage Projects’ page and from here you can see the project you’ve just added.

Click Active Crawls


Related Content

You may also be interested in the guides below that are also within the project management section of our support documentation.