How to Add a Project
In this guide we explain how to add a project to Raptor’s web crawler software. A project is a container for sites that are related. Typically, projects have a main site and some competitors contained in them.
The number of URLs we must crawl, and the load times of those URLs are the two biggest factors in how long it will take to crawl a project. With that in mind, we typically crawl at around 10 URLs per second per site being crawled.
Step 1: Login
Open a web browser and navigate to:
Once there, login with your Raptor Username and Password:
You can click on the ‘Remember Me’ tick box to save your details for future access on that device.
Then click on ‘Sign In’.
Step 2: Click 'Add Project'
As illustrated in the screenshot below, click on the ‘Add Project’ link in the left-hand side menu:
Step 3: Add Site Details
This will take you to the ‘add project’ page, from here you can add details:
For the main site, you must add the following details:
Project Name: Create a name for your project.
Site Name: Create a name for the site.
URL: Add the complete TLD (Top Level Domain) of the site you wish to add. Include the full URL, for example:
https://raptor-dmt.com/ = Correct
raptor-dmt.com/ = Wrong
Once these have been added, you do not need to complete any other field in order to save and create a project. You can also come back and add any of these details in at a late stage should you wish.
Sitemap URLs: If the site has an XML sitemap with an unusual filename or URL and is not detailed in the robots.txt file, you can add the URL of it here. If you have an index sitemap that lists all other sitemaps, you can add that URL here.
You can also add competitors into the web crawler, you can use the crawl data from this to benchmark against your own site/s. Currently you can add up to four competitors when creating a project, but you can add more after the project is created.
Competitor Name: Typically use the brand name of the competitor, but feel free to be creative!
URL: Same as the ‘Main URL’ described above, add the competitor’s URL into this field.
Step 4: Save Project
Once you have added the site details, click on the ‘Save Project’ button at the bottom of the page, as shown in the screenshot below:
The charts and data are updated every second.
The process for our crawler is the following:
- Find xml sitemap/s
- Crawl xml sitemap/s
- Build list of URLs from sitemaps
- Crawl list of URLs
- Add new URLs found during crawl to the list
- Continue until all URLs that we can find have been crawled
- Analyse data in preparation for reports
If usage is very high you may see that some sites have been queued before crawling.
Step 5: See Project In Manage Projects
Once you click the save button, you will be taken back to the ‘Manage Projects’ page and from here you can see the project you’ve just added.
You may also be interested in the guides below that are also within the project management section of our support documentation.