Step 1.10: Robots
Core Concepts | Search Engines | Robots, Crawlers & Spiders
Also called bots, crawlers, web crawlers, & spiders.
These are programs that crawl web pages and collect data from them or about them. Robots, like Google’s ‘googlebot’ collect hundreds of pieces of information about every single web page that they visit.
It’s not only Google that use these, all search engines use crawlers or robots to collect data.