Crawlers (or bots) are used to gather information out there on the internet. By utilizing web site navigation menus, and reading inside and exterior links, the bots start to understand the context of a web page. Of course, the words, images, and other information on pages additionally help search engines like google and yahoo perceive what they’r