A web crawler, also known as a web spider or search engine bot, is a bot that visits and indexes the content of web pages all over the Internet. A search engine will be able to present its users relevant information in the search results.