McDonald Pontiac Cadillac GMC

Mid-Michigan's Bigger Dealer With Better Deals

A robots.txt file provides search engines with the necessary information to properly crawl and index a website. Search engines such as Google, Bing, Yahoo, etc all have bots that crawl websites on a periodic basis in order to collect existing and/or new information such as web pages, blog articles, images, etc. Once these resources are published via the website it is up to the search engines to determine what will be indexed.

A robots.txt file can help you better define what you want the search bots to crawl and therefore index. Doing this is useful for a variety of reasons including controlling crawl traffic to help ensure that the crawler does not overwhelm your server. The robots.txt file, however, should not be used to hide web pages from Google search results.

Views: 7


You need to be a member of McDonald Pontiac Cadillac GMC to add comments!

Join McDonald Pontiac Cadillac GMC


Jeff Simonton created this Ning Network.

© 2021   Created by Jeff Simonton.   Powered by

Badges  |  Report an Issue  |  Terms of Service