robots.txt

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt