Robots.txt is really a file that exists on the foundation directory of each Web site and can be used to instruct search engines like google and yahoo on which directories/data files of the website they will crawl and consist of inside their index. When composing out your approach, look at https://desawisatabunder.com