Robots.txt is often a file that exists on the root Listing of every website and may be used to instruct engines like google on which directories/files of the web site they can crawl and incorporate within their index. You might also consider setting up different pillar pages close to seed https://technik.forum