Create robots.txt Tool
Choose a type of web robot to manage by clicking on a tab:
All robots
These options will disallow all robots of any kind.
Disallow these paths:
Web search
Web search robots are spiders from search engines such as Google, Yahoo, Bing and Ask. They index your site for search results.
Disallow these paths:
Image search
These robots are similar to the web search bots except they index the images on your site instead of the content. You should ban these bots if your image content is valuable, for example technical drawings or copyrighted material.
Disallow these paths:
Contextual ads
Contextual ad services like Google Adsense and Yahoo YPN use a robot that looks at your site to determine its topic and choose suitable ads for it.
There are few legitimate reasons for you to want to block these robots.
Disallow these paths:
Web archivers
Web archivers like the Internet Archive (archive.org) use robots that take copies or "snapshots" of your site so that future surfers may view what your site looked like at a specific time.
Disallow these paths:
Bad robots
There are many "bad robots" which serve no useful purpose, including many "data scrapers" and other nefarious activities.
Most bad bots do not obey the Robots Exclusion Standard but a surprising number do.