Create robots.txt Tool

When you're done, copy and paste this into a file called robots.txt and upload it to the root of your website. More info is available at The Web Robots Pages.

Choose a type of web robot to manage by clicking on a tab:

All robots

These options will disallow all robots of any kind.

Disallow these paths:

    Web search

    Web search robots are spiders from search engines such as Google, Yahoo, Bing and Ask. They index your site for search results.

    Disallow these paths:

    Image search

    These robots are similar to the web search bots except they index the images on your site instead of the content. You should ban these bots if your image content is valuable, for example technical drawings or copyrighted material.

    Disallow these paths:

      Contextual ads

      Contextual ad services like Google Adsense and Yahoo YPN use a robot that looks at your site to determine its topic and choose suitable ads for it.

      There are few legitimate reasons for you to want to block these robots.

      Disallow these paths:

        Web archivers

        Web archivers like the Internet Archive (archive.org) use robots that take copies or "snapshots" of your site so that future surfers may view what your site looked like at a specific time.

        Disallow these paths:

          Bad robots

          There are many "bad robots" which serve no useful purpose, including many "data scrapers" and other nefarious activities.

          Most bad bots do not obey the Robots Exclusion Standard but a surprising number do.

          Disallow these paths: