Home / robots.txt / Good robots.txt
  • How to Good robots.txt

    good-robots-txt

    This is a good example of a robots.txt to use with your wordpress installation. Bellow the first example is one from smashing magazine. Yes yes I know, I know, not the best robot drawing in the world! have a better one? send it along ill replace my robot with yours.

    ( click code to copy )

    wordpress snippet : robots.txt

    <>
    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    
    ( click code to copy )

    wordpress snippet : robots.txt - smashingmagazine.com

    <>
    Sitemap: http://smashingmagazine.com/sitemap.xml
    User-agent: *
    Disallow: /cc/
    Disallow: /wp-admin/
    Disallow: /tag/
    Disallow: /wp-rss.php
    Disallow: /wp-rss2.php
    Disallow: /*?
    Disallow: /*comment-page
    User-agent: MSIECrawler
    Disallow: /
    User-agent: psbot
    Disallow: /
    User-agent: Fasterfox
    Disallow: /
    User-agent: Xenu
    Disallow: /
    User-agent: SiteSucker
    Disallow: /
    

    ( WordPress codex functions, hooks, in this snippet. )

    wp,