SEO Tip – Meta Tags – Part 4 – Robots

  • Allows or disallows indexing into search engines by robots or crawlers on a page-by-page basis
  • This is very different from the robots.txt file. The Robots Meta tags will not be seen if the robots.txt file blocks indexing, since the crawler will never get that far.
  • You can find out more about this meta tag at :
  • Here is a list of all the attributes for the robots tag:
    • NOINDEX – prevents the page from being included in the index.
    • NOFOLLOW – prevents crawlers from following any links on the page. (Note that this is different from the link-level NOFOLLOW attribute, which prevents crawlers from following an individual link.)
    • NOARCHIVE – prevents a cached copy of this page from being available in the search results.
    • NOSNIPPET – prevents a description from appearing below the page in the search results, as well as prevents caching of the page.
    • NOODP – blocks the Open Directory Project description of the page from being used in the description that appears below the page in the search results.
    • NOYDIR – sell Yahoo to not use Yahoo Directory information to make a title and/or description for your web page listings
  • Here is a standard sample of a tag that allows all robots to index the page (Note, this is actually the default if the tag was not included on the page)
    • META name=”ROBOTS” content=”INDEX,FOLLOW”
  • Here is a sample of robots tags that are targeted at each of the four major search crawlers:
    • meta name=”TEOMA” content=”NOINDEX”
    • meta name=”GOOGLEBOT” content=”NOARCHIVE”
    • meta name=”MSNBOT” content=”NOODP”
    • meta name=”SLURP” content=”NOFOLLOW”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.