- Allows or disallows indexing into search engines by robots or crawlers on a page-by-page basis
- This is very different from the robots.txt file. The Robots Meta tags will not be seen if the robots.txt file blocks indexing, since the crawler will never get that far.
- You can find out more about this meta tag at :
- Here is a list of all the attributes for the robots tag:
- NOINDEX – prevents the page from being included in the index.
- NOFOLLOW – prevents crawlers from following any links on the page. (Note that this is different from the link-level NOFOLLOW attribute, which prevents crawlers from following an individual link.)
- NOARCHIVE – prevents a cached copy of this page from being available in the search results.
- NOSNIPPET – prevents a description from appearing below the page in the search results, as well as prevents caching of the page.
- NOODP – blocks the Open Directory Project description of the page from being used in the description that appears below the page in the search results.
- NOYDIR – sell Yahoo to not use Yahoo Directory information to make a title and/or description for your web page listings
- Here is a standard sample of a tag that allows all robots to index the page (Note, this is actually the default if the tag was not included on the page)
- META name=”ROBOTS” content=”INDEX,FOLLOW”
- Here is a sample of robots tags that are targeted at each of the four major search crawlers:
- meta name=”TEOMA” content=”NOINDEX”
- meta name=”GOOGLEBOT” content=”NOARCHIVE”
- meta name=”MSNBOT” content=”NOODP”
- meta name=”SLURP” content=”NOFOLLOW”
SEO Tip – Meta Tags – Part 4 – Robots
Leave a reply
