An online journal to share my personal journey in technology, photography, and travel. Opinions expressed within this blog are solely my own and do not represent the views or opinions of people, institutions, or organizations that I may or may not be associated with in a professional or personal capacity.
SEO Tip – Use robots.txt file
Robots.txt files tell Search Engines what should and should not be crawled
NOTE – This is very different from the Robots Meta Tag. The crawler will see this file before it tries to call the page, so this file will override the Robots Meta tags on the pages.
Robots.txt files should be stored in the root directory
Remember, the point of the robots.txt file is to exclude pages from being crawled. So if a page or directory is banned, it will never even get to see what code is on those page(s). Accordingly, no code of those pages could change the bots behavior to re-index the page. So, robots.txt will overwrite the meta and robots tags on the page.