So, how does one Increase Search Engine rankings, bring in increased Search Engine Traffic and also prevent duplicate content on your blog or website using a single robots.txt file?
What is a robots.txt file?
robots.txt protocol also called the Robots Exclusion standard is used by web spiders and other web robots such as Search engine robots, blog robots, splog robots, adsense robot etc and prevents them from accessing all or part of a blog or a website which is otherwise, publicly viewable.
By placing a robots.txt file on your blog or website you are telling the specified robots to ignore and prevent their indexing of the specified files or directories.
Why is the robots.txt important for my blog?
Search Engines penalize duplicate content by placing that particular post or in some instances an entire blog or website in their supplemental index. Also known as Search Engine Hell, if your post goes into the supplemental index of a search engine, it might take almost 6-8 months for it to come out into the regular index and that too after you removed the duplicate content from that particular post.
Okay, so a Robots.txt file might prevent duplicate content. How does it increase my search engine rankings?
Simple, having a clean blog makes search engines think that you do not host spammy content and if you write good content, the robots will keep coming and index your site everyday instead of once every week. Also by preventing duplicate content, you prevent articles from going into the supplemental index which in turn means that people can find that post when they search on Google or any other search engine, which in turn means you get more traffic! Prominent blogger Neil Patel wrote on his Link Building blog that by using a robots.txt file, his web site traffic went up by 11.3%.
What content does one need to prevent the search engine robots on a website from accessing using a robots.txt file?
a. We need to deny the search engine bots and robots from accessing our WordPress folders or if you have a Movable type blog, your movable type folder. For example, in the picture below, you can see that Google is trying to access a file within my WordPress blog folder and throwing out a 403 (not found) error. This is bad

b. Remove comment feeds from the search results. You should never block the robots from accessing comments on your posts, but block the search engines from accessing your comment feeds as this would result in duplicate content.
Read more »
If you liked this article, click here to buy me a coffee!Popularity: 15% [?]