Any website when get crawled by search engine spider, most often it initiate with robot.txt file present in root domain file. After identify that robot.txt it reads the file in order to identify all the files and directories which could be blocked.
The robot.txt file exists in your root directory and it is a very simple file in form of .txt... its path may be defines simply as e.g www.anydomainname.com/robot.txt. robot.txt file allows the search engines and other robots to visit your website and index it. Always all the spambot of search engines do not make it respectable but all other unique search engine spiders make this respectable and work on these robot.txt file. For the security purpose one should have to put your all file within a protected directory in place of putting only one robot.txt. With the help of robot.txt your website visits could also be controlled for various reasons.
Firstly robot.txt prevents the visit of those pages which are of no use, Its also gives protection when any one if include robot.txt not to access your files through search engines they could not do this if you do not want them to visit because they try to enter into your directories directly, which gets typical in presence of robot.txt.
Robot.txt also cleans the logs and cookies all time whenever your website visited by search engine. And if the files robot.txt in not there then its starts generating "404 not found" errors. Robot.txt also protects website from duplicate contents spam. which will increase your website speed also. Robot.txt can also protect your website indexing. It is a good work to include robot.txt in your directory. Many developers use this file but some do not feel any need of such files google webmaster tool cannot be possible to get without it, if you want to validate your website in google then its necessary to include robot.txt…
ITS VERY NECESSARY TO HAVE ONE ROBOT.TXT WORKING ON YOUR DIRECTROY.
Robot.txt mainly works as a medium which tell search engines about those parts of websites which you want not to visit by search engines. Its very profitable in such cases. Websites could be prevented by robot.txt to get indexed accidently.
Sometimes error generating when one had done any updation on their website but the robot.txt is not being updated, it causes the error appearance in search engines. Its resolved by recreating your robot.txt file and check it everytime whenever you do any major changes on your websites.