Robots.txt file for Search Engines
What are Robots? Robots are the bots that is used in search engines to read and index the webpages. Robots are also called bots/spider/crawler and with much more names. Name of Search Engine Robots? Slurp: Yahoo’s robot name Googlebot: Google’s robot name Bingbot: Bing’s robot name What is robots.txt? Robots.txt, remember it is not “robot.txt” that correct file...