What are Robots?
Robots are the bots that is used in search engines to read and index the webpages. Robots are also called bots/spider/crawler and with much more names.

Name of Search Engine Robots?
Slurp: Yahoo’s robot name
Googlebot: Google’s robot name
Bingbot: Bing’s robot name

What is robots.txt?
Robots.txt, remember it is not “robot.txt” that correct file name is “robots.txt”. This robots.txt file is used to tell search engine bots that what pages it should index or what shouldn’t.
Robots.txt is a file that is used to tell the search engine that the what pages it should index and what shouldn’t.

Robots.txt in SEO?
Robots.txt in Search Engine Optimization is the key part. In general, SEO prefers that the webpages are indexed by search engines. For this, a document is notepad is created and the file is saved with the name of “robots.txt” file.
To allow bot to read all the pages of your website.
The code mentioned below will allow all the pages.
User-agent: *
allow: /

To disallow bot to any pages/directory of your website.
The code below will allow all the pages except about.asp page
User-agent: *
disallow: /about.asp

Robots.txt file test.
It is better to use webmaster tools. Search Google webmaster tools and then signup there. This webmaster tools are much great to learn more about the tools of the webmaster.
Written by Arsalan Tariq.