What are Robots?
Robots are the bots that is used in search engines to read and index the webpages. Robots are also called
bots/spider/crawler and with much more names.
Name of Search Engine Robots?
Slurp: Yahoo’s
robot name
Googlebot: Google’s
robot name
Bingbot: Bing’s robot name
What is robots.txt?
Robots.txt, remember it is not “robot.txt”
that correct file name is “robots.txt”. This robots.txt file is used to tell
search engine bots that what pages it should index or what shouldn’t.
Robots.txt is a file that is used to
tell the search engine that the what pages it should index and what shouldn’t.
Robots.txt in SEO?
Robots.txt in Search Engine Optimization is the key part. In general, SEO prefers that the webpages are
indexed by search engines. For this, a document is notepad is created and the
file is saved with the name of “robots.txt” file.
To allow bot to read all the pages of your website.
The code mentioned below will allow
all the pages.
User-agent: *
allow: /
To disallow bot to any pages/directory of your website.
The code below will allow all the
pages except about.asp page
User-agent: *
disallow: /about.asp
Robots.txt file test.
It is better to use webmaster tools.
Search Google webmaster tools and then signup there. This webmaster tools are
much great to learn more about the tools of the webmaster.
Written
by Arsalan Tariq.
