What is a Robots.txt File and Why Should I Care?

If you are running a small company, building your online presence should be one of your top priorities.

Contrary to popular belief, your job isn’t done as soon as you manage to roll out your website.

Even after you complete this first important step, various other elements will still demand your full attention in the long run.

Onsite and off-site SEO strategies are only a few factors that you should focus on regularly.

If you can’t afford to hire a SEO consultant, as least try to maintain a close relationship with your webmaster to get unlimited access to important information and get more familiar with concepts like robots.txt.

What is a robots.txt file and how can it impact your website?

Let’s find out.

Everything You Need to Know About Robots.txt

Website owners rely on robots.txt files to give a set of instructions related to their website to web robots.

The whole process depends on REP (Robots Extension Protocol).

Here is how it works: a web robot decides to visit a certain website URL, like for instance http://www.demo.com/example.html.

Before accessing the website, it starts inspecting http://www.demo.com/robots.txt.

Here, the web robot will stumble across

User-agent: *

or

Disallow: /

“User-agent: *” lets the web robot deduce that this particular section is applicable to all web robots.

On the other hand, “Disallow: /” tells web robots that they should not check out any of the webpages comprised by a certain website.

If you plan to use /robots.txt you should know two important things about this tactic.

First of all, web robots can neglect your /robots.txt file.

Email address collectors set in place by spammers and malware web robots that always check websites and scan different pages in an attempt to detect security threats are more likely to ignore your robots.txt files.

Secondly, you should be fully aware of the fact that the robots.txt file is available to the grand public.

In a nutshell, anyone can check out your robots.txt file and determine which sections and what kind of information you are trying to conceal via this tactic.

What we’re trying to say is that you shouldn’t rely on /robots.txt to conceal sensitive data that you may feel the need to include in your web copy.

How Do I Create a Robots.txt File, and Where Do I Put It?

So now you know that robots.txt also known as robots exclusion protocol represents a text file that webmasters can use to instruct search engine robots when it comes to indexing and crawling web pages.

So the next logical question that may be on your mind is this: how can you actually create your own robots.txt file and where should you include it?

The shortest answer to your question is this: robots.txt files should be placed in the top level directories of web servers.

Basically, when a web robot is checking out the robot.txt file for a certain URL, it replaces the URL’s path component with the “/robots.txt” element.

For a better understanding of this phenomenon, let’s analyze the following example.

When it comes to reading the robots.txt file of an URL like “http://www.demo.com/visit/index.html”, web robots will ditch the /visit/index.html section and use “/robots.txt” as its substitute.

In the end, the robot will analyze http://www.demo.com/robots.txt.

In order to make sure that the final URL is fully functional and will be interpreted by search engines robots in a correct manner, it should occupy the right spot on the web server that you are utilizing.

In most cases, this spot may coincide with the place where you or your webmaster have placed the main index.html page of your website.

What Should I Include In My Robots.txt File?

At the end of the day, whether you want to take care of this assignment on your own or would rather let your webmaster do the heavy lifting, you should know that “/robots.txt” file actually constitutes a text file comprising one or more text records that usually look something like this:

User-agent: *
Disallow: /tmp/
Disallow: /cgi-bin/
Disallow: /~joe/

In this case, “Disallow: /” lets us know that 3 directories have been excluded.

There are many good practices that you should pay attention to.

For instance, you can’t put Disallow: /~joe/ and Disallow: /cgi-bin/ should not be placed on the same line.

At the same time, you should make sure that there are no blank lines inside your record, since blank spaces can only be utilized to separate multiple records.

The good news is that you can test your robots.txt files and see if they function properly by simply using Google Webmaster Tools.

Testers will let you know whether or not your robots.txt files are blocking Googlebot from directories or files on your website.

To find all the additional details that you may need on how to use robots.txt to block URLs, just click here and follow Google’s precise set of instructions.

All in all, if your robots.txt files are giving you a hard time, or if you’d want to learn more about website design, development and Google-friendly SEO tactics, just give us a call or write us an email and we’ll offer you the shortest answer to your website-related questions.