
As you know, the majority of the webmasters upload a file called robots.txt to their servers in order to give instructions to the crawlers like Google, Yahoo, Bing... about what pages mustn't be indexed.
Example: http://behindthefirewalls.blogspot.com.es/robots.txt
Why does the webmaster want to hide some URLs? One of the first things the hackers...