Page 1 of 1
Robots.txt
Posted: Tue Nov 21, 2006 9:35 am
by Will C
I dont want any of the pages to be blocked from the Search Engines. Do i need to place a blank robots.txt file
please advise
thanks
Hi
Posted: Fri Feb 02, 2007 9:03 pm
by razitun
SE can't block any of your page until you put some scamming contents on your page... so, why do you want to put rebot.txt file on your server?
Posted: Mon Mar 26, 2007 8:12 pm
by siteadmin
Blank text file saved as robots.txt is recommended since every search engine first searches for your robots.txt before crawling your site.
Also from time to time your robots.txt file will be fetched by crawlers to prevent search engines from hitting 404s on your site we recommend adding a blank robots file to your root folder
Posted: Fri Apr 13, 2012 12:13 pm
by lupinesteve
The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are absolutely different to your absolute web website content. Also, This gives you a little more control over the robots.
Posted: Mon Apr 16, 2012 6:33 am
by susan_jane
Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.Google specifically follows instructions given in this robots.txt file.
Posted: Fri Apr 20, 2012 10:10 am
by heenakapoor
Google doesn't read blank pages. So You can inform to Google by use the code of allow all URLs for crawling in robots.txt file. It helps to crawl all pages of your website in Google.