Potential reasons for Sites Crawled But Not Indexed - search engine optimization tip - march 15th 2005
Written by power
@ 3:14 AM
permanent link on
Wednesday, April 13, 2005
| Post a Comment |
For a more detailed list of search engine crawlers contact us and we will send it to you for free,
Many of us have noticed sometimes search engines keep visiting the site regularly but dont index the site nor show it in the site: command,
There are various reasons for this to happen,
1. The domain is an expired domain, if the domain expires and not registered for a certain period of time google imposes a expired domain penalty on that domain, That domain is left to suffer for certain number of months, in that period googlebot keeps visiting the site but they don't index it and they don't show the site in site: command too,
2. An other reason is the domain is a new domain and if the domain is a new domain sometimes the crawler regularly visits the site and it doesn't show up in the index for a long time, There is nothing wrong with this, probably google index is taking longer time to expand, you just have to wait till google updates its index,
3. An other possible reason is the site is banned from the search engines for any particular onpage factor, In that case search engines periodically checks to see whether the onpage spam tactics is removed and as soon as they see the spam being removed they might reinclude the site into the index, So for people who are complaining that their site was previously indexed and listed in google but suddenly it disappeared from the index and googlebot keeps visiting the site it is good to look at your onpage work and see if there is any spam tactics like hidden text, cloaking, keyword stuffing etc,
4. An other important reason could be that the site was permanently banned from the search engines, Even here search engine crawlers visit the site following existing links to the site but they don't index the site because the site is banned, This is common with Yahoo slurp yahoo's robot, yahoo slurp is known to visit the site and don't index the site if the site is banned,
Many of us have noticed sometimes search engines keep visiting the site regularly but dont index the site nor show it in the site: command,
There are various reasons for this to happen,
1. The domain is an expired domain, if the domain expires and not registered for a certain period of time google imposes a expired domain penalty on that domain, That domain is left to suffer for certain number of months, in that period googlebot keeps visiting the site but they don't index it and they don't show the site in site: command too,
2. An other reason is the domain is a new domain and if the domain is a new domain sometimes the crawler regularly visits the site and it doesn't show up in the index for a long time, There is nothing wrong with this, probably google index is taking longer time to expand, you just have to wait till google updates its index,
3. An other possible reason is the site is banned from the search engines for any particular onpage factor, In that case search engines periodically checks to see whether the onpage spam tactics is removed and as soon as they see the spam being removed they might reinclude the site into the index, So for people who are complaining that their site was previously indexed and listed in google but suddenly it disappeared from the index and googlebot keeps visiting the site it is good to look at your onpage work and see if there is any spam tactics like hidden text, cloaking, keyword stuffing etc,
4. An other important reason could be that the site was permanently banned from the search engines, Even here search engine crawlers visit the site following existing links to the site but they don't index the site because the site is banned, This is common with Yahoo slurp yahoo's robot, yahoo slurp is known to visit the site and don't index the site if the site is banned,
2 Comments:
Slurp only pulls the homepage and robots.txt if you're banned
I also have a site that has been crawled a few times but not yet indexed.
I dont think i have any spamming in my site and my code is validated. I hope it is just a timing issue. Wish me luck!
Cheers,
David.
BigRedBall website design, Birmingham
Post a Comment
Links to this post:
Create a Link
<< SEO Blog Home