The Joomla! Extensions Directory ™

CraftyJon

Reviews(1)
 
byCraftyJon, August 5, 2010
JCrawler
Simple to set up and use. A great timesaver.

The only thing I seem to be having a problem with is my robots.txt file. I have pages blocked by the robots file but Jcrawler is still including them in the sitemap. No real problem at the moment as they are easy to find and I can delete them manually.

The file is readable and while the line:
Disallow: /index.php?*page=shop.ask*

Works in Google it doesn't in the component.

Thanks