- sitemap generation on the fly, no additional plugins needed
- submit sitemap to 4 searchengines (google, Bing, ask.com, moreover)
- Automatic priority calculation based on internal PageRank (experimental)
- shows bad links, or not crawlable sites
- with curl: max 250 parallel connections to spider urls (configurable)
- exclude list
- modification of robots.txt with the location of the sitemap
- crawling with curl of fopen
- saves config in a xml-file
There is a new forum and a new dedicated page for JCrawler!
JCrawler is under redeveloppment to create with even more features.
- fopen or Curl is required to crawl the site
this is very good extention like very much. I tried on small page it worked excellent. But I have bigger portal there was fatal error.
I tried to crawl but memory was not enough, so how will I solve it I do not know.
I really like this extention and want to use on my portal.
I have yet to see any of the other search engines make an appearance... but nevertheless google works fine for me!
There was 1 tiny glitch, if you want your site map to perfectly pass validation you will want to switch the following lines of code...
[review system strips tags out from code; code removed]
NOTE: This is addressing release 1.7Beta. Depending on when you read and download this extension this may have been addressed. Check the file against the above before and after to see.
Thanks for your cool review!
The validation issue is fixed in the latest build available on the jcrawler website.
Soon I'll release the next version!
Of course the sitemap.xml file that is generated is very empty though.
Hope it will work with PHP, because this is what i am looking for.
Hey Morten, thx for your review.
Unfortunatly has JCrawler in version 1.7 a problem with relative links. So thats why your sitemap is empty. It will be fixed in version 1.8
Thank you! The next version will be more compatible with different PHP versions/configurations.