- sitemap generation on the fly, no additional plugins needed
- submit sitemap to 4 searchengines (google, Bing, ask.com, moreover)
- Automatic priority calculation based on internal PageRank (experimental)
- shows bad links, or not crawlable sites
- with curl: max 250 parallel connections to spider urls (configurable)
- exclude list
- modification of robots.txt with the location of the sitemap
- crawling with curl of fopen
- saves config in a xml-file
There is a new forum and a new dedicated page for JCrawler!
JCrawler is under redeveloppment to create with even more features.
- fopen or Curl is required to crawl the site
thank you for your review, you're the first with this issue. i found the bug and it will be fixed for release 1.7 greets Patrick
support is great, quick, personal and friendly.
the component is getting more and more functions all the time, and i hope it will keep getting better so fast.