JCrawler does the same as the Google crawler. It crawls your Joomla site and stores all the urls in a XML file without any plugins. After crawling you can ping Google, Ask.com and Bing.
- sitemap generation on the fly, no additional plugins needed
- submit sitemap to 4 searchengines (google, Bing, ask.com, moreover)
- Automatic priority calculation based on internal PageRank (experimental)
- shows bad links, or not crawlable sites
- with curl: max 250 parallel connections to spider urls (configurable)
- exclude list
- modification of robots.txt with the location of the sitemap
- crawling with curl of fopen
- saves config in a xml-file
There is a new forum and a new dedicated page for JCrawler!
JCrawler is under redeveloppment to create with even more features.
- fopen or Curl is required to crawl the site