JCrawler Component

A perfect XML-Sitemap generator.

This component does the same as the Google crawler. It crawls your Joomla site and stores all the urls in a XML file. After the crawling you can submit the XML-file to Google, Ask.com, Moreover, Yahoo and MSN.

Features:

- sitemap generation on the fly, no additional plugins needed
- submit sitemap to 5 searchengines (google, yahoo, msn, ask.com, moreover)
- shows bad links, or not crawlable sites
- with curl: max 200 parallel connections to spider urls (configurable)
- exclude list
- modification of robots.txt with the location of the sitemap
- crawling with curl of fopen
- saves config in a xml-file


Note:
- fopen or Curl is required to crawl the site
- PHP parameter allow_url_fopen=ON, required to submit the XML-file

Now Beta 1.5 is out!

Changes:

- fixed the "administr" issue
- added useragent: "JCrawler" (only with curl)
- the config is now saved in a xml-file
- extended the excludelist feature
- changed the validator link (validome doesnt accept umlaut-signs)
- max. connections is now 200

Report

Thank you for this component. I tested many sitemap generating applications for joomla. Yours helped me to make all that work with one button pressing, without needing to search documentation or help forums. It's time saving application.
bym107 on December 1, 2008
thank you
JCrawler do all you want for you site map.
i had problems with others but this one satisfied me.

thanks in advance
byZatriX on November 14, 2008
thanks for a much needed sitemap component. As intuitive as they get, yet very valuable.

Still have to see about them submissions, though ;-)
byyandos on November 14, 2008
I really enjoyed using this component. it develops fast and might be the next generation in sitemap creation. no plugins. no fuss...

support is great, quick, personal and friendly.

the component is getting more and more functions all the time, and i hope it will keep getting better so fast.
Tried a couple of sitemap generators, a few seemed to work but google subsequently complained about formats etc. JCRAWLER WORKED FIRST TIME. Google liked it, so I like it.
Very clean page with configurable options.

Top comp and thank you very much.
Absolutely the easiest sitemap component I've ever used. The automatic submission to search engines is nice too, although time will tell how well that actually works.
byhiddink on October 14, 2008
Very good plugin, a must have in my opinion. Would be nice for future versions to give different priorities to sections or categories.
Owner's reply

yes, you're absolutly right. actually im developping an algorythm for finding the priorities.

bymustardstache on October 14, 2008
Easy install, does what it says. I was planning on spending all morning generating and submitting sitemaps to different places but with this tool I submitted to the top 5 in about 5 minutes.
bynoodlema on September 29, 2008
This works great. Easy as butter. I just ran into one glitch as one of my sites and I doubt it is related to the extension: In the Adapt to your site area my "HTTP host (readonly)" is coming up with a subdirectory that doesn't even exist, this is causing an error in the subsequent sitemap.xml file and errors within Google. Any ideas?
bygreg716 on September 24, 2008
Excellent extension! Does exactly what it's supposed to do, dead-simple install, runs quickly, even submits the sitemap for you to google, yahoo, msn, and 2 others! It actually took me longer to get my account registered here to be able to write this review than it did to install and run the extension! Highly recommend!
byJoeHana on August 31, 2008
Very nice, and helpful component. Was really easy to installand you dont even have to configure it much - just click start - generate and submit.

Looks really clean and does his Job.

Well done
After trying a couple different extensions for creating Google sitemaps with little real success, I came across this little gem. It wasn't rated yet so I figured I better take care of that because it does exactly what it says. Other sitemap components generated files with errors, but not this one. You can even validate the sitemap from the admin panel. Installed in a snap and we were up and running with the Google sitemap. I did notice that there is an option to link to JCrawler from the Joomla menu items, but that seems to just throw an error, so I'm not sure what that is for, but who cares...
byozwest on July 14, 2008
nifty little tool. It installed with an error but the component was there, so I tried it and it generated a complete sitemap which is great. I also uploaded but I do not know if sometings is going to happen.

Thanks