The Joomla! Extensions Directory ™


JCrawler Component

JCrawler does the same as the Google crawler. It crawls your Joomla site and stores all the urls in a XML file without any plugins. After crawling you can ping Google, Ask.com and Bing.

Features:

- sitemap generation on the fly, no additional plugins needed
- submit sitemap to 4 searchengines (google, Bing, ask.com, moreover)
- Automatic priority calculation based on internal PageRank (experimental)
- shows bad links, or not crawlable sites
- with curl: max 250 parallel connections to spider urls (configurable)
- exclude list
- modification of robots.txt with the location of the sitemap
- crawling with curl of fopen
- saves config in a xml-file

There is a new forum and a new dedicated page for JCrawler!

JCrawler is under redeveloppment to create with even more features.


Note:
- fopen or Curl is required to crawl the site

Report Extension

 

 

2010-07-08
Reviews: 7
It was never so Easy to create Sitemaps for Google. Needs no Plugins for 3th-party-extensions.
Theres just a little wish: Please provide the possibility to start the crawling from the frontend.
2010-06-17
Reviews: 3
a really great component. simple, not much work, effective.

thumbs up!
2010-06-08
Reviews: 9
This is a great component, one of my favourites. I think that any serious webmaster should have it. It is especially good when you set up a new website and you wan to spread the word via the search engines, it does the job!
Thank you to the developer.
2010-06-03
Reviews: 1
This is such a great extension! However, my site really needs a plug in for Zoo component as its the majority contents! Does anyone have xmap plugin for Zoo 2.0.1?
2010-05-04
Reviews: 4
Worked first time, created every single link on the site whereas xmap would only create the menu links for some reason. This is a great little extension. Well done to the developer, clean, neat and tidy xml file produced. That to me is the BEST part, it produces a physical XML file and not a lengthy component link name.
2010-04-08
Reviews: 47
I just can't be unfair while rating this tool, however i've not been successful to implement it. it seems to just sit there and keep crawling. Is it because my site is very large, it host 21368 links as of now.

May be i am having hard luck :(
2010-03-20
Reviews: 4
Hi,
this is very good extention like very much. I tried on small page it worked excellent. But I have bigger portal there was fatal error.
I tried to crawl but memory was not enough, so how will I solve it I do not know.
I really like this extention and want to use on my portal.
Thank you.
2010-03-10
Reviews: 1
This was extremely easy to install and set up. Sitemap was submitted to the search engines within minutes and Google crawled all the site pages the same day.

I have yet to see any of the other search engines make an appearance... but nevertheless google works fine for me!
2010-01-14
Reviews: 4
Worked like a charm! Real easy setup, generated sitemap for a small site in a minute and submitted to google and all! great extension.
2009-09-26
Reviews: 7
This works great
2009-09-17
Reviews: 1
This is a first rate component! Creates a beautiful xml site map. You can easily specify what you want the crawler to ignore and with 1 click of the mouse it will submit your site map to selectable list of major search engines. This will be standard in all of my builds from now on.

There was 1 tiny glitch, if you want your site map to perfectly pass validation you will want to switch the following lines of code...

[review system strips tags out from code; code removed]

NOTE: This is addressing release 1.7Beta. Depending on when you read and download this extension this may have been addressed. Check the file against the above before and after to see.
Owner's reply

Thanks for your cool review!

The validation issue is fixed in the latest build available on the jcrawler website.

Soon I'll release the next version!

2009-08-28
Reviews: 1
Easy to install, easy to use. Makes a great sitemap.xml, and what a great logo.
2009-07-30
Reviews: 1
You install it, do a slight bit of config (had to make a directory and file writable) and then you run it. It's the way software should be built.

Thanks for writing this plugin.
2009-07-18
Reviews: 2
simly installed. simply customized. simply executed. perfect results. what to add? only one more thank you for the sitemap submission feature.
2009-06-17
Reviews: 1
No further words needed before the title.
Made my sitemap in few secconds, much better than the most used tools arround the web.

Congratulations.
2009-06-06
Reviews: 10
I recently tried this on my site of 1500+ links and had no issues whatsoever.

Note: It did take about 5 minutes to crawl my whole site (as I held my breath). But, 5 minutes later, I was a happy camper.

Thanks for a great addition!
2009-05-29
Reviews: 20
Engines couldn't find my website, but thanks to this component this problem is now solved. Even the Beta worked fast and made the job easy. Thank you!
2009-05-20
Reviews: 12
This seems like a very useful extension. I have only one problem / question.

Is it normal for my sitemap.xml to have just 7-8 URLs ? I mean, I have a lot of articles on my website but only a few URLs.
2009-05-14
Reviews: 4
Since i am not on php4 i cannot say an opinion on that, but i am on PHP5, and what is confusing is that it installs fine, and also it crawls and says everything is a sucsess.

Of course the sitemap.xml file that is generated is very empty though.

Hope it will work with PHP, because this is what i am looking for.

Morten
Owner's reply

Hey Morten, thx for your review.
Unfortunatly has JCrawler in version 1.7 a problem with relative links. So thats why your sitemap is empty. It will be fixed in version 1.8

Greets Patrick

2009-04-07
Reviews: 3
Nice component! Unfortunately it does not works for me with PHP5 enabled. But if I disable and switch back to PHP4.x, the sitemap generation works like a charm!
Owner's reply

Thank you! The next version will be more compatible with different PHP versions/configurations.

Page 3 of 4