The Joomla! Extensions Directory ™


JCrawler Component

JCrawler does the same as the Google crawler. It crawls your Joomla site and stores all the urls in a XML file without any plugins. After crawling you can ping Google, Ask.com and Bing.

Features:

- sitemap generation on the fly, no additional plugins needed
- submit sitemap to 4 searchengines (google, Bing, ask.com, moreover)
- Automatic priority calculation based on internal PageRank (experimental)
- shows bad links, or not crawlable sites
- with curl: max 250 parallel connections to spider urls (configurable)
- exclude list
- modification of robots.txt with the location of the sitemap
- crawling with curl of fopen
- saves config in a xml-file

There is a new forum and a new dedicated page for JCrawler!

JCrawler is under redeveloppment to create with even more features.


Note:
- fopen or Curl is required to crawl the site

Report Extension

 

 

2012-01-20
Reviews: 2
JCrawler able to build a sitemap, even for sobipro which failed XMAP. Excellent, bravo to the developers!
2012-01-10
Reviews: 3
I tried the other big ones and I was very frustrated. I wanted to do one simple thing which was exclude some links which this let me do very easily. If you just want a nice sitemap for content articles this is a great extension.
2011-12-05
Reviews: 6
This is a great extension that enables you with:
+ 1 click create an xml sitemap and the 2nd click submit to google, bing and ask.com

Saves time and is much better than XMAP
2011-12-01
Reviews: 2
Installation was easy, no problems.

I used standard settings and after few minutes I got "Internal Server Error" page. When I checked my CPU usage on a hosting server, it reached quickly max allowed limit, hold for a while, then goes to zero and then up again. So far I noticed, it looks like crash because of lack of CPU power.

So I tried again but with lower settings "Max parallel connections" from 50 to 30 (I know it's about RAM not CPU but thought that with less connection CPU will be not so overwhelmed), result the same, too hight CPU usage, everything crashes for a while, the same CPU usage pattern.

Again I lowered settings. This time: priority to 0.3, Max parallel connections to 5. Result: an error site with: "Fatal error: Cannot pass parameter 2 by reference in /home/xxxxx/public_html/administrator/components/com_jcrawler/admin.jcrawler.php on line 189. CPU usage again hits maximum and crashes. Memory usage is ok, far from reaching limits.

I checked FTP and did find sitemap.xml but given above, I don't trust it is done well or till the end.

I have a payable hosting account and it suppose to be one of the best in given area, yet it can't handle JCrawler. So either hosting is overrated or JCrawler simply works poorly.

Thanks, but I am deinstalling JCrawler, doesn't work for me :(.
2011-11-16
Reviews: 1
Very nice module, but I was surprised not to see other search engines like yahoo, AOL and some others. But in general great component.

Thx a lot
Owner's reply

Hey,

Thanks for your review.

Yahoo's search results are powered by Bing (earlier MSN) from Microsoft and AOL's search results are comming from Google.

Jcrawler covers both search engines to submit to, maybe you got other ideas?

Greets Patrick

2011-10-28
Reviews: 3
The xmap stuff literally destroyed my site, I did not have access to my own front end site. I downloaded this, installed and 30 seconds later the sitemap was submitted! Wow! Fantastic, this does what it´s suppose to do.
2011-10-27
Reviews: 1
I've tested several other methods to produce sitemaps. This is the absolute winner with several advantage rounds to the rest. If you can't get it to work no doubt it's your fault, but their excellent support will be willing to help.
Also, this is the only one which actually crawls your site, producing a complete sitemap. Extremely useful if you want to clean your sh404SEF database and rebuilt it with the proper URL.
2011-10-23
Reviews: 11
There are some extensions which EVERY project should have. For me these include JCE, Abivia autocopy, PlaceHere and now JCrawler. Talk about making life easy. This is particularly useful when you have a HUGE site with a stack of data. Life is SO much simpler with this extension.

Simply a "Must Have" extension.
2011-10-20
Reviews: 6
after installing this component I thought "easy as pie", click 'start' and that's it. But it made my complete website crash. No FTP, No Email, no WWW, nothing.

Now, hours later I'm still hoping that my provider can fix this for me as there is nothing I can do myself from here.

NO component should be able to have such drastic consequences for a website, so that's clearly 'very poor'.
Owner's reply

Hy,

Thanks for your review.

If you choose too many parallel connections the webserver can be overloaded/going down.

But JCrawler does not delete/change/add any files (except of the sitemap.xml)/MySQL data or E-Mails. It just reads every page of your Website and looks for links. Afterwards it writes the links into an file. That's it.

Actually i'm developing JCrawler 2.0, maybe you can get in touch in our forum to resolve your issues.

Thanks, Patrick

2011-09-02
Reviews: 6
Works like a charm, nothing to say about it really.
Except don't forget to look for the latest release.
1.8 fixes a conflict with sh404sef.
great support from the forum.
good job
2011-08-29
Reviews: 1
Had some issues with XMAP on 1.5 native. Came here, snagged this component and had my sitemap created and submitted and confirmed to 4 major search engines in seconds. Nicely done.
2011-08-16
Reviews: 3
I love this small app.All we have to do is just click the button it will do the rest.we dont have to make any changes. i really love this app
2011-07-04
Reviews: 1
I have installed this "little app" it works perfectly. How ever I think it should be made to submit to more than just 5 search (google,msn,ask,yahoo and moreover")
I would suggest the developer make provision for added more search engine.
In all a good work. Thumbs Up.
2011-05-19
Reviews: 10
Ideal for a newbie like me, very little to set up, easy to read and understand.

Thank You.
2011-04-21
Reviews: 1
Great component!!! THANK YOU!!! I´m now waiting for the plugin, which recreates the sitemap after adding new content.
2011-02-01
Reviews: 1
Hi
I use several sitemap extensions but it is different.
special thank to developers.

Regards
Amirhosein Soltani
2010-08-05
Reviews: 1
Simple to set up and use. A great timesaver.

The only thing I seem to be having a problem with is my robots.txt file. I have pages blocked by the robots file but Jcrawler is still including them in the sitemap. No real problem at the moment as they are easy to find and I can delete them manually.

The file is readable and while the line:
Disallow: /index.php?*page=shop.ask*

Works in Google it doesn't in the component.

Thanks
2010-07-30
Reviews: 1
Hi,

I have installed JCrawler on my joomla site, but I see the sitemap.xml file wasn't updated since long time. Isn't this JCrawler runs automatically and update sitemap.xml file? What can cause my sitemap.xml to not get updated?

Any help is greatly appreciated.

Thanks,

Samir Patel
Owner's reply

...the sitemap.xml is not get updated automatically.

But i'm working on a plugin, which recreates the sitemap after adding new content.

Patrick

2010-07-10
Reviews: 31
Hi,
it's easy to use. Thanks for doing and sharing this.
Regards
Rudolf Aigner
2010-07-08
Reviews: 7
It was never so Easy to create Sitemaps for Google. Needs no Plugins for 3th-party-extensions.
Theres just a little wish: Please provide the possibility to start the crawling from the frontend.
Page 2 of 4