Comments on: WhatDoTheyKnow growing pains (and Ruby memory leaks)

By: Francis Irving

Francis Irving — Thu, 11 Feb 2010 00:51:02 +0000

I decided to try Ruby on Rails, mainly because I hoped it would make it easier for other people to contribute code. This is because it has a standard file structure, and way to run the code.

PHP/MySQL applications are very hard for people to configure and get going. e.g. No standard way for database migrations etc.

Yes, I could have used Cake or Django – but 2.5 years ago they weren’t as obvious a choice as trying Rails. I think it was a good choice. I’ve learnt a lot about frameworks from using Rails. I don’t think it is especially good, but then again I haven’t found anything I think is better (Django is in some ways, not in others).

In practice, it hasn’t created more code contributions. Partly the Rails app is still hard to install (getting example data, setting up Xapian and various other bits). Partly it is just had to get code contributions to web applications.

By: Mathew Anderson

Mathew Anderson — Wed, 10 Feb 2010 22:22:46 +0000

Is it possible to give the reason for using Ruby for this DB ? for example could you not have used php mysql ?
Was there a special reason for using Ruby ? for a production site as this.

By: Michael Mullany

Michael Mullany — Tue, 06 Oct 2009 16:55:20 +0000

Francis,

You should read:

http://www.engineyard.com/blog/2009/thats-not-a-memory-leak-its-bloat/

And definitely reach for New Relic or Scout.

– michael

By: Francis Irving

Francis Irving — Wed, 30 Sep 2009 23:12:00 +0000

Thanks Charles for that vote for trying JRuby. I think I’m unnecessarily put off because I’m not used to Java (from the days when it wasn’t open source, so didn’t have nice Debian packages).

I’m also worried that it’ll be hard to set up an environment as nice as mod_ruby (e.g. running Rails as a particular Unix user).

I’ll give it a go if I get stuck again in future!

By: Charles Oliver Nutter

Charles Oliver Nutter — Sat, 26 Sep 2009 20:46:23 +0000

You really should try JRuby just to see if it solves your memory issues. We make much more efficient use of memory than the standard implementation, and there are sites running JRuby on multi-GB heaps without any issues.

By: Francis Irving

Francis Irving — Fri, 18 Sep 2009 14:53:52 +0000

Dave – I agree that the weak Ruby virtual machine was a large cause of my troubles. I’m not totally convinced the debugging tools for Perl, Python or PHP are substantially better than Ruby, even if they are slightly better. Certainly we had lots of trouble in the early days with, say, FastCGI and those languages.

Perhaps what I was trying to say was that things could be better organised across LAMP-style stacks in general. At the moment for such stacks there are lots of adhoc programs wired together that you can sort of debug things with. I’d like a stable platform that tells you simply exactly the information you need to know.

Peter – We’re using Xapian and acts_as_xapian. I wrote the latter for WhatDoTheyKnow, because acts_as_solr wasn’t good enough. Details in this blog post http://www.mysociety.org/2008/07/17/acts_as_xapian/ – the Ruby Xapian bindings did cause problems with Passenger’s spawning model, which we had to make conservative. It doesn’t seem to cause memory trouble though (Xapian doesn’t allocate much memory, it uses memory mapped files and relies on the kernel for in-RAM speed caching).

Dan – Django was not as clearly good 2 years ago when we started writing WhatDoTheyKnow, although it is what I’d use if I started again now. Rails was genuinely innovative, and well marketed with a community that has lots of good things about it. On the other hand I think Ruby, Rails, and 3rd party libraries and tools are in general much more poorly software engineered than the Python equivalents.

Obviously we’re not going to rewrite it now. That would be a crazy waste of time (second systems effect). It might look like a simple website, but inside it is surprisingly complicated – there is lots going on that happens invisibly. And lots and lots of slowly accumulated knowledge now embodied in the source code.

By: Dan

Dan — Fri, 18 Sep 2009 14:37:42 +0000

1 Comment: Dump Rails, try Django.

By: Peter Morris

Peter Morris — Fri, 18 Sep 2009 10:29:51 +0000

Are you using ferret for the search side?
I have seen that ferret can be very memory hungry and using the ferret gem in a rails app can make it bloat out.

By: Dave Page

Dave Page — Fri, 18 Sep 2009 09:08:58 +0000

I’m not sure why you claim this is a disaster for Perl, Python, PHP and other free software languages…

By: Francis Irving

Francis Irving — Thu, 17 Sep 2009 20:42:00 +0000

One thing I should add that we did early on... Lots of the traffic was, as ever, search engine robots. To try and minimise problems for real uses, we slowed down Google's bot with webmaster tools and added a Crawl-delay to robots.txt to slow down bing's bot. I also added lots of pages to robots.txt, to stop search engines crawling them. These were pages that had no real new content, usually because they were pages for performing an action. e.g. the page to add an annotation to each requset (/annotate/*). I'm just undoing the crawl delays now...