Dear developers,
The website is pretty much back to speed. Some help from OSUOSL was vital in understanding so a big thanks to them.
What went wrong was a couple of points. The first was the mo file issue where django was trying to open files from the disk hundreds of times per request (and failing). Provisioning it with the files stopped that.
Then there was the caching. I installed memcached on the server and enabled it's use. This cut down on the amount of memory because wsgi processes are able to share the same cache and reduced mysql load because we didn't have to cache things there any more.
I put the number of parallel requests back up to 4, since 2 doesn't really help with all requests and created some backlog.
Please continue to test. You will notice gallery pages, user pages are slower than cms pages and this is mostly because those apps don't use cache (something to look into for the next version).
Sorry for the disruption.
Martin,