I've been continuing to investigate the problems we've been having with the website. The root issue is that we get so many hits to our website that it is overwhelming apache. Apache has been hitting its MAX_CLIENTS limit.

Unfortunately, simply increasing MAX_CLIENTS isn't doable because we only have 1G of memory available. We've also been maxing out our ip_conntrack_count limit periodically (which is really bad, because it disables all the other sites sharing the box with us at OSUOSL).

I've hooked up squid as a transparent caching proxy, which hopefully should help a good deal, as it'll cache our php pages and avoid reexecuting those scripts. Should make the site work faster too. Whether it'll actually solve the problem remains to be seen though, so please let me know if you notice the site down again.

Bryce