What I did to reduce memory usage for WordPress was to use PHP 5.3 instead of FastCGI. FastCGI accelerates sites a lot at the expense of more memory — way more memory. PHP 5.3 is also far better at handling memory than 5.2 (better garbage collection and fewer memory leaks). And then I turned off the extra Web security and Google Web page performance things as well — they’re great for increased performance, but, yes, they consume extra memory as well.
So you might think that this leads to far less responsive website, even though memory consumption is at a minimum. The answer? Put CloudFlare on top of everything. CloudFlare is a free reverse proxy alternative which works similarly to a CDN but is far easier to configure (and they will serve your content from 12 different data centres located around the world). Besides working as a reverse proxy, they also do a security scan for incoming connections, filtering out spammers and crackers with their nasty MySQL/PHP injections, and adds on-the-fly compression, as well as Google Analytics and so forth (if you don’t have it installed on your site). They have a lot of extra paid features, but I just use the basic services which are for free.
To get WP better integrated with CloudFlare, you can use some plugins; in my case, I use W3 Total Cache as my “master” cache plugin, which interfaces with CloudFlare nicely and does pretty much everything I can think of. It’s hard to configure, though, and that’s why Dreamhost recommends the much simpler WP Super Cache instead. But none of this is actually required — CloudFlare can definitely accelerate your whole site without needing any extra “help” from WordPress. I use CloudFlare on all the sites that have more than a few hundred page views per day