Here is one of my site urls: http://goo.gl/ju2PH
I'm not using any caching other than .htaccess to have client browsers cache the page:
Header set Cache-Control "max-age=290304000, public"
I did enable Xcache earlier but it seemed like that made the site even worse, so I turned it off for now. All of my sites are migrated into the VPS, but only 2 of them (like the one in the link above) are the heavy PHP/MySQL sites. The rest only get maybe 200 visits/day combined.
All sites have PHP set for: 5.4 FastCGI
I tried disabling mod_php, but my thumbnail generator stopped working due to an internal server error... Susposedly the dreamhost wiki says we can turn off mod_php if all domains are configured with FastCGI.
I'm not even sure if the VPS takes a few days for everything to "settle" or what.
Also, here is what my memory currently looks like after I set the VPS memory to 300MB: http://s9.postimg.org/q4dskq0wv/vps_memory_2.jpg
The cached memory went down to a nice level, but the actual memory and CPU load went up. It's still above 300MB.