A bit of a bizarre experience. I don't run a plugin-intensive site, but I included WP Super Cache and when the site was slow I was troubleshooting it. Bizarrely, I think the site is faster when I turn it OFF - I don't get cached pages, but it seems sometimes pages would take forever to generate, wherein now they generate rather well.
The rough status seems to be:
With WP Super Cache I get occasional serious slowdown mixed with pages generating fast (from 8 seconds to under a second) - and of course the advantages of the cache.
Without WP Super Cache, I get consistent good page generation (1-2 second)
On the server I'm on a VPS, doing PHP 5.3, XCache, server and database in the same data center.
WP Super Cache settings were:
Using PHP to serve files
3600 Second Cache timeout
Preload All content with 720 min timeout, all posts (about 5000).
A bit odd, so I figured I'd throw this one out there. I've been debating moving to DreamPress, but confess I savor the challenge of doing this on my own as it keeps me technical (became a technology manager ten years ago and I fear getting out of touch).