I experience the same problem. I am also on VPS and VPS MySQL. Some pages on my forums would take over 3 seconds to load, then after that, all the subsequent pages would load very quickly. If I go away for 15 minutes and come back to try that same page again, it would again take 3 seconds. I tried to see what it was doing by looking at it with Firefox Firebug. And it seems like it was stuck in "Waiting Time" for 3 seconds, Connecting and Receiving was only less than 100ms.
I, too, am trying to diagnose this problem..