Yes, you are right ! I would have loved someone to tell me this before.
I moved from VPS to dedicated and I'm so happy to have done it.
Websites load a lot faster and even under attack they hardly are slower.
To tell you the truth I even think that VPS is a scam compared to dedicated. I had so many problems with VPS that I hate Dreamhost for not being able to tell me earlier that everything would be better with a dedicated.
I hope you will have the same experience as me and that you won't regret it, anyway you can always go back.
This info is not only based on my feeling but we are different people working on websites moved to dedicated and everybody agrees.
Also, even with VPS set to max I had "websites down alerts" from different services almost everyday, and now it almost never happens.