It could be. I am not on the same server as you, but I have not noticed any speed pronlems with my sites lately and the Panel, while still a little slow, is much faster than it was a few weeks ago.
I would suggest submitting a support request (via the panel) and outlining your concerns. There may be a issue with your particular server that has not yet been reported.
Save [color=#CC0000][b]$50[/b][/color] on DreamHost hosting using promo code [color=#CC0000][b]SAVEMONEY[/b][/color] ( Click for promo code details )