Five minutes does seem to be excessive. The panel can be slow at times, but I have personally never experienced more than (approx) a minutes delay.
According to some recent posts by Michael from DreamHost, they have a team working on improving response times for the panel. It might be worth lodging a support request, detailing the date/time, which part of the panel you were accessing and the delays experienced. This might help them to further optimise the panel.
Save [color=#CC0000][b]$50[/b][/color] on DreamHost hosting using promo code [color=#CC0000][b]SAVEMONEY[/b][/color] ( Click for promo code details )