vBulletin 3.x can be tweaked pretty well to load up relatively quickly, but I do find that phpBB3 loads quite a bit faster irregardless. Vbulletin just offers a lot more features out-of-the-box that phpBB3 does not. So it's really up to you.
I personally had a custom PHP5 install going with eAccelerator and some template compression tweaks to load vBulletin at a good speed - and that was with fairly low traffic.
Chips N Cheese - Custom PHP installs and the like!