There are a lot of factors that go into site speed, one being the actual speed of your server (which is impacted by server load, yours is low, this is good). The other ones can be crazy nebulous, because it could be the code you're running, the size of images and files, the traffic, the settings, and so on.
It looks like you're running Concrete 5, so I would review http://andrewembler.com/posts/5-easy-ways-to-speed-up-a-concrete5-site/ and other similar tutorials to speed that up. I'd also upgrade PHP from 5.2 to 5.3 (or 5.4) and make sure you're on the latest version of Concrete5 (since they know the older versions are slow). You may be at the limit of that app on shared hosting, though, as the developers did note that it's not the best on shared hosting.