I don't know if this is the best way to do it, but this is how I would:
First make a new user, I would place them inside your user directory so that if you wanted to change any of the forum files you wouldn't have to use their pw, but don't worry they can't see your stuff.
Two options from here do you want the forum to be a sub-domain (forum.domain.com) or a folder (domain.com/forum/)
sub domain: Make a new subdomain called forum.domain.com (or whatever you want) - Assigned the subdomain to be owned by the new user.
folder: you can use the "Remap Sub-Dir" tool DH has now to redirect it to that new users folder (on the server)