First, you should review the documentation on installing vanilla.
You can upload the (numerous) files that are part of the vanilla package via an FTP client program (see the DH Wiki article on FTP) or via the DH "WebFTP" feature in the Control Panel.
"Installation", however, involves more than just uploading the files. In addition to uploading the files, you need to create a MySQL database for use by the vanilla program. This is done from within the Control Panel, using the Control Panel -> Goodies -> Manage MySQL screen(s).
Once you have done that. and have uploaded all the files to the proper location on your server, you can proceed by following the instructions provided by the vanilla folks to complete the installation.
One note: On DreamHost, you should not set file permissions to "777" on the "./conf" directory. Because DreamHost uses suEXEC, "755" is the appropriate setting for this directory's permissions.
That almost sounds as though you have a "registration only" account. Do you have a hosting plan at DreamHost? If so, you will not need to pay anything else to set up and run/host a vanilla forum. If, however, you only have a domain registration at DreamHost, and have not established a "hosting plan", you will need to do that first, and you will have to pay for the hosting plan.
Registration and Hosting are two separate services, and they have their own charges associated with them (though every hosting plan comes with a "free" domain registration, it doesn't work the "other way around".)