The system does not know, and (basically) it doesn't matter. Without setting up a domain, and having the registrar pointing at Dreamhost's name servers you'll never be able to access it via that domain name.
There are ways that you can set up your computer so that when you type UnregisteredDomain.com your computer will know to go to the IP address of your server. Ie. you type in UnregisteredDomain.com and your computer sends out the webpage request to 154.456.123.5/domain.com (without doing a lookup as it normally would)
However if you should add a domain name that is allready registered and hosted else where, there have been some (extremly limted and minute) cases where outgoing mail from dreamhost followed the info from their nameservers, rather than looking it up to send on to the proper server. Through as before you'll never get web traffic becuase there is no record in the official registrar pointing to Dreamhost's Name servers.