I have used a Visa debit-card at DreamHost in the past with no problems. MasterCard is shown in the billing section of the panel as a valid card-type, so it should work fine.
The 'card already in system' is not an uncommon occurrence when something goes wrong during the first sign-up attempt. The only way I know of that this can be rectified is by contacting DreamHost sales and having them remove the card number from the system, which you have already done.
As for 'error 12', I have no idea what that is telling you. Perhaps you should write (again) to DreamHost sales and they might be able to shed some light on the situation.
Save [color=#CC0000]$50[/color] on DreamHost hosting using promo code [color=#CC0000]SAVEMONEY[/color] ( Click for promo code details )