Dreamhost forwarding my mail, and nothing else?

I have perhaps an unusual situation. I have a Dreamhost $10/month account (Crazy Domain Insane), and I have a non-dreamhost account which basically gives me a static IP and root on a virtual box. I’m using the latter to run a process on port 80 that is my web page. But I’d rather not deal with setting up something like Postfix on this server, and instead just have Dreamhost deal with it. So what I did was change my nameservers to all point to the virtual box. On my dreamhost, I just have one email address which is a “forwarding-only” account.

This all works (http://www.example.com loads my web page, and rob@example.com gets forwarded appropriately) EXCEPT, around 24+ hours after I send a test email to rob@example.com, I get an email message saying

"Delivery to the following recipient has been delayed:


Message will be retried for 2 more day(s)

Technical details of temporary failure:
TEMP_FAILURE: Could not initiate SMTP conversation with any hosts:
[example.com (1): Connection refused]"

But note, I already have successfully received this message. Any way to avoid this last part without running a mail server on the virtual box and handling all the mail from there?