I never messed with it, but other claim to have successfully setup custom nameservers here. Rlparker mentions in the thread that there are also reasons not to do it. I suppose an IP change could screw things up.
You still have to register them with your registrar and follow the steps in those threads.
I'm surprised DH doesn't offer a simple custom nameserver setup option. That seems like one of the few things that most places offer (especially reseller accounts) that DH doesn't. It has probably been explained at some point why they don't... my guess would be some sort of complication with their panel backend, but who knows.
Save up to $96 at Dreamhost with ALMOST97 promo code (I get $1).
Or save $97 with THEFULL97.