First, change all your important passwords. Shell/FTP, MySQL, any CMS software you're running, etc. Make them long and random.
Second, find out who their ISP is (traceroute it if you have to) and contact them, letting them know you were threatened by one of their users, and include all relevant log info.
Thirdly, make sure you're not running any scripts that could lead to vulnerabilities. Software like Moveable Type, PHPbb, etc tends to be fairly well tested in this area.
Fourth, pray, if that's your thing.