I use it all the time but not like you're using it. I normally don't use the same database as a live server/website for my testing domain...
I remote in using MySQL Workbench if I need to run some update queries or something but that's about it.
From what you are doing it could be your ISP connection, what's ping say? How about tracert?
Rule of thumb, from what I've read, is remote is ~4 times your connection latency.