It seems to me that there is a problem with FTP being slow, but not the transfers themselves, but with the "negotiation" part, i.e. when going into a directory, or starting an upload. This is very apparent when you try to upload or download a large number of small files or use a tool that syncs directories via FTP.
I'm not an expert, but I've encountered a very similar problem before and it turned out to be a configuration problem with the FTP server software. I'll be glad to provide extra information or perform extra tests if necessary.
Please help, good FTP access is very important to me, and the affected scenarios are 99% of what I do via FTP. Thanks.