Couple things. Can you give an approximation of the size of your forum? Appropriate answers could be "wee", "decent", or "frickin huge!" Are you attempting to export the script as an archived file, such as zipped? Evidently zipped archives have to be completely assembled in server memory, and could exceed the amount permitted by a standard PHP installation. This seems unlikely to be the culprit based on your command shell example though.
When using the export functionality of phpMyAdmin, are you selecting the boxes marked "Enclose export in a transaction" and "Disable foreign key checks"? The first option can help to ensure minimal risk of corruption should things continue to fail, and the second option should speed the process up even if you run the risk of losing a minimal number of posts created at the exact time of your export.
Keep in mind that you're ALWAYS going to risk losing a few posts if you're unwilling to shut the forum down for maintenance during export. Wish I had something a little more concrete for you right now.