- DreamObjects are moving clusters (from us-west-1 to us-east-1).
- us-west-1 is in lock-down mode right now, and will be completely removed come October 2018.
- DreamHost sent emails asking users to migrate their data to us-east-1
- DreamHost provides a Migration Tool to move data from us-west-1 to us-east-1. The migration downloads your us-west-1 objects and uploads them to us-east-1.
Problem: Data Corruption
I migrated many videos in MP4 format. The migration tool does not copy ACLs, so I had to update the permissions to all of my objects after uploading to the new cluster.
After the migration, the videos on my site stopped working.
Specifically: The videos stream fine for about 1 minute, then stop. I cannot jump/seek to any point in the video. The streaming functionality is now broken after migration.
I downloaded a few files from DreamObjects to my local machine, and VLC media player also stops playback after about 1 minute.
VLC’s debug log shows these warnings:
avcodec warning: cannot decode one frame (4163 bytes) faad warning: Channel coupling not yet implemented faad warning: Channel coupling not yet implemented avcodec warning: cannot decode one frame (187 bytes) faad warning: Maximum number of bitstream elements exceeded faad warning: Unexpected fill element with SBR data
These same videos worked & played fine before the migration.
I also migrated some large .zip files and the same issue is happening when the file is > 4MB.
The migration corrupted objects > 4MB.
Luckily I have local backups of the files, so there was no data loss. But now I have to reupload everything and change the ACLs (again).
Knowing what I know now, I would have avoided using the automated migration tool completely and done the migration myself.
The migrated (corrupted) files are the exact same size (number of bytes) as the old files.
And the first 4MB of the corrupted files (up to
0x00400000) are identical to the old files.
But after 4MB, the data completely changes (the rest of the file is completely different, not shifted or anything).
Maybe this was a chunking issue / encoding issue or something in the migration tool?