Data corruption after DreamObjects migration (west-to-east)


#1

Background

  • DreamObjects are moving clusters (from us-west-1 to us-east-1).
  • us-west-1 is in lock-down mode right now, and will be completely removed come October 2018.
  • DreamHost sent emails asking users to migrate their data to us-east-1
  • DreamHost provides a Migration Tool to move data from us-west-1 to us-east-1. The migration downloads your us-west-1 objects and uploads them to us-east-1.

Problem: Data Corruption

I migrated many videos in MP4 format. The migration tool does not copy ACLs, so I had to update the permissions to all of my objects after uploading to the new cluster.

After the migration, the videos on my site stopped working.

Specifically: The videos stream fine for about 1 minute, then stop. I cannot jump/seek to any point in the video. The streaming functionality is now broken after migration.

I downloaded a few files from DreamObjects to my local machine, and VLC media player also stops playback after about 1 minute.

VLC’s debug log shows these warnings:

avcodec warning: cannot decode one frame (4163 bytes)
faad warning: Channel coupling not yet implemented
faad warning: Channel coupling not yet implemented
avcodec warning: cannot decode one frame (187 bytes)
faad warning: Maximum number of bitstream elements exceeded
faad warning: Unexpected fill element with SBR data

These same videos worked & played fine before the migration.

I also migrated some large .zip files and the same issue is happening when the file is > 4MB.

The migration corrupted objects > 4MB.

Conclusion

Luckily I have local backups of the files, so there was no data loss. But now I have to reupload everything and change the ACLs (again).

Knowing what I know now, I would have avoided using the automated migration tool completely and done the migration myself.

More Details

The migrated (corrupted) files are the exact same size (number of bytes) as the old files.

And the first 4MB of the corrupted files (up to 0x00400000) are identical to the old files.

But after 4MB, the data completely changes (the rest of the file is completely different, not shifted or anything).

Maybe this was a chunking issue / encoding issue or something in the migration tool?


#2

Sounds like a serious bug. I’m about half way through my migration, started 10 days ago. I’m hoping your bug only affects large files, over 4MB, most of my files are much smaller. I’ll have to check them all I guess.
Please keep us informed of how this gets fixed.
-mort