3 hours of data disappeared


#1

At around 6:45 PM Pacific time today, all of my files reverted to almost exactly 3 hours earlier. Any updates to my files after 3:45 PM are gone, and any files that were created after 3:45 PM are also gone. Server logs also show a gap from 3:45 to 6:45. There was no downtime and the server has been up for almost 2 days. I’m waiting for a response from DH support. Has anyone ever heard of this happening or experienced this before?


#2

That’s a new one on me, and I’m very interested to hear what support says in response. :wink:

Just out of curiosity, do you have anything in your .snapshot directories for that time frame (I’m suspecting probably not)?

–rlparker


#3

This is crazy, my files just reverted again to two hours ago. My snapshot directory only has one directory in it: nightly.0, and the files in it are all messed up. The other directories that are supposed to be there such as hourly.0 seem messed up as well. They are not showing up as directories and are 0 bytes.


#4

Sounds very bad, maybe a filer problem. Did you get a response from DH yet? Normally you should have two nightlies, two hourlies and two weeklies. Did support respond yet?


Promo-Code: [color=#CC0000] SAVEMONEY97 [/color] - Save [color=#CC0000]$97[/color] on your first year of hosting.


#5

No response from DH yet, but fortunately my files have not had another “time warp”. What happened was at 6:45 pm, my files reverted to 3:45 pm. Then at 8:45 pm, my files reverted back to 6:45 pm and the three hours of missing data was back, but are now missing two hours of data from 6:45 to 8:45. I was afraid this process would keep repeating all night but luckily it did not. This is a serious issue and I thought I would have received a response by now.


#6

In case anyone is interested, here is the response I got:

“I’m sorry for the problems you experienced with this. I spoke with
one of our admins and he believes this is because we were in the process of offloading a bunch of users accounts to a new faster file server at around the time you wrote in. We were doing this to help prevent any one file server from being overloaded. It’s possible that the files that were reverted were a result of the initial copy completing and the final rsync running. I checked in your directories and saw that files were uploaded shortly after you wrote in, so it doesn’t look like this has happened again. I don’t think you should have any further issues with this, as all of your data has been moved. If you do experience these issues again though, please don’t hesitate to let us know.”

Still not happy that two hours of data is gone for good.