Email lost through a series of accidents

“I was a victim of a series of accidents, as are we all…” - Winston Niles Rumfoord, Sirens of Titan.

Oh, what a week! Monday, website down! Tuesday, emails lost!

I maintain a website on DH for my little agency. We used to have DH handle our email as well, but then 18- 24 months ago when DH email had a series of problems, the management wanted me to take the option of having Google manage mail for our domain. Enabled Google Apps at the DH web panel, moved all the users mailboxes. Cost quite a bit but they wanted the extra reliability.

4 -5 months ago, they wanted some more info on web page usage, so I tried out Google Analytics from the DH panel. Needed to also have Page Speed Optimization turned on to use this feature. The results were interesting, and I just left Analytics on.

On 4/22, suddenly none of my sites pages were loading, even down in some subdomains for testing. Everything was coming back quickly with “Server reset” messages. My agency people got real excited, blaming me for not preventing hacking, or for screwing around with testing CMSes in subdomains. But it didn’t look like hacking from my scans, and I tried to convince everyone that it wouldn’t be coming from separate subdomains, no exploits of WordPress since I was up to date on versions. On 4/23, while waiting for DH support to get back to me, I happened to notice something from looking at the failed loads “View Page Source”: That Page Speed Optimization was inserting a few lines calling some javascript on the bottom of all the failed pages. So I went back to the DH Panel, turned off Page Speed and Google Analytics, and bingo! the pages started loading again.

Well, I was thinking I was pretty smart to have figured this out until a couple hours into the morning I noticed I wasn’t receiving any email. I tried some test emails from another account, but they didn’t show up. No bounce back either. Then I checked with other folks and they noticed they weren’t getting any emails either. We could send email, but no email was coming in. To eliminate the possibility of problems with the email client I was using on my PC, I checked the inbox at Google Web Mail and it too had the last email received at 10:34.

In coming email was down for all of us from 10:34 4/23 to about 17:20 4/23.

No confirmation from DH support yet, but I’m guessing that when I turned off Google Analytics, all of the mx records that direct our e-mail to google were deleted from the server and purged from all active DNS servers throughout the internet, then took a while to re-propagate. I didn’t expect that Google Analytics would screw up Google Apps for Business like this. I hope someone on my team didn’t also toggle Google Apps for Business on the DH Web Panel and isn’t telling me.

Is there any hope of retrieving the lost email for this period? Google Support just told me they don’t have them. A key piece of evidence was no bounce messages. The Google servers never had them. Can I get the messages back from the DH servers? Hope to hear back from DH support before the messages get automatically purged.