Huntress Status


#1

I have had a hosting account at Dreamhost since 2008 according to the panel.

According to Dreamhost there was a hardware failure with Huntress and they have worked to restore the server. They said that they have reloaded date from a backup, but for cases where data is not up to date, they could copy the data from the old hardware.

5 of my sites are back up and running correctly.

I have two fairly urgent problems:

  1. My Subversion server hosted by Dreamhost is no longer working because the files are out of date. I asked Dreamhost to restore the files from the disk, but I have not heard anything. This is critical because my remote team cannot work with SVN.

  2. Another site on my account SherTremonte.llp throws a bad_httpd_conf error. Although I believe that the site is probably properly restored, a simple DNS problem is keeping the site from working. I have checked the DNS and it resolves to the correct IP.

I understand that servers crash, but what frustrates me is that there is no communication. A couple stock meaningless emails. The site: http://www.dreamhoststatus.com/ has not been updated. Nobody has responded to any of the tickets that I have opened. I paid for chat, but it is not available. I paid for phone support, but it is not available. Dreamhost now appears to be a black hole and I have no idea when I can resume work.

If anyone has any information about the current status of Huntress, what has been restored, how many accounts were on the server, how many tickets they have to get through. The order they are processing the tickets, etc…

Other hosting companies keep their clients informed during disasters and it helps. Step by step updates with an expected timeline really helps explain to clients what is happening.

Thanks,
Todd


#2

I’m not sure what you mean. What files are out of date? I’m not sure how it is related to SVN.

Some people have suggested to re-add your domain to your hosting plan by DH panel --> Manage Domains --> Domains --> Edit --> Fully Hosting This Domain with Dreamhost. Just save and hope DH will refresh the record. I’ll also suggest you to check errors in the log files.


#3
  1. Two of my subversion repositories:

http://svn.shifteight.com/awp

and

http://svn.shifteight.com/artremba

Throw the following error when they are accessed:

<D:error xmlns:D=“DAV:” xmlns:m=“http://apache.org/dav/xmlns” xmlns:C=“svn:”>
<C:error/>
<m:human-readable errcode=“160006”>No such revision 720</m:human-readable>
</D:error>

I am hoping that when the latest files from the salvaged disk are copied back to the restored server, that the repositories return to functioning as they had been, and any updates that developers have made locally can be checked in without problem.

  1. I have re-saved my domain with dreamhost and this did not fix the bad http conf error. I found the page on the wiki and followed all of the steps. I am using an external DNS server, but it is unclear where I should point it to, I have it currently pointing to the A record listed under the Dreamhost DNS section.

I really appreciate your response, it is nice to get feedback.

The last update from Dreamhost was 13 hours ago … they were 63% complete at that moment but did not give us an estimate of how much was left. They also did not tell us how many users had requested that they manually restore files from the salvaged drive or how long that would take.

Thanks again,
Todd


#4

Your IP probably changed. Did you check it now and update your A-record?


#5

I followed the instructions on the WIKI and here is the information that I found. I updated my A-Record so it points to the same A record as on the Dreamhost DNS. I also tried a few other IPs, such as the a record for other sites on Huntress and they did not appear to work.

C:\Users\User>nslookup shertremonte.com
Server: home
Address: 192.168.1.254

Non-authoritative answer:
Name: shertremonte.com
Address: 69.163.163.7

C:\Users\User>nslookup shertremonte.com ns1.dreamhost.com
Server: ns1.dreamhost.com
Address: 66.33.206.206

Name: shertremonte.com
Address: 69.163.163.7


#6

Go to manage domains in the panel and click the DNS link right under the domain as listed, check the first A record listed in the non-editable entries there. It should be the same as what you found using NS1.dreamhost.com.

bad_httpd_conf is definitely about the DNS not matching the server config.


#7

Non-editable DreamHost DNS records for shertremonte.com:
These records were added automatically by DreamHost to provide services (such as web hosting or mail) which are active for this domain.

Record (shertremonte.com zone) Type Value
A 69.163.163.7
NS ns1.dreamhost.com.
NS ns2.dreamhost.com.
NS ns3.dreamhost.com.
ftp A 69.163.163.7
www A 69.163.163.7
[hr]
POSTED AROUND MIDNIGHT EST LATE TUESDAY NIGHT:

Current Outage: DreamHost web server ‘huntress’ being moved to new hardware (updated 12 hours 51 mins ago)
Update: This is an update to let you know that the restore is currently at 63% complete, continues to remain stable on your server, and your sites should be loading as expected. However, if you need assistance with files missing or being outdated and you don�t have local backups, or if your sites aren�t performing as expected other than being somewhat slow or timing out occasionally due to the high restore load, please contact us with the specifics. You can do so by either replying to the original announcement message from your inbox or by submitting a ticket through the DreamHost panel (https://panel.dreamhost.com/), under Support/Contact Support.
We will update this post again tomorrow morning unless there is a change in status which necessitates additional notification.

As a reminder: You are receiving this message because your web hosting services are being affected by hardware maintenance due to issues with your shared web hosting server huntress.
Due to issues with the RAID controller, we are replacing the hardware for this server and will be restoring all data from our backup server to the new hardware. We’ve completed all the work to get the server mounted and the file restore started, and Apache services are starting up now to bring sites back online. In the meantime you may be experiencing downtime with your websites or may be unable to access your server. We apologize for the inconvenience, and we�ll update this post in the next hour or as soon as progress is made, whichever is sooner.
A little bit about the restore process: once the backup connection is established, we will reload your web service and will serve your sites again using the data on the backup server. The data will be mounted over the network to allow your hosting services to remain online for the duration of the process.
Please note, if those backups are older than the most current version of your site, you will have to upload/change those files again. It�s safe to upload your own backup files during the restore process, as we have engineered the process such that we will not overwrite the files you�re adding yourself. Let us know if you�re unable to do so, and when / if we�re able to recover the old hardware that crashed, we�ll be happy to manually transfer your most recent files for you to the new drive.
During the restore process, there may be periods of high load on the server due to the large amount of data being sent to the new server. We will be closely monitoring the server and mitigating that load whenever possible.
Please note, we�re only able to monitor the status of the server and that of the web services responsible for your sites and not individual sites and users. If you�re still having trouble loading your sites or access your server, please let us know the specifics, so we can take a look for you. You may contact us either by replying to this message from your inbox, or by submitting a support ticket in the DreamHost panel (https://panel.dreamhost.com/), under Support/Contact Support.
As per the above, if you need assistance with files missing or being outdated and you don�t have local backups, or if your sites aren�t performing as expected other than being somewhat slow or timing out occasionally due to the high restore load, please contact us with the specifics by either replying to this message from your inbox or by submitting a ticket through the DreamHost panel (https://panel.dreamhost.com/), under Support/Contact Support.
This notice will also appear on that page in your DreamHost Panel and will be updated throughout the process with the latest details we can provide.
This incident is only related to the shared web server huntress, and no other servers or services (such as MySQL or email) are affected.
Thank you for your patience!
The Happy DreamHost Server Fixing Team
[hr]
POSTED ~ 1:30 PM ON WEDNESDAY

Update: We are experiencing some difficulties with the backup server, which is causing a higher than normal load on the new machine. This in turn is affecting your site performance: you may be down for extended period of times or seeing time-out errors more often than expected. We are looking into alleviating the load. We apologize for the inconvenience this is causing, and we appreciate your continued patience during this process. We will update this notice as soon as we make progress or within a few hours, whichever is sooner.

As a reminder: You are receiving this message because your web hosting services are being affected by hardware maintenance due to issues with your shared web hosting server huntress.
Due to issues with the RAID controller, we are replacing the hardware for this server and will be restoring all data from our backup server to the new hardware. We’ve completed all the work to get the server mounted and the file restore started, and Apache services are starting up now to bring sites back online. In the meantime you may be experiencing downtime with your websites or may be unable to access your server. We apologize for the inconvenience, and we�ll update this post in the next hour or as soon as progress is made, whichever is sooner.
A little bit about the restore process: once the backup connection is established, we will reload your web service and will serve your sites again using the data on the backup server. The data will be mounted over the network to allow your hosting services to remain online for the duration of the process.
Please note, if those backups are older than the most current version of your site, you will have to upload/change those files again. It�s safe to upload your own backup files during the restore process, as we have engineered the process such that we will not overwrite the files you�re adding yourself. Let us know if you�re unable to do so, and when / if we�re able to recover the old hardware that crashed, we�ll be happy to manually transfer your most recent files for you to the new drive.
During the restore process, there may be periods of high load on the server due to the large amount of data being sent to the new server. We will be closely monitoring the server and mitigating that load whenever possible.
Please note, we�re only able to monitor the status of the server and that of the web services responsible for your sites and not individual sites and users. If you�re still having trouble loading your sites or access your server, please let us know the specifics, so we can take a look for you. You may contact us either by replying to this message from your inbox, or by submitting a support ticket in the DreamHost panel (https://panel.dreamhost.com/), under Support/Contact Support.
As per the above, if you need assistance with files missing or being outdated and you don�t have local backups, or if your sites aren�t performing as expected other than being somewhat slow or timing out occasionally due to the high restore load, please contact us with the specifics by either replying to this message from your inbox or by submitting a ticket through the DreamHost panel (https://panel.dreamhost.com/), under Support/Contact Support.
This notice will also appear on that page in your DreamHost Panel and will be updated throughout the process with the latest details we can provide.
This incident is only related to the shared web server huntress, and no other servers or services (such as MySQL or email) are affected.
Thank you for your patience!
The Happy DreamHost Server Fixing Team


#8

It definitly appears that 69.163.163.7 is the correct ip and i get the same using nslookup from my location. I’m not getting the bad_http_conf error for your site tho, I’m just getting a timeout.


#9

That’s probably because they just said they are having problems with the server and requests might timeout or be slow.

But they fail to mention what % complete they are. They were at 63% 14 hours ago… now they just tell us there is a problem and things are slow.

So frustrating being in the dark.


#10

Have you mentioned the bad_http_conf error to support? I know they are horrible about answering messages until the server is restored, but it seems like they may need to handle that for you.


#11

Yes, I opened a ticket specifically for the bad_http_conf error, but none of the tickets are being responded to.

I understand that the SVN issue cannot be fixed until the restore is complete, but they are not responding at all.