Current time: 04-24-2014, 11:12 PM Hello There, Guest! (LoginRegister)

Post Reply 
Question about blingy cluster issue
04-05-2008, 03:18 PM
Post: #31
Question about blingy cluster issue
Quote:RE unreliable hardware:

DH reports that

* The server was recovering from the initial problem.
* The majority of the current problem is due to insufficient disk space, and the server will work as it should when disks aren't full.
* A significant part of the long-term solution is adding more storage.

These three points imply that the current problem is not due to unreliable hardware, but rather to insufficient disk space.
I read the dreamhoststatus updates and understand their problem at a high level. Obviously, there are specific details that aren't in the post that would perhaps be interesting to some who are keen to hear war stories about running a big filer cluster. The original problem was a crash of the filer. Jordan doesn't go into detail about why moving people off of the filer that crashed will help alleviate problems, but I can guess that the load on the server was too great or that the growth rate of the accounts assigned to the device was too much and that it was viewed as prudent to move the accounts to another device. It's not obvious to me that the conclusion of "insufficient disk space" is the underlying problem on blingy, but the statement toward the end of that update that mentiond that more storage was on the way belies that.

Quote:RE two doing the job of one:

DH has said that the server will work as it should when it has more space. Is not load sharing one of the primary goals of distributed computing or storage?
In this context, what do you mean by "load sharing"?

Quote:Am I incorrect in my understanding that the storage network is scalable in a live setting? DH's hosting plans offer continuously-expanding storage allowances. Since I doubt that they have a 1TB disk hanging out for each customer waiting for the customer's allowance to fill it, I'm inferring that there is a distributed storage architecture that allows them to add storage without taking down customer accounts, maybe something analogous to JBOD. No?
How familiar are you with managing multiple multi-terabyte NAS filers for thousands of account holders? Accounts are assigned to individual mounted NAS volumes. These volumes are created from farms of individual RAIDed disks and are allocated to various servers. These disks are themselves mounted into the physical NAS server (presumably one of their NetApps), but perhaps with some constraints in configuring physical drives into logical devices, either due to actual hardware/software constraints or due to policy to keep the data center organized.

Quote:Thus, the question of why it takes so long...
Ultimately, you might get the most satisfaction by learning how to manage data center storage yourself. Start with the wikipedia article on NAS and move on to the more complicated issues associated with configuring NAS to large clusters of Linux servers. It would also be helpful to make sure you're up on the latest on networking as it applies to data centers, as I imagine that it applies to issues like the problem they had on the spunky cluster where I think their switch-to-switch backbone connection got saturated, but they probably alleviated this by allocating another link to the channel-bonded backbone connection. And again, I dunno whether their backbones are channel-bonded gigabit links or if they've taken the plunge with 10GbE links. If they're doing file serving over the cross-connect, I'd guess that they're using 10GbE but I don't have enough real-world experience to guess accurately - though I do know that 10GbE ports are pretty darned expensive.

Use the 3DOM50 promo code for 3 extra lifetime domains and $50 off
More Dreamhost coupons here!
Find all posts by this user
Quote this message in a reply
04-05-2008, 08:44 PM
Post: #32
Question about blingy cluster issue
Quote:More pressing is whether or not I can trust DH for clients sites in the future, ya know?
I'm in a similar position. I have nothing mission-critical here myself, but I do have people pestering me about wanting to move here asap.

I think next week after the new equipment has been installed will be a prime time to setup accounts here at DH. Possible teething problems aside, the new gear should afford some comfort that the structure will remain stable and we'll see services jump straight back up to the high standards we know and love.

Maximum Cash Discount on any plan with MAXCASH
Visit this user's website Find all posts by this user
Quote this message in a reply
04-05-2008, 09:50 PM
Post: #33
Question about blingy cluster issue
Quote:How familiar are you with managing multiple multi-terabyte NAS filers for thousands of account holders?

Not hardly. That's why I'm askin'.

Thanks for your informative replies and pointer to add'l resources.
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump: