DreamCompute Network Upgrades && Downtime


#1

Had to reboot DreamCompute instances after today’s Ceph changes. Last network update I had to reboot them as well. Why not do reboot on auto without x hrs of downtime?


#2

Hey rhand, sorry to hear that your instances needed rebooting in such short timeframe. As you can imagine, the real solution would be not to force any reboot :slight_smile: We can’t really reboot customer’s VMs automatically as that would be bad. What most likely happened to your VM is that the Ceph storage was blocking IO for some blocks for more time than the operating system liked. Some OS/configurations handle that alright, yours seemed to have suffered. Even that shouldn’t have happened and we’re sorry.

We’re working to change Ceph’s configuration so that upgrades won’t create any IO hiccups. We already made some progress, since in past upgrades we had some customers suffer damaged filesystems that required running fsck but we still have some ways to improve things. Thank you for your suggestions!