Accessing instance launched from a snapshot

dreamcompute

#1

Given that DreamCompute accrues usage costs anytime an instance exists, even if it is suspended or shutdown, it would often be nice to be able to:

[list]
[]launch an instance
[
]do work on that instance
[]shutdown the instance
[
]create a snapshot while it is shutdown
[]terminate that instance
[
]launch a new instance at some later time from the snapshot
[/list]

The GUI supports doing all of that easily and I would expect that’s a common pattern and reason for having snapshots. However, I’ve been unable to access the second instance that is booted from the snapshot. At least not from SSH. I can access it from the web console.

I believe it’s because the instance network configuration is configured with a static IP, which is then the wrong IP when the new instance is launched.

I’m wondering if/how others are dealing with this? I’m sure it’s possible to modify the network configuration manually via the console in the new instance, and maybe even possible to do it automatically via some sort of boot script. (Although I haven’t yet been able to get either approach to work myself yet.)

It seems like there should be an easier way to launch new instances from snapshots and have them just work though. Ideas? Am I missing something?

In case it matters, I’ve been launching instances from the Ubuntu 16.04 images.


#2

You’re not missing anything and in a perfect world what you describe should work but in practice we noticed that rebooting an instance from a snapshot rarely works flawlessly. There are too many moving parts and variations for that process to be reliable.

So to answer your question ‘how are others dealing with this’, there are different ways and all depend on how your applications work. One way is to separate the data from the software. For example, let’s say you need to run some ETL batch scripts daily on a VM and those tests only last 1h, and the scripts run on a bulk of data that keeps on growing: every day you upload new data, run the ETL scripts, send the results to a datawarehouse and you’re done. You may want to save cash and shut down that machine, right?

One way to address this is by creating a volume to store the data and rely on either a custom virtual image with the software I need to run for the ETL and configurations, or a configuration management tool like Ansible. Either way, will give you a solid, always replicable starting point for your software. You have separated the ETL-side from the data: you’ll be able to create the machine exactly the way you need it to be, attach the volume with your data and the execute the ETL processes. That would be a more robust way to address the problem than relying on snapshots.

What kind of applications do you need to run few hours at the time?


#3

A common use as a compile, build, and/or test server. I have often have projects that require significant customization and setup simply to do the compile/build/test. These projects go in and out of activity and it’s convenient to do the necessary setup on a vm, and then just the vm as needed.

I’m familiar with chef/ansible/puppet, etc., but these all take setup work. They are great and represent a more robust approach, but sometimes it’s convenient to simply setup a machine manually and then snapshot/thaw it periodically as needed. And in some cases, especially in setting up more esoteric environments, it can be especially difficult to use the automatic provisioning tools.

Another use case is for part-time development projects. For example, I have a project that requires loading a large postgreSQL database. I only work on this project periodically so can’t justify running a machine 24x7. At the same time, it takes so many long to get the data loaded that if I had to do that every time I worked on the project, even if using chef/ansible/etc, I’d never make progress on it.

These are use-cases for which I currently use AWS EC2 instances. Usually, I just shutdown the EC2 instance when not needed at which point I’m only paying for the very small EBS storage costs. I can then restart as needed. I’ve never given much thought to how they handle the OS network configuration upon restarting an instance because it just works. I believe in the past I’ve also launched new EC2 instances from volume snapshots that I’ve taken without any additional effort, but I’d have to try that again to be sure. It’s possible I was creating and launching AMIs instead.

If it’s difficult to launch a new DreamCompute/OpenStack instance from a snapshot, then I’m curious what is the main value of snapshots? Backups?


#4

I found that if I launch from the ubuntu-14.04 image, everything seems to work the way I would expect. Specifically:

[list]
[] I can shutdown instance, create a snapshot, terminate instance, and then later launch a new instance off that snapshot and immediately access it via SSH. Looking at network config, I can see that something updated the config to use new ip address assigned to the instance.
[
] I can also, alternatively, shutdown instance, create a volume-snapshot, terminate instance, and then later launch a new instance on the volume to continue work persistently.
[/list]

This matches with what I would expect of a cloud service like DreamCompute.

So one guess for my earlier problems is that there is something wrong with the ubuntu-16.04 image such that it doesn’t auto configure itself upon instance relaunch, or perhaps it is due to fact that I always applied updates immediately after launching before creating the snapshot, and there’s something in the updates that breaks things. I wanted to verify that idea but I ran in to a recurring bug where the snapshot creation takes forever and/or never seems to complete.


#5

For anyone interested, I just confirmed that it is the ubuntu-16.04 image that is making this difficult, even without updates. Ubuntu-14.04 works as expected.


#6

Thanks for letting us know. I and other cloud colleagues were just trying things out in order to isolate the issue and we were converging on Ubuntu 14.04 acting fine. As you noticed, snapshotting 80GB drives takes time :slight_smile:

I’ve opened a bug internally. Thanks.


#7

Same problem with Centos6 image (cannot launch a new instance from a snapshot of an instance based on this image).
Can confirm as well that the Ubuntu 14 image works as expected.


#8

I’ve been able to get Ubuntu 16.04 (and Ubuntu 15.10, which suffers from same problem) based images launched from snapshots/volumes accessible from SSH again by running the following from the web console after launch:
[list]
[]sudo ifdown eth0
[
]sudo ifup eth0
[/list]
Bringing eth0 down usually results in an error message, but bringing it back up seems to work. After I do that it’s immediately accessible via SSH again. Of course, to use web console, you’ll need to have created a separate username/password login for yourself during the first launch. i.e.
[list]
[]sudo adduser myusername
[
]sudo adduser myusername sudo
[/list]

It seems like bouncing the network via a post-creation script might work and not require logging in manually via web-console, but I was unable to get it to work on 1st attempt. Not sure what sort of script is expected.

Looking at /var/log/syslog I can seem some errors from cloud-init, such as “Failed to start Initial cloud-init job (pre-networking).” I’m not familiar enough with OpenStack to decipher it.

In any case, the ifdown/up workaround makes it possible to use newer images with snapshot/volume launches, even if not super convenient.


#9

Thanks for the info. The Cloud team is working on updating the images. I don’t have a time for the fix yet, I’ll do my best to keep you posted.


#10

Snapshots are just not ideal for any kind of backup and restore purposes

Because they image the whole thing byte for byte a restore takes very long

A deploy using terraform can take < 1 minute while a restore of a 100GB image takes what an hour?

The only case I see snapshots being useful is for Block storage where i want to store my data and then use it later on, but the system is seperate from the block storage.


#11

FYI, there are new images on DreamHost Cloud and the snapshots work as expected now. If you have issues booting from a snapshot please open a ticket on https://dreamhost.com/support

[closing the thread]