How to automate backups, uploads/downloads of your website files, and make them compatible between Linux & Windows platforms


#1

Preface

Over years of hosting my small website on various shared web hosting platforms I’ve struggled with the following seemingly mundane issues:

  • How can I automate backups of my website files to my local computer?

  • When moving my files from one web hosting company to another, how can I automate downloading and uploading of my website files between hosting platforms?

Most of the time I’ve tackled it with the head-on approach – I used an FTP Client, such as WinCSP, or FileZilla to manually select all my files on the server and then drag them to my local folder and have the FTP software do the transfer.

But that created the following major issues:

  • First, it was obviously a manual operation.

  • And most importantly, my local computer was running Windows operating system, while the servers I was hosting on were running some flavor of Linux. And whoever knows those two operating systems can tell that their file systems are very different.

Moving files and directories from one operating system, Linux, to another, Windows, will most certainly create conflicts. For instance, Linux allows way more leniency with its file names, while Windows does not. In Windows, filenames containing any of the following characters are illegal: \ / : ? " < > | while on Linux they are not. Additionally, Linux filenames are case-sensitive when Windows filenames are not.

To illustrate let’s take a simple file structure that you may have on your Linux/PHP server. Say, I wanted to have the following URLs to refer to the same payment page on my site independent of how a user typed it into the address bar:

  • example.com/pay
  • example.com/Pay
  • example.com/PAY

For that I would create three separate directories in my main directory on the server, i.e. "pay", "Pay" and "PAY" and put my index.php into each of them, that will handle the PHP redirection.

Well, if I was to download my website files onto my local Windows computer using any of the aforementioned FTP clients I’d get an error of matching folder names. That was because Windows treats "pay", "Pay" and "PAY" as the same folder. So at best you’d get your FTP client rename those folders into "pay (1)", "Pay (2)" and "PAY (3)", or at worst, it will smoosh them all into one location by overwriting previously downloaded index.php files.

All of that was BAD if you needed to restore your files back to the same or another Linux server later. Which eventually translated into a lot of manual work, that I was obviously trying to avoid.

Solution

I did some digging around, hoping to find a solution, and to the best of my knowledge I failed to find an already made software that could handle this situation. It almost looked like I was the only one who was moving files between Linux and Windows. So I had to come up with my own solution.

Luckily, I’m a software developer and I know how to handle this. It was just the matter of finding time to do it. And this time, when I was moving my website to Dreamhost, I felt like I was done doing it manually. I sat down and wrote a small tool based on the open-source SSH.NET project that can automate all that.

The tool is called FtpsTransfer.

It is free and can be downloaded from my web site. The download contains a technical INFO.txt file that will give you a full Windows command line syntax. Below I’ll give a couple of examples how you can use it.

Prerequisites

First off, you will need to obtain the following from your (web hosting) server:

  • Does web hosting server support SFTP? If yes, then good. We’ll use it. Otherwise, check if it supports FTPS (which is FTP over secure sockets, or TLS.) If it does, then we’ll use it as a second option. Lastly, if your web hosting provider does not support neither SFTP, nor FTPS, it is probably time for you to look for a new web hosting provider. Seriously. Because the last option of using an insecure FTP is a really BAD idea. So use it only to transfer out your website files from an archaic server.

  • Enable SFTP (SSH) connection on the server. It is usually possible to do via Control Panel (CP.)

  • Obtain your SFTP (or FTP) username and password. Those can be found in your CP. By default the password is usually the same as your CP login, although some web hosts, like Dreamhost, allow to change that. (Which I will certainly recommend doing!)

  • You will need your (web hosting) server’s FTP host name, or IP address. That can be found in the CP under FTP section.

  • For added security, see if you can get the (web hosting) server’s fingerprint keys. Those are specific for each server and can be used to verify that you’re uploading your files to the server where you actually intended to upload them to. Just as a second precaution. (For Dreamhost you can find it here.) You will need an RSA md5 server fingerprint key.

  • Lastly download the FtpsTransfer tool, un-ZIP the FtpsTransfer folder into some location on your local hard drive. We’ll run it from there.

PS. As a prerequisite to running FtpsTransfer tool, you will need to have the .NET Framework version 4 installed, which most computers do by default. But in case of an older system, you can download it from Microsoft’s site.

Securing Your Password

This step is optional. It always makes me feel uneasy to save my login credentials, and namely my password, in a plaintext file. That is why the FtpsTransfer tool supports ways to scramble it. Here’s how:

  • First download one more of my free tools, called ScramblePwd. (It comes as a part of the RunAsUser tool.)

  • Un-zip it into some location in your hard drive.

  • Then start up Windows command line (go to Start and search for cmd and then run it.)

  • And cd to the folder where you downloaded the ScramblePwd tool.

  • Lastly, let’s scramble your password. Assuming that your password was something like Plw$ud5eNmC6ux%wR run your command line call as such:

ScramblePwd "Plw$ud5eNmC6ux%wR"

This will scramble your password and show it in the output. Additionally it will also copy it to the Clipboard. So paste it somewhere so that you can refer to it later:

ScramblePwd Output

There’s one important thing to remember about how ScramblePwd tool scrambles passwords. The scrambled password will be available for de-scrambling back into its original form only on the same computer and under the same Windows user account that the ScramblePwd tool was run under! In other words, you will not be able to take the scrambled password and de-scramble it on a different system. So keep this in mind. This is the part that makes it secure!

PS. For technical details on how ScramblePwd tool encrypts passwords, check Windows Data Protection, or DPAPI, and CryptProtectData API in particular.

Automating Downloads

Configure your command line for the FtpsTransfer tool. The easiest way on Windows is to open Notepad, then add the following commands into it, and save it as a Windows batch file into the same folder where you downloaded the FtpsTransfer tool. To do that, when saving in Notepad, make sure to call the file something akin to auto_download.bat (the important part is to end it with the .bat file extension.)

The following commands will download all files from your example.com website hosted at Dreamhost under the user name UserName and the password that we scrambled above with the ScramblePwd tool. It will transmit all data using secure/encrypted SFTP connection. The tool will place all files into the example-dot-com backup folder on your local Windows desktop. The name of the folder will have the current date at the end (because of the %DT% special word used in the path.)

To ensure that downloaded files can be later ZIP’ed on the Windows side, we’ll use the safeNm command line parameter that will force FtpsTransfer tool to use file names that are compatible with Windows Explorer.

Additionally the connection will succeed only if the server fingerprint key matches 1e:d1:5a:72:c4:8e:a3:ff:01:0a:3b:56:17:6f:e1:42 md5 RSA key. (Which you will have to change for your actual key.)

FtpsTransfer op "d" sftp host "signature.dreamhost.com" srvdir "example.com" dir "%UserProfile%\Desktop\example-dot-com backup %DT%" usr "UserName" pwd "xpd:'+CgBAAAA0Iyd3wEV0RGMegDAT8KX6wEAAAB8+GG36Oe8T4VbbD55psYdBAAAAAIAAAAAABBmAAAAAQAAIAAAAHbYE4mfJEcZw/wtT1YhsZ1aCC0i6NyjryX9pm8nEX/rAAAAAA6AAAAAAgAAIAAAAFs+bHaNI6lfdx9geRT2fwANDnttRr9zLnwfXsfXOGUzIAAAAPDs7idd0oNdN77UZWRaVaBZbuPWuAaomNB9qdN1raozQAAAAB/Jycvw174DE2pgeiTaqvewp2HnDnC5JtKTuIgywsIl3AMiouMm0FFOhc//bfVGFuAXE66t+s0T/b1N/GR8ATjO2w=='" sshKFP "1e:d1:5a:72:c4:8e:a3:ff:01:0a:3b:56:17:6f:e1:42" safeNm

Optionally, if you don’t want to deal with scrambling your password, you can paste it in plaintext as such:

FtpsTransfer op "d" sftp host "signature.dreamhost.com" srvdir "example.com" dir "%UserProfile%\Desktop\example-dot-com backup %DT%" usr "UserName" pwd "Plw$ud5eNmC6ux%wR" sshKFP "1e:d1:5a:72:c4:8e:a3:ff:01:0a:3b:56:17:6f:e1:42" safeNm

Or, if you don’t care about checking the server fingerprint key, just omit it as well:

FtpsTransfer op "d" sftp host "signature.dreamhost.com" srvdir "example.com" dir "%UserProfile%\Desktop\example-dot-com backup %DT%" usr "UserName" pwd "Plw$ud5eNmC6ux%wR" safeNm

Then you can run this batch file by either double-clicking it in Windows Explorer. Or if you want to employ it for regular backups of your website files to your local system, you can run this batch file from a scheduler, such as Windows Task Scheduler, or by using my own WakeupOnStandBy tool.

Additionally, to preserve the output of the FtpsTransfer tool, you may want to redirect its output into a file by calling the batch file you created above as such:

auto_download.bat>log.txt

When started, give FtpsTransfer tool some time to transfer your requested files. It may take some time, depending on the size of your files and the speed of your connection to the server:

FtpsTransfer Download Output

Note that when placing downloaded files into Windows folders on your local machine, the FtpsTransfer tool creates two auxiliary files in each folder: __FtpsTransfer_app_dirLst__ and __FtpsTransfer_app_nameAssoc__. (These files are local and are not uploaded to the web server.) The first file contains directory listing as it was received from the server, and the second file contains file permissions & associations between filenames on the server and the filenames that the tool used on your local Windows machine. As I described above, Windows does not support the same gamut of characters that are available on Linux. The __FtpsTransfer_app_nameAssoc__ auxiliary file will be instrumental if you decide to upload the same website files to another Linux server, as described below.

The __FtpsTransfer_app_nameAssoc__ file can be opened and edited with any text editor, such as Notepad, if your local directory layout is changed after its files were downloaded by the FtpsTransfer tool.

Automating Uploads

Once you have your website files previously downloaded using the FtpsTransfer tool (as was shown above) you can then upload them to a new (or same) web hosting platform using the following command line:

FtpsTransfer op "u" sftp host "signature.dreamhost.com" srvdir "example.com" dir "%UserProfile%\Desktop\example-dot-com backup" usr "UserName" pwd "xpd:'+CgBAAAA0Iyd3wEV0RGMegDAT8KX6wEAAAB8+GG36Oe8T4VbbD55psYdBAAAAAIAAAAAABBmAAAAAQAAIAAAAHbYE4mfJEcZw/wtT1YhsZ1aCC0i6NyjryX9pm8nEX/rAAAAAA6AAAAAAgAAIAAAAFs+bHaNI6lfdx9geRT2fwANDnttRr9zLnwfXsfXOGUzIAAAAPDs7idd0oNdN77UZWRaVaBZbuPWuAaomNB9qdN1raozQAAAAB/Jycvw174DE2pgeiTaqvewp2HnDnC5JtKTuIgywsIl3AMiouMm0FFOhc//bfVGFuAXE66t+s0T/b1N/GR8ATjO2w=='" sshKFP "1e:d1:5a:72:c4:8e:a3:ff:01:0a:3b:56:17:6f:e1:42"

As before, if you don’t want to mess with a scrambled password, you can specify it in plaintext:

FtpsTransfer op "u" sftp host "signature.dreamhost.com" srvdir "example.com" dir "%UserProfile%\Desktop\example-dot-com backup" usr "UserName" pwd "Plw$ud5eNmC6ux%wR" sshKFP "1e:d1:5a:72:c4:8e:a3:ff:01:0a:3b:56:17:6f:e1:42"

and if you don’t care to verify server fingerprint before uploading your files, use this command line:

FtpsTransfer op "u" sftp host "signature.dreamhost.com" srvdir "example.com" dir "%UserProfile%\Desktop\example-dot-com backup" usr "UserName" pwd "Plw$ud5eNmC6ux%wR"

As with downloads, give FtpsTransfer tool some time to upload your files:

FtpsTransfer Download Output

Uploading your files using SFTP transfer will also set your file and directory permissions. (If you want to modify permissions before uploading files, you can edit them in __FtpsTransfer_app_nameAssoc__ files in each directory in your downloaded folder. For that use any text editor, such as Notepad.)

Upload Any Windows Folder

In case you want to upload contents of a Windows folder that was not previously downloaded by the FtpsTransfer tool, use the following command line:

FtpsTransfer op "ua" sftp host "signature.dreamhost.com" srvdir "example.com/backups/%UDT%" dir "D:\Backups\Last Backup" usr "UserName" pwd "xpd:'+CgBAAAA0Iyd3wEV0RGMegDAT8KX6wEAAAB8+GG36Oe8T4VbbD55psYdBAAAAAIAAAAAABBmAAAAAQAAIAAAAHbYE4mfJEcZw/wtT1YhsZ1aCC0i6NyjryX9pm8nEX/rAAAAAA6AAAAAAgAAIAAAAFs+bHaNI6lfdx9geRT2fwANDnttRr9zLnwfXsfXOGUzIAAAAPDs7idd0oNdN77UZWRaVaBZbuPWuAaomNB9qdN1raozQAAAAB/Jycvw174DE2pgeiTaqvewp2HnDnC5JtKTuIgywsIl3AMiouMm0FFOhc//bfVGFuAXE66t+s0T/b1N/GR8ATjO2w=='" sshKFP "1e:d1:5a:72:c4:8e:a3:ff:01:0a:3b:56:17:6f:e1:42"

The command line above will upload the folder "D:\Backups\Last Backup" from your local Windows computer onto your web hosting account into the "~/backups/<DATE>" directory, where <DATE> is the current UTC date. The last parameter was specified by adding the %UDT% special word to the server path. (For more details on special words used by the FtpsTransfer tool, refer to the INFO.txt file available in the download package.)

Limitations

Note that the method described above will not be able to backup your (MySQL, or other) database files. (I plan on adding this functionality in the future.) It may not be able to backup your email data either if your web hosting provider does not allow FTP access to the mail directory.