Well one of the programs I am trying to run for my server requires a background window to open which won't work without either an X server or a VNC server. I have tried VNC but it keeps telling me connection refused so now I am trying X and it is pretty much telling me I need the graphics drivers.
Edit: Never mind, I got it working. I had to install XMing and set X11 forwarding flags. My biggest issue was that when I would first login, I would login as a root user so the user I would switch to after wouldn't have the xauth list propagated with the keys. After signing in with the correct user, I was able to start the programs with no issues.