That error message indicates that the application you're trying to run is trying to launch some sort of graphical "splash screen", which isn't possible over a terminal connection. Check the configuration to see if there's some way to disable that.
That being said, it's generally been our experience that Java web applications are not suitable for use on shared hosting. Either they must run as a persistent server process, which we do not permit on shared hosting, or their resource requirements are simply too high for the environment. You will likely need to upgrade to a DreamHost VPS or Dedicated Server to run this application, or find a lighter-weight alternative.