Follow the instructions on Clozure.com to rebuild the Lisp kernel. Since I’m on shared hosting, I ran each step with “nice” to avoid impacting other users.
I put this in ~/bin/ccl:
exec $HOME/ccl/lx86cl64 "$@"
To compile, I use Xach’s BuildApp and Quicklisp, so I went through the three or so steps to set up Quicklisp (quicklisp.org) and installed BuildApp through it, putting the binary in ~/bin
To deploy, I do a git checkout of my app code and run (“nice”) BuildApp over SSH, then move the file into the server root.
Alternatively, I had tried SBCL, but ran into problems where it’s confused with the address-space randomization used (I think). Running SBCL yields an error message about mmap failing. I haven’t tried any other compilers (Allegro, Lispworks, etc).
I have considered building a local VM to match Dreamhost’s library configs, etc. so that I can reliably compile locally and deploy remotely, but my applications (at least) tend to be dependent on the system libraries and naïvely building on a default Ubuntu 12.04 image here didn’t get me a usable executable.
(Note to Dreamhost staff, it’d be nifty if there were some way for us folks who compile our programs to get a reliable local clone of a shared host …)
I’m very interested! I’ve been gathering information on how to do this for the past two days. Unfortunately, I haven’t been able to find much. Just a lot of dead links.
I have an app built on Clack that I’ve been developing on a local machine, using Hunchentoot as the server. I would now like to deploy that application to my DH site.
Like you, I ran intro trouble with sbcl, but have successfully gotten ccl running.
Can you share some more information on how you set [f]cgi up to accomplish this? Say I want my application deployed under /my-app. Where does the executable application go, and what goes in .htaccess? I’ve been looking at using mod_rewrite to send all requests to the application, but I’m still a little fuzzy on how it is all going to work (I’m looking at mod_rewrite because the directives for mod_fastcgi generally don’t appear to be applicable at the directory level).
Finally, what does the application (toplevel function) look like? I tried building an application similar to the last code listing in this article. If I just try executing it from the command line, it appears my application starts up and immediately exits (maybe that has to do with expecting fd 0 to be a bi-directional stream?)
So, in the interim since my prior post, I’ve changed my ways a little.
Personally: I just happen to like SBCL a little more, so I built a Docker to compile SBCL in, with the following lame Dockerfile. Note DHUSER matches the Dreamhost user account under which I deploy it, just in case any paths end up hard-coded into the corefile or something:
MAINTAINER Bruce-Robert Fenn Pocock <firstname.lastname@example.org>
RUN sh -c 'dhclient eth0 &'
RUN apt-get update
RUN apt-get install -y make sbcl curl gnupg unzip
RUN cat /etc/passwd
RUN echo DHUSER:x:1000:0::/home/DHUSER:/bin/bash >> /etc/passwd
RUN cp -a /etc/skel/ /home/DHUSER
RUN chown -R DHUSER /home/DHUSER
RUN echo 'DHUSER:$6$norealpassword:16615:0:9999:7:::' >> /etc/shadow
RUN su - DHUSER -c 'curl -OL https://github.com/sbcl/sbcl/archive/master.zip'
RUN su - DHUSER -c 'unzip master.zip'
COPY map-patch.patch /home/DHUSER/sbcl/map-patch.patch
RUN apt-get install -y gcc patch
RUN su - DHUSER -c 'cd sbcl-master; ls src/runtime/linux-os.c ; patch -p1 < ~/sbcl/map-patch.patch'
RUN su - DHUSER -c 'cd sbcl-master; echo "\"1.3.0x\"" > version.lisp-expr'
RUN apt-get install -y time
RUN su - DHUSER -c 'cd sbcl-master; ./make.sh --prefix=/home/DHUSER/sbcl'
#RUN apt-get install -y ed
#RUN su - DHUSER -c 'cd sbcl-master/tests; ./run-tests.sh'
RUN chown DHUSER -R /home/DHUSER/sbcl
RUN su - DHUSER -c 'cd sbcl-master; ./install.sh --prefix=/home/DHUSER/sbcl'
RUN apt-get remove -y sbcl
RUN su - DHUSER -c 'curl -LO https://beta.quicklisp.org/quicklisp.lisp'
COPY quicklisp.lisp.asc /home/DHUSER/quicklisp.lisp.asc
#RUN su - DHUSER -c 'gpg --verify quicklisp.lisp.asc quicklisp.lisp'
RUN su - DHUSER -c "./sbcl/bin/sbcl --load ./quicklisp.lisp --eval '(quicklisp-quickstart:install)' \
--eval '(ql:add-to-init-file)' --eval '(quit)'"
RUN su - DHUSER -c "./sbcl/bin/sbcl --load ~/quicklisp/setup.lisp --eval '(quicklisp:quickload :caveman2)' --eval '(quit)'"
Yes, that’s not super-efficient, running apt-get multiple times, … it grew on me. The run-tests.sh fails, BTW.
The associated patch for address-randomization corrections is:
Pretty typical stuff, I suppose. Note particularly that BuildApp cannot overwrite an executable while it’s running, so building a new copy then moving it into place is required if you ever need to do an in-production patch job; BuildApp won’t do that bit for you. (The --backup=t means you’ll end up with myapp.cgi.~1~ for ever-increasing values of ~1~ over time, which is helpful if you mess up but you’ll want to clean up behind yourself periodically.)
Then: Clack entry point #'FASTCGI-ENTRY: (I have this in src/main.lisp right after Caveman’s generated #'START for Hunchentoot)
(defun fastcgi-entry (argv)
(format t "Content-Type: text/plain~c~c~c~c
Starting FastCGI is half-working if you see this. But, only half.~%"
#\Return #\Linefeed #\Return #\Linefeed)
(format *error-output* "~%FastCGI started with ARGV=~s~%" argv)
(let ((ql:*quickload-explain* nil) (ql:*quickload-verbose* nil))
(clackup *appfile-path* :server :fcgi :port nil :use-thread nil :fd 0)))
The #'FORMAT calls are hopefully irrelevant now, but I had needed them for debugging this.
It’s going to take me a little while to absorb it all. In the mean time, I sort of got my application working last night. Some of the routes work, while others result in an internal server error (Premature end of script headers). I wonder if it has to do with the fact that I am not currently passing the requested uri to the script? Something along the lines of what you did here:
More on that in a bit.
While my application still needs work, I thought I would post a very pared down example that might point somebody in the right direction (feedback welcome!)
Step 1 - Build an application
For my example, i’m still using ccl, and the built-in function ccl:save-application. Below is a lisp file that can be loaded. It will: ensure clack (insert your additional dependencies here) is loaded into the lisp image, define an entry point that builds a simple application, and dump the lisp image into an executable file that will begin in main.
This is a really simple application, but it does two things I found helpful:
It maintains some state (number of visits). I wanted to verify that the application process was persisting between calls to the web server
2.) It outputs the request headers. I wanted to make sure the proper request uri was coming in. This is why I wasn’t sure that I needed to be passing any additional parameter(s) when executing the script, as mentioned above.
Step 2 - Drop the application into your document root
This is easy, just copy or sftp my-app.fcgi up
Step 3 - Configure .htaccess
Below is what I have. Just the minimum to get going. My intention with this is to have any request for /my-app or /my-app/whatever-else result in my fcgi process being used. (However, a request to /my-application doesn’t get handled that way)
Contents of .htaccess in the document root
RewriteRule ^my-app(/.*)?$ my-app.fcgi [L]
So that did it! Below is sanitized, sample output from a request to /my-app/its-working
With my real application, this is what I’ve determined:
For now, I’m placing the app at my web root. I have the following routes defined (using lucerne)
Trying to hit the top level routes (/ /foo /bar /baz), only /foo returns something. All others result in an error log entry “Premature end of script headers”. Now what /foo returns is interesting. It returns what should be returned for “/”! And further, the rest of my site is available under /foo! So /foo/foo returns what I would expect for /foo, and /foo/barb/this returns what I would expect for /bar/this.
I’m not sure what to make of this. Based on the test application from above, I have every reason to believe the proper request URIs are being passed to the application. I’m using clack from quicklisp 2016-06-28. It is possible it is an issue with the fcgi backend? It is very odd. Running with the hunchentoot backend works as expected.
I’ve edited by .htaccess file to be very similar to yours. At one point I kept getting too many internal redirect errors. I lost track of which update to .htaccess fixed it, but I seem to be in the clear now.
The odd routing behavior that I encountered had to do with how the framework I was using on top of clack (lucerne) was parsing the request uri. It was assuming that whatever can in with the request under script-name was also a prefix of the request uri. Therefore, it was attempting to grab a substring starting at the end of the prefix. The relevant code from the current version looks like this
(defun strip-app-prefix-and-query-string (url app-prefix)
(max 0 (1- (length app-prefix)))
(position #\? url)))
(defmethod clack:call ((app lucerne.app:base-app) env)
"Routes the request determined by @cl:param(env) on the application
(let* ((req (make-request env))
(method (request-method req))
(prefix (script-name req))
(uri (request-uri req))
(final-uri (strip-app-prefix-and-query-string uri prefix))
;; Now, we actually do the dispatching
(route (myway:dispatch (lucerne.app:routes app)
;; We have a hit
(funcall route req)
;; Not found
(let ((lucerne.http:*request* req))
The utility function at the top does the parsing. It turned out that the request uri was not actually coming in with the script name as a prefix.
The odd behavior that i described was happening because the uri prefix that I found “worked” happened to have the same number of characters as the script name, so grabbing the suffix did the right thing.
I edited a local copy of the code above to not modify the request uri and now I’m in business.