FYI: Clozure Common Lisp for [f]cgi works

Since it seems not to have been mentioned on this forum before, but I did see a few other random souls asking:

If you have a Common Lisp app that you’d like to compile & deploy on Dreamhost, Clozure CL does run just fine.

If anybody’s deadly interested in the details, I could post more, but in essence, you can

 svn checkout ~/ccl

Follow the instructions on to rebuild the Lisp kernel. Since I’m on shared hosting, I ran each step with “nice” to avoid impacting other users.

I put this in ~/bin/ccl:

 exec $HOME/ccl/lx86cl64 "$@"

To compile, I use Xach’s BuildApp and Quicklisp, so I went through the three or so steps to set up Quicklisp ( and installed BuildApp through it, putting the binary in ~/bin

To deploy, I do a git checkout of my app code and run (“nice”) BuildApp over SSH, then move the file into the server root.

Alternatively, I had tried SBCL, but ran into problems where it’s confused with the address-space randomization used (I think). Running SBCL yields an error message about mmap failing. I haven’t tried any other compilers (Allegro, Lispworks, etc).

I have considered building a local VM to match Dreamhost’s library configs, etc. so that I can reliably compile locally and deploy remotely, but my applications (at least) tend to be dependent on the system libraries and naïvely building on a default Ubuntu 12.04 image here didn’t get me a usable executable.

(Note to Dreamhost staff, it’d be nifty if there were some way for us folks who compile our programs to get a reliable local clone of a shared host …)

I’m very interested! I’ve been gathering information on how to do this for the past two days. Unfortunately, I haven’t been able to find much. Just a lot of dead links.

I have an app built on Clack that I’ve been developing on a local machine, using Hunchentoot as the server. I would now like to deploy that application to my DH site.

Like you, I ran intro trouble with sbcl, but have successfully gotten ccl running.

Can you share some more information on how you set [f]cgi up to accomplish this? Say I want my application deployed under /my-app. Where does the executable application go, and what goes in .htaccess? I’ve been looking at using mod_rewrite to send all requests to the application, but I’m still a little fuzzy on how it is all going to work (I’m looking at mod_rewrite because the directives for mod_fastcgi generally don’t appear to be applicable at the directory level).

Finally, what does the application (toplevel function) look like? I tried building an application similar to the last code listing in this article. If I just try executing it from the command line, it appears my application starts up and immediately exits (maybe that has to do with expecting fd 0 to be a bi-directional stream?)

First bit: running SBCL.

So, in the interim since my prior post, I’ve changed my ways a little.

Personally: I just happen to like SBCL a little more, so I built a Docker to compile SBCL in, with the following lame Dockerfile. Note DHUSER matches the Dreamhost user account under which I deploy it, just in case any paths end up hard-coded into the corefile or something:

FROM ubuntu:12.04 MAINTAINER Bruce-Robert Fenn Pocock <> RUN sh -c 'dhclient eth0 &' RUN apt-get update RUN apt-get install -y make sbcl curl gnupg unzip RUN cat /etc/passwd RUN echo DHUSER:x:1000:0::/home/DHUSER:/bin/bash >> /etc/passwd RUN cp -a /etc/skel/ /home/DHUSER RUN chown -R DHUSER /home/DHUSER RUN echo 'DHUSER:$6$norealpassword:16615:0:9999:7:::' >> /etc/shadow RUN su - DHUSER -c 'curl -OL' RUN su - DHUSER -c 'unzip' COPY map-patch.patch /home/DHUSER/sbcl/map-patch.patch RUN apt-get install -y gcc patch RUN su - DHUSER -c 'cd sbcl-master; ls src/runtime/linux-os.c ; patch -p1 < ~/sbcl/map-patch.patch' RUN su - DHUSER -c 'cd sbcl-master; echo "\"1.3.0x\"" > version.lisp-expr' RUN apt-get install -y time RUN su - DHUSER -c 'cd sbcl-master; ./ --prefix=/home/DHUSER/sbcl' #RUN apt-get install -y ed #RUN su - DHUSER -c 'cd sbcl-master/tests; ./' RUN chown DHUSER -R /home/DHUSER/sbcl RUN su - DHUSER -c 'cd sbcl-master; ./ --prefix=/home/DHUSER/sbcl' RUN apt-get remove -y sbcl RUN su - DHUSER -c 'curl -LO' COPY quicklisp.lisp.asc /home/DHUSER/quicklisp.lisp.asc #RUN su - DHUSER -c 'gpg --verify quicklisp.lisp.asc quicklisp.lisp' RUN su - DHUSER -c "./sbcl/bin/sbcl --load ./quicklisp.lisp --eval '(quicklisp-quickstart:install)' \ --eval '(ql:add-to-init-file)' --eval '(quit)'" RUN su - DHUSER -c "./sbcl/bin/sbcl --load ~/quicklisp/setup.lisp --eval '(quicklisp:quickload :caveman2)' --eval '(quit)'"

Yes, that’s not super-efficient, running apt-get multiple times, … it grew on me. The fails, BTW.

The associated patch for address-randomization corrections is:

# Patch mmap for Dreamhost's address-randomization code
--- master/src/runtime/linux-os.c
+++ patched/src/runtime/linux-os.c
@@ -337,6 +337,9 @@ is_validate(os_vm_address_t addr, os_vm_size_t len)
+    if (addr) flags |= MAP_FIXED;
     actual = mmap(addr, len, OS_VM_PROT_ALL, flags, -1, 0);
     if (actual == MAP_FAILED) {

That’s against the master branch of SBCL as of summer 2016; I haven’t probed the #sbcl cabal about getting it checked in (or why it isn’t already) yet.

So, the above yields inside the Docker a working SBCL at /home/DHUSER/sbcl that can be used to compile. SFTP those up to Dreamhost as well.

Next bit: CLACK main function and .htaccess. This took some facepalming trial-and-error, sadly, and is probably not the most efficient either, but:

Options ExecCGI SymLinksIfOwnerMatch
AddHandler fastcgi-script .cgi

AcceptPathInfo On
RewriteEngine On

RewriteCond %{HTTPS} !=on
RewriteRule (.*) https://|your-domain-name|%{REQUEST_URI} [R=301,L]

RewriteRule ^|prefix|/(.*)$ |executable|.cgi/%1 [QSA,L]
RewriteRule ^[Pp][Ll][Aa][Yy].* |executable|.cgi/login [QSA,L]

Header always set X-Frame-Options SAMEORIGIN

Here |your-domain-name| is obvious and that bit enforces TLS-only. Drop those two lines if you don’t care.

The next two lines map |prefix| to |executable|.cgi and pass the path-info so (DEFROUTE) will work. (Or whatever Ningle routing you use.) And remap a “pretty” URI pattern to a specific point.

That gets you into the FCGI executable.

Here’s my top-level Makefile:

all:	|your-app-name|.cgi
|your-app-name|.cgi:	|your-app-name|.asd $(shell find . -name \*.lisp -and -not -path '**/lib/**')
	PATH=~/bin:${PATH} ~/bin/buildapp --output |your-app-name| \
		--load ~/quicklisp/setup.lisp \
		--asdf-path . \
		--eval '(ql:quickload :|your-app-name|)' \
		--load-system |your-app-name| \
		--entry |your-app-name|:fastcgi-entry
	mv --backup=t |your-app-name| |your-app-name|.cgi

src/lib/jscl.js:	$(shell find src/lib/jscl -name \*.lisp -or -name \*.lisp)
	cd src/lib/jscl; ./

Pretty typical stuff, I suppose. Note particularly that BuildApp cannot overwrite an executable while it’s running, so building a new copy then moving it into place is required if you ever need to do an in-production patch job; BuildApp won’t do that bit for you. (The --backup=t means you’ll end up with myapp.cgi.~1~ for ever-increasing values of ~1~ over time, which is helpful if you mess up but you’ll want to clean up behind yourself periodically.)

Then: Clack entry point #'FASTCGI-ENTRY: (I have this in src/main.lisp right after Caveman’s generated #'START for Hunchentoot)

(defun fastcgi-entry (argv)
  (format t "Content-Type: text/plain~c~c~c~c
  Starting FastCGI is half-working if you see this. But, only half.~%"
  #\Return #\Linefeed #\Return #\Linefeed)
  (format *error-output* "~%FastCGI started with ARGV=~s~%" argv)
  (let ((ql:*quickload-explain* nil) (ql:*quickload-verbose* nil))
    (clackup *appfile-path* :server :fcgi :port nil :use-thread nil :fd 0)))

The #'FORMAT calls are hopefully irrelevant now, but I had needed them for debugging this.

(defun fastcgi-entry (argv)
  (declare (ignore argv))
  (clackup *appfile-path* :server :fcgi :port nil :use-thread nil :fd 0))

… that should really do it. NB I “had to” pass the :fd 0 explicitly, even though it seems like it would be the default. (FD 0 = STANDARD-INPUT)

Hope that helps!

PS. The pipe chars in the examples are metasyntactic. So eg

RewriteRule (.*){REQUEST_URI} [R=301,L]

This is great read folks! If any of you wants to write up a tutorial on how to do this on DreamCompute, I’d be glad to provide you free accounts. Let me know!

Thanks brpocock, this is great stuff!

It’s going to take me a little while to absorb it all. In the mean time, I sort of got my application working last night. Some of the routes work, while others result in an internal server error (Premature end of script headers). I wonder if it has to do with the fact that I am not currently passing the requested uri to the script? Something along the lines of what you did here:

More on that in a bit.

While my application still needs work, I thought I would post a very pared down example that might point somebody in the right direction (feedback welcome!)

Step 1 - Build an application
For my example, i’m still using ccl, and the built-in function ccl:save-application. Below is a lisp file that can be loaded. It will: ensure clack (insert your additional dependencies here) is loaded into the lisp image, define an entry point that builds a simple application, and dump the lisp image into an executable file that will begin in main.

(ql:quickload "clack")
(ql:quickload "clack-handler-fcgi")

(defun main ()
   (let ((visits 0))
     (lambda (env)
         (:content-type "text/plain")
         (,(format nil "Hello visitor ~a!~%You are requesting: ~a"
                   (incf visits) env)))))
   :server :fcgi
   :use-thread nil
   :silent t
   :fd 0))

(ccl:save-application "my-app.fcgi"
                      :toplevel-function #'main 
                      :prepend-kernel t)

You can load this from a repl

cl-user> (load "~/my-app.lisp")

or run it through ccl from the command line:

$ ccl64 -l my-app.lisp

This is a really simple application, but it does two things I found helpful:

  1. It maintains some state (number of visits). I wanted to verify that the application process was persisting between calls to the web server
    2.) It outputs the request headers. I wanted to make sure the proper request uri was coming in. This is why I wasn’t sure that I needed to be passing any additional parameter(s) when executing the script, as mentioned above.

Step 2 - Drop the application into your document root
This is easy, just copy or sftp my-app.fcgi up

Step 3 - Configure .htaccess
Below is what I have. Just the minimum to get going. My intention with this is to have any request for /my-app or /my-app/whatever-else result in my fcgi process being used. (However, a request to /my-application doesn’t get handled that way)

Contents of .htaccess in the document root

Options +FollowSymLinks
RewriteEngine On
RewriteRule ^my-app(/.*)?$ my-app.fcgi [L] 

So that did it! Below is sanitized, sample output from a request to /my-app/its-working

Hello visitor 1!

From here, I still need/want to:

  • Figure out why my real application is not working
  • Beef up my .htaccess
  • Review your build/deploy notes to get a process in place
  • See if there is a way to increase how long the fcgi process is kept around

Any suggestions for improvement to example above are welcome.

With my real application, this is what I’ve determined:

For now, I’m placing the app at my web root. I have the following routes defined (using lucerne)

Trying to hit the top level routes (/ /foo /bar /baz), only /foo returns something. All others result in an error log entry “Premature end of script headers”. Now what /foo returns is interesting. It returns what should be returned for “/”! And further, the rest of my site is available under /foo! So /foo/foo returns what I would expect for /foo, and /foo/barb/this returns what I would expect for /bar/this.

I’m not sure what to make of this. Based on the test application from above, I have every reason to believe the proper request URIs are being passed to the application. I’m using clack from quicklisp 2016-06-28. It is possible it is an issue with the fcgi backend? It is very odd. Running with the hunchentoot backend works as expected.

More digging to be done.

I’ve edited by .htaccess file to be very similar to yours. At one point I kept getting too many internal redirect errors. I lost track of which update to .htaccess fixed it, but I seem to be in the clear now.

The odd routing behavior that I encountered had to do with how the framework I was using on top of clack (lucerne) was parsing the request uri. It was assuming that whatever can in with the request under script-name was also a prefix of the request uri. Therefore, it was attempting to grab a substring starting at the end of the prefix. The relevant code from the current version looks like this

(defun strip-app-prefix-and-query-string (url app-prefix)
  (subseq url
          (max 0 (1- (length app-prefix)))
          (position #\? url)))

(defmethod clack:call ((app env)
  "Routes the request determined by @cl:param(env) on the application
  (let* ((req    (make-request env))
         (method (request-method req))
         (prefix (script-name req))
         (uri    (request-uri req))
         (final-uri (strip-app-prefix-and-query-string uri prefix))
         ;; Now, we actually do the dispatching
         (route (myway:dispatch ( app)
                                :method method)))
    (if route
        ;; We have a hit
        (funcall route req)
        ;; Not found
        (let ((lucerne.http:*request* req))
          (not-found app)))))

The utility function at the top does the parsing. It turned out that the request uri was not actually coming in with the script name as a prefix.

The odd behavior that i described was happening because the uri prefix that I found “worked” happened to have the same number of characters as the script name, so grabbing the suffix did the right thing.

I edited a local copy of the code above to not modify the request uri and now I’m in business.

Glad to be of some help.
Sorry I didn’t see the intermediate messages until now …

Re: staff: drop me an e-mail or something? I probably will write up the above in a blog or sommat, regardless.

We have published the details of the program to get contributions in exchange of Cloud credits this week. You can find them on the github repository for cloud documentation:

Basically for every tutorial merged (at least 250 words long), we remove $100 from dreamhost cloud’s bill (dreacompute, dreamobjects, dreamspeed).

It was assuming that whatever can in with the request under script-name was also a prefix of the request uri.