FYI: Clozure Common Lisp for [f]cgi works

software development

#1

Since it seems not to have been mentioned on this forum before, but I did see a few other random souls asking:

If you have a Common Lisp app that you’d like to compile & deploy on Dreamhost, Clozure CL does run just fine.

If anybody’s deadly interested in the details, I could post more, but in essence, you can

 svn checkout http://svn.clozure.com/publicsvn/openmcl/release/1.10/linuxx86/ccl ~/ccl

Follow the instructions on Clozure.com to rebuild the Lisp kernel. Since I’m on shared hosting, I ran each step with “nice” to avoid impacting other users.

I put this in ~/bin/ccl:

 #!/bin/sh
 exec $HOME/ccl/lx86cl64 "$@"

To compile, I use Xach’s BuildApp and Quicklisp, so I went through the three or so steps to set up Quicklisp (quicklisp.org) and installed BuildApp through it, putting the binary in ~/bin

To deploy, I do a git checkout of my app code and run (“nice”) BuildApp over SSH, then move the file into the server root.

Alternatively, I had tried SBCL, but ran into problems where it’s confused with the address-space randomization used (I think). Running SBCL yields an error message about mmap failing. I haven’t tried any other compilers (Allegro, Lispworks, etc).

I have considered building a local VM to match Dreamhost’s library configs, etc. so that I can reliably compile locally and deploy remotely, but my applications (at least) tend to be dependent on the system libraries and naïvely building on a default Ubuntu 12.04 image here didn’t get me a usable executable.

(Note to Dreamhost staff, it’d be nifty if there were some way for us folks who compile our programs to get a reliable local clone of a shared host …)


#2

I’m very interested! I’ve been gathering information on how to do this for the past two days. Unfortunately, I haven’t been able to find much. Just a lot of dead links.

I have an app built on Clack that I’ve been developing on a local machine, using Hunchentoot as the server. I would now like to deploy that application to my DH site.

Like you, I ran intro trouble with sbcl, but have successfully gotten ccl running.

Can you share some more information on how you set [f]cgi up to accomplish this? Say I want my application deployed under /my-app. Where does the executable application go, and what goes in .htaccess? I’ve been looking at using mod_rewrite to send all requests to the application, but I’m still a little fuzzy on how it is all going to work (I’m looking at mod_rewrite because the directives for mod_fastcgi generally don’t appear to be applicable at the directory level).

Finally, what does the application (toplevel function) look like? I tried building an application similar to the last code listing in this article. If I just try executing it from the command line, it appears my application starts up and immediately exits (maybe that has to do with expecting fd 0 to be a bi-directional stream?)


#3

First bit: running SBCL.

So, in the interim since my prior post, I’ve changed my ways a little.

Personally: I just happen to like SBCL a little more, so I built a Docker to compile SBCL in, with the following lame Dockerfile. Note DHUSER matches the Dreamhost user account under which I deploy it, just in case any paths end up hard-coded into the corefile or something:

FROM ubuntu:12.04 MAINTAINER Bruce-Robert Fenn Pocock <brfennpocock@star-hope.org> RUN sh -c 'dhclient eth0 &' RUN apt-get update RUN apt-get install -y make sbcl curl gnupg unzip RUN cat /etc/passwd RUN echo DHUSER:x:1000:0::/home/DHUSER:/bin/bash >> /etc/passwd RUN cp -a /etc/skel/ /home/DHUSER RUN chown -R DHUSER /home/DHUSER RUN echo 'DHUSER:$6$norealpassword:16615:0:9999:7:::' >> /etc/shadow RUN su - DHUSER -c 'curl -OL https://github.com/sbcl/sbcl/archive/master.zip' RUN su - DHUSER -c 'unzip master.zip' COPY map-patch.patch /home/DHUSER/sbcl/map-patch.patch RUN apt-get install -y gcc patch RUN su - DHUSER -c 'cd sbcl-master; ls src/runtime/linux-os.c ; patch -p1 < ~/sbcl/map-patch.patch' RUN su - DHUSER -c 'cd sbcl-master; echo "\"1.3.0x\"" > version.lisp-expr' RUN apt-get install -y time RUN su - DHUSER -c 'cd sbcl-master; ./make.sh --prefix=/home/DHUSER/sbcl' #RUN apt-get install -y ed #RUN su - DHUSER -c 'cd sbcl-master/tests; ./run-tests.sh' RUN chown DHUSER -R /home/DHUSER/sbcl RUN su - DHUSER -c 'cd sbcl-master; ./install.sh --prefix=/home/DHUSER/sbcl' RUN apt-get remove -y sbcl RUN su - DHUSER -c 'curl -LO https://beta.quicklisp.org/quicklisp.lisp' COPY quicklisp.lisp.asc /home/DHUSER/quicklisp.lisp.asc #RUN su - DHUSER -c 'gpg --verify quicklisp.lisp.asc quicklisp.lisp' RUN su - DHUSER -c "./sbcl/bin/sbcl --load ./quicklisp.lisp --eval '(quicklisp-quickstart:install)' \ --eval '(ql:add-to-init-file)' --eval '(quit)'" RUN su - DHUSER -c "./sbcl/bin/sbcl --load ~/quicklisp/setup.lisp --eval '(quicklisp:quickload :caveman2)' --eval '(quit)'"

Yes, that’s not super-efficient, running apt-get multiple times, … it grew on me. The run-tests.sh fails, BTW.

The associated patch for address-randomization corrections is:

# Patch mmap for Dreamhost's address-randomization code
--- master/src/runtime/linux-os.c
+++ patched/src/runtime/linux-os.c
@@ -337,6 +337,9 @@ is_validate(os_vm_address_t addr, os_vm_size_t len)
         addr=under_2gb_free_pointer;
     }
 #endif
+
+    if (addr) flags |= MAP_FIXED;
+
     actual = mmap(addr, len, OS_VM_PROT_ALL, flags, -1, 0);
     if (actual == MAP_FAILED) {
         perror("mmap");

That’s against the master branch of SBCL as of summer 2016; I haven’t probed the #sbcl cabal about getting it checked in (or why it isn’t already) yet.

So, the above yields inside the Docker a working SBCL at /home/DHUSER/sbcl that can be used to compile. SFTP those up to Dreamhost as well.

Next bit: CLACK main function and .htaccess. This took some facepalming trial-and-error, sadly, and is probably not the most efficient either, but:

Options ExecCGI SymLinksIfOwnerMatch
AddHandler fastcgi-script .cgi

AcceptPathInfo On
RewriteEngine On

RewriteCond %{HTTPS} !=on
RewriteRule (.*) https://|your-domain-name|%{REQUEST_URI} [R=301,L]

RewriteRule ^|prefix|/(.*)$ |executable|.cgi/%1 [QSA,L]
RewriteRule ^[Pp][Ll][Aa][Yy].* |executable|.cgi/login [QSA,L]

Header always set X-Frame-Options SAMEORIGIN

Here |your-domain-name| is obvious and that bit enforces TLS-only. Drop those two lines if you don’t care.

The next two lines map |prefix| to |executable|.cgi and pass the path-info so (DEFROUTE) will work. (Or whatever Ningle routing you use.) And remap a “pretty” URI pattern to a specific point.

That gets you into the FCGI executable.

Here’s my top-level Makefile:

all:	|your-app-name|.cgi
|your-app-name|.cgi:	|your-app-name|.asd $(shell find . -name \*.lisp -and -not -path '**/lib/**')
	PATH=~/bin:${PATH} ~/bin/buildapp --output |your-app-name|.cgi.new \
		--load ~/quicklisp/setup.lisp \
		--asdf-path . \
		--eval '(ql:quickload :|your-app-name|)' \
		--load-system |your-app-name| \
		--entry |your-app-name|:fastcgi-entry
	mv --backup=t |your-app-name|.cgi.new |your-app-name|.cgi

src/lib/jscl.js:	$(shell find src/lib/jscl -name \*.lisp -or -name \*.lisp)
	cd src/lib/jscl; ./make.sh

Pretty typical stuff, I suppose. Note particularly that BuildApp cannot overwrite an executable while it’s running, so building a new copy then moving it into place is required if you ever need to do an in-production patch job; BuildApp won’t do that bit for you. (The --backup=t means you’ll end up with myapp.cgi.~1~ for ever-increasing values of ~1~ over time, which is helpful if you mess up but you’ll want to clean up behind yourself periodically.)

Then: Clack entry point #'FASTCGI-ENTRY: (I have this in src/main.lisp right after Caveman’s generated #'START for Hunchentoot)

(defun fastcgi-entry (argv)
  (format t "Content-Type: text/plain~c~c~c~c
  
  Starting FastCGI is half-working if you see this. But, only half.~%"
  #\Return #\Linefeed #\Return #\Linefeed)
  (format *error-output* "~%FastCGI started with ARGV=~s~%" argv)
  (let ((ql:*quickload-explain* nil) (ql:*quickload-verbose* nil))
    (clackup *appfile-path* :server :fcgi :port nil :use-thread nil :fd 0)))

The #'FORMAT calls are hopefully irrelevant now, but I had needed them for debugging this.

(defun fastcgi-entry (argv)
  (declare (ignore argv))
  (clackup *appfile-path* :server :fcgi :port nil :use-thread nil :fd 0))

… that should really do it. NB I “had to” pass the :fd 0 explicitly, even though it seems like it would be the default. (FD 0 = STANDARD-INPUT)

Hope that helps!
[hr]

PS. The pipe chars in the examples are metasyntactic. So eg

RewriteRule (.*) https://example.com%{REQUEST_URI} [R=301,L]

#4

This is great read folks! If any of you wants to write up a tutorial on how to do this on DreamCompute, I’d be glad to provide you free accounts. Let me know!


#5

Thanks brpocock, this is great stuff!

It’s going to take me a little while to absorb it all. In the mean time, I sort of got my application working last night. Some of the routes work, while others result in an internal server error (Premature end of script headers). I wonder if it has to do with the fact that I am not currently passing the requested uri to the script? Something along the lines of what you did here:

More on that in a bit.

While my application still needs work, I thought I would post a very pared down example that might point somebody in the right direction (feedback welcome!)

Step 1 - Build an application
For my example, i’m still using ccl, and the built-in function ccl:save-application. Below is a lisp file that can be loaded. It will: ensure clack (insert your additional dependencies here) is loaded into the lisp image, define an entry point that builds a simple application, and dump the lisp image into an executable file that will begin in main.

(ql:quickload "clack")
(ql:quickload "clack-handler-fcgi")

(defun main ()
  (clack:clackup
   (let ((visits 0))
     (lambda (env)
       `(200
         (:content-type "text/plain")
         (,(format nil "Hello visitor ~a!~%You are requesting: ~a"
                   (incf visits) env)))))
   :server :fcgi
   :use-thread nil
   :silent t
   :fd 0))

(ccl:save-application "my-app.fcgi"
                      :toplevel-function #'main 
                      :prepend-kernel t)

You can load this from a repl

cl-user> (load "~/my-app.lisp")

or run it through ccl from the command line:

$ ccl64 -l my-app.lisp

This is a really simple application, but it does two things I found helpful:

  1. It maintains some state (number of visits). I wanted to verify that the application process was persisting between calls to the web server
    2.) It outputs the request headers. I wanted to make sure the proper request uri was coming in. This is why I wasn’t sure that I needed to be passing any additional parameter(s) when executing the script, as mentioned above.

Step 2 - Drop the application into your document root
This is easy, just copy or sftp my-app.fcgi up

Step 3 - Configure .htaccess
Below is what I have. Just the minimum to get going. My intention with this is to have any request for /my-app or /my-app/whatever-else result in my fcgi process being used. (However, a request to /my-application doesn’t get handled that way)

Contents of .htaccess in the document root

Options +FollowSymLinks
RewriteEngine On
RewriteRule ^my-app(/.*)?$ my-app.fcgi [L] 

So that did it! Below is sanitized, sample output from a request to /my-app/its-working

Hello visitor 1!
You are requesting: (SCRIPT-NAME /my-app.fcgi REQUEST-URI /my-app/its-working QUERY-STRING  REQUEST-METHOD GET SERVER-PROTOCOL HTTP/1.1 REMOTE-PORT 12345 REMOTE-ADDR 12.34.56.78 SERVER-PORT 80 SERVER-NAME my.examplesite.com HEADERS #<HASH-TABLE :TEST EQUAL size 6/60 #x2B16D308C5D> PATH-INFO /my-app/its-working URL-SCHEME HTTP RAW-BODY #<VECTOR-INPUT-STREAM #x2B16D30436D> CLACK.STREAMING T)

From here, I still need/want to:

  • Figure out why my real application is not working
  • Beef up my .htaccess
  • Review your build/deploy notes to get a process in place
  • See if there is a way to increase how long the fcgi process is kept around

Any suggestions for improvement to example above are welcome.


#6

With my real application, this is what I’ve determined:

For now, I’m placing the app at my web root. I have the following routes defined (using lucerne)
/
/foo
/foo/thing1
/foo/thing2
/bar
/bar/this
/bar/that
/baz

Trying to hit the top level routes (/ /foo /bar /baz), only /foo returns something. All others result in an error log entry “Premature end of script headers”. Now what /foo returns is interesting. It returns what should be returned for “/”! And further, the rest of my site is available under /foo! So /foo/foo returns what I would expect for /foo, and /foo/barb/this returns what I would expect for /bar/this.

I’m not sure what to make of this. Based on the test application from above, I have every reason to believe the proper request URIs are being passed to the application. I’m using clack from quicklisp 2016-06-28. It is possible it is an issue with the fcgi backend? It is very odd. Running with the hunchentoot backend works as expected.

More digging to be done.


#7

I’ve edited by .htaccess file to be very similar to yours. At one point I kept getting too many internal redirect errors. I lost track of which update to .htaccess fixed it, but I seem to be in the clear now.

The odd routing behavior that I encountered had to do with how the framework I was using on top of clack (lucerne) was parsing the request uri. It was assuming that whatever can in with the request under script-name was also a prefix of the request uri. Therefore, it was attempting to grab a substring starting at the end of the prefix. The relevant code from the current version looks like this

(defun strip-app-prefix-and-query-string (url app-prefix)
  (subseq url
          (max 0 (1- (length app-prefix)))
          (position #\? url)))

(defmethod clack:call ((app lucerne.app:base-app) env)
  "Routes the request determined by @cl:param(env) on the application
@cl:param(app)."
  (let* ((req    (make-request env))
         (method (request-method req))
         (prefix (script-name req))
         (uri    (request-uri req))
         (final-uri (strip-app-prefix-and-query-string uri prefix))
         ;; Now, we actually do the dispatching
         (route (myway:dispatch (lucerne.app:routes app)
                                final-uri
                                :method method)))
    (if route
        ;; We have a hit
        (funcall route req)
        ;; Not found
        (let ((lucerne.http:*request* req))
          (not-found app)))))

The utility function at the top does the parsing. It turned out that the request uri was not actually coming in with the script name as a prefix.

The odd behavior that i described was happening because the uri prefix that I found “worked” happened to have the same number of characters as the script name, so grabbing the suffix did the right thing.

I edited a local copy of the code above to not modify the request uri and now I’m in business.


#8

Glad to be of some help.
Sorry I didn’t see the intermediate messages until now …

Re: staff: drop me an e-mail or something? I probably will write up the above in a blog or sommat, regardless.


#9

We have published the details of the program to get contributions in exchange of Cloud credits this week. You can find them on the github repository for cloud documentation:

https://github.com/dreamhost/dreamcloud-docs/blob/master/CONTRIBUTING.rst

Basically for every tutorial merged (at least 250 words long), we remove $100 from dreamhost cloud’s bill (dreacompute, dreamobjects, dreamspeed).


#10

It was assuming that whatever can in with the request under script-name was also a prefix of the request uri.