Lisp Web Server From Scratch using Hunchentoot and Nginx

**************************I tried my setup on Ubuntu 13.10 server and had some issues so I decided to update this article, unfortunately I dont have the time to rewrite it so I am just hacking the bits that where issues on Ubuntu 13.10**********************

So you want your own lisp web server to hack away at but where do you start? Well don’t despair, what follows is a recipe for getting such a server running in no time. (Don’t worry you won’t be coding it from scratch you will be building it with a “software lego” set.)

Much of what you will see here was taken from different websites especially these two great articles http://blog.ponto-dot.com/2010/08/15/setting-up-common-lisp-on-a-web-server and http://blog.ponto-dot.com/2009/08/18/hunchentoot-behind-proxy-server. These articles do a better job of explaining what is afoot than my posting here. Please do go to these articles and show your appreciation for the hard work that went into them by leaving a comment.

All that I am doing is giving you an A to Z recipe in one place to get you going.

There are a lot of good articles in the slicehost articles collection as well that are worth a read on their own if you want to understand more about the iptables and stuff.

Just a note, when I supply a command for the command line I will prefix it with ~# for the server or ~local# for the local machine

First things first, lets set-up a secure server:

1. Install your slice from www.slicehost.com. A Ubuntu 10.10 (maverick meerkat) 32bit 512 slice should be enough. (Initially I thought a 256 slice would be enough but once sbcl gets used it reserves memory and if you don’t have enough spare for the GC to run then you end up with problems. Maybe one of the experts can enlighten us on what the reasons for this is and what the best way is to handle it. If budget is an issue you can limit the amount of memory sbcl reserves when you load it up. I suppose you need to experiment till you find then optimal settings for your set up.)

2. ssh into your new slice with the supplied password.

  ~local#ssh root@ip_address_of_server

3. Set up your locale.

  ~#locale-gen en_ZA.UTF-8
  ~#update-locale LANG=en_ZA.UTF-8
  ~#dpkg-reconfigure locales
  #Set the local time
  ~#dpkg-reconfigure tzdata

3. Update the server.

  ~#apt-get update
  ~#apt-get upgrade

4. Change the root password. Do I have to remind you to choose something sensible but secure enough?

  ~#passwd

5. Add a new user called admin and add it to the sudoers list. This is the user you will be using in the future to ssh into the server. We are going to disallow logging in as root later so you cant skip this step unless you want to experience locking your self out of the server later.

  ~#adduser admin
  ~#usermod -a -G sudo admin

6. Create a ssh key and upload your public key to the server. Dont leave the ssh passphrase blank!

  ~local#ssh-keygen -t rsa
  ~local#scp ~/.ssh/id_rsa.pub admin@ip_address_of_server:

7. Create the authorized_keys file and set its permissions correctly. ssh is very pedantic when it comes to permissions!

  ~#mkdir ~admin/.ssh
  ~#mv ~admin/id_rsa.pub ~admin/.ssh/authorized_keys
  ~#chown -R admin:admin ~admin/.ssh
  ~#chmod 700 ~admin/.ssh
  ~#chmod 600 ~admin/.ssh/authorized_keys

8. Next, let’s edit the /etc/ssh/sshd_config file and disable some defaults. Here are the adjustments you should make:

  ~#nano /etc/ssh/sshd_config

  Port 22222                           <--- change to a port of your choosing
  Protocol 2
  PermitRootLogin no
  PasswordAuthentication no
  X11Forwarding no
  UsePAM no
  AllowUsers admin

9. Set up IPTABLES to make server more secure.

/sbin/iptables -F
nano /etc/iptables.up.rules

Paste the following into the file:

*filter

#  Allows all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT

#  Accepts all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#  Allows all outbound traffic
#  You can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

# Allows HTTP and HTTPS connections from anywhere (the normal ports for websites)
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

#  Allows SSH connections
#
# THE -dport NUMBER IS THE SAME ONE YOU SET UP IN THE SSHD_CONFIG FILE
#
-A INPUT -p tcp -m state --state NEW --dport 22222 -j ACCEPT

# Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

# log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

# Reject all other inbound - default deny unless explicitly allowed policy
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT

10. Tell the server to use the new rules.

  ~#/sbin/iptables-restore < /etc/iptables.up.rules

11. To make sure the iptable rules are automatically loaded in the future create the following script.

  ~#nano /etc/network/if-pre-up.d/iptables

#!/bin/sh
/sbin/iptables-restore < /etc/iptables.up.rules

  #Make the file executable
  ~#chmod +x /etc/network/if-pre-up.d/iptables

12. Now that we have our security in place lets reset ssh

  ~#/etc/init.d/ssh reload
  *****or for Ubuntu 13.10****
  ~#restart ssh

13. To test the ssh security settings run the following command from your local machine without closing the current ssh connection. If this command does not log you in automatically you have fluffed something along the way and need to recheck your settings!

  ~local#ssh -p 22222 admin@ip_address_of_server

Install and set up nginx:

(From here on we will be using the session where we logged in as admin so that is why there are sudo’s in the commands now.)

  ~#sudo apt-get install nginx
  ~#sudo nano /etc/nginx/nginx.conf

user www-data;
worker_processes  4;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
    # multi_accept on;
}

http {
    include       /etc/nginx/mime.types;

    access_log  /var/log/nginx/access.log;

    sendfile        on;
    tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  3;
    tcp_nodelay        off;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
    gzip_comp_level 2;
    gzip_proxied any;
    gzip_types  text/plain text/css application/x-javascript text/xml
                application/xml application/xml+rss text/javascript;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Install and set up SBCL:
(Note: if you did not chose a 32 bit slice you must adjust the sbcl-1.0.43-x86-linux-binary.tar.bz2 download to the correct version.)

  ~#wget http://prdownloads.sourceforge.net/sbcl/sbcl-1.1.15-x86-64-linux-binary.tar.bz2
  ~#bzip2 -cd sbcl-1.1.15-x86-64-linux-binary.tar.bz2 | tar xvf -

  ~#cd sbcl-1.1.15-x86-64-linux

  ~#sudo apt-get install build-essential

  ~#sudo sh install.sh



To test sbcl run the following.
  ~#sbcl

Set up a hunchentoot user that will be used to run the hunchentoot/sbcl server process in detachtty:

  ~#sudo adduser --system --disabled-password --home /home/hunchentoot --shell /bin/bash hunchentoot

Install and setup Quicklisp as your lisp library repository and setup the infrastructure for your own projects.:

1. Set up Quicklisp.

~#curl -O http://beta.quicklisp.org/quicklisp.lisp

~#sbcl --load quicklisp.lisp

(quicklisp-quickstart:install)
(ql:add-to-init-file)
(ql:quickload "swank")
(ql:quickload "hunchentoot")
(ql:quickload "cl-who")

(quit)

  ~#sudo ln -s /home/admin/quicklisp/ /home/hunchentoot/quicklisp

Configure the web server as a process that can be started and stoped:

1. Set up the web server process.

Telnet is used to kill hunchentoot server
  ~#sudo apt-get install telnet 

  ~#sudo nano /etc/init.d/hunchentoot

(You can download the following script from http://blog.ponto-dot.com/2010/08/15/setting-up-common-lisp-on-a-web-server. 
Once again the cudos for this must go to the original author.)

#!/bin/sh
### BEGIN INIT INFO
# Provides:          hunchentoot
# Required-Start:    $all
# Required-Stop:     $
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Launch Hunchentoot
# Description:       Launch Hunchentoot with start-stop-daemon
### END INIT INFO
NAME=hunchentoot

. /lib/lsb/init-functions

hunchentoot_start()
{
    echo -n "Starting Lisp Hunchentoot Server process..."
    echo ""
    su - hunchentoot -c ~hunchentoot/start-hunchentoot
}

hunchentoot_stop ()
{
    echo -n "Stopping Lisp Hunchentoot Server process..."
    echo ""

    # 6200 is the port to signal end of lisp process
    (telnet 127.0.0.1 6200 &) > /dev/null
    (sleep 7)
}

hunchentoot_usage()
{
    echo "Usage: /etc/init.d/hunchentoot {start|stop|restart|status}"
    exit 1
}

hunchentoot_status()
{
    status_of_proc -p "/home/hunchentoot/run/$NAME.pid" "$NAME" $NAME
}

case "$1" in
    start)
        hunchentoot_start
        ;;

    stop)
        hunchentoot_stop
        ;;

    restart)
        hunchentoot_stop
        hunchentoot_start
        ;;

    status)
        hunchentoot_status
        ;;

    *)
        hunchentoot_usage
        ;;

esac

exit 0

  ~#sudo chmod +x /etc/init.d/hunchentoot

  ~#wget https://github.com/huetsch/detachtty/archive/master.zip
  ~#sudo apt-get install unzip
  ~#mkdir source
  ~#unzip master.zip -d source/
  ~#cd source/detachtty-master/
  ~#make
  ~#sudo make install
  ~#sudo install detachtty attachtty /usr/local/bin

  ~#cd ~hunchentoot
  ~#sudo mkdir log run
  ~#sudo chown hunchentoot:nogroup log run
 

  ~#sudo nano /home/hunchentoot/start-hunchentoot

#!/bin/bash

detachtty --dribble-file /home/hunchentoot/log/hunchentoot.dribble \
          --log-file /home/hunchentoot/log/hunchentoot.detachtty \
          --pid-file /home/hunchentoot/run/hunchentoot.pid \
          /home/hunchentoot/run/hunchentoot.sock \
          /usr/local/bin/sbcl --load /home/hunchentoot/init.lisp --userinit /home/hunchentoot/.sbclrc

  ~#sudo chown hunchentoot:nogroup start-hunchentoot
  ~#sudo chmod u+x start-hunchentoot

2. Set up how sbcl and hunchentoot will be initialised.
(Go to http://blog.ponto-dot.com/2010/08/15/setting-up-common-lisp-on-a-web-server to see the source of the script used here. Again thanx to Alex for his hard work.)

  ~#sudo nano /home/hunchentoot/init.lisp

(ql:quickload 'swank)

;; define some parameters for easier update
(defparameter *shutdown-port* 6200)  ; The port SBCL will be listening for shutdown
                                     ; this port is the same used in /etc/init.d/hunchentoot
(defparameter *swank-port* 4016)     ; The port used for remote interaction with slime

;; Start the Swank server
(defparameter *swank-server*
  (swank:create-server :port *swank-port* :dont-close t))

;;;Load websites
(ql:quickload 'my-home-project)

;;; We need a way to actually kill this baby so we
;;; setup a socket listening on a specific port.
;;; When we want to stop the lisp process we simply
;;; telnet to that port as run by the stop section
;;; of the /etc/init.d/hunchentoot script.
;;; This thread will block execution until the
;;; connection comes in on the specified port,
(let ((socket (make-instance 'sb-bsd-sockets:inet-socket
                             :type :stream :protocol :tcp)))

  ;; Listen on a local port for a TCP connection
  (sb-bsd-sockets:socket-bind socket #(127 0 0 1) *shutdown-port*)
  (sb-bsd-sockets:socket-listen socket 1)

  ;; When it comes, close the sockets and continue
  (multiple-value-bind (client-socket addr port)
      (sb-bsd-sockets:socket-accept socket)
    (sb-bsd-sockets:socket-close client-socket)

    (sb-bsd-sockets:socket-close socket)))

;;; Here we go about closing all the running threads
;;; including the Swank Server we created.
(dolist (thread (sb-thread:list-all-threads))
  (unless (equal sb-thread:*current-thread* thread)
    (sb-thread:terminate-thread thread)))
(sleep 1)
(sb-ext:quit)

  ~#sudo chown hunchentoot:nogroup init.lisp

Set up nginx to know about hunchentoot:

Here I had to tweak the code given in http://blog.ponto-dot.com/2009/08/18/hunchentoot-behind-proxy-server to get nginx to serve multiple virtual hosts correctly. This was mostly because hunchentoot does not have a concept for virtual hosts. The way to get one hunchentoot instance to serve multiple “sites” is to prepend the uri’s registered with hunchentoot with an identifier for the site. So when you register the home page for the domain myhome.com you would use /myhome/home.html. This is where nginx comes in because now we can tell nginx that if I get a request for myhome.com/home.html it should request /myhome/home.html from the hunchentoot server.

  ~#sudo rm /etc/nginx/sites-enabled/default

  ~#sudo nano /etc/nginx/sites-available/myhome.com

# Here we define a cluster of hunchentoot servers
# this can later be extend for load-balancing
# if we had more instances of hunchentoot. In this
# case i only have one instance running.
upstream hunchentoot {
    server 127.0.0.1:8000;
}

server {
    listen 80;
    server_name myhome.com;

    rewrite ^(.*)/$ $1/home.html;

    # General request handling this will match all locations
    location / {

        root /var/www/myhome.com/public/;

        # Define custom HTTP Headers to be used when proxying
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;


        # if the requested file does not exist then
        # rewrite to the correct hunchentoot path
        # so that the proxy_pass can catch and pass on
        # to hunchentoot correctly - proxy_pass
        # cannot have anything but the servername/ip in it
        # else nginx complains so we have to rewrite and then
        # catch in another location later

        if (!-f $request_filename) {
                rewrite ^/(.*)$ /myhome/$1 last;
                break;
        }

    }

    location /myhome/ {
        # proxy to the hunchentoot server cluster
         proxy_pass http://hunchentoot;
     }

}

  ~#sudo ln -si /etc/nginx/sites-available/myhome.com /etc/nginx/sites-enabled/myhome.com

  ~#sudo /etc/init.d/nginx restart

Set up my home project as an example:

  ~#sudo mkdir ~admin/source/my-home-project/
  ~#sudo nano ~admin/source/my-home-project/defpackage.lisp

(in-package :common-lisp-user)

(defpackage :my-home-project
  (:nicknames :my-home-project)
  (:use :cl :hunchentoot :cl-who)
  )

  ~#sudo nano ~admin/source/my-home-project/my-home-project.asd

(defpackage #:my-home-project-asd
  (:use :cl :asdf))

(in-package :my-home-project-asd)

(defsystem my-home-project
  :name "my-home-project"
  :version "0.1"
  :depends-on (:hunchentoot
	       :cl-who)
  :serial t
  :components ((:file "defpackage")	    
	       (:file "home-page" )))

  ~#sudo nano ~admin/source/my-home-project/home-page.lisp

(in-package :my-home-project)

(define-easy-handler (home-page :uri "/myhome/home.html")
    ()
  (with-html-output-to-string (*standard-output*)
      (str "This is my home page."))) 

  ~#sudo ln -s ~admin/source/my-home-project/my-home-project.asd ~admin/quicklisp/local-projects/my-home-project.asd

Start the server and view your handy work:

  ~#sudo /etc/init.d/hunchentoot start

Go to your browser and browse to http://myhome.com/ and you should see your home page.

When things go wrong!

To see if the server started up correctly see if sbcl and dettachy is running.

  ~#ps aux

The most likely place that things will go wrong is in your /home/hunchentoot/init.lisp file so check out the dribble file to see what happened in the start up.

  ~#sudo tail -n100 -f /home/hunchentoot/log/hunchentoot.dribble

If you can’t stop or start the server again after something went wrong do the following to clean up before starting the server again:

  ~#sudo tail /home/hunchentoot/run/hunchentoot.pid
  ~#sudo kill -9 
  ~#sudo rm /home/hunchentoot/log/hunchentoot.dribble
  ~#sudo rm /home/hunchentoot/run/*

Exstras:

For those of you that want to install a database do the following:
~#sudo apt-get install postgresql
~#sudo -u postgres createuser -D -A -P mynewuser
~#sudo -u postgres createdb -O mynewuser mydatabase

To use the db in lisp with postmodern have a look at http://marijnhaverbeke.nl/postmodern/

Home work:

Do your self a favour and read about how to connect to the sbcl instance remotely http://blog.ponto-dot.com/2010/08/15/setting-up-common-lisp-on-a-web-server.

Additional resources:

http://marijnhaverbeke.nl/postmodern/
http://weitz.de/hunchentoot/
http://blog.ponto-dot.com/2010/08/15/setting-up-common-lisp-on-a-web-server
http://blog.ponto-dot.com/2009/08/18/hunchentoot-behind-proxy-server
http://blog.ponto-dot.com/2009/08/18/slime-tramp-filename-translation
http://www.sbcl.org/
http://marijnhaverbeke.nl/postmodern/
http://weitz.de/hunchentoot/
http://www.quicklisp.org/
http://www.lisp.org
http://planet.lisp.org
http://gigamonkeys.com/book/

Advertisement

Tags: hunchentoot, maverick, meerkat, nginx, postgresql, postmodern, sbcl, slicehost, Ubuntu, virtual hosts