Common Lisp Hunchentoot Quicklisp on Ubuntu Server @ Rackspace VM

So after six years I finally get round to revisiting this subject, a lot has changed over the years, but not much either, the basics are still the same.

So you want your own lisp web server to hack away at but where do you start? Well don’t despair, what follows is a recipe for getting such a server running in no time.

SERVER CREATION

So I am using Rackspace servers a bit like you would partition your favourite linux install. We are going to store data on one Rackspace volume, have the live data on another and use a different volume for the server. The basic idea is that you can disconnect your data volumes, spin up a new server and reconnect them if you should need to. This way you can also choose which volumes you need as SSD and which not.

You can find a lot of good “how to” stuff on Rackspace at https://support.rackspace.com/how-to/

1. Create data and backup volumes separately (in the same Region).

create-data-cut

I used a bit more for my backup volume.

create-backup-cut

2.Create new server, these are old pics so the version of Ubuntu could be different.

create-server-1-cut

The application I run most uses an in memory database so I select plenty of RAM

create-server-2-cut

create-server-3-cut

After creating your sever you can attach the data and backup volumes to the server.

attach-volume-to-server-cut

SERVER CONFIGURATION BASICS

1. ssh into your new server with the password that was given to you during the server creation.

ssh root@ip_address_of_server

2. Configure the data and backup volumes using the instructions here https://support.rackspace.com/how-to/prepare-your-cloud-block-storage-volume/

 NOTE to SELF: If you fluff your FSBTAB file you might have to start all over again so concentrate.

3. Set up your locale.

locale-gen "en_ZA.UTF-8"
update-locale LANG=en_ZA.UTF-8
dpkg-reconfigure locales

#Set the local time
echo 'Africa/Johannesburg' > /etc/timezone
dpkg-reconfigure --frontend noninteractive tzdata

4. Update the server.

apt-get -y update
apt-get -y upgrade

5. Change the root password. Do I have to remind you to choose something sensible but secure enough?

  passwd

6. Add a new user called “theadmin” and add it to the sudoers list. This is the user you will be using in the future to ssh into the server. We are going to disallow logging in as root later so you cant skip this step unless you want to experience locking your self out of the server later.

We are also creating a dedicated user to run hunchentoot with, called hunchentoot.

#Add users & groups
groupadd hunchweb
adduser theadmin
usermod -a -G sudo theadmin 
usermod -a -G hunchweb theadmin 
adduser --system --disabled-password --home /home/hunchentoot --shell /bin/bash  hunchentoot

#Set hunchentoot primary group to hunchweb
usermod -g hunchweb hunchentoot

#Set default file permissions.
cat > hunchprofile <<'_EOFX'
umask 002
_EOFX

mv hunchprofile /home/hunchentoot/.profile
chown hunchentoot:hunchweb /home/hunchentoot/.profile

7. Create an ssh key on your local machine and upload your public key to the server. Don’t leave the ssh passphrase blank!

ssh-keygen -t rsa
scp ~/.ssh/id_rsa.pub theadmin@ip_address_of_server:

8. Create the authorized_keys file and set its permissions correctly. ssh is very pedantic when it comes to permissions!

mkdir ~theadmin/.ssh
mv ~theadmin/id_rsa.pub ~theadmin/.ssh/authorized_keys
chown -R theadmin:theadmin ~theadmin/.ssh
chmod 700 ~theadmin/.ssh
chmod 600 ~theadmin/.ssh/authorized_keys

9. Next, let’s edit the /etc/ssh/sshd_config file and disable some defaults. Here are the adjustments you should make:

 #Change ssh settings for the server.
sed -i -e "s/Port 22/Port 22222/" /etc/ssh/sshd_config
sed -i -e "s/PermitRootLogin yes/PermitRootLogin no/" /etc/ssh/sshd_config
sed -i -e "s/X11Forwarding yes/X11Forwarding no/" /etc/ssh/sshd_config
sed -i -e "s/UsePAM yes/UsePAM no/" /etc/ssh/sshd_config
echo -e '\nAllowUsers theadmin' >> /etc/ssh/sshd_config

10. Set up IPTABLES to make server more secure. Read https://www.digitalocean.com/community/tutorials/how-to-implement-a-basic-firewall-template-with-iptables-on-ubuntu-14-04

#Clear IP tables
/sbin/iptables -F

#Setup IP tables -- 
cat > /etc/iptables.up.rules <<'_EOF'

*filter
# Allow all outgoing, but drop incoming and forwarding packets by default
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]

# Custom per-protocol chains
:UDP - [0:0]
:TCP - [0:0]
:ICMP - [0:0]

# Acceptable UDP traffic

# Acceptable TCP traffic
-A TCP -p tcp --dport 22222 -j ACCEPT
-A TCP -p tcp --dport 80 -j ACCEPT

# Acceptable ICMP traffic

# Boilerplate acceptance policy
-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-A INPUT -i lo -j ACCEPT

# Drop invalid packets
-A INPUT -m conntrack --ctstate INVALID -j DROP

# Pass traffic to protocol-specific chains
## Only allow new connections (established and related should already be handled)
## For TCP, additionally only allow new SYN packets since that is the only valid
## method for establishing a new TCP connection
-A INPUT -p udp -m conntrack --ctstate NEW -j UDP
-A INPUT -p tcp --syn -m conntrack --ctstate NEW -j TCP
-A INPUT -p icmp -m conntrack --ctstate NEW -j ICMP

# Reject anything that's fallen through to this point
## Try to be protocol-specific w/ rejection message
-A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
-A INPUT -p tcp -j REJECT --reject-with tcp-reset
-A INPUT -j REJECT --reject-with icmp-proto-unreachable

# Commit the changes
COMMIT

*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT

*security
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
_EOF

11. Tell the server to use the new rules.

#Set iptables to new rules.
/sbin/iptables-restore < /etc/iptables.up.rules

12. To make sure the iptable rules are automatically loaded in the future create the following script.

#Create script to run at startup to re-apply the new rules.
cat > /etc/network/if-pre-up.d/iptables <<'_EOFX'
#!/bin/sh
/sbin/iptables-restore < /etc/iptables.up.rules
_EOFX

#Make script executable
chmod +x /etc/network/if-pre-up.d/iptables

13. Now that we have our security in place lets reset ssh

systemctl restart ssh

14. To test the ssh security settings run the following command from your local machine without closing the current ssh connection(ie open a new terminal). If this command does not log you in automatically you have fluffed something along the way and need to recheck your settings!

ssh -p 22222 theadmin@ip_address_of_server

(note from here on we work as theadmin user and not root!)

NGINX

We use nginx as a proxy in front of hunchentoot.

sudo apt-get install nginx -y

#Set up nginx to know about hunchentoot
cat > mysite.com <<'_EOFXXX'
# Here we define a cluster of hunchentoot servers
# this can later be extend for load-balancing
# if we had more instances of hunchentoot. In this
# case i only have one instance running.

upstream hunchentoot {
    server 127.0.0.1:4242;
}

server {
    listen 80;

    server_name app.mysite.com;
    proxy_read_timeout 2500;
    client_max_body_size 20M;
    rewrite ^(.*)/$ $1/login;

    root /home/hunchentoot/my-static-files/;

    # General request handling this will match all locations
    location / {

        # Define custom HTTP Headers to be used when proxying
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        # if the requested file does not exist then
        # rewrite to the correct hunchentoot path
        # so that the proxy_pass can catch and pass on
        # to hunchentoot correctly - proxy_pass
        # cannot have anything but the servername/ip in it
        # cannot have anything but the servername/ip in it
        # else nginx complains so we have to rewrite and then
        # catch in another location later

        if (!-f $request_filename) {
                rewrite ^/(.*)$ /myapp/$1 last;
                break;
        }

    }

    location /myapp/ {
        # proxy to the hunchentoot server cluster
         proxy_pass http://hunchentoot;
     }

}
_EOFXXX

sudo mv mysite.com /etc/nginx/sites-available/
sudo ln -s /etc/nginx/sites-available/mysite.com /etc/nginx/sites-enabled/mysite.com

#If you are going to use static files point nginx in the right direction
sudo mkdir /var/www
sudo ln -s /home/hunchentoot/my-static-files/ /var/www/mysite.com

sudo systemctl restart nginx

SBCL

sudo apt-get install git -y

sudo apt-get install build-essential -y

wget http://prdownloads.sourceforge.net/sbcl/sbcl-1.3.11-x86-64-linux-binary.tar.bz2
bzip2 -cd sbcl-1.3.11-x86-64-linux-binary.tar.bz2 | tar xvf -

cd sbcl-1.3.11-x86-64-linux

sudo sh install.sh

QUICKLISP

1. Log in as the hunchentoot user and install quicklisp

sudo -iu hunchentoot

#Optional: Set up ssh key to use for github
ssh-keygen -t rsa -f /home/hunchentoot/.ssh/id_rsa -q -P ""

cat /home/hunchentoot/.ssh/id_rsa.pub

2. Set up Quicklisp.

curl -O http://beta.quicklisp.org/quicklisp.lisp

sbcl --load quicklisp.lisp

(quicklisp-quickstart:install)
(ql:add-to-init-file)

(quit)

exit # to get back to theadmin user session

HUNCHENTOOT

1. Setup an init.lisp file to be used to load your website/app

sudo -iu hunchentoot

cat > init.lisp <<'_EOFX'
(ql:quickload 'swank)

(defparameter *swank-port* 4016)     ; The port used for remote 
                                     ;interaction with slime

;; Start the Swank server
(defparameter *swank-server*
  (swank:create-server :port *swank-port* :dont-close t))

;;For testing purposes remove if you are loading your own app
#start test
(ql:quicload 'hunchentoot)
(hunchentoot:start (make-instance 'hunchentoot:easy-acceptor :port 4242))

(hunchentoot:define-easy-handler (say-yo :uri "/myapp/yo") (name)
  (setf (hunchentoot:content-type*) "text/plain")
  (format nil "Hey~@[ ~A~]!" name))
#end test

;;;Load websites/app
;;(ql:quickload 'some-great-app-that-i-wrote)

 _EOFX  

exit # to get back to theadmin user session

SERVER DAEMON

1. Create system daemon that runs sbcl in a screen session loading an init.lisp file, more reading here https://lwn.net/Articles/534389/

sudo apt-get install screen -y

cat > hunchentoot.service <<'_EOFXX'
[Unit]
Description=Hunchentoot webserver session on a screen tty.
After = syslog.target network.target

[Service]
User=hunchentoot
Group=hunchweb
Type=simple
ExecStart=/usr/bin/screen -D -m /usr/local/bin/sbcl 
    --dynamic-space-size 15000 
    --load /home/hunchentoot/init.lisp 
     --userinit /home/hunchentoot/.sbclrc 

[Install]
WantedBy=multi-user.target
_EOFXX

sudo mv hunchentoot.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable hunchentoot.service

RUNNING THE SERVER IN SCREEN

sudo systemctl start hunchentoot.service
#check if the server started
sudo systemctl status hunchentoot.service

#To see what SBCL says
sudo -iu hunchentoot
script /dev/null
screen -r #Ctrl A D to get out of screen
exit # to get out of script
exit # to get back to theadmin user session

Now go to your browser and browse to http://app.mysite.com/myapp/yo and you should see the default hunchentoot page. If that does not work have a look at http://my-server-ip-address:4242 to see if hunchentoot is loaded.

Bits and pieces

# I did my fstab this way to facilitate backups and file uploads
 
sudo printf "\nUUID=oauyrsntoarsnt /home/hunchentoot/external       ext4    defaults        0       2\n" >> /etc/fstab
sudo printf "UUID=arstonrstarstrst /home/hunchentoot/data           ext4    defaults        0       2\n" >> /etc/fstab

sudo printf "/home/hunchentoot/data/hunchentoot-upload /home/hunchentoot/hunchentoot-upload    none    bind           0       0\n" >> /etc/fstab
sudo printf "/home/hunchentoot/data/db /home/hunchentoot/db                                none    bind           0       0\n" >> /etc/fstab

#A backup script that can be run by cron.
cat > backup.sh <<'_EOFX'
#!/bin/bash          

for i in /home/hunchentoot/data/* ; do
  if [ -d "$i" ]; then
        dir=/home/hunchentoot/external/backups/$(date +%Y%m%d)/$(basename "$i")
        znm=$dir/$(basename "$i")-$(date +%Y%m%d-%H%M).tar.gz
        mkdir -p $dir
        tar -zPcf $znm "$i" --exclude "$i/backup"
   # echo "$i"
#   echo "$znm"
  fi
done
_EOFX
Advertisements

Tags: , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: