Thursday, February 17, 2011

How to create a reverse HTTP(s) failover web proxy using nginx & heartbeat

Update: There is a Virtual Machine Image available at the bottom of this post
Currently we use a pair of very expensive F5 load balancers to manage our highly available SaaS application, providing SSL offloading and round robin load balanacing with failover in the event one of the SaaS nodes fails.  I decided I wanted to see if I could replace all of the functionality we use from those F5's with a custom built (and free) solution.  I spoke to my coworker JW who works on the SaaS app and has recently been working with other vendors such as Barracuda to find a cheaper device (Our F5's are aging) to replace the F5's and got a list of requirements from him so I would know exactly what we needed.  I spent some time researching and came across nginx which seemed like it would be a suitable package as it provided web proxying, SSL offloading, failover, and sticky sessions (but only in more recent builds).  It took me about half an hour to create a proof of concept with simple web HTTP traffic and another hour to get SSL working and refine/test the configuration.  These are instructions on how to create a set of failover load balancer devices using heartbeat and nginx on Debian Lenny.  If you don't need the heartbeat failover cluster then skip all of the instructions that involve heartbeat.  Configuration for different linux platforms should be similar as well.

Start with a base install of Debian (two if you want clustering)
Configure the network interface with a static IP Address
create a static ip address:
/etc/network/interfaces:
allow-hotplug eth0
iface eth0 inet static
address 192.168.11.250
netamsk 255.255.255.0
gateway 192.168.11.1




Configure Apt Package Manager

The version of nginx we need is in the testing tree, so we need to configure apt by adding these lines to sources.list.
/etc/apt/sources.list:
deb http://debian.osuosl.org/debian testing main contrib non-free
deb-src http://debian.osuosl.org/debian testing main contrib non-free

In order to tell apt to install nginx from the testing tree we hard code the system to use the testing tree for nginx.
/etc/apt/preferences:
Package: *
Pin: release a=stable
Pin-Priority: 900

Package: nginx
Pin: release a=testing
Pin-Priority : 1000

Install Packages

apt-get update
apt-get install heartbeat nginx

Configure Heartbeat
On each server you need to modify ha.cf to look like this.  On node one you put the IP of node two in OtherNodeStaticIP, and on node two you use node 1's ip for OtherNodeStaticIP.  


/etc/ha.d/ha.cf
udpport 694
bcast eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 <OtherNodeStaticIP>
node NODENAME1
node NODENAME2
auto_failback off
logfacility local0

/etc/ha.d/haresources
NODENAME1 CLUSTERIP nginx
cp /etc/init.d/nginx /etc/ha.d/resource.d/nginx




Certificates
If you need reverse SSL support to provide SSL encryption for a web service that is not SSL, or you would like to offload the SSL process from your webserver to another device, you will need to configure SSL certificates.  I created a new folder, /etc/nginx/sslcerts and did all of my work in there.

We use the entrust certification authority, so we got the certificate and key file for the certificate we wanted and put it in the /etc/nginx/sslcerts directory.  Additionally, you need to concatenate the middle certificate from your Certificate Signing Authority to teh end of your .crt file.  

copy your crt and key to /etc/nginx/sslcerts/CERTIFICATE.key/crt
cat EntrustMiddleCert.crt >> /etc/nginx/sslcerts/CERTIFICATE.crt


Modify /etc/ha.d/authkeys on both boxes:

auth 2
2 sha1 SOMETHING_UNIQUE


Configure nginx
In our configuration we have 2 clusters, one for the main site, one for reporting, and a final server that hosts a simple page saying the site is down.  This configuration provides all of these features, and in the event that both nodes of the cluster are down the user will get the error page defined instead of a 404 error or something similar.  This configuration example includes HTTP & HTTPS reverse proxy configurations, although you may or may not want to provide both.  Edit to your liking.

/etc/nginx/sites-enabled/default
upstream  SITENAME_cluster  {
  ip_hash; #Sticky sessions
  server   10.32.41.230:20780          max_fails=3  fail_timeout=31s;
  server   10.32.41.231:20780          max_fails=3  fail_timeout=31s;
}
upstream  inetsoft  {
  ip_hash; #Sticky sessions
  server   10.32.41.113:8080          max_fails=3  fail_timeout=31s;
  server   10.32.41.114:8080          max_fails=3  fail_timeout=31s;
}

#the server that servers our notice that the application is unavailable
upstream  errorpage  {
  server   status.localdomain.com          max_fails=3  fail_timeout=31s;
}


#If you want to redirect all HTTP traffic to the HTTPS site, use this instead of  following the port 80 declaration below
#server {
#        listen   80;
#        server_name  SITENAME;
#        rewrite ^(.*) https://$host$1 permanent;
#}


server {
        listen   80;
        server_name  SITENAME;

        location /Reporting {
                proxy_pass http://inetsoft$request_uri;
        }

        location / {
                proxy_pass  http://SITENAME_cluster$request_uri;
                proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

                ### Set headers ####
                proxy_set_header X-Real-IP $remote_addr;
        }

        error_page 502 /error.html;
        location /error.html
        {
                proxy_pass http://errorpage/error.php?Novell;
                proxy_set_header X-Real-IP $remote_addr;
        }
}

server {
        listen   443;
        server_name  SITENAME;

        ssl  on;
        ssl_certificate  /etc/nginx/sslcerts/CERTIFICATE.crt;
        ssl_certificate_key  /etc/nginx/sslcerts/ CERTIFICATE.key;

        ssl_session_timeout  90m;

        ssl_protocols  SSLv2 SSLv3 TLSv1;
        ssl_ciphers  ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
        ssl_prefer_server_ciphers   on;

        location /Reporting {
                proxy_pass http://inetsoft$request_uri;
        }

        location / {
                proxy_pass  http://SITENAME_cluster$request_uri;
                proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

                ### Set headers ####
                proxy_set_header X-Real-IP $remote_addr;
        }

        error_page 502 /error.html;
        location /error.html
        {
                proxy_pass http://errorpage/error.php?SITENAME;
                proxy_set_header X-Real-IP $remote_addr;
        }
}





You will need to make sure you keep your configurations the same on both servers, using a tool like rsync can automate this process for you but for the purposes of this tutorial I'm going to leave it up to you to decide how to keep your nodes in sync.

Test your nginx configuration for errors, /etc/init.d/nginx start.  If you get any error messages be sure to resolve those, the http://wiki.nginx.org site was very useful for me.  If everything works properly you are good.  If you are using clustering, now you need to start and test your Linux HA cluster.
/etc/init.d/heartbeat start

Congratuations, you have a Reverse HTTP(s) web proxy with failover clustering, and it didn't cost you any body parts.

2 comments:

  1. Were you actually able to replace the F5s completely with this set up?

    ReplyDelete
  2. It works perfectly for our situation, although we didn't replace the F5's as our business needs changed.

    The F5 is very flexible and powerful, and we primarily used this reverse-proxy feature so it would have easily replaced the F5's had we decided to go ahead and do it.

    If I were starting from scratch I would use a pair of HA enabled nginx machines in our clustered VM environment providing another layer of protection.

    ReplyDelete