Currently we use a pair of very expensive F5 load balancers to manage our highly available SaaS application, providing SSL offloading and round robin load balanacing with failover in the event one of the SaaS nodes fails. I decided I wanted to see if I could replace all of the functionality we use from those F5's with a custom built (and free) solution. I spoke to my coworker JW who works on the SaaS app and has recently been working with other vendors such as Barracuda to find a cheaper device (Our F5's are aging) to replace the F5's and got a list of requirements from him so I would know exactly what we needed. I spent some time researching and came across nginx which seemed like it would be a suitable package as it provided web proxying, SSL offloading, failover, and sticky sessions (but only in more recent builds). It took me about half an hour to create a proof of concept with simple web HTTP traffic and another hour to get SSL working and refine/test the configuration. These are instructions on how to create a set of failover load balancer devices using heartbeat and nginx on Debian Lenny. If you don't need the heartbeat failover cluster then skip all of the instructions that involve heartbeat. Configuration for different linux platforms should be similar as well.
Start with a base install of Debian (two if you want clustering)
Configure the network interface with a static IP Address
create a static ip address:
/etc/network/interfaces:
allow-hotplug eth0
iface eth0 inet static
address 192.168.11.250
netamsk 255.255.255.0
gateway 192.168.11.1
The version of nginx we need is in the testing tree, so we need to configure apt by adding these lines to sources.list.
/etc/apt/sources.list:
deb http://debian.osuosl.org/debian testing main contrib non-free
deb-src http://debian.osuosl.org/debian testing main contrib non-free
/etc/apt/preferences:
Package: *
Pin: release a=stable
Pin-Priority: 900
Package: nginx
Pin: release a=testing
Pin-Priority : 1000
Install Packages
apt-get update
apt-get install heartbeat nginx
On each server you need to modify ha.cf to look like this. On node one you put the IP of node two in OtherNodeStaticIP, and on node two you use node 1's ip for OtherNodeStaticIP.
/etc/ha.d/ha.cf
udpport 694
bcast eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 <OtherNodeStaticIP>
node NODENAME1
node NODENAME2
auto_failback off
logfacility local0
/etc/ha.d/haresources
NODENAME1 CLUSTERIP nginx
cp /etc/init.d/nginx /etc/ha.d/resource.d/nginx
Certificates
If you need reverse SSL support to provide SSL encryption for a web service that is not SSL, or you would like to offload the SSL process from your webserver to another device, you will need to configure SSL certificates. I created a new folder, /etc/nginx/sslcerts and did all of my work in there.
We use the entrust certification authority, so we got the certificate and key file for the certificate we wanted and put it in the /etc/nginx/sslcerts directory. Additionally, you need to concatenate the middle certificate from your Certificate Signing Authority to teh end of your .crt file.
copy your crt and key to /etc/nginx/sslcerts/CERTIFICATE.key/crt
cat EntrustMiddleCert.crt >> /etc/nginx/sslcerts/CERTIFICATE.crt
Modify /etc/ha.d/authkeys on both boxes:
auth 2
2 sha1 SOMETHING_UNIQUE
Configure nginx
In our configuration we have 2 clusters, one for the main site, one for reporting, and a final server that hosts a simple page saying the site is down. This configuration provides all of these features, and in the event that both nodes of the cluster are down the user will get the error page defined instead of a 404 error or something similar. This configuration example includes HTTP & HTTPS reverse proxy configurations, although you may or may not want to provide both. Edit to your liking.
/etc/nginx/sites-enabled/default
upstream SITENAME_cluster {
ip_hash; #Sticky sessions
server 10.32.41.230:20780 max_fails=3 fail_timeout=31s;
server 10.32.41.231:20780 max_fails=3 fail_timeout=31s;
}
upstream inetsoft {
ip_hash; #Sticky sessions
server 10.32.41.113:8080 max_fails=3 fail_timeout=31s;
server 10.32.41.114:8080 max_fails=3 fail_timeout=31s;
}
#the server that servers our notice that the application is unavailable
upstream errorpage {
server status.localdomain.com max_fails=3 fail_timeout=31s;
}
#If you want to redirect all HTTP traffic to the HTTPS site, use this instead of following the port 80 declaration below
#server {
# listen 80;
# server_name SITENAME;
# rewrite ^(.*) https://$host$1 permanent;
#}
server {
listen 80;
server_name SITENAME;
location /Reporting {
proxy_pass http://inetsoft$request_uri;
}
location / {
proxy_pass http://SITENAME_cluster$request_uri;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
### Set headers ####
proxy_set_header X-Real-IP $remote_addr;
}
error_page 502 /error.html;
location /error.html
{
proxy_pass http://errorpage/error.php?Novell;
proxy_set_header X-Real-IP $remote_addr;
}
}
server {
listen 443;
server_name SITENAME;
ssl on;
ssl_certificate /etc/nginx/sslcerts/CERTIFICATE.crt;
ssl_certificate_key /etc/nginx/sslcerts/ CERTIFICATE.key;
ssl_session_timeout 90m;
ssl_protocols SSLv2 SSLv3 TLSv1;
ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
ssl_prefer_server_ciphers on;
location /Reporting {
proxy_pass http://inetsoft$request_uri;
}
location / {
proxy_pass http://SITENAME_cluster$request_uri;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
### Set headers ####
proxy_set_header X-Real-IP $remote_addr;
}
error_page 502 /error.html;
location /error.html
{
proxy_pass http://errorpage/error.php?SITENAME;
proxy_set_header X-Real-IP $remote_addr;
}
}
You will need to make sure you keep your configurations the same on both servers, using a tool like rsync can automate this process for you but for the purposes of this tutorial I'm going to leave it up to you to decide how to keep your nodes in sync.
Test your nginx configuration for errors, /etc/init.d/nginx start. If you get any error messages be sure to resolve those, the http://wiki.nginx.org site was very useful for me. If everything works properly you are good. If you are using clustering, now you need to start and test your Linux HA cluster.
/etc/init.d/heartbeat start
Congratuations, you have a Reverse HTTP(s) web proxy with failover clustering, and it didn't cost you any body parts.
