Thursday, February 17, 2011

How to create a reverse HTTP(s) failover web proxy using nginx & heartbeat

Update: There is a Virtual Machine Image available at the bottom of this post
Currently we use a pair of very expensive F5 load balancers to manage our highly available SaaS application, providing SSL offloading and round robin load balanacing with failover in the event one of the SaaS nodes fails.  I decided I wanted to see if I could replace all of the functionality we use from those F5's with a custom built (and free) solution.  I spoke to my coworker JW who works on the SaaS app and has recently been working with other vendors such as Barracuda to find a cheaper device (Our F5's are aging) to replace the F5's and got a list of requirements from him so I would know exactly what we needed.  I spent some time researching and came across nginx which seemed like it would be a suitable package as it provided web proxying, SSL offloading, failover, and sticky sessions (but only in more recent builds).  It took me about half an hour to create a proof of concept with simple web HTTP traffic and another hour to get SSL working and refine/test the configuration.  These are instructions on how to create a set of failover load balancer devices using heartbeat and nginx on Debian Lenny.  If you don't need the heartbeat failover cluster then skip all of the instructions that involve heartbeat.  Configuration for different linux platforms should be similar as well.

Start with a base install of Debian (two if you want clustering)
Configure the network interface with a static IP Address
create a static ip address:
allow-hotplug eth0
iface eth0 inet static

Configure Apt Package Manager

The version of nginx we need is in the testing tree, so we need to configure apt by adding these lines to sources.list.
deb testing main contrib non-free
deb-src testing main contrib non-free

In order to tell apt to install nginx from the testing tree we hard code the system to use the testing tree for nginx.
Package: *
Pin: release a=stable
Pin-Priority: 900

Package: nginx
Pin: release a=testing
Pin-Priority : 1000

Install Packages

apt-get update
apt-get install heartbeat nginx

Configure Heartbeat
On each server you need to modify to look like this.  On node one you put the IP of node two in OtherNodeStaticIP, and on node two you use node 1's ip for OtherNodeStaticIP.  

udpport 694
bcast eth0
mcast eth0 694 1 0
ucast eth0 <OtherNodeStaticIP>
auto_failback off
logfacility local0

cp /etc/init.d/nginx /etc/ha.d/resource.d/nginx

If you need reverse SSL support to provide SSL encryption for a web service that is not SSL, or you would like to offload the SSL process from your webserver to another device, you will need to configure SSL certificates.  I created a new folder, /etc/nginx/sslcerts and did all of my work in there.

We use the entrust certification authority, so we got the certificate and key file for the certificate we wanted and put it in the /etc/nginx/sslcerts directory.  Additionally, you need to concatenate the middle certificate from your Certificate Signing Authority to teh end of your .crt file.  

copy your crt and key to /etc/nginx/sslcerts/CERTIFICATE.key/crt
cat EntrustMiddleCert.crt >> /etc/nginx/sslcerts/CERTIFICATE.crt

Modify /etc/ha.d/authkeys on both boxes:

auth 2

Configure nginx
In our configuration we have 2 clusters, one for the main site, one for reporting, and a final server that hosts a simple page saying the site is down.  This configuration provides all of these features, and in the event that both nodes of the cluster are down the user will get the error page defined instead of a 404 error or something similar.  This configuration example includes HTTP & HTTPS reverse proxy configurations, although you may or may not want to provide both.  Edit to your liking.

upstream  SITENAME_cluster  {
  ip_hash; #Sticky sessions
  server          max_fails=3  fail_timeout=31s;
  server          max_fails=3  fail_timeout=31s;
upstream  inetsoft  {
  ip_hash; #Sticky sessions
  server          max_fails=3  fail_timeout=31s;
  server          max_fails=3  fail_timeout=31s;

#the server that servers our notice that the application is unavailable
upstream  errorpage  {
  server          max_fails=3  fail_timeout=31s;

#If you want to redirect all HTTP traffic to the HTTPS site, use this instead of  following the port 80 declaration below
#server {
#        listen   80;
#        server_name  SITENAME;
#        rewrite ^(.*) https://$host$1 permanent;

server {
        listen   80;
        server_name  SITENAME;

        location /Reporting {
                proxy_pass http://inetsoft$request_uri;

        location / {
                proxy_pass  http://SITENAME_cluster$request_uri;
                proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

                ### Set headers ####
                proxy_set_header X-Real-IP $remote_addr;

        error_page 502 /error.html;
        location /error.html
                proxy_pass http://errorpage/error.php?Novell;
                proxy_set_header X-Real-IP $remote_addr;

server {
        listen   443;
        server_name  SITENAME;

        ssl  on;
        ssl_certificate  /etc/nginx/sslcerts/CERTIFICATE.crt;
        ssl_certificate_key  /etc/nginx/sslcerts/ CERTIFICATE.key;

        ssl_session_timeout  90m;

        ssl_protocols  SSLv2 SSLv3 TLSv1;
        ssl_ciphers  ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
        ssl_prefer_server_ciphers   on;

        location /Reporting {
                proxy_pass http://inetsoft$request_uri;

        location / {
                proxy_pass  http://SITENAME_cluster$request_uri;
                proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

                ### Set headers ####
                proxy_set_header X-Real-IP $remote_addr;

        error_page 502 /error.html;
        location /error.html
                proxy_pass http://errorpage/error.php?SITENAME;
                proxy_set_header X-Real-IP $remote_addr;

You will need to make sure you keep your configurations the same on both servers, using a tool like rsync can automate this process for you but for the purposes of this tutorial I'm going to leave it up to you to decide how to keep your nodes in sync.

Test your nginx configuration for errors, /etc/init.d/nginx start.  If you get any error messages be sure to resolve those, the site was very useful for me.  If everything works properly you are good.  If you are using clustering, now you need to start and test your Linux HA cluster.
/etc/init.d/heartbeat start

Congratuations, you have a Reverse HTTP(s) web proxy with failover clustering, and it didn't cost you any body parts.

Wednesday, February 9, 2011

Asterisk with Exchange 2010 SP1 Unified Messaging

Today I decided I wanted to play with using Microsoft Exchange Server Unified Messaging for voicemail in our Digium Asterisk 1.6.2 deployment.  Already having an Exchange 2010 Deployment all I needed to do was install the UM Component in our Exchange infrastructure and configure everything.

Unified messaging provides voicemail transcription support and automatically delivers the voicemail to the users mailbox with as much information about the caller as possible.  If it is another Unified Messaging enabled user it will include all of their contact info based on their callerid. For example when I call somebody from extension 1593 in asterisk and leave another UM user a voicemail, and I have 1593 configured in UM, they will get my cell phone number, email address, and title along with the transcription of the email and an MP3 of the voicemail.  Slick.

One thing to note is the transcription uses a fair amount of CPU power, it took about 30 seconds for it to transcribe a 30 second voicemail, while fully utilizing a core of a Xeon E5430 processor and 250 megabytes of memory.  If you have a lot of voicemail messages being recorded you may need to watch your system utilization closely.

Install Unified Messaging Component
  • UM Requires the desktop experience pack in Windows Server 2008 R2, so I installed that and rebooted
  • Next, you need to have the Microsoft Server Speech Platform Runtime (X64) installed.  If you run the installer it will tell you to install it, and provides you this link to download and install it.  You must use this link, if you just search for and download the platform run time you will not get the right version and the check will still fail.
  • Finally, if the Windows Firewall service is disabled the install will fail with the following message so make sure the service is set to automatic or manual start.  Thanks to this blog post for helping me figure this one out:
    • The following error was generated when "$error.Clear(); install-UMService " was run: "There are no more endpoints available from the endpoint mapper. (Exception from HRESULT: 0x800706D9)".

    • There are no more endpoints available from the endpoint mapper. (Exception from HRESULT: 0x800706D9)
    • Click here for help...
  • Once you have the prerequisites installed you can install the UM component using your installation media.
  • Once the installation is complete you need to configure UM.
Configure Unified Messaging
  • First you need to create a Dial Plan
  • We use 4 digit extensions and we are in the United States, so I configured the settings accordingly.

  • Now create a UM IP Gateway to point at your Asterisk server, entering the IP Address of your Asterisk box and selecting the Dial Plan you just created.
  • Next I configured the UM Mailbox Policies to use 4 digit pin length instead of the default 6.
  • Finally I configured my Auto Attendant with an extension to provide directory lookups, etc.
Configure Asterisk
  • Next configure your Asterisk server.  You need to add a SIP Peer to sip.conf:
    • [exchangeum]
    • host= <-- IP of your exchange server w/ UM Installed
    • type=friend
    • insecure=very
    • transport=tcp
    • port=5065
    • context=from-ocs <-- Context you want calls that come from exchange to go to.  I pointed it at the same context we use for our OCS/Lync deployment
  • Configure your dial plan to use Unified Messaging for voicemail instead of Asterisk's Voicemail() App.
  • This dialplan will call my phone whenever somebody dials 1593, and if I don't answer on my desk phone, or on Lync (OCS_TRUNK_R2) it will send the call to my unified messaging mailbox.
    • exten => 1593,1,Dial(SIP/1593&SIP/OCS_TRUNK_R2/+2593,25)
    • exten => 1593,n,SipAddHeader(Diversion:<tel:1593>)
    • exten => 1593,n,Dial(SIP/1593@exchangeum)
    • exten => 1593,n,Hangup
Configure Users
  • You can manually configure one user at a time in the Exchange Console, or you can use a powershell script to do it in bulk. I found a useful tool available here to allow you to bulk add users to Unified Messaging.  
  • I modified his tool a little bit to make it simplify deployment, making it take username,extension in a CSV delimited file.  
  • You can see the sample CSV here, and download the modified powershell script here