Friday, September 30, 2011

Sending email with Amazon SES Email Service from PHP

We recently started using Amazon's Simple Email Service, a feature of Amazon Web Services to deliver messaging to our CIMLS Commercial Real Estate user database of 270,000 members.  We haven't done any form of newsletter or update and now that we've spent the last 8 months redesigning the site and giving it a major face lift we decided it would be a good opportunity for us to say hey, we're back, we're working on the site and provide examples of the many new features we have added.

We decided to use the AWS SES platform to send emails to our users in bulk because it provides statistics on bounce rates, complaints, abuse reports, and a trusted stable bulk mail platform.  Some of our members have been gone for 3 or 4 years and we've found some of them simply file a complaint about the email even though they signed up for us and we include the unsubscribe link in the message.  As we've received each complaint we simply go through and unsubscribe them from future communication from us, no harm no foul.

I downloaded the AWS SDK for PHP and looked at the implementation.  It includes classes for many of the AWS services but I'm only interested in the Simple Email Service, so I went and found the ses.php class and chose to use just that.

The ses class provides many functions and digging through the class helped unravel what all could be done with it, this is a simplified version of the function I wrote to deliver email.

I made one modification to the ses class, if there were an error I wanted it to be returned to me so I could handle it.  On line 335 of ses.php I changed the line from "return false" to "return $rest->error"

Our true code takes a CSV of emailaddress,fullname and will send the email to every user in the CSV.  This is a stripped down example of that so you can use it however you please, but that's why we do the %%USERNAME%% replace and handle errors the way we do.


$ses = new SimpleEmailService("ACCESSKEY","SECRETKEY");
$m = new SimpleEmailServiceMessage();
$m->setReturnPath(''); //the return path defines where any complaints or abuse messages will go -- this is important to have configured so you can clean out your mailing list of members that do not want to receive messages
$usermsg = str_replace("%%USERNAME%%",$username,$msg); //we use a token to replace with the users name to make the message more personalized

//setMessageFromString takes the first parameter for a TEXT message, and an 
//optional second for an HTML message.  If you would like to provide both
// formats at the same time that is possible, however it isn't something we have
// been doing so I wrote this little code to handle our use case.
if($_REQUEST['content'] == "HTML")
$result = $ses->sendEmail($m);

//check and see if there is an error, print it to the console.  In most cases
// you probably want to reattempt delivery to any address that errors out, 
//but the throttling error is the only one i've encountered so far.
print("Message not sent to {$name[0]}, {$result['Error']['Code']} - {$result['Error']['Message']}<br />\r\n");
if($result['Error']['Code'] == "Throttling")
$failedemails .= "$to,$username<br>\r\n";
else print("Email sent to {$name[0]}<br>\r\n");
print $failedemails; //i do all of this in a for loop, and spit out a list of emails,names that need to be re-sent due to quota limitations

What I've Learned:
After five days of sending 1,000 emails a day our Max24HourSend quota is now 1360, and our MaxSendRate is no 5. The documentation suggests that it should take 3 days to get to 10,000 but our rate may be increasing more slowly due to higher bounce rates (10 to 17%).  Last night I sent out a large batch of emails to members that haven't been around from 2006-2008 so our bounce rates are pretty high, as are the complaint rates.  I expect that to tail off as we start emailing members that are more current.  

Here's the example output of the command:
Timestamp               DeliveryAttempts        Rejects Bounces Complaints
2011-09-30T05:33:00Z    414                     0       97      2
2011-09-30T06:33:00Z    67                      0       8       0
2011-09-30T06:48:00Z    631                     0       110     3
2011-09-30T07:18:00Z    93                      0       14      0

Going Forward
I would like to build a daemon to monitor the Amazon SES send quota and then as quota becomes available automatically deliver the appropriate number of emails.  This could help efficiently distribute the email load and make sure that we keep running up the Amazon SES quota.  In order to effectively implement this I want to create an email table that stores all of the outbound emails that are queued up and then let the daemon work through the list automatically, simplifying delivery of email newsletters. 

The documentation I've found has said that it takes several days of maxing out your 1,000 message per day quota before your quota will be raised.  We've been hammering ours for 2 days and haven't seen an increase, even though we've been lightly using our account for the last several months (under 50 per day).

Complaints and Abuse Messages
We have received a few complaints and a few abuse messages for which we have removed those users from our email campaigns.  Even though there is an unsubscribe link in the email people don't use it, probably because they've been trained to never use an unsubscribe link in an email you think is spam.  My guess is they forgot they registered for us and are now treating email from us as unsolicited spam.  It is important to set the returnpath in the email to an account that can receive the complaints and abuse messages, although the email address used must by verified by Amazon SES before it can be used.  Using a tool like saasbouncer to filter your bounces and spam complaints can definitely simplify your life if you are sending out thousands of emails or more.

Thursday, September 29, 2011

MozyPro Isn't So CozyTho (Feature Review)

I help small businesses with their IT needs and this last June I found myself replacing a very costly backup solution for a small law firm. At work I use Microsoft Data Protection Manager and sometimes Symantec Backup Exec, but both of those are overkill and not natively offsite, which was a requirement.  They have a Microsoft Small Business Server 2008 Premium server, and we needed a backup solution to handle the following things:
  1. SQL Server Database
  2. Exchange Server Database
  3. Files
  4. Offsite Backup

After some research and testing I ended up selecting MozyPro to replace the old backup solution because it's very cost effective at $7 per server and 50 cents per gigabyte per month, and it filled all of our requirements.  With approximately 85GB of data the total cost comes out to under $50/mo which is very manageable for the security of an offsite backup.  We started out backing up the majority of our data with it, using VSS for SQL and Exchange and file backups for the rest.

This is where we hit our first bump in the road.  If you select the VSS backup for exchange (which is the way to back it up), and you accidentally select the EDB itself for backup, then the backup will inexplicably fail over and over.  After a few hundred GB of failed attempts at uploading the database I called Mozy and they pointed this out.  Problem solved.  One would think if this were a problem they knew about they would fix  the software to prevent this from happening.  Apparently that would be too easy.

Fast forward to today where I'm finally getting around to testing the ability to restore all of that data to a different server in the event of disaster recovery.  Mozy provides 3 seperate methods to restore your data, direct download, archive, and by media restore.  The final recovery method is in the Mozy client on the protected server, but that's not the focus of this review.

"Direct Download" as they call it allows you to select your files, click direct download, and you install the mozy restore manager which will do the downloading starting within a couple minutes of clicking restore.  We restored 85GB in about 48 hours, although the restore manager client has a bug in it and will stop transferring files saying: "Stopped: need private key"
Mozy support provided me with a beta version of  the restore manager that picked up where the previous version had left off.  The restore speed is quite abysmal, the test was performed in a datacenter with a 100megabit connection.  Watching the progress bar it seemed to move around 200-300K/s much of the time.

Here is the graph of the connection, the restore started sunday night and finished tuesday night:

Archive is where it compresses all of your files and provides links to download them.  When restoring 85gb it took 22.5 hours for it to archive everything, and provided 89 seperate links to download the files.  There is no hierarchy so you need to know where each file goes.  All 128,000 of them.  Good luck!  Archive is better suited for downloading a single file, or a directory.  Archiving 1GB of data in 3 files takes around 30 minutes, although according to the timestamps it took 71 minutes, it only took about 30.

Media Restore is the final option.  In my case restoring 85GB of data it would cost $112.45, and apparently take 2-5 days for processing and then they overnight it to you on DVD's.  If you're in a hurry this isn't a very good option unless you have dialup.  Otherwise you should hopefully be able to download 85GB quicker than that.

The Good
The restore process using the restore manager put most of the files back where they were on the original file system.  The exchange backup worked as I expected it would.

The Bad (Why I Wouldn't Recommend Mozy)
When I went to test the restore of our data I started out restoring 85GB using the Direct Download / Restore Manager. I went to bed and the next morning the download had stopped saying I needed to enter my private key (what private key?).  I did a search and found others had this problem as well,, so I contacted support and they gave me a beta version of the client which solved the problem.  

Fast forward 36 hours and the restore is complete.  I went to restore the exchange database and it's in a dirty shutdown state which is fairly typical for backup software.  After I used eseutil to repair the database everything there worked fine, I was able to mount the database in a Recovery Storage group and use the powershell to restore mailboxes to a test account.  

Next up I needed to restore the SQL database.  Unfortunately the transaction log was in the restore folder, but the main database file, the MDF, isn't there.  I  found it in the error list in Mozy so I went and selected just the SQL database files using the web interface and redownloaded them.  After several attempts the restore manager gave up saying 3 of 3 files complete, but 2 failed:

Okay, no problem.  I went back and chose the archive option.  Thirty minutes later the web interface showed 1 of 3 files... the other 2 missing, and it says it had recovered 1 of 1 files... I started another chat session (I'm good at this by now), and the guy said odd, did something and half an hour later the archive download works.  I downloaded the files (at 300 to 600K/s), extracted them, went in to SQL Server Management Studio and attempted to attach the database.
For some reason the MDF and LDF files do not match, so the database can't be attached. 

Having failed so far I decided to do one last thing.  Attempt to restore the database on the production server using the VSS restore feature of the full Mozy client from the source server. Yep.  You guessed it.  That failed too.  Why wouldn't it?
I contacted Mozy again and the chat agent escalated this to Tier2 and informed me I should get a response in 24 to 48 hours.  When pressed about how long a response typically takes the answer was "it depends how busy they are", no kidding, thanks genius, that's very insightful.  It's a good thing I'm simply attempting to test the restore process because at this point I'm not certain that our SQL data is protected by Mozy.  If this were a real outage where we needed to restore from backups I would now be 72 hours in to the process, have encountered many errors in the Mozy system, and be expecting a 24 to 48 hour expected response time from a Mozy Tier2 technician.

The next day I bugged Mozy and said I was going to cancel service if I didn't get a response ASAP, 30 minutes later I had a new archive download to try, which actually worked!

As it turns out, the Mozy software does not always take a snapshot of the LDF (Transaction Log) and the MDF (Core Database) at the same time, and if the files aren't a matching set they are useless.

Here is a screenshot they provided showing the timestamps for each file.  I've highlighted each of the rows that resulted in a successful backup by taking a snapshot of both files at the same time.  The rest of the backups are useless.  Good job MozyPro!

Awesome.  I guess I won't be planning to use the Mozy backup software for SQL server anymore as it only works half of the time.  From now on I plan to use the built in SQL Backup, which will create a .bak file on the server, and then let Mozy upload the 1GB file every night.  It's not very efficient because it can't be a differential backup, but it's better safe than lose your data.

Mozy works well for file backups, and exchange backups, but I definitely wouldn't recommend them if you need to do any SQL backups.  As always, you should test your backup system, but given the questionable reliability of the Mozy backups I would recommend testing it several times over the course of a couple days or weeks.  Additionally, Mozy support is very slow to respond taking several hours to respond to any emails even after the Tier 2 tech has finally contacted you.

Test your backups.  DO NOT use Mozy if quick and easy access to your data is important to you.

Update 2/21/2013

I decided it was time to do another test restore to make sure everything was still going okay.  Not so surprisingly the website restore functionality still doesn't work.  It constantly says it does not detect the restore manager software even though it is installed.  Eventually I simply launched the restore manager by hand and logged in, and it started to restore the data.  *phew*.  

Performance seems to be different, it now manages 0-25megabit (on a 35megabit connection), and uses 3 simultaneous streams.  In 3 hours it downloaded 8,714 files for a total of 4.5GB, approximately 4megabit.  The tool gets stuck downloading files sometimes, so the download rate is jumpy.  The tool says it restored 170gb of 118gb (175,000 files), and then crashed.  Checking the size of the restored files reads 78GB, and the Exchange databases were not restored.  Ugh.  What the heck.

I figured after a year and a half my review would be outdated, but it appears as though Mozy still hasn't worked out all of the kinks.  Maybe in mid 2015 they will have gotten all of their stuff together. :)