Faster, Higher, Stronger: The Secrets of Keeping up with Ever More Campaigns and Higher Data Volumes

Find out how the FastStats team work hard to deliver at an Olympic speed across the board. The phrase “Faster, Higher, Stronger” is, in addition to being the Olympic motto, something that you should be demanding of your data analysis software.

Deep down, the whole development team working on FastStats knows that application performance is one of the most important things we have to care about.  “Fast” has been in the name of the product since the beginning and so we have nowhere to hide if things start to slow down.  We even have a special high performance group who work on eking more out of the engine inside FastStats.  To try and avoid surprises when people show us the things that they ask our software to do we also develop new ways of monitoring how FastStats performs in the real world to help us get the edge on the challenges ahead of us.

Part of this development process is talking to our users. We saw at the FastStats User Group last November that most people were already dealing with large amounts of data and, what is more, for a lot of these people their data volumes were going to increase.  Of the roughly 250 attendees that knew about their data volumes, about a quarter reported having 100s of millions of transactions in their FastStats systems and over 15% were dealing with billions of transactions.  Then, of those that had knowledge of data growth, over 20% thought their data volumes would grow by half again over this year and even more thought their data would at least double in size.

So how do we make sure we can cope with these data volumes?  One way is to develop monitoring tools to look more intelligently at real life systems and see where the bottlenecks start to appear.  The last couple of FastStats releases have included an unheralded new utility called the Job Visualiser:

This shows when jobs run within FastStats as well as providing some simple stats on how long they are taking on average.  We have used this tool on real-world and test systems to see how FastStats behaves under different loads.  If you are interested in how your system is performing and have access to your FastStats Service, try it out and let us know how you get on.

As well as monitoring tools, those of us working on PeopleStage have also developed performance and load testing tools to simulate running campaigns in different situations and with different numbers of users:

These tools have helped us reproduce the demanding scenarios that people expect FastStats to cope with and improve the software’s performance accordingly.

To list a couple of examples of the improvements we have made:

• The Q1 2015 release of PeopleStage can process a simple campaign with 1 million people roughly 25% faster than the August 2014 release.   This has been due to some streamlined handling of campaign constraints and database access.  
• We are working on ways to make it quicker to interact with PeopleStage and the Q2 2015 release should have a much reduced login time and quicker library refresh. This is something the guys here have always nagged me about and so I’m pleased to make an improvement in this area.
• All of this comes on top of general improvements in the FastStats engine resulting in improved query times and system load times.  On one of our benchmark systems running on a reasonably fast desktop machine the load time for a system with about 750 million rows of input data was reduced by over a quarter from 3 hours 24 minutes to 2 hours 29.

But there is always more to do, and we are now looking at how we can further integrate these tools in to our day to day development processes.  The Olympic motto of “Faster, Higher, Stronger” seems apt here as whilst data volumes keep increasing, FastStats performance will have to keep up - but it’s a challenge that will never end. This constant improvement enables you and your business to create data driven marketing campaigns that target exactly who you need in a manner they want - quickly.