Rayid Ghani was the Chief Data Scientist at Obama for America 2012, bringing analytics, technology and data mining to the political realm, helping the Obama for America re-election campaign to raise funds while getting people out to vote in compelling new ways. With Obama back in the White House for another four years, Rayid will now use the same skills at the University of Chicago to address complex social problems. ADMA’s Kim Carter finds out more.
You worked for 10 years in the research lab at Accenture. What was the attraction of becoming the chief data scientist for the Obama for America campaign?
I spent 10 years at Accenture Labs working on Analytics R&D, living between the academic and business worlds – writing papers, giving talks at conferences, working with universities and at the same time applying my skills to solve problems for businesses (retailers, manufacturers, hospitals, insurers and banks). But eventually the intellectual and technical challenges were not satisfying enough. I was getting frustrated that I wasn’t able to make more direct social impact with my work. So I decided to leave my job at Accenture and focus on finding something related to non-profits or organisations that are making a social impact, which, interestingly, was when Obama’s campaign people approached me about taking on the chief data scientist job. Chicago can be a small town in that way.
What attracted me to the campaign was Obama and his policy record and the chance to make an impact. He is great at emotionally connecting with and motivating people. But what was important to me is what he had accomplished in his first term and I felt there was still more he could do with a second term. I couldn’t think of a better place to make the kind of impact I was hoping to make in that period of time.
The majority of the work I was doing at Accenture was looking at large amounts of data to help companies make better business decisions and I could apply this experience in the political space by making sense of the data to help the team make better decisions throughout the campaign.
People have said your very-well run analytics strategy was what won the campaign for Obama. What’re your thoughts about this?
Analytics certainly helped us to make better decisions and be more efficient and effective, but they don’t win an election on their own. We had an excellent candidate in Barack Obama and better policies for the country (and the larger world). We also had extremely supportive leadership from Jim Messina who realised the importance of analytics and wanted it embedded in every aspect of the campaign.
We also had an energetic, motivated staff of thousands and over two million volunteers who did the hard work in the field -- organising and making over 100 million door-knocks and phone calls. Those people were the ones doing the hard work that was critical to making the analytics actionable and ensuring the campaign was effective.
Where did analytics make the biggest contribution -- pre-election analysis, fundraising or perhaps mobilising volunteers?
We were part of every campaign function ranging from fundraising, volunteer recruiting and mobilising, email messaging, polling, social media, TV and online ads. We worked with every department in the campaign and helped them to be more efficient and effective. Data was used to decide everything from what ad spots do we buy in a market to who we target on social media.
What data did you use?
We used voter registration data that is publicly available and data about people who have supported Obama by volunteering or making campaign contributions. Most of the rest of the data we had was collected from interacting with people, which meant we had called them, they had donated money, or were active with us in some online or offline channel.
Looking at the data allowed the team to find the best ways to message our targeted voters. The media analytics team also used the same targeting to inform the hundreds of millions of dollars spent on TV ads for the campaign.
What’s the advantage in using data?
It makes you more efficient. In the old days you might have picked up the phone book and started calling people or used simple heuristics such as party registration, but you’d be wasting a lot of time and money on people who you won’t convince to vote your way or you’d be preaching to the converted. With a data-driven strategy, you can target those voters who are more likely to be persuaded by your call and pick up more votes more efficiently.
How did you make the data work for marketing purposes?
The first thing our analytics technology team did was merge all the information from voting records, fundraising information, volunteer data, and any other interactions to provide a common data store. The modelling team then used that data to build predictive models that predicted the behaviour of each voter.
Through all this data and models, we began creating multi-channel marketing campaigns, connecting offline and online marketing to build stronger connections with people and to get them out of the house to vote on voting day.
Data was also used to create frequent, personalised email messages to supporters and potential supporters. These messages included talking directly with recipients and using their friends’ names and pictures to build stronger connections. The individuals could also connect with their friends and family to get them to register to vote and go to the polls on Election Day. Data was also helpful in figuring out the best ways to ask people for money and how much we should ask them for.
Five billion emails were sent out to potential donors and voters. Surprisingly, through constant testing, we found the more emails we sent, the more money we were able to raise. As we got closer to election day, our experiments convinced us to send up to four or five emails a day to every person without any negative outcomes.
What challenges did you face during the campaign?
Hiring was the first challenge – getting people with the right technical skills and not being able to pay them a lot. Also it was hard not being able to tell them a great deal in the interview about what we were going to do during the campaign.
The second challenge was we had too many databases all over the place from the 2008 campaign and they didn’t talk to each other. The 2010 team had done an excellent job of building out a starting infrastructure and the current team took over from there. They created a single massive system that could merge the information collected from pollsters, fundraisers and field workers as well as from social media and other contacts. This took a lot of time and effort. But it paid off: this system told us how to find voters and get their interest, and allowed us to run tests to predict which people would be persuaded to vote Democrat by certain types of appeals. We could also make predictive models of who the typical volunteer could be. In all, it made the campaign spend its time more efficiently.
The third challenge was our initial struggle to reach younger people. The usual marketing channels like TV ads and door knocking don’t reach people under 25 because they’re not watching TV, they’re on their computers, they don’t always have landline phones and they’re not likely to be home during the day to talk to a volunteer. So we used social media and we built a special Facebook tool called Targeted Sharing to engage them that was extremely effective in using our predictive models and getting people to take action on our behalf.
I think people coming to ADMA Global Forum would be interested to hear what analytics/data mining tools were used in the Obama 2012 campaign. Could you tell us a bit about that?
We used a combination of commercial, open-source and internally built tools for voter turnout, fundraising, advertising, door knocking, social media and email campaigns. We developed a common data platform which was supported by various tools and applications like Hadoop, R and Stata. We used some commercial tools like Vertica for our database, Tableau for visualisation, and KXEN for some of the predictive models. We didn’t have a lot of time to train people in new tools so we often relied on tools they were familiar with and experts in and we coded a lot of our own software.
Targeted Sharing using Facebook’s social graph was one example of building our own software. We asked people to authorise our Facebook App, and asked permission to access their feed and friend list. We then matched their list of friends to the voter database to use additional information about the friends and infer their volunteering, donation and voting behaviour. Based on our predictive influence models and that additional information, we could then suggest 10 friends this person should ask to do something on our behalf. We ended up getting over a million people to authorise that app and help us get our message out in a very targeted manner through the social graph.
Then Facebook was an extremely important tool for the campaign?
Definitely. Our offline field strategy was to get our supporters to talk to their friends, family and neighbours who are most likely to help us win the election. Targeted Sharing allowed us to do the same in the online world.
There’s a lot of talk now about big data. Is this just a buzz word? How was Big Data used in the campaign?
It’s certainly a buzzword right now but I think that’s a short-term issue. It’s less about big data and more about big infrastructure and big implementation to make big decisions. The campaign didn’t have a lot of data, compared to a lot of commercial organisations. In fact, the data we had at the campaign was smaller in size than most data I’ve worked with before. What was different was that we had to take that data and make big decisions from it. We needed a big infrastructure that was able to put all their data together quickly and reliably, and make recommendations that guided the spending of hundreds of millions of dollars, and affected what millions of people did.
In Australia we have many data analysts and many marketers, but there is a shortage of analysts who understand marketing and marketers who understand analytics. Any advice for the industry and for would-be data analysts?
I think we’re seeing a convergence in skills. Marketers are learning to be more analytical and analysts are getting deeper in domains such as marketing. I don’t think you need to be an expert in the two but you need to have some basic training. Marketers not only need to be more analytical but also understand how to consume the results of statistical algorithms. They don’t need to know how to do it themselves but they need to know what it means and what to do with it. I hear so many business professionals say, ‘I’m not good at math.’ That’s just not acceptable. I would say, sit down with the analysts and try to understand the maths and ask lots of questions so you really understand where the analysts are coming from.
As for data analysts they not only need to get the analysis done but also understand the theory behind the tools they’re using so they understand their capabilities and limitations. They shouldn’t just focus on the technical aspect of their work but learn how to communicate well. They need to be able to clearly articulate the impact of their analysis, as well as its limitations, not just present graphs and charts which can be incomprehensible to the marketing team and clients. And then the analysts need to be able to map their analysis to actions that marketers can take. This link makes the marketers a lot more effective and helps them understand the value of analytics. The more a data analyst can communicate, the more likely it is that their work will get used more effectively.
You’ve made it so data will now be a key component of political campaigns in the future. How will data use evolve???
Data has been a component of campaigns for a while but I think we’re still at the beginning of using data-driven approaches to political campaigns. I see the future use of data as enabling more personalised and relevant interactions with voters in a scalable way, to get them more educated about the issues and more involved in political and policy discussions. And it doesn’t just stop with elections. Data can be used by governments to help inform its citizens about policies that matter to them, get them more educated about their impact and get them involved in shaping future policy.
The Obama for America campaign ended six months ago. What have you been up to since?
First I had to catch up on a year and a half of my life. I don’t think anyone at the campaign slept during that time, responded to personal emails or read a book. It was exhausting. We worked 18+ hour days with no weekends which wasn’t easy. After the campaign, I wanted to catch up not only on sleep, but also on friends and family.
Now I’m busy again. I have a startup called Edgeflip with a couple of my colleagues (Kit Rodolfa and Matthew Rattigan) from the campaign. We are extending one of the social media tools we built in the campaign, Targeted Sharing, to help non-profits and social good organisations use social networks more effectively to do fundraising, volunteer recruiting, and targeted outreach and advocacy.
I’m also at the University of Chicago, at the Computation Institute and the Harris School of Public Policy. My work there is focused on applying my analytics skills to issues that I care deeply about: education, healthcare, and related areas. The University of Chicago is offering me a chance to do this. We’ll be launching a new master’s program in data-driven and computational public policy at the University of Chicago next year. I’m also traveling a little bit, giving talks, coming to ADMA Global Forum in Australia.
This summer, I’m also running a three-month fellowship program on Data Science for Social Good (http://dssg.io). The goal is to use data analysis to address the world’s complex social problems in education, health care, public safety, transportation and energy. We had over 600 applications and have taken on 36 students from computer science, statistics, economics and public policy who we’re hoping to train up to be a community of scientists who want to apply statistics and data to aid society’s greatest challenges and who are passionate about making a social impact.
Using analytics I want to try to find ways to stop kids dropping out of school by identifying the ones at risk and also look at using analytics in issues like childhood obesity so we can plan interventions earlier.
Life will be busy, but I doubt it will get as crazy as working on the campaign!
Like the insights from Rayid? IAPA has secured ten seats at the Real-time Data & Analytics Masterclass on 7 August, featuring Rayid Ghani, for just $495 each (saving up to $300 off normal registration).
Take advantage of the limited IAPA Masterclass tickets by registering here.
The Masterclass is part of the ADMA Global Forum taking place in Sydney on 7 to 9 August, 2013.
See more about ADMA Forum here.