The Playbook

next

How we built an A/B testing framework for mobile apps in 24 hours at AngelHack DC

how-we-built-an-ab-testing-framework-for-mobile-apps-in-24-hours-at-angelhack-dc

Michele Walk

Here at Engage, we A/B test websites, subject lines, copy — pretty much anything and everything — without a second thought. But surprisingly, A/B testing in mobile apps is much, much more difficult and is years behind A/B testing tools for the web.

However, there are a couple solutions out there if you want to A/B test your iOS app. Unfortunately, many of them require you to write tests into your code or create multiple storyboards and then resubmit to the App Store (like Clutch.io and Amazon’s still-in-beta mobile A/B testing service) or mess with various tools to perform just a simple test (like pathmapp.com).  But these methods seemed inefficient and onerous — having to re-submit your app to the App Store every time you want to run a test means you not only have to wait for approval from Apple, but for your users to update the app as well. With such a long lag time and inability to tweak tests while they’re running, many of the benefits of A/B testing are lost. And it seemed ridiculous that you should have to paw through your code every time you want to run a test.

Thus, we thought there must be a better way to do things. So, we decided to use this past weekend’s AngelHack hackathon to try out a new approach. AngelHack is a worldwide series of hackathons that is being held in 35 cities this year and culminates in a final competition in San Francisco. This was DC’s second year as a host city, with the event held at the 1776 incubator space on 15th Street. I was joined by Engage’s Michele Hansen (who served as Product Manager/Idea Person) and Will Summers, a Graphic Designer at Interface Media Group, who we met at the hackathon.

We had 24 hours to build. Here is what we learned.

We started out by defining our requirements: it should be easy to implement into an existing iOS app without re-submitting to the App Store once the SDK is added and it should be possible to create tests on the fly – even for non tech-savvy people.

This led us to building a web interface inspired by Optimizely and Unbounce. We call our framework FlipTest.

The FlipTest framework builds a full overview of the app’s different screens (ViewControllers) and view hierarchies and then sends this information to the FlipTest.io website via a simple REST API. This allows the website to visualize the different ViewControllers and provides an easy way for users to A/B test changes to elements such as labels and buttons.

The view-hierarchy generation (to be sent to the server) and the view-injection process (making changes based on the A/B testing profile) is implemented using method swizzling. This allows us to very lightly inject into the existing code base of the app at runtime and perform view updates and traversal in a very efficient way. And since the code isn’t executable, it doesn’t violate App Store policies.

The FlipTest integrated app will then periodically ask for a testing profile via the REST API and make any changes to the UI if necessary. This also enables the app to track unique views and clicks so a conversion rate can be calculated and visualized on the website.

We currently show simple conversion rates visualized on a line chart. We’re looking to improve this along the way by adding more intelligent feedback for the collected data.

For a quick overview of what all this looks like, check this awesome video that @willsummers made just before we handed in the project:

We didn’t end up winning any prizes, but considering we went into the weekend without knowing if what we wanted to do was technologically possible (nevermind something we could build in a weekend), we’re incredibly proud of what we built and hope to release a version of FlipTest in the near future. You can sign up for updates about FlipTest here.

Engage’s Mathias Hansen (Lead Developer) and Michele Hansen (Operations Manager) pitching FlipTest at AngelHack DC. Photo taken by Evan Burfield

So what do you think about this approach? And is this something you would like to be able to use in your app? We would love to hear some feedback and comments.

FlipTest.io was built by @MicheleWalk, @MathiasHansen and @willsummers at AngelHack DC. (Cross-posted from FlipTest.io)

Innovative targeting highlighted in CRNC report

innovative-targeting-highlighted-in-crnc-report

Stephanie Stouffer

During the 2012 election cycle, we had the opportunity to experiment different methods for reaching the unique target audience for the Crossroads Generation super PAC — 18-29 year-old voters. The recently released postmortem report from the College Republican National Committee — “Grand Old Party for a Brand New Generation” — highlights the success of XG’s innovative tactics, such as hyper-targeted mobile ads at college football games, promoted tweets during the MTV Movie Awards, and hashtag campaigns like #PaulRyanFacts. The methods of advertising set XG apart from any other right-leaning political group or campaign.

In the CRNC report, a survey of millennials showed 9 out of 10 young voters use a cell phone at least once a week. This presented us with the opportunity to capitalize on their behavior and put XG’s message in front of them. We used hyper-targeted mobile ads that reached mobile users within a one-mile radius of a central location, like a football stadium or city park. Beginning with the Lollapalooza music festival in Chicago and carrying into football season, we targeted locations where millennials were gathering and served them XG banner ads on their mobile devices. Time magazine highlighted the campaign as one of the ways technology is changing political advertising.

To continue capitalizing on events, we ran a series of promoted tweets during the MTV Movie Awards that were specifically written to connect the nominated movies to facts millennials could relate with. For example, “It’s easier to win the Hunger Games than it is to get a job in the Obama economy.” The campaign received a record level of engagement for XG.

Looking forward, we now apply what we learned from XG’s success to other digital campaigns. We always consider the potential return on a mobile ad campaign and are constantly hunting for breaking news events or pieces of pop culture we can use to increase engagement on social channels. Let us innovate with your next digital campaign. The sky’s the limit.

Engage Welcomes Shane Helm as Design Director

engage-welcomes-shane-helm-as-design-director

Shane Helm

Posted in on May 29th, 2013

Write a blog post?  Wait.  What?  I’m a designer, not a writer …  I love design; live and breathe it, in fact!  Design is my passion.  Design is me, and at times, I am at one with it.

Design is about choosing color, type, shape, and balance to convey a mood.  Design is about keeping a message clear and concise by stripping any unnecessary clutter from the design.  So I ask, “Why muddy up a page with a bunch of words?  Couldn’t I just design a graphic to express my passion for design and my excitement to join Engage?”

But if I must choose words, I would say how impressed I have been working with Engage’s team over the last couple of months as a freelancer and with the team’s ability to affect policy through messaging for their clientele.  I am excited now to be full-time and to add my talents to Engage’s already heavy arsenal of services.

I come to Engage with over 15 years experience in interactive, motion, and print graphic design and creative direction. My work over the last few years has brought about great opportunity in several verticals including movies, product design, and redesigns of social networks.  However, I have a passion for using my talent to help a cause bigger than myself, and that is why I have joined Engage. Our mission is aligned and I believe Engage is about something bigger than itself. I am excited to add my talent to the mix and join in that great cause, albeit bigger than us.

Now, if I had been asked to use design to express my excitement, you would’ve gotten this message 2 minutes ago, and that is clearly why you need Engage.  Your interactive traffic is just that, traffic of people passing by.  You must make your message clear and engaging in the few seconds you have to grab their core viscerally.  The clock is running, time to get Engage.

Check out Shane’s work at shanehelm.com and/or dribbble.com/shanehelm.

When the right target matters

when-the-right-target-matters

Patrick Ruffini

According to the Obama campaign’s digital director, targeted sharing to friends on Facebook was the “most groundbreaking piece of technology” they developed for the 2012 campaign.

Engage is proud to announce that we’re delivering this technology to campaigns, organizations, and causes of all sizes — right now.

For this weekend’s Republican Party of Virginia convention, supporters of Pete Snyder for Lt. Governor can connect to Facebook and see which of their friends are among the 13,600 registered convention delegates. We then email them reminders about the big convention this weekend and highlight specific friends to contact — through Facebook or any way they choose.

 

Targeted sharing is the ideal tool for when personal contact to the right target matters. Whether you’re trying to mobilize people for a campaign or get people to out to an event, there’s nothing more influential than personal outreach from a friend.

The best part? It works even if you don’t have a predefined list of contacts for your supporters to reach out to. Using social data, we apply predictive modeling to your supporters’ friend lists to figure out who is most likely to be persuaded by your message.

Engage is proud to be driving innovation in politics and advocacy where it’s needed most — technology and data science.

(We also happen to think this is pretty cool.)

Let’s talk about what targeted sharing can do for you. I’d love to chat.

Applying “Inside the Cave” Beyond Campaigns

applying-inside-the-cave-beyond-campaigns

Patrick Ruffini

Posted in on Apr 17th, 2013

After reading our report on the Obama campaign’s digital and analytics operations, Inside the Cave, you may have thought to yourself, “That’s fascinating, but how can all this help my organization?”

To answer that question for non-profits, I teamed up with Obama for America’s Michael Slaby to produce “From Politics to Public Policy: How Campaign Lessons Can Amplify Your Work.” The paper was sponsored by the Joyce Foundation, a leading foundation based in Chicago.

We’re quickly seeing the differences in digital methods employed by campaigns, corporations, and non-profits disappear as everyone zeroes in on the same question: “What can I do to make sure the right message hits the right people, and only the right people, at the right time?” With more tools than ever at our disposal, the answer to this question might seem simple. But, as we saw in 2012, tools cannot reach their potential without insight and rigorous testing — even when it means putting sacrosanct gut feelings up to the test.

You can read the full paper here or read summaries in the Standford Social Innovation Review.

As always, feel free to get in touch if you have questions. You can reach me at patrick (at) engagedc (dot) com.