A/B Test everything. Even your tests.

How we automated and optimized A/B testing for continuous growth.
Market insight

Introduction: Why A/B testing matters

In today’s competitive tech landscape, data-driven experimentation is no longer optional, it’s essential. Studies show that businesses using A/B testing effectively can achieve up to 30% higher conversion rates. At our studio, we’ve embraced this ethos by automating and refining our A/B testing processes to drive growth for our apps, including Dygest (our non-fiction book summary app designed to empower readers with actionable insights 😎).

The never-ending experimentation process

A/B testing is often seen as a one-off activity, but we believe it’s a continuous journey. For Dygest, we’ve adopted an iterative approach that doesn’t just test features, it tests the tests themselves. This philosophy has allowed us to uncover hidden opportunities and optimize every aspect of our user experience.

Step 0 to 1: Building the foundation

When launching Dygest, we focused on critical early-stage experiments:

  • App Store CPP Screenshots: We tested various designs and messaging strategies to maximize click-through rates. For example, screenshots emphasizing “actionable insights” performed 15% better than generic designs.
  • Paywall Optimization: Subscription models are pivotal for apps like Dygest. Through rigorous testing of price points, offer layouts, and timing (e.g., when to display the paywall), we identified that offering a free trial upfront increased conversions by 20%.

Automating the process

To scale our efforts, we developed an internal automation system for A/B testing. This system allows us to:

  • Run multiple experiments simultaneously without manual oversight.
  • Iterate faster by feeding winning variants back into new tests.

Lessons learned

  1. Test Early and Often: Starting tests at Step 0 ensures you’re building on validated assumptions.
  2. Don’t Fear Failure: Over half (even more) of our experiments didn’t yield significant results, but each failure provided valuable insights.
  3. Don’t give up and iterate again, and again.

Conclusion: The future of experimentation‍

At our studio, we see experimentation as the backbone of innovation. By automating and optimizing A/B testing processes, we’ve not only improved our apps but also fostered a culture of continuous learning.

The key is simple: test everything, and then test it again.