How to make testing and optimization part of your digital roadmap.

As part of most of our engagements, we’re often helping our clients create a culture of testing and optimization. While analytics has become the cost of entry, most organizations simply don’t have data warehouses or data stores in enormous servers, or the resources and expertise to make analytics a proactive method of driving continual improvement. But you don’t need to have “big data” in order to start testing. Even with large amounts of data, many organizations struggle to make use of this available asset. Often, data is passively acquired and stored with no real action taken. They have it. It’s there. But now what?

While immense data is clearly valuable, it is often more effective to focus on smaller sets of manageable, quality data, allowing for actionable insights to be generated. Actionable meaning: let’s understand the data and let’s put a plan in place to act on the data, to make changes based on what we’re seeing, so we can create a positive result of some kind. Testing enables this behavior because, rather than passively acquiring and storing data, testing allows an organization to actively generate data directly from its customers. This in turn allows teams to iterate quickly and optimize, taking action based on the insights.

Let’s start with a simple definition.

Testing is the comparing of two or more versions of something to understand which one performs better. However, we believe there is far more to testing than enabling conversion rate optimization. While much of testing is focused on conversions, a better definition is “behavior management” which is about identifying a target behavior and determining whether you’re looking to increase or decrease that behavior.

So, why should you care about testing anyway?

For most organizations, buying into the idea of testing is about understanding its benefits. Often, testing assumes some level of increased effort for the production team. While this is a valid concern, as more work means more costs, there are many benefits that justify the increased effort that comes with a culture of testing and optimization. Here are just a few:

1. Stop relying on gut instinct alone - How often is feedback based solely on personal perspectives and not based on data-driven, provable metrics? Usually feedback that begins with “I think…because…” is a good indicator that decisions are being made on gut instinct alone. With testing, you don’t have to solely rely on gut instinct or the opinion of the most senior person in the room because you have objective, data-backed results. Having data to validate decisions minimizes (or eliminates) the back and forth discussions and justifications and helps to remove the subjectivity of decision making.
2. Get to a decision quicker by failing fast - While sometimes less palatable in the enterprise business world, failing fast is a start-up mentality and it means that you’re learning and iterating quickly. Testing generates the data and insight to help you quickly learn whether a feature, a design approach, or a pricing strategy is working or not. It allows teams to avoid pouring hours into a concept only to learn after launch that the conversion rate is much lower than expected. Fail fast. Test often. Do better.
3. Let more ideas see the light of day - Critics of testing claim that it stifles creativity and inhibits designers from doing their work based on their inherent skill, knowledge, and craft. However, in the same vein as failing fast, testing allows more of your team’s work to actually see the light of day rather than fizzle away on the boardroom table. Testing requires iteration and iteration requires multiple variations and versions of designs and features. So, the more the team produces, the more tests you can run.

If you’ve read this far, you’re likely already convinced that enabling a culture of testing and optimization is clearly all upside to your organization. If you’re interested in introducing testing into your organization, you’ll likely have some hurdles to get through, as we noted above, this extra effort often requires additional resources and in turn, new costs. If you’re up for the challenge, here are a few considerations to keep in mind:

1. Get buy in from the top - Focussing on the benefits of a culture of testing and optimization is how to get the green light from Senior Management. They are going to be most interested to know how increased efforts and cost will drive results that are clear, measurable, and linked to concrete business goals. Be able to answer two questions clearly: how does it increase revenues or lower costs, and how does it create a better experience for customers. If test-driven improvements clearly lead to positive outcomes, it’s a no brainer.
2. Commit resources - A testing and optimization strategy requires commitment. Creating a budget for this work ensures that the planning, testing, and analysis process is thoughtfully and properly executed. While an Analyst might lead the charge in developing the testing approach, pulling reports, and presenting insights, a broader team that includes User Experience, Design, Content, Engineering, and Product Management will typically be part of the evaluation and recommendations process, as data can impact all aspects of a digital product, service, or platform, and each of these team members will view data through a slightly different lens. While Google Analytics is a great starting point, don’t overlook the additional fees tied to additional SaaS services that might be required.
3. Make testing an ongoing, integrated part of your process - It’s important that testing isn’t perceived as a roadblock to the regular flow of your process. But it’s also important to give testing the time it needs to be successful. We typically use testing as a method to validate our assumptions early in the Concepting and Design stage of a project, to ensure we’re creating the right solution for users. We also test after deployment as an optimization initiative. Ongoing, we use analytics to understand how users are engaging with an experience and whether, over time, we might need to make adjustments to increase performance, engagement, or usability. For most of the work we produce, ongoing measurement is a key aspect of the support we provide in helping our clients fully realize their investment in new digital products and services.
4. Document everything - It may sound like a lot of extra work, but the value in clearly documenting and sharing testing ideas, plans, efforts, and outcomes will absolutely pay off. We use a simple Google Sheet approach to organize our test plan and backlog of tests, as well as all past tests and their results/conclusions. Documentation also helps the project team to consider testing opportunities and capture test ideas at any stage in the process. That feature that might seem a little risky? Plan it, design it, but assume you’ll be testing it to make sure it’s actually desirable to users. For our clients, we use documentation as a way to transparently share our recommendations, helping to prioritize the product roadmap and deliver continual improvements.
5. Pick your partners - Testing is absolutely a behavior that requires internal buy in. If it’s going to create real value for your business, you and your team must make it a priority, it needs to be factored into your bigger roadmap of digital initiatives, and you probably need to change some internal behaviors. But in reality, you likely have partners at the table that will also play a critical role in making testing and optimization feasible. First, make sure your partners in fact use data throughout their process. Ask about their testing methodology to validate design decisions and understand your role in the process. Also, ensure any digital initiative you undertake with your partners includes a post-deployment testing and optimization plan. With the access we have to measurement today, there’s no reason to overlook this as an important part of the process, and you want to make sure your partners equally believe in the value that comes from ongoing optimization.

• Challenge assumptions - Question what you know or what your stakeholders think they know. Be skeptical and open to other perspectives!
• No pointless tests - There’s always something to be learned. If a variation is unsuccessful, you know not to do it again. If your assumption is successful, it’s extra validation.
• Continuous iteration - Tests drive incremental changes and doing one-offs won’t deliver significant results. Testing must be a continuous and consistent process.
• Know your customers - Understanding the motivations, desires, and behavioral differences of your customers will help guide a more informed hypothesis.
• Collaborate and share - From test ideas, to hypotheses, to variation development, testing requires input from across the team and organization.

For those just starting to implement a true testing and optimization approach across your digital products, platforms, and services, we hope this primer will come in handy. If you’re looking for a partner to help you define your approach, reach out, we’d love to hear from you.