Ab testing

A/B Testing Fundamentals

A/B testing is a method of comparing two versions of a webpage or element to determine which performs better. This guide covers the fundamental concepts you need to understand before running your first test.

What is A/B Testing?

A/B testing, also known as split testing, is a controlled experiment where you:

  1. Create two versions of a page or element (A and B)
  2. Split your traffic randomly between the two versions
  3. Measure performance using predefined metrics
  4. Analyze results to determine which version performs better

Why A/B Test?

Remove Guesswork

Instead of making decisions based on opinions or assumptions, A/B testing provides data-driven insights into what actually works with your audience.

Increase Conversions

Even small improvements in conversion rates can have significant impacts on your business. A 1% improvement could mean thousands of additional customers.

Reduce Risk

Testing changes on a portion of your traffic before rolling them out to everyone minimizes the risk of negative impacts.

Learn About Your Audience

A/B tests reveal preferences and behaviors of your specific audience, not just general best practices.

Key Components of an A/B Test

Hypothesis

Every test should start with a clear hypothesis:

"If I change [variable] to [variation], then [metric] will [increase/decrease] because [reasoning]."

Example: "If I change the CTA button text from 'Learn More' to 'Start Free Trial', then click-through rate will increase because it creates clearer value and urgency."

Variables

The element you're testing. Common variables include:

  • Headlines and copy
  • Call-to-action buttons
  • Images and videos
  • Page layouts
  • Forms
  • Colors and fonts

Metrics

How you measure success. Metrics fall into two categories:

Primary Metrics (what you're optimizing):

  • Conversion rate
  • Click-through rate
  • Sign-up rate
  • Revenue per visitor

Secondary Metrics (additional insights):

  • Bounce rate
  • Time on page
  • Pages per session

Sample Size

The number of visitors needed to detect a meaningful difference. This depends on:

  • Current conversion rate
  • Minimum detectable effect
  • Statistical significance level (95% is standard)
  • Statistical power (80% is typical)

Types of A/B Tests

Simple A/B Test

Compares two versions of a single element.

Example: Testing two different headlines on a landing page.

Pros:

  • Easy to set up and analyze
  • Clear, actionable results
  • Fast to implement

Cons:

  • Tests only one element at a time
  • May miss interaction effects

Multi-Element Test

Test multiple elements on the same page, where each element is tested independently.

Example: Testing headline variations AND button variations on the same page as separate tests.

Pros:

  • Faster than running tests sequentially
  • Independent analysis for each element
  • Maximizes testing velocity

Cons:

  • Doesn't capture interaction effects between elements
  • Requires more traffic than single-element tests

Setting Up Your Test

1. Define Your Goal

What specific business outcome are you trying to improve?

  • More sign-ups?
  • Higher purchase rate?
  • Increased engagement?
  • Better click-through?

2. Develop Your Hypothesis

Based on user research, analytics data, or conversion optimization best practices.

Good hypothesis: "Changing the CTA from 'Submit' to 'Get Started Free' will increase conversions because it emphasizes value and removes friction."

Bad hypothesis: "Let's try a different button color and see what happens."

3. Choose Your Variable

Start with elements likely to have the biggest impact:

  • Headlines (often highest impact)
  • Call-to-action buttons
  • Value propositions
  • Hero images

4. Create Your Variations

Make meaningful differences that your audience will notice:

Good:

  • "Start Your Free Trial" vs "Get Started Free Today"
  • Blue button vs High-contrast orange button
  • Benefit-focused headline vs Feature-focused headline

Not recommended:

  • "Start Your Free Trial" vs "Begin Your Free Trial" (too similar)
  • Minor color shade changes (not noticeable)
  • Small copy tweaks (not impactful)

5. Set Success Metrics

Choose metrics that:

  • Align with your business goals
  • Can be measured reliably
  • Reflect real user value

6. Launch and Monitor

Let Keak handle the statistical analysis. The test will run until:

  • Significance is reached (winner found)
  • Futility is detected (no meaningful difference)
  • You manually stop it

Common Mistakes to Avoid

Testing Too Many Things

Problem: Testing headline, image, AND button all at once makes it impossible to know what drove results.

Solution: Test one primary change per test for clear insights.

Stopping Tests Too Early

Problem: Declaring a winner after 100 visitors because one variant is ahead.

Solution: Wait for Keak to declare statistical significance based on SPRT.

Ignoring External Factors

Problem: Running a test during Black Friday and assuming results will hold year-round.

Solution: Consider seasonality, campaigns, and other factors. Run tests during typical periods.

Not Having a Clear Hypothesis

Problem: Random testing without reasoning rarely leads to insights.

Solution: Always start with "I believe X will improve Y because Z."

Testing Insignificant Elements

Problem: Testing elements that don't impact user behavior or conversions.

Solution: Focus on high-impact areas: headlines, CTAs, value props, hero sections.

How Long Should Tests Run?

Test duration depends on your traffic and conversion rate:

Traffic Volume

  • High traffic sites: Days to weeks
  • Medium traffic sites: Weeks to months
  • Low traffic sites: Months to quarters

Effect Size

  • Large changes: Detect faster (bold redesigns, major copy changes)
  • Small changes: Need more time (minor tweaks, subtle differences)

Baseline Conversion Rate

  • High conversion rates (>10%): Detect changes faster
  • Low conversion rates (<2%): Need significantly more visitors

Best Practices

Start Simple

Begin with high-impact, easy-to-implement tests before moving to complex experiments.

First tests:

  • Homepage headline
  • Primary CTA button
  • Hero image
  • Value proposition

Later tests:

  • Navigation structure
  • Page layout
  • Multi-step forms
  • Checkout flow

Test Continuously

A/B testing should be an ongoing process, not a one-time activity.

Build a testing culture:

  • Always have tests running
  • Test new pages and features
  • Revisit old tests with new insights
  • Build on previous learnings

Document Everything

Keep detailed records:

  • Test hypothesis and reasoning
  • Launch and end dates
  • Results and significance levels
  • Implementation decisions
  • Learnings and insights

Share Results

Communicate findings across your organization:

  • Build a testing knowledge base
  • Share wins AND losses
  • Educate stakeholders on methodology
  • Create a culture of experimentation

Learn from Failures

"Failed" tests (where neither variant wins) often provide valuable insights:

  • Your audience might care about different things than you thought
  • The change might not be impactful enough
  • External factors might be interfering
  • You might need a more radical variation

When to Trust Your Results

Trust results when:

✅ Keak declares statistical significance

✅ Minimum sample size reached (1000+ visitors)

✅ Test ran for complete business cycle (at least 1-2 weeks)

✅ Both weekdays and weekends included

✅ No major external events occurred

When to Be Cautious

Be skeptical when:

❌ Very low sample size (<500 visitors total)

❌ Test ran less than one week

❌ Major marketing campaign occurred during test

❌ Seasonal event (holiday, sale) affected traffic

❌ Technical issues occurred (site downtime, tracking errors)

Practical Significance vs Statistical Significance

A result can be statistically significant but not worth implementing.

Example:

  • Variation B wins with 99% confidence
  • Improvement is 0.1% conversion rate increase
  • Implementation requires major development work
  • Projected revenue gain: $200/year
  • Development cost: $5,000

Decision: Statistically significant, but not practically worth it.

Always consider:

  • Business impact vs implementation cost
  • Long-term sustainability
  • Brand alignment
  • User experience implications

Next Steps

Now that you understand A/B testing fundamentals:

  1. Launch Your First Test - Get started with testing
  2. Learn About SPRT - Understand how Keak determines winners
  3. Understand Test Types - Choose the right test for your goals

Ready to start testing? Open the Keak extension on your website and create your first variation.