Optimizely A/B Testing: 15% Growth by 2026

Getting started with effective growth hacking techniques can feel overwhelming, but the right tools make all the difference. We’re talking about rapid experimentation, data-driven decisions, and a relentless focus on scalable user acquisition and retention. Ready to stop guessing and start growing?

1. Setting Up Your First Experiment in Optimizely Web Experimentation

When I advise clients on growth, I always tell them: start small, but start smart. For web-based growth hacking, there’s no better place to begin than with A/B testing. We’ll use Optimizely Web Experimentation (formerly Optimizely X Web) because it’s powerful, flexible, and gives you deep insights. Forget those free tools; they just don’t cut it for serious growth.

1.1. Creating a New Experiment

First things first, log into your Optimizely Web Experimentation dashboard. On the left-hand navigation bar, click on “Experiments”. You’ll see a big, friendly button that says “+ Create New Experiment”. Click it.

A modal will pop up. For our purposes, select “A/B Test”. Give your experiment a clear, descriptive name – something like “Homepage CTA Button Color Test” or “Pricing Page Headline Refinement”. Clarity here prevents future headaches, trust me. I had a client last year who named everything “Test 1,” “Test 2,” and we wasted hours trying to figure out what was what.

Next, enter the URL of the page you want to test. If you want to test across multiple similar pages, you can use wildcards (e.g., https://yourdomain.com/products/*). For a beginner, though, stick to a single, high-traffic page.

Pro Tip: Always start with pages that have significant traffic. Testing a page with 10 visitors a day will take months to reach statistical significance, if ever. Aim for pages getting at least 500-1000 daily unique visitors for meaningful results within a few weeks.

Common Mistake: Not defining a clear hypothesis before starting. Before you even touch Optimizely, ask yourself: “What do I expect to happen, and why?” For instance, “I believe changing the CTA button color to green will increase clicks by 10% because green often signifies ‘go’ or ‘success’.”

Expected Outcome: A new experiment draft is created, ready for you to define your variations and goals.

1.2. Defining Variations and Traffic Allocation

Once your experiment is created, you’ll land in the visual editor. This is where the magic happens. On the left panel, under “Variations”, you’ll see your “Original” (Control) and “Variation 1”.

1. To edit “Variation 1”, click the pencil icon next to its name. The visual editor will load your page.
2. Hover over the element you want to change (e.g., your “Sign Up Now” button). A blue box will appear. Click it.
3. A context menu will pop up. You can choose to “Edit Text”, “Edit HTML”, “Change Image”, or “Change Style”. For a button color change, select “Change Style”.
4. Find the background-color property and enter your desired hex code (e.g., #4CAF50 for a vibrant green). Click “Apply”.
5. Repeat for any other changes you want in this variation.
6. To add another variation, click “+ Add Variation” in the left panel and repeat the editing process.

Next, look for the “Traffic Allocation” section, usually found below your variations. By default, Optimizely splits traffic evenly. For your first test, I recommend a simple 50/50 split between your Original and Variation 1. If you have multiple variations, say three, you might do 33/33/34. You can adjust this by dragging the sliders.

Pro Tip: Don’t split traffic too thinly across too many variations if your page traffic is low. You’ll dilute your data and need much longer to reach significance. Two variations (Control vs. one new idea) are often best for beginners.

Common Mistake: Making too many changes in a single variation. If you change the headline, button color, and image all at once, and your variation wins, you won’t know which change caused the improvement. Stick to one primary change per variation for clear insights.

Expected Outcome: Your experiment now has defined variations, and traffic is allocated. You can preview your changes directly in the Optimizely editor.

2. Defining Goals and Audiences

An experiment without clear goals is just random clicking. A growth hacker knows exactly what they’re trying to achieve.

2.1. Setting Up Experiment Goals

In the Optimizely experiment editor, navigate to the “Goals” tab. This is where you tell Optimizely what success looks like. You’ll see options to add existing goals or create new ones.

1. Click “+ Add Goal”.
2. For most initial growth hacks, you’ll want a “Click” goal, a “Pageview” goal, or a “Custom Event”.
3. If you’re testing a button, choose “Click”. Then, use the visual editor to select the specific button or element whose clicks you want to track.
4. If you’re looking for deeper engagement, a “Pageview” goal to a “Thank You” page after a conversion is excellent. Enter the exact URL of that page.
5. For more advanced tracking (like form submissions without a redirect), you might need a “Custom Event”. This usually requires a developer to implement a small snippet of JavaScript on your site. For now, stick to clicks or pageviews.

Always set a primary goal. This is the single metric that will determine if your experiment is a winner. You can add secondary goals too, but don’t let them muddy the waters regarding your main objective.

Pro Tip: Think about the entire user journey. If you’re testing a headline on a product page, your primary goal might be “Add to Cart” clicks, but a good secondary goal would be “Product Page View” to ensure your headline isn’t actually deterring initial engagement.

Common Mistake: Setting too many primary goals. Pick one. Your experiment should answer one big question, even if it sheds light on others.

Expected Outcome: Your experiment now has clear success metrics, allowing Optimizely to accurately measure performance differences between your variations.

2.2. Segmenting Your Audience

Not all users are created equal, and smart growth hacking recognizes this. In the Optimizely experiment editor, go to the “Audiences” tab. Here, you can specify who sees your experiment.

1. Click “+ Add Audience”.
2. You’ll see a list of pre-defined audience types: “Geographic”, “Device”, “Traffic Source”, “Custom Attributes”, and more.
3. For example, if you’re a local business in Atlanta, Georgia, you might add a “Geographic” audience and specify “United States > Georgia > Atlanta”. This ensures only local users see your specific test.
4. If your site has a mobile app, you might create a “Device” audience for “Mobile” users to test a mobile-specific layout.
5. You can combine conditions using “AND” or “OR” logic. For instance, “Mobile users AND from California”.

This is where Optimizely shines. We ran an experiment last year where we saw no overall difference, but when we segmented the results by device, we found the mobile variation was crushing it, while the desktop version was underperforming. Without audience segmentation, we would have missed a massive opportunity.

Pro Tip: Don’t over-segment initially unless you have a very specific reason. The more niche your audience, the longer it takes to gather enough data. Start broad, then refine. However, if you know a particular segment behaves differently, target them directly!

Common Mistake: Not considering audience implications. Running a test designed for first-time visitors on returning customers might yield confusing results. Think about your user personas.

Expected Outcome: Your experiment is now configured to show only to a specific, relevant segment of your website visitors, increasing the relevance and impact of your test.

3. Launching and Analyzing Your Experiment

The launch is exciting, but the analysis is where you learn and grow. This is the heart of marketing experimentation.

3.1. Launching Your Experiment

Once you’ve defined your variations, goals, and audiences, you’re almost ready. Before launching, go to the “Review” tab in Optimizely. It will highlight any potential issues or warnings.

1. Click “QA Your Changes” to preview your variations live on your site without exposing them to real traffic. This is critical for catching visual bugs or broken functionality. I always QA aggressively; a broken button is worse than no test at all.
2. Once you’re satisfied, head back to the “Overview” tab of your experiment.
3. In the top right corner, you’ll see a button that says “Start Experiment”. Click it.
4. Confirm the launch in the pop-up.

Your experiment is now live! Optimizely will begin allocating traffic to your original and variation(s) according to your settings.

Pro Tip: Notify your team when an experiment goes live, especially if it affects a high-visibility page. Transparency prevents confusion and ensures everyone understands why certain users are seeing different versions of the site.

Common Mistake: Launching without proper QA. A small visual glitch can completely invalidate your results because users might interact with a broken element differently, regardless of your hypothesis.

Expected Outcome: Your experiment is live and collecting data from real users. You can see the experiment status change from “Draft” to “Running.”

3.2. Monitoring and Interpreting Results

This is where you become a data detective. In your Optimizely dashboard, click on your running experiment, then navigate to the “Results” tab. Optimizely’s results page is incredibly user-friendly.

You’ll see a breakdown for each goal you set, showing performance for your original and each variation. Key metrics to watch:

• Conversion Rate: The percentage of users who completed your goal.
• Improvement: The percentage lift or drop compared to the original.
• Statistical Significance: This is paramount. Optimizely displays a p-value or a confidence level. We’re looking for a p-value less than 0.05 (or a confidence level > 95%). This means there’s less than a 5% chance your observed results are due to random luck. If your p-value is higher, you don’t have enough data yet, or there’s no real difference.
• Visitors and Conversions: Raw numbers to ensure you have enough data volume.

Don’t stop an experiment just because one variation is “winning” after a day or two. You need to reach statistical significance and ensure the results are consistent over time. My rule of thumb: wait at least one full business cycle (e.g., a week if your business has weekly patterns, or two weeks if you see strong weekend/weekday differences) and until you have at least 1,000 conversions per variation for high-traffic pages, or as much data as you can reasonably get for lower-traffic ones, all while maintaining statistical significance.

Pro Tip: Segment your results by audience, device, or traffic source within the “Results” tab. Sometimes, an overall “loser” variation is actually a huge winner for a specific, high-value segment. This is gold for personalization strategies.

Common Mistake: Stopping an experiment too early. This is called “peeking” and it leads to false positives. Resist the urge to declare a winner before statistical significance is achieved and maintained.

Expected Outcome: You identify a statistically significant winning variation, providing clear direction for implementing changes to your website and improving your marketing efforts. Conversely, you might find no significant difference, which is also a valuable insight – it tells you that particular change wasn’t impactful, and you need to try something else.

Mastering growth hacking isn’t about finding one silver bullet; it’s about building a robust system of continuous experimentation, learning, and iteration. By systematically applying these techniques with tools like Optimizely, you’ll uncover what truly drives your audience and achieve sustainable marketing growth.