Does split-testing actually maximize business revenues, or is it a waste of time?
I was asked by a reader (Regev) to make a comparison post on different split-testing tools. He was concerned about their pros and cons, and whether they performed best for your sites and businesses. And he’s not the only one asking.
There are so many split-testing tools today, allowing you to test every aspect of your website:
- text & copywriting (headlines, descriptions…)
- design elements (background colors, button colors, images…)
- timing (day of week, frequency…)
So what is best? What features are most important? How can you optimize split-testing? Will split-testing really help you get those extra clicks, extra sign-ups, or extra purchases?
You can already guess my challenge by the title. I’m not here to tell you which is my favorite, I’m here to tell you if I think split-testing is even worth your time.
Is split-testing worth the effort?
It’s a lot of time and energy spent on things that don’t maximize revenue.
What I reeeeeally wanted to say was…”split-testing is a waste of time!”…but decided on something less abrasive. It’s not that I don’t think it’s useful. It’s that I think your time is best spent elsewhere. Among all possible tasks to be done for your business…aside from product delivery, service execution, customer management, marketing, administration, HR…split-testing is like the “cleaning the bottom of your shoes” of marketing tasks.
How the idea of split-testing even gets brought up is a testament to regular marketing itself (not split-testing). You were a business owner scoping out ideas on how to grow your business, and perhaps stumbled on a site or overheard some growth-hacking nerd talking about the grand potentials of split-testing.
They probably piqued your curiosity by playing up fantasies of better maximizing the brand attention you have…or tugged at your insecurities by questioning how much potential sales you lost.
The idea of SPLIT-TESTING says…
- “Hey, you’ve got a fantastic product…but people didn’t buy it because you didn’t use the right word, the right button color, or tell them at the right time of day.”
…and I (mostly) disagree with that narrative.
1. Split-testing is not practical
Large sample-size requirement
This right here is the first disqualifier for most people. You need a decent amount of traffic to see reliable differences between your control groups. If your new website gets 1000 hits/month, of which only 46 of those click through, and only 3 make a purchase…how much improvement can you really expect with split-testing?
How many tests would you need for confidence in the findings? How long can you even run the tests for?
High cost offsets any gains
Some of the split-testing tools out there are too expensive. It’s just not realistic for a typical small business. You have to pay for the tool, learn how to use the tool, then spend time testing random things on your site…and then hope for some ROI that’s better than all the effort you spent on split-testing. (This is assuming you split-tested the right things and was able to find actionable improvements.) Split-testing tools cost $30-100/month for very low sample sizes. I think that’s such a ripoff.
But there’s also the cost of testing. The time you spend trying random things and potentially hurting your business. Instead of acting on more tried and proven tactics. Split-testing only compares the output of different hypotheses, it doesn’t actually tell where your marketing issues are.
Do you even know what to split-test?
I’d say the average business owner hasn’t a goddamn clue. Go ahead and imagine it. Let’s say you bought the tool and learned how to use it. Ok…sit down and tell me, what keywords are you going to test? Do you even have copywriting experience to know which keywords are best? Do you know how to research which keywords to try?
Same goes with design elements, sizing, colors, etc. Do you have enough design knowledge to know what to try? It’s like shopping for clothes…you don’t have time to try on everything. You walk in and can tell at a glance which things are more likely to fit you or not.
Too many unknown factors
Split-testing (unsuccessfully) attempts to make a controlled environment in an unpredictable real world. You can fantasize all you want…oh, let’s change some text and test out colors and images, and see which gets the most engagement! But it doesn’t work like that.
Your audience’s behaviors are often influenced by external factors you weren’t aware of. Your competitors’ change in pricing can change how your pricing is perceived. Your competitors’ design styling or marketing positioning can change how your brand promotion is perceived. Which could mean what scores higher in one month, could be the worse option a few months later. So why even bother to control anything once you realize how little control you have over it all?
I prefer an approach more towards awareness rather than control…
2. You don’t need to do split-testing
You can learn and apply it naturally already.
Let’s say you’re dropping newsletters or YouTube videos every week. Why kill yourself split-testing each one…essentially creating multiple variants for each release? Instead try something a little different each week. Sure sure…there’s no control and variable group to compare but you can get a general idea over time of what works and what does better.
Keep a regular pulse on your business. If you’re working at your business everyday, you’ll instinctively already have an idea of what works or not. You should also pay attention to things outside of your business as well. Industry trends, things other competitors are doing. All that together gives you the information for you to be your own AI (aka “intuition”).
Maximizing user actions at what cost?
I’d even go as far as to say you shouldn’t act on every split-testing improvement. Examples below:
- Ok sure…the ugly orange button gets slightly more clicks than the smooth pink button that better-complements your brand. Are you really gonna wreck your site design harmony for 8% gains?
- Even better…those annoying clickbait titles. Sure, you got the click but did you really provide a better user experience? I think it works for a little while and then people catch on and develop distrust for your brand.
So much of split-testing only cares about the now. It doesn’t care about all that underlying brand trust and user experience. It’s the crying toddler of attention. While real brands are built in much more calculated ways, giving a more comprehensive user experience from beginning to end.
Building a real brand requires a seamless longterm user experience, rather than different trendy brand jolts every week. Constant change is uncomfortable, jarring. Consistency builds trust.
3. Your biggest growth opportunities are elsewhere
Your time is better spent on other things.
I’m a strong believer in putting more traffic into your funnel, rather than micro-tweaking that funnel. And give me a second…I’m not saying increasing initial traffic is more valuable than conversion rate.
It’s that I believe the things you do to increase traffic will also likely be the things that increase conversions. Being in touch with how you get traffic, will help you be in touch with how you convert traffic. And understanding why it did or didn’t convert.
Some people argue you’re missing out on tons of potential conversions because of small easy tweaks. I’d say that’s just analytics nerds preying on your insecurities. The reality is…the conversion point is only one point along an entire chain of user engagement. I think if your branding is consistent from beginning to end, the conversion should be smooth and predictable. Fallouts are more likely due to more obvious reasons, rather than simply visual changes.
So how about focusing on building a better product and better service? Let existing clients do the selling for you. Tweak the actual marketing, branding and copy, rather than micro-tweaking subtle changes that don’t substantially change the overall branding. Worry about actually being better, rather than being perceived as better.
Focus on out-of-the-box thinking
I’m a strong believer in bringing in real changes. That is…if you’re serious about wanting to test changes. Instead of playing with different advertisement copy, how about sponsoring a live event? Instead of seeing which colors fit better, how about creating a second brand with different positioning?
There’s so much opportunity in trying something totally different, and could actually bring new value. Rather than the massaging the same stuff you already have. Try something totally weird and random, that’s always been far more impactful for my businesses. Try something new, rather than new versions of the old.
4. If I was to do split-testing
Yes, split-testing still has benefits and is worth trying.
Many people will still on insist on trying every opportunity to grow their business, leaving no stone unturned. And I applaud that. If you got time and money, already did all other conventional marketing methods, and feel tech-savvy enough to play with something new…ok sure. Why not?
Whether split-testing really gives you an ROI is debatable, but for sure you can learn a lot about what works and doesn’t work. This is how I would recommend you go about it…
Minimum sample size for split-testing.
You need a minimum amount of traffic to reliably compare numbers. Minimum sample size is 100 for me, 1000 is better, 10k and beyond is great. Ideally, your results vary enough where you get something like 53 conversions for version A, and 8 conversions for version B. You want a clear difference between winner and loser.
How often you repeat the test to verify it is your call. I’ve found many businesses have different customer segments, which respond differently to variations. Yes, that makes things complicated. Imagine the work spent to discover that customer type A likes variant A, and customer type B likes variant B. Hey, it’s great that you figured it out…now you have to keep it in mind for future decisions.
But hahahaha, this is your problem to deal with now—not mine!
Different methods of split-testing.
- 2-option split-testing is A vs B (e.g. dogs vs cats)
- 4-option split-testing is A vs B vs C vs D (e.g. dogs vs cats vs horses vs birds)
Definitely, you should test more options if you can. Especially if you have enough traffic to get useful data. But there’s another way to use split-testing.
- I like to do A1 vs A2 vs B (e.g. big dogs vs small dogs vs cats)
- Other example would be (dogs vs cats vs houses)
You can think of this as like a split-test for your split…to see if you’re even testing the right things. Very often, my options were too narrow without realizing they ddidn’t make much difference…and that something else totally unrelated would perform much better.
The problems with multivariate testing.
- Multivariate is having multiple variables and multiple options simultaneously.
- Imagine have 2 colors (red & blue) and 3 animals (dogs, cats, birds)
- This would provide 6 possible choices: red dog, red cat, red bird, blue dog, blue cat, blue bird.
Honestly, I wouldn’t even recommend it. Not that it can’t be useful but it requires a lot of traffic to get clear meaningful results. I also get the feeling if you’re at that point, you’re just being random and don’t even know what you should be testing.
Start with simple tests
Don’t get swayed by fancy test tools offering every feature. Stick to simpler cheaper ones. Test different home page designs, then conversion pages. Try different headlines, section colors, button placements & design.
Testing the most obvious visual things should be more than enough for you to get a sense. Remember…the average person is probably clicking onto your page and scrolling a few seconds before leaving. Many don’t stay past 10-15 seconds.
Simple tests will give you plenty ideas to move forward with. You really don’t have to test every possible thing. Take an instinctual guess and move on with your life!
Option inspiration
Many people don’t even know what to test. They aren’t trained copywriters to know all the buzzwords out there, and they’re also not trained designers to know how best to test visual elements.
A good way to find test inspirations is to pay attention to your industry. What’s everybody else doing? What keywords or design elements are they using? When do they send their promotional emails? Even better is to look at what other industries are doing. Why copy barbershop marketing tactics, when men’s luxury clothing brands are so much more refined?
But of course, copying the industry guarantees you’ll always be a step behind it. Perhaps you could use something like Google Trends site to lookup what recent words are being searched. You could also go reddit or Facebook groups to see what new things are happening in the industry. You could also join designer groups and portfolio sharing sites to see future design trends.
Best of all…you would kinda instinctually know these things if you’ve been on top of your business. Working day in and day out with it, that you wouldn’t even have to look it up.
Be careful who you hire
Many people think “split-testing takes so much time to learn, how about I let my marketing person handle that”. It’s not so simple. This person doesn’t
First consideration is this person’s personality and expertise. Are they simply a numbers-oriented analyst or do they care about personal non-numbers “human” details as well?
The last thing you want is some data nerd with no knowledge of your industry, demanding you test wild crazy things that hurt your brand. I’ve had this happen before. He insisted I test every keyword he suggested no matter how cringe or abrasive, or he couldn’t promise results. His suggestions were so off-putting, I went to an even more expensive well-known expert who felt my existing copy was already good enough as is. That even he himself couldn’t do too much better.
Anyway, I’m just glad I didn’t waste the time testing crazy shit, that would only make my brand look cheesy and desperate. I felt vindicated to follow my instinct and I hope you do, too.
Branding has so much value, even on people who DON’T buy your product. You should protect it no matter what. I would even put brand value above sales.
Actually, you know what? Why don’t you focus on your branding?
At the end of the day, it’s all about the feeling. A simple word change or different design tweak isn’t going to make anywhere near the impact as a clean clear branding improvement.
I think it’s crazy how many people waste time on split-testing when their branding sucks in the first place. It’s like deciding whether to put swiss cheese or cheddar cheese on a shitty sandwich.
Go make your website better. Improve the design. Improve the message. You’ll KNOW when it’s right because your website will literally feel sooooo good to you. That it makes you wanna spend more time on it, clicking around and exploring. It makes you want to buy BEFORE you’ve even read any text.
….oh, and please share your experiences of split-testing and split-testing tools you tried (because I didn’t cover that in this post 😂😂😂)
Hey Johnny –
Longtime fan of your WordPress speed optimization posts. I usually enjoy your contrarian takes, but man, this one felt off. When so many areas of marketing aren’t accountable to results, attacking the one discipline that is explicitly measurable as low-ROI and overhyped felt…misguided.
This line made me laugh out loud and probably explains our disagreement on the value of split testing: “You’ll KNOW when it’s right because your website will literally feel sooooo good to you.”
I get it. You’re a “common sense”, you’ll know it when you see it kind of guy. Unfortunately, those blind spots, biases, and sample-size-of-one attitude are exactly why A/B testing is necessary.
That mindset leads to a false choice: that you can either invest in A/B testing or focus on activities like product, branding, or traffic. The best marketers know you do both. Proper CRO and split testing are high-leverage activities because they validate and amplify the impact of your other efforts.
I agree that many businesses lack the traffic, context, or experience to run meaningful tests. And yes, there are plenty of amateur growth hackers who chase cheap wins at the expense of long-term brand equity. But there are also real CRO specialists who know how to balance brand, UX, and revenue, and can pay for themselves many times over.
This post misrepresents what A/B testing is, when it’s valuable, and how to use it strategically. So here’s a bit of perspective:
I’ve been doing CRO and A/B testing since 2009 (both agency and brand side). I’ve run thousands of tests (winners, losers, and inconclusive) and compiled many multi-year case studies with statistically significant results. Not a “growth hacker guru”, rather a practitioner.
To respond to a few key points:
1. A/B testing is not about button colors
It’s about testing solutions to real conversion problems, not random changes. Use analytics tools to find drop-off points, session recording tools to observe user friction, and A/B testing to validate which hypotheses actually move the needle. The goal isn’t to tweak aesthetics, it’s to fix what’s broken.
2. You don’t need “massive” traffic. You need meaningful differences.
Statistical significance depends on the magnitude of the impact. Big changes in behavior (like responses to new pricing models, lead gen flows, or core messaging) often require far less traffic to validate than minor aesthetic tweaks. If you’re working with limited traffic, don’t waste it on marginal changes. Test bold, high-leverage hypotheses that can produce measurable lifts.
3. “Going with your gut” introduces more risk, not less
Intuition is useful for generating ideas but without testing, you don’t actually know if your changes are helping or hurting. Worse, you might attribute success to the wrong thing, mistaking seasonality or external factors for the impact of your changes. That’s how you end up with false confidence and flawed decisions.
4. Branding and CRO/split testing are not mutually exclusive
In fact, the best A/B tests validate which branding, messaging, and UX choices build trust and resonance with your actual audience, not your internal team. Not everything needs testing. This is where your “common sense” approach should actually apply. Test things that matter.
5. Tools won’t save you from bad methodology
You’re absolutely right that many split testing tools are overpriced or overkill. But that’s a tooling problem, not a testing problem. The value comes from what you test, what you learn, and how you apply it.
Split-testing isn’t for every business at every stage.
But dismissing it as “cleaning the bottom of your shoes” might be the most credibility-damaging thing I’ve seen you write. And I’ve seen you post some spicy takes over the years, lol.
Hey Nathan,
I so appreciate your long detailed comment. And to share your split-testing experiences with all the readers on here. I actually agree very much with you. Every single point of yours! Indeed split-testing is more than just growth-hacking, it’s actually a way of objectively proving changes.
I still don’t think it’s realistic, feasible, or should be prioritized for the everyday business. At least not in the most detailed way, and for the 1-man show type of readers I have. For a big enough company with big enough traffic, and time and budget to do it correctly as you mentioned…I think it’s great.
Going with your gut is a real thing for me. Everyone should develop their business sensitivity. You can base it on whatever information you want, but you need to have a sense. While that sense can’t (and shouldn’t) answer all your questions, it can definitely guide you in the right direction.
Thank you for coming by, and if you do have a nicely-written post about split-testing, please let me know so I can link it here. As much as I champion my own opinions, I love sharing opposing takes as well.
Johnny, my man. I love your work beyond words. You’ve helped me tremendously along the years, especially in selecting hiqh-quality plugins and themes and hosting solutions, and have helped me create Usain Bolt’s types of sites. You’ve driven my conversions up. I know this because I TESTED 🙂
Now, I’m an obsessed student of Direct Response (copywriting, marketing, advertising), learned all my chops from the greatest of the past: Joseph Sugarman, Eugene Schwartz, Jay Abraham, etc. (All three Jewish by the way – like me xD). I’ve moved millions of dollars of merchandise through the printed word, and through testing. I know the value of high-quality copywriting – I’ve multiplied the income of existing businesses, by simply reworking the text. I’ve also multiplied the income of existing sales pages from cold traffic (on my own businsses). For instance, I once tested a page that was bringing lots of traffic against itself, one with the header and bottom navigation bars, and one without. The latter resulted in x2.5 the income, measured and judged statistically significant. I would argue that the latter also results in higher branding value, because the more people buy, the more people talk on forums, and recommend to each other, and buy again in the future. I see branding as a positive side effect of high-converting copy, not the opposite.
I see what you mean for smaller websites bringing smaller traffic. By all means, if anybody reading this is an owner of a smaller operation, focus on creating a high-quality site and an awesome offering, and when you get a sizable amount of traffic, start testing (big ideas first, for bigger changes right away, and smaller tweaks later when you’re experienced).
However, this is not my case 🙂 As one example, I own a tour operation in the US. We built a bar to support it, have employees and tour guides and what not. You said if I would keep an ugly orange button for 8% extra conversion. 8%?????? Of course I would. That’s a lot. Specifically for this tour company, it could buy you whatever new car you wanted every year, or be thrown into SPY or SMH to work for you and increase 10-20% a year. But often, by testing change after change after change, you get these small incremental improvements that compound over time. Like an athlete, you make your copy better, stronger, faster.
You live a good life, and then you die. But before that: I hope to find a good A/B testing tool for WP, cause I’m tired of using Excel spreadsheets and doing acrobatics to serve two variants.
Hahaha, yes I’m with you. 8% is a lot for those producing thousands of conversions. For the tiny business with hardly any traction, I’d say focus on getting more exposure. I totally get the idea of scraping out tiny gains that add up collectively. I’d just prefer to do my scraping in other areas. Seems to me, split-testing worked for you because you had the traffic to justify it.
I took a look at split-testing tools out there and mostly, I preferred the simpler ones that weren’t charging by traffic volume. Multi-variate is massive overkill for what I’m doing. Honestly, I think split-testers should be paid like celebrity copywriters….that is a small base fee and then percentage of sales increase. Win-win for everyone.
Which simple tools did you stumble upon?
The ones that I found that I was most attracted to were Stellar (“https://www.gostellar.app/”) and A/B Split Test (“https://absplittest.com/”). The former you install a script on your site, and manage the A/B testing offsite on their dashboard. The latter is an actual plugin you install.
If you have any opinion on one of them or any further info or recommendations, by all means fire it our way. Thank you so much for taking the time to create this post for me. I deeply appreciate it. Big fan of your work, through and through.