agentcocoon

Guides

How do you test ad headlines without changing the image?

By Drew Amato · Published

Run every ad with the identical image and card, change only the headline text, and send each ad to its own tagged link so site visits and signups tie back to one headline. Give each angle the same daily budget, wait for about 200 clicks per angle before calling it, and judge on cost per landed visit, not click rate alone. A headline can win on clicks and still lose most of them before your page loads.

From October 28 to November 5, we ran a headline test on X for AgentCocoon: 11 angles, one image, 27 ads at launch. Every ad was the same post card with the same picture and the same card title. The only thing that changed was the text above it. It cost $482.49, served 219,340 impressions and drew 788 link clicks.

This guide covers how the test was set up, what it showed, and the mistakes worth avoiding.

How do you set up a headline-only test?

Freeze everything except the words. Meta's testing guidance says results are more conclusive when the versions are identical except for the one variable you are testing (Meta Business Help Center). Our setup:

  1. One image, no text on it. The card image carried the logo and a button, but no headline, so the image could not compete with the copy.
  2. Same card title on every ad. Ours just said the brand name.
  3. Group headlines into angles. Nine angles with three headlines each at launch, such as "done for you," "tried it yourself" and "built for our brands." We added two more angles during the test.
  4. Same daily budget per angle. Ours was $11 a day each, so spend never decided the winner.
  5. A unique link per ad. Each ad pointed to its own short address that forwarded to the home page with UTM tags naming the angle and headline, so every visit and signup carried the ad's code.
  6. Match the page headline to the angle. Visitors from each angle saw that angle's headline on the home page, and kept seeing it on return visits.

What did the results show?

Two angles beat everything else on click rate by more than four times. Final numbers by angle:

Angle Click rate Cost per click Site visits Clicks that landed Cost per visit
T11 Your outsourced AI hire 1.97% $0.23 81 38% $0.59
T10 Four agencies or one 1.22% $0.25 57 29% $0.87
T9 Built for our brands 0.32% $0.82 47 84% $0.97
T8 Ask like a teammate 0.26% $0.74 54 87% $0.85
T3 Unlimited agents 0.26% $1.08 26 67% $1.62
T2 Done for you 0.21% $1.04 31 76% $1.38
T1 AI agents for your store 0.12% $1.20 23 68% $1.77
All 11 angles 0.36% $0.61 409 52% $1.18

The other four angles finished between 0.15% and 0.20%. Both winners came from angles we added mid-test, framed around replacing a hire or a stack of agencies.

Why isn't the click rate winner automatically the best ad?

Because most of its clicks never reached the site. X counts a link click the moment someone taps, while site analytics only counts visitors who stay long enough for the page to load, and X says slow loads and low-intent taps both widen that gap (X Help Center). Only 29% of T10's clicks and 38% of T11's became visits. The plainer angles, T8 and T9, landed 84% to 87%.

Cost per landed visit settles it. T11 still won at $0.59, but T10, second on click rate, was no cheaper per visit than T8. Read the click rate first, then check it against your own visit count before you scale anything.

How long should you wait before picking a winner?

Until each angle has a couple of hundred clicks. After the first half day, the leader was "Built for our brands" at 0.32% on just 7 clicks. It finished third on click rate, far behind the two angles added later. The final leaders each passed 200 clicks, which is when the gap stopped moving. Meta recommends at least 7 days for its own A/B tests and warns that shorter tests can be inconclusive (Meta Business Help Center).

Plan for the platform to underspend, too. In the first days, X spent only $4 to $6.50 of each angle's $11 budget, so we extended the end date by three days instead of raising budgets.

What can spoil a headline test?

Changing the page while the test runs. We made about a dozen changes to the home page during those eight days, and the comparison angle got an extra block on the page that the others did not. That makes the visit and signup numbers harder to compare. Freeze the landing page, or at least change it for every angle at once.

Three more to watch:

  1. Clicks are not customers. The 409 visits produced one Join click, which opened a checkout, and no paid signups. A headline test tells you what earns attention, not what sells.
  2. Know which click metric you are reading. X's "Clicks" includes likes, profile clicks and other engagement, while "Link clicks" counts only the link or website button (X Help Center). Use link clicks.
  3. Redirects can strip the click ID. X warns that redirect URLs remove its twclid parameter, which its conversion tracking relies on (X Help Center). Our short links forward to the home page, so we track visits ourselves instead. If you rely on X's pixel, link straight to the page.

For how the ad code follows a visitor all the way into checkout, see how to know which ad a Stripe checkout came from.

Common questions

How many headlines should you test at once?

As many as your budget can get to a few hundred clicks each. We ran 11 angles at the same daily budget for about a week, and only the two leaders cleared 200 clicks. With a smaller budget, test three or four angles, not eleven.

Should the landing page headline match the ad?

We matched them, so each angle's visitors saw that angle's headline on the home page. It usually helps conversion, but it means you are testing the ad and the page together. Once you have a winner, test page headlines on their own.

Why did X report more clicks than my site recorded?

X logs a link click the moment someone taps, before the page loads. Your analytics only counts visitors who stay long enough for the page and its tracking to load. Slow pages and curiosity taps both widen the gap.

How long should a headline test run?

At least a week. Meta recommends a minimum of 7 days for its own A/B tests. Our day-one leader finished in the middle of the pack, so early reads are mostly noise.

Sources

  1. Meta Business Help Center: Best practices for A/B testing
  2. X Help Center: Common analytics discrepancies
  3. X Help Center: Export CSV definitions
  4. X Help Center: Create a website conversions campaign

About the author. Drew Amato is the founder of Mully, a golf apparel brand in Detroit, and of AgentCocoon. He builds and runs the AI agents behind Mully's own operations. Follow on X.