Jeff Brokaw

24 Agency AI Case Studies. Two Showed the Before and After.

Jeff Brokaw · Chief AI and Marketing Officer · Published · Updated · Agency AI series, part 4 of 5

PUSH says the share of leads that became sales went from 1.2% to 14%. I'd take that meeting.

I'd ask whether the AI videos brought in better leads or the new checks kept bad leads out of the count.

In my October review of 24 selected agency AI case studies, 13 reported a measured client outcome. Only two printed a starting and ending number for the same metric. One of those baselines still needs an explanation.

If you're pitching AI work for 2027, put the starting number and how you'll measure it in the proposal. Let the buyer see what would justify spending more before they spend anything.

What should an agency AI case study include?

An agency AI case study should show what the agency changed, how the client measured it, the dates, the comparison and what else changed during the same period. Give the actual starting and ending values when the client can share them. If you claim AI caused the improvement, show how you estimated what would have happened without it.

What did the 24 agency AI case studies report?

Scroll the table sideways to see all columns.

What the public account contained Case groups
Selected cases where AI or machine learning was part of delivery 24
Reported at least one measured client outcome 13
Published a numeric starting and ending value for at least one metric 2

The 24 case groups came from 21 agencies. I grouped pages that described the same work so a case study and a second write-up didn't count twice. The 13 with measured outcomes came from 11 agencies. I checked what the publishers reported. One of the two cases with numeric endpoints had a baseline I couldn't reconcile from the published figures. The counts describe the selected accounts. Public research method and aggregate data.

What changed when PUSH switched to AI video?

Which Health Insurer had been getting over 300 leads a month from static Meta ads. Only 1.2% converted to a sale. Some people were asking about travel insurance. PUSH made testimonial-style ads with Arcads, had legal review the scripts and set up three Meta campaigns to test, rotate and scale the creative. The team reviewed performance three times a week, verified contact details, checked for duplicates and monitored the travel insurance inquiries. PUSH's health insurance case study.

PUSH published three comparisons. Each answers a different question.

Scroll the table sideways to see all columns.

What PUSH compared What it reported What I'd ask on the call
Campaign vs. the period before it Lead-to-sale conversion rose from 1.2% to 14%. Cost per acquisition fell 31%. Which dates and how many leads went into each rate? What counted as an acquisition?
AI video vs. other formats in creative tests Click-through rate was 1.41% for AI video and 0.94% for static images. How did you allocate impressions and budget across formats?
Leads vs. the forecast 6,070 leads from September 2024 through January 2026, against a projection of 3,600. What assumptions went into the forecast?

The 31% compares the whole campaign with its earlier baseline. The creative tests have their own comparators. The 3,600 was a projection. PUSH spells out the comparisons on the page. That's worth copying. PUSH's results and comparison notes.

Here's what I'd ask for before putting more money into the videos. Show me sales per 100 raw inquiries and sales per 100 verified leads, for both periods. Give each cohort the same amount of time to buy. If you changed what counted as a lead halfway through, I want to see how much of the improvement holds up when we use the same definition.

How do you measure what AI actually contributed?

AdVenture Media worked with AdSkate to pick YouTube channels for a consumer electronics advertiser. The agency compared a 45-day campaign with historical traffic and revenue and reported a 44.1% incremental revenue estimate from its analysis. I'd ask which other changes went into the regression and how it handled the client's other campaigns. That detail would help me decide how much weight to put on the estimate before moving budget. AdVenture Media's YouTube case study.

If the decision is whether to pay for AI creative, compare it with the creative you'd otherwise run. Keep the offer, audience rules and conversion definition steady. Use a randomized comparison where volume and platform access support it, and agree on when you'll read the results. A holdout from the whole campaign answers what the campaign added. A creative test answers what changing the creative added. Those are two different spending decisions. Google Ads Conversion Lift and The Test Before the A/B Test.

I use the same standard on my own work. On an electrical-distribution quoting product, we rebuilt the catalog and matching process. In our internal research, item-level match accuracy went from 80% to 98%+. A rep still had to review alternatives and check compatibility. If I were budgeting for a rollout, I'd want a second number. How much time does a rep spend getting a full quote ready to send? The case study reports the matching result. Electrical-distribution AI rebuild.

A new tool may only have a first measurement. WebSpero reported an average response time of less than a second for ChatNT, a tourism voice assistant. That's useful if you're buying a voice assistant. I'd ask when the timer starts, when it stops and what happens on the slowest calls. WebSpero's ChatNT case study.

How should you measure AI search results?

I reviewed 39 AI-search case groups separately. Thirty-two, across 18 agency brands, reported a measured outcome. Those cases concern visibility and other results from AI search work. They have their own denominator.

Siege Media reported that Hebbia's visibility for finance AI queries tracked in Peec AI rose from 9.94% in November 2025 to 42.17% in August 2026. Delante reported that Culligan Water's ChatGPT share of voice in Belgium went from 5.23% to 26.33% over six months. Its table also shows AI traffic rising from 116 to 232, alongside a reported 192% rise in conversions from the channel. Delante gives the traffic counts and the conversion percentage. I'd ask for the actual conversion counts and what qualified as a conversion. Siege Media's Hebbia case study and Delante's Culligan Water case study.

Start with the prompt list. Ask which questions were tracked in which engines and markets, how often they ran and whether the same list was used at both ends. Then pull observed visits, qualified inquiries and closed deals separately. If the case gives me more mentions, I know where to look next. I still need to know what happened when somebody clicked. Choosing AEO prompts, AI referral conversion by engine and Proving AI Pipeline to Your Board.

Put these seven lines in your next AI proposal

Scroll the table sideways to see all columns.

Line What I'd write down
Work What will change, including changes to the process, budget or targeting?
Metric What counts as a success? What goes in the denominator?
Comparison Which earlier period, concurrent test, holdout or forecast will we use?
Dates When do we start and stop counting? How long do leads have to become sales?
Source Which system supplies each count? Who pulls and checks it?
Responsibility Which parts does the agency own? Which belong to the client or a platform partner?
Limits Which decisions can the buyer make with this evidence? What still needs measuring?

This week, give the case study you lead with to someone who didn't work on the account. Ask them to fill in the seven lines using the page alone. Take the blank lines to the account lead. By Friday, see which ones you can fill from the source and which need a new measurement. For the next engagement, start with the same seven lines in the proposal and save the baseline in the marketing decision record before you change the work. For a separate test of who did the work and what they contributed, see what agency AI proof should show.

What I'd fund for 2027

I'd fund a pilot of one existing AI service with a recorded baseline, a credible comparison and a clear decision about what happens next. Tell me what the pilot must show before you ask for another dollar.

Frequently asked questions

What does the two-of-24 finding mean?

Two of 24 selected agency AI and machine-learning delivery case groups in the October 2026 review published numeric before-and-after values for at least one metric. Thirteen reported a measured client outcome. One of the two endpoint cases contained a baseline that couldn't be reconciled from the public material.

Does a before-and-after result show how much AI contributed?

A before-and-after result shows what changed over time. To estimate what AI contributed, compare with what would likely have happened without that intervention. Use a randomized comparison where practical. Record other changes and describe the assumptions behind any observational estimate.

How do you report AI results for a new system without a baseline?

State what you measured after launch, the period and how you counted it. A voice assistant's response time or a new system's first-month usage can be useful on its own. To report improvement later, repeat the measurement under comparable conditions.

Can an anonymized AI case study be useful?

Yes. Ask the client what can be published, then include the agency's role, the metric, comparison, dates and measurement source. If exact values are private, an approved range or indexed values can show the scale of change. Explain how you calculated them.

Research method and recheck

The public research summary covers 350 selected agency brand and domain records reviewed October 15 to 20, 2026. The delivery-case and AI-search counts above come from separate selected case sets. The outcomes are figures published by the agencies. The review checked published sources and flagged figures that did not reconcile. It did not independently verify the reported performance.

In October 2027, I'll recheck the same 24 case groups. I'll count a case when its public account states a numeric starting and ending value for at least one metric, using the same rule that produced the original count of two. I'll log unavailable accounts separately and record the dates and comparison details in their own fields. I expect more than two to meet the original rule. Any new cases will get a separate count.

Agency AI series, part 4 of 5. Earlier articles cover AI services marketing agencies advertise, client AI systems and support and what buyers get from an agency AI tool. Next in the series: what agencies should sell with AI in 2027.

← All articles

Tell me what's stuck.