Jeff Brokaw

The Marketing Learning Loop: How to Record What AI Actually Learned

Jeff Brokaw · CMO and Certified Chief AI Officer ·

Ask your team why the budget moved last spring. Then watch everyone open Slack.

Someone finds the deck. Someone remembers a conversation differently. The person who approved it has left. You can recover the spend, but the reason for spending it takes an afternoon.

Add AI and you can produce more campaigns, variants and reports before anyone writes down what the last decision taught you. Six months later, the assistant has plenty to summarize and no reliable record of why the money moved.

An AI marketing learning loop should make that decision recoverable and useful the next time you face it. This is the final article in my AI Marketing Operating Model series. The test is whether a past result changes a later recommendation, with evidence you can inspect.

What belongs in a marketing decision record?

A marketing decision record contains the prior belief, what changed, what stayed fixed, the audience, success metric, stop rule and lesson carried forward. Record the first six before launch. Leave the lesson pending until the outcome is ready, then add what the evidence supports and where it applies.

Keep a named owner, decision date, record ID and planned review date alongside those seven fields. Link the approved creative, audience rules, data definitions and relevant AI configuration as they existed when the decision was made. A link to a dashboard that keeps changing cannot preserve last spring's context by itself.

There is a long-established precedent for keeping a short record of a decision and its consequences: Michael Nygard described architecture decision records in November 2011. The procedure below is my proposed adaptation for marketing, not a claim that this record format has been proven to increase revenue.

Seven fields, with enough detail to disagree later

  1. Prior belief. What do you think will happen, and why? Name the earlier evidence or write that this is an untested assumption. “AI recommended it” tells the next person nothing about the reason.
  2. What changed. Identify the actual change: the form field, offer, bid rule or message. Save the version. If several things changed together, record them together and limit the conclusion accordingly.
  3. What stayed fixed. State the control and the conditions you intend to hold steady. Record departures when they happen. A new price halfway through the test belongs in the readout.
  4. Audience. Define who was eligible, who was excluded and how people were assigned. New prospects, returning buyers and existing customers should not disappear into one convenient average.
  5. Success metric. Define the primary outcome, denominator, source and observation window. Include the size of improvement worth acting on and the measures that must not worsen beyond agreed limits. Submitted forms and paid customers are different outcomes.
  6. Stop rule. Set the planned duration or sample requirement, the analysis method and the conditions that pause or end the work. Include spend, customer harm and broken measurement. Name who can restart it.
  7. Lesson carried forward. After the readout, state the supported finding, its limits and what it changes next time. “Inconclusive” is a usable entry. “Short forms always win” is usually a claim much bigger than the test.

The test before the A/B test covers whether an idea deserves traffic in the first place. My Experiment Preflight can help you draft that plan. Save it before launch; completing a planning tool does not record the eventual outcome for you.

Sort out the different definitions across analytics, advertising, sales and Finance before filling in the success metric. Otherwise you will spend the readout arguing about what a conversion meant.

The procedure: Freeze, Attach, Compare

Use one record for one consequential decision. A document, a database row or an experiment platform can hold it. I would start wherever the team will maintain it, with version history and access appropriate to the information it contains.

Freeze the plan before launch

Save a dated version with the first six fields and the approver. The named owner and software authority should be settled already. Record what the owner authorized, including any limit on automatic action.

If the plan changes, append the change, reason, date and approver. Keep the original readable. Rewriting the original hypothesis after seeing the result makes it impossible to tell a prediction from an explanation.

GrowthBook's August 7, 2026 experiment-lifecycle guide puts hypothesis, metrics, guardrails and runtime ahead of launch, and data-quality checks ahead of interpreting lift. It also preserves stopped experiments and their decisions. Those are useful requirements for the record, whichever tool holds it.

Attach the outcome when it is ready

Add the result at the planned readout. Include the dates, eligible population, sample sizes and uncertainty, the primary metric, the downside checks and any measurement failures. Link the supporting data or retain a dated export under the right access controls. Keep sensitive customer information out of a broadly shared experiment library.

Say whether the evidence supported the change, rejected it, remained inconclusive or could not be interpreted. Retain abandoned and stopped tests too. A library containing only winners will teach the next assistant to repeat your selection bias.

A pause for broken tracking or excessive spend is a stop, not a winning result. For a fixed-horizon test, repeatedly checking for significance and stopping when it appears can invalidate the inference. Use the chosen analysis plan. Statsig's sequential-testing documentation explains the adjustment needed for repeated looks; an early result on the primary metric also does not guarantee adequate evidence on every downside measure.

Give delayed outcomes their agreed time. A form can arrive today and become a qualified opportunity later. Keep that window in the plan. If refunds or later revenue change the readout, append a dated update and explain the revision.

Compare only cases that can answer the next question

Before reusing a lesson, check the audience, offer, channel, metric definitions and timing. Check the business conditions too. A campaign that worked with an empty sales queue may be a bad recommendation when the team is full. That is the capacity check from the previous article.

For AI-search work, keep the actual prompts, capture dates and observed answers. Re-running a different set of questions does not show whether the earlier work improved coverage. Choosing searches worth pursuing comes before comparing them.

A decision record does not turn a before-and-after report into a controlled experiment. Preserve the study design. A valid randomized test can support a causal conclusion within its scope. An observational result needs its alternative explanations kept visible.

What the next recommendation should say

Take a hypothetical form test. Returning visitors are randomly assigned to two versions. One requires a phone number; the other makes it optional. The offer and follow-up process stay fixed. The team defines a qualified opportunity as the primary outcome and records the qualification window before launch.

The optional-phone version produces more completed forms. At readout, the qualified-opportunity result is inconclusive.

The record should say that. It should not turn “more forms” into “more pipeline” because somebody wants a winner for the slide.

When AI recommends the next budget increase, I want it to retrieve this record and explain the limit: the team observed more forms, but did not establish an improvement in qualified opportunities. The recommendation might be another adequately designed test. It cannot cite this one as proof that increasing spend will produce more qualified opportunities.

The owner may still choose to keep the easier form for another reason. Record that reason separately from the experiment's finding. A business decision can be reasonable without pretending the test proved it.

What AI is allowed to say it learned

Saving a result does not automatically train the underlying model. Retrieving a past record gives the workflow evidence to use in its next answer. Changing an operating rule changes what the workflow does. Updating model weights is a separate technical process. Name which one happened.

I would require an AI recommendation to identify the records it used, explain why they apply and show what changed in the proposed action. Keep the referenced versions with the recommendation. If a person corrects it, save the correction with its reason and scope.

You can check whether the workflow uses a lesson by giving it a comparable case and inspecting its recommendation. Include cases where the old lesson should be rejected. A workflow that repeats the same advice for every audience has failed that check, even if it quotes the record correctly.

Agencies can do this well. The team that remembers why a client rejected an offer, which audience responded and what failed qualification can make a better-informed recommendation at the next planning meeting. My agency operating model article calls that retained decision history a client-owned learning asset. The value comes from using it, including when the evidence gives you a reason to change your own advice.

Re-check the method on November 12, 2026

WFA's January 27, 2026 benchmark summary reported that three in four marketing procurement teams lacked KPIs for AI/GenAI impact. Its public summary gives no sample size, so I would not generalize that figure to every marketing department. It does give procurement teams a reason to ask what evidence will support the next AI investment.

This procedure needs its own test. If I were introducing it to a team today, I would review the next five consecutive AI-assisted decisions that change spend, an offer or a customer-facing page on November 12, 2026. Select them as they occur, including work that loses, stops or never launches. Do not choose five attractive examples afterward.

For each decision, ask a teammate who was not in the approval meeting to reconstruct what was authorized from the record. Can they identify the first six fields, the owner and the approved version without chasing a Slack thread? Record each missing answer.

Then check whether an outcome is due, whether the evidence is attached and whether the lesson stays within that evidence. Finally, inspect the next comparable recommendation. It should cite the earlier record and explain what it reused or why it rejected the old lesson.

My pass rule for this small trial would be all five decisions reconstructable, every due readout attached, and every later comparable recommendation explaining what it reused or rejected. Count each failure. If there are fewer than five decisions, outcomes still inside their observation window or no subsequent comparable recommendation, report the unfinished part as pending. Keep the original outcome windows, even when they run past November 12.

That checks whether the procedure preserves decisions and affects later recommendations. It does not establish better campaign performance. Testing that requires a separate evaluation with a credible comparison.

Start with the next budget change. Write down why you expect it to work, keep the approved version and put a date on the readout. When the next recommendation arrives, make the person or software proposing it account for what happened last time.

Frequently asked questions

What belongs in a marketing experiment decision record?

Record the prior belief, what changed, what stayed fixed, audience, success metric, stop rule and lesson carried forward. Complete the first six before launch and leave the lesson pending until the readout. Keep the owner, dates, approved versions and supporting evidence alongside the record.

How do you know what an AI marketing system actually learned?

Inspect the records supplied to the workflow, any approved rule changes and its next recommendation on a comparable case. Require it to explain how the past outcome changed that recommendation or why the lesson does not apply. Saving or retrieving a record does not by itself update model weights or prove better performance.

Why does a decision record matter after a test ends?

It preserves the reason for the decision, conditions, result and limits so another person can assess whether the finding applies later. Retain losses, stopped work and inconclusive results as well as winners. Append corrections without erasing the original plan or readout.

How often should a marketing team re-check past learnings?

Check applicability before reusing a lesson. Set a dated review when recording it, based on the outcome window and how quickly the audience, offer or business conditions change. This article proposes November 12, 2026 for an initial five-decision process review; that date is a trial rule, not a universal review frequency.

Sources