// THE LAB / MARKETING EXPERIMENT PREFLIGHT
An A/B Test Can Be Broken Before It Spends a Dollar
A campaign can produce more leads and still leave sales with less usable pipeline. I built this private marketing and revenue lab to inspect the audience, comparison, and measurement plan before committing traffic.
I want to ask these questions while changing the test is still cheap. Once the ad and page are live, the CRM is already collecting a stage change that means one thing to sales and another thing to finance. Then someone asks whether the result was significant, whether the audience was consistent, or whether the test ever had a fair control.
I want the hard questions to arrive before spend. The application stays private. This page explains what the lab holds, what it can do today, what it refuses to call a result, and where I think the system can become useful across MarTech, RevOps, and sales.
The fictional CTA test behind this field guide
The public walkthrough uses a fictional B2B landing-page test. The challenger changes the CTA from “Book a demo” to “Get the sample plan” and gives the sample plan a route into demo scheduling. A download never counts as a qualified demo request. The fictional packet defines completion of that event within seven days as the primary outcome, along with the denominator, measurement window, and stopping rule.
That is a completed synthetic rehearsal, not a campaign result or a predicted winner. It shows the kind of measurement definition I want settled before traffic moves.
What is marketing experiment preflight?
Marketing experiment preflight is the work of defining the audience, control, challenger, evidence, outcome, and stopping rule before traffic or media spend enters a test. It does not pick a winning ad. It gives a team a question that can be answered, then points out when the test has no chance of answering it.
A useful preflight asks simple questions that become expensive when nobody asks them:
- What changes between control and challenger?
- Who sees each version, and how do we know the groups are comparable?
- What exact outcome matters, over what period, and with what denominator?
- What evidence is observed, modeled, synthetic, stale, or missing?
- When does the test stop, and who decides what happens next?
That sounds basic because it is. The work gets difficult when the answer lives across an ad account, an analytics tool, a CRM, a finance system, a spreadsheet, and the memory of the person who set up the campaign. The lab is my attempt to keep those decisions from dissolving the moment the campaign starts.
Why I built a private marketing and revenue lab
Marketing has a memory problem. A team can spend real money testing a landing page, an offer, an audience, a sales sequence, or an onboarding flow. A few weeks later, the result is reduced to a sentence in Slack: “That did not work.” The original version is gone. The alternative is gone. The source data is stale. The metric changed. Nobody remembers whether the audience changed with the creative.
That is how the same question gets bought twice.
The first version of this lab is a place to preserve the decision itself. A record can hold the original asset, a proposed alternative, the audience context, the evidence attached to the decision, the intended outcome, the test constraints, and a human-recorded outcome when one exists. It can be reopened later with a timestamp cutoff so the team can inspect what was known before a result was available.
Hindsight makes every result look obvious. A decision record lets you see the actual evidence, uncertainty, and assumptions that were available at the time.
What is in the build today
The current private build is deliberately narrower than the eventual thesis. It has an imported aggregate context layer and working decision records. It does not have connected provider data, campaign outcomes, or an automatic learning engine.
| Current capability | What it does | What to inspect |
|---|---|---|
| Aggregate audience context | Organizes 35,892,370 company-source records into 117,341 aggregate groups using industry, country, and company-size context. | Coverage, grouping, and relevance to the proposed audience. The count describes company-source records. |
| Quality disclosure | Shows that 1,006 malformed source records were excluded and leaves unknown freshness visible. | Excluded records and source freshness. Freshness remains unknown. |
| Decision record | Holds a marketing asset, optional challenger, audience context, evidence notes, test constraints, and an outcome field. | Which asset changed, the comparison, and the intended outcome. |
| What-if planning | Runs deterministic capacity and break-even scenarios to expose assumptions before a test starts. | Conversion assumptions, cost, value, and capacity limits. |
| Point-in-time replay | Can reopen a saved decision as it looked before a later outcome existed. | The timestamp cutoff and the evidence available at that time. |
| Human outcome entry | Records an outcome value, unit, period, action taken, and optional notes. | The outcome value, unit, measurement period, action, and supporting notes. No observed campaign outcomes have been recorded. |
The audience layer gives the decision context. It does not make the decision for me. The globe in the private interface is a visual way to understand aggregate context. It is not a map of people and it does not predict reactions. The distinction sounds fussy until someone tries to turn a company-source record into a claim about demand.
How the build handles evidence
The lab distinguishes six evidence categories: observed, modeled, synthetic, assumed, benchmark, and unknown. An assumption about conversion rate needs a different label from a measured conversion rate.
| Evidence state | Meaning in the lab |
|---|---|
| Observed | A real outcome from an admitted source, with a period and definition attached. None exist in the lab yet. |
| Modeled | Deterministic what-if math or model advice. Useful for deciding what to test. It is never measured response. |
| Synthetic | A fictional fixture or rehearsal used to test whether the machinery behaves as intended. It does not prove lift. |
| Assumed | An input chosen for planning that has not been established by observation. It needs to stay visible when the scenario changes. |
| Benchmark | A reference value used for comparison. Its source and relevance need review before applying it to this experiment. |
| Unknown | A missing, unauthorized, stale, or unverified fact. Unknown stays visible. It does not quietly become zero. |
A model can propose a sharper question, identify a missing definition, or point out a control problem. The decision contract allows a winner claim only for observed causal evidence. The output-safety layer filters unqualified winner, lift, revenue-impact, and customer-response language from generated recommendations. That guard is not evidence of campaign performance. A person must still admit an observed outcome with a source and period. A private source should never turn into a public claim.
That boundary follows the same principle I use in my work on AI connector governance: a system may inspect evidence for a defined job, but the evidence does not get to expand the system's authority. Connecting a source and trusting a source are different decisions.
How I built the first version
I built the lab around a few simple pieces instead of starting with a giant predictive model.
- A context layer. The first private dataset is profiled into aggregate groups, with exclusions and freshness limits visible. It helps frame the test audience without pretending that data can see inside a buyer's head.
- A decision layer. The system takes a piece of marketing or revenue work such as an ad, page, email, offer, sales outreach, onboarding step, or campaign journey. It adds the original version, an optional challenger, the intended audience, evidence, a CTA, proof, an outcome definition, and the rules for a fair comparison.
- A deterministic planning layer. Capacity and break-even scenarios make their assumptions inspectable. If the math requires an implausible conversion rate, qualified-opportunity rate, or margin to pay back, I would rather learn that before launching a test.
- A replay and outcome layer. A saved decision can be reviewed as it was known at a given time. When a real outcome exists, a person has to admit it with a source and period. The system does not manufacture observed history.
- A separate AI boundary. Model output can be useful as advice or a list of unanswered questions. It is labelled as modeled output. It does not become proof because it sounds fluent.
I also built a synthetic truth test because statistical language can look convincing long before a method has earned trust. The test plants known conditions in fictional data, locks the hypothesis and measurement boundary before the hidden condition is revealed, then checks whether the method sees the direction it was designed to detect. The saved prediction record includes a SHA-256 hash, which lets me check whether its definition changed after the reveal.
The method sandbox includes CausalPy synthetic-control checks, Spotify's Confidence library as a reference implementation, and separate synthetic allocation work with Ax, Optuna, and Nevergrad. Those are engineering checks, not live campaign optimization. No visitor traffic is allocated. No customer outcome is reported.
What a preflight can catch before media spend
The lab is more useful when it catches a bad question than when it produces a polished recommendation. Here are the failure modes I want it to expose early:
- Control drift. The creative, audience, offer, landing page, and follow-up all change at once. A result cannot tell you which change mattered.
- An undefined outcome. “More leads” is not a metric. A test needs a counted event, denominator, window, and owner.
- A missing stopping rule. Somebody will want to stop when the chart looks good or bad. Decide the threshold and the time window before the first click arrives.
- An evidence gap hidden as confidence. The source is stale, the audience match is loose, or the CRM definition is disputed. The right answer may be to collect a fact before testing.
- A decision nobody can reconstruct. If a later team cannot see the version, the audience, the evidence, and the assumptions, it cannot use the result responsibly.
The learning loop I am building toward
My working hypothesis: the next campaign should start with a record of what changed, what happened, and whether those conditions still apply.
The future loop is straightforward: a team frames a decision, locks the comparison, admits the right evidence, executes the real work in its source systems, records the observed outcome, then makes that record available as context for a later, comparable decision. A similarity match is still context, not proof. A past win may have come from a different market, message, offer, period, or sales motion. The record has to show that difference.
The private lab has the decision-record and outcome-entry foundation. It does not yet have an active automatic corpus of past results. I will not claim compounding learning until a real outcome changes a later decision for a reason someone else can inspect.
A better system should never let one source overwrite another because the dashboard needs a clean answer. The ad platform should remain authoritative for delivery. Analytics should remain authoritative for behavior. The CRM should remain authoritative for lifecycle stage. Finance should remain authoritative for recognized revenue and margin. When those values disagree, the decision record needs to keep the conflict visible and name the owner who resolves it.
Where the lab can expand across MarTech, RevOps, and sales
The same record format applies far beyond a landing-page test. The object is always the same: a decision, its evidence, its constraints, its authority boundary, and the outcome that eventually earns a place in the record.
| Commercial team | A preflight question | What a stronger future record would connect |
|---|---|---|
| Marketing | Should this message, offer, audience, or page be tested, and what would a fair comparison require? | Campaign delivery, creative version, click identifiers, consent state, page and CTA version, and defined conversion events. |
| RevOps | Do the lifecycle definition, attribution window, and handoff rules make this result usable? | Analytics behavior, CRM stage history, routing rules, qualification definitions, and offline outcome reconciliation. |
| Sales | What should change in this outreach, discovery, offer, or onboarding motion, and what evidence would show it helped? | Approved account context, message version, meeting or opportunity stage, human review, and an owned outcome period. |
| Leadership | Which assumptions are carrying the commercial plan, and what would change the spend decision? | Scenario inputs, capacity limits, pipeline definitions, finance-owned revenue and margin, and the decision owner. |
There is a sharper future version of this same machinery for account-based work. A connector could frame a preflight around one named company and its public footprint, so the first conversation begins with a specific, inspectable commercial question instead of a generic pitch. The decision record does not change. Only the permitted context does, and that remains a connector decision with a defined source, scope, and owner.
I am intentionally keeping future activation out of the first connector plan. The lab can prepare a better question before it should be allowed to change a bid, update a CRM record, enroll a contact, or publish a claim. Read, review, and write are different jobs. That boundary is the subject of my AI governance work for commercial teams.
The connector roadmap starts with read-only evidence
Seven external evidence sources are represented in the private build as planned read-only contracts: PageSpeed Insights and Chrome UX Report, Google Search Console, Google Analytics 4, Bing Webmaster Tools, Google Ads Keyword Planner, Google Ads reporting, and Meta Marketing Ads Insights. None are connected. There are zero provider calls.
That is deliberate. A connector becomes useful only when the job, source fields, freshness expectation, owner, retention, and approval path are clear. I would rather show a planned source honestly than create the familiar dashboard theatre where every logo implies data is moving.
Search and paid-media sources are the first planned connections. CRM and finance evidence would require additional integrations. These are the questions I want that expansion to answer:
- Search and site quality: Which query, page, and experience signals could change how a landing-page hypothesis is framed?
- Paid media: Which delivery, keyword, creative, and cost signals belong beside a proposed audience and offer test?
- Revenue operations: Which lifecycle and qualification facts need a shared definition before marketing can claim a campaign influenced pipeline?
- Finance: Which revenue and margin facts should be attached before anyone says a commercial motion paid back?
Each connection will stay read-only until it proves its value. A source can inform a decision without having permission to act on it.
What I will share, and what will stay private
I am not opening the application as a public demo. The lab holds private work, internal source details, and experimental machinery that does not belong behind a public URL. I will share the useful parts: the data model, sanitized screenshots, short walkthroughs, failure cases, connector decisions, and the actual lessons that survive contact with real outcomes.
Every public result will say whether its evidence is observed, modeled, synthetic, or unknown. If I connect a source, I will say what it is allowed to do. If an assumption changes, I will update the decision record instead of rewriting history. The goal is a system that gets more useful because it can be inspected, challenged, and repaired.
That is also the common thread with Brokaw Bets. In football, a number only becomes useful when the player, matchup, source, and time context stay attached. Commercial data has the same problem. The hard part is rarely finding another number. It is keeping the decision context intact long enough for the number to matter.
Frequently asked questions
What is marketing experiment preflight?
Marketing experiment preflight is the work of defining the audience, control, challenger, evidence, outcome, and stopping rule before traffic or media spend enters a test. It does not predict a winner. It makes a test question specific enough to answer and exposes defects that would otherwise waste budget.
What does this private marketing and revenue lab do today?
The private lab organizes aggregate audience context, marketing material, a proposed alternative, evidence notes, test constraints, deterministic what-if scenarios, timestamp-cutoff replay, and a human-recorded outcome when one exists. Its seven external evidence sources are currently planned read-only contracts, not connected provider feeds.
Does the lab use live campaign data?
No admitted provider, campaign, visitor, conversion, or revenue outcomes are in the lab today. Synthetic and modeled work stays labelled as such. A real outcome must be explicitly recorded with its source and period before it can be described as observed.
How does the lab distinguish modeled, synthetic, and observed evidence?
The lab uses six categories. Observed evidence has a source and period. Modeled evidence is scenario math or model advice. Synthetic evidence uses fictional inputs to test mechanics. Assumed evidence is a planning input awaiting verification. Benchmark evidence supplies a reference comparison. Unknown marks a missing or unverified fact. A benchmark or assumption does not become an observed result.
How could this expand into MarTech, RevOps, and sales?
The same decision record can support campaign and landing-page preflight in marketing, lifecycle and attribution definitions in RevOps, and outreach or onboarding hypotheses in sales. Any future connector would remain read-only until its job, fields, authority, freshness, owner, and human approval path are defined.
Build status
Current private build: aggregate context, local audience and ad-history imports, decision records, deterministic what-if planning, break-even scenarios, timestamp-cutoff replay, and human outcome entry. Not connected: seven planned read-only sources. Not claimed: live campaign performance, customer response, revenue impact, automatic learning, or a public application.
About the author
Jeff Brokaw is a CMO and Certified Chief AI Officer who builds commercial systems around AI, data, measurement, and the decisions that make those systems useful. He has been building AI systems commercially since 2016.