// MARKETING EXPERIMENT PREFLIGHT / TECHNICAL FIELD GUIDE
How the Synthetic Truth Test Works
A method should not influence a marketing experiment until it can handle an answer I already know. The private lab’s synthetic truth test gives it a fictional answer, hides that answer, fixes the prediction record, then reveals the truth.
The test rehearses the machinery. It does not backtest a campaign, model customer response, or prove that a real treatment will work.
What is a synthetic truth test?
A synthetic truth test is a controlled fictional dataset with a known condition hidden from the method being tested. The method makes its prediction first. The planted truth is revealed second. The exercise shows whether the method can recognize the kind of condition it claims to handle, and whether it abstains when it should.
A marketing dashboard can hide this problem for years. It can present a clean chart, a confidence interval, or a recommendation without anyone knowing whether the method would notice a null result, respect a holdout boundary, or keep unknown data separate from zero.
The first public field guide explains the commercial use case: how to preflight a marketing experiment before spending media budget. This companion explains the test bench underneath it.
Why I built it before connecting real data
Real campaign data creates pressure to tell a story. The budget was real. The team wants an answer. The executive meeting is on the calendar. That is the worst moment to learn that the method had no defined denominator, treated missing outcomes as zero, or quietly changed its assumption after seeing the data.
Fictional data removes that pressure. It lets me plant a condition, introduce known complications, and see whether the system reports a limitation instead of a conclusion it did not earn.
The Lab uses the test before a method becomes part of a live decision path. This does not certify a method for production. It gives me a concrete failure mode to repair before I ask a marketer or revenue leader to trust it.
The prediction is locked before truth is revealed
The core protection is preregistration. Before the fictional holdout is revealed, the Lab records the hypothesis, metric, denominator, threshold, declared segment, and holdout boundary. It then produces a SHA-256 hash of that record.
The saved hash provides a reference for checking whether the question, metric, threshold, or audience definition changed after the answer was revealed. It does not establish that the experiment is correct, and the original record must be retained for that comparison.
That is a useful discipline for ordinary commercial work too. A marketing team does not need to use a cryptographic hash for every landing-page test. It does need to define the comparison before the graph starts moving.
The three arms in the current rehearsal
The current known-truth comparison uses three intentionally different approaches. Their purpose is not to crown a winning method. Each gives the Lab a different way to fail visibly.
| Arm | What it does | What it is there to catch |
|---|---|---|
| Null | Fixes the effect at no change, with a true effect of 0.00 percentage points. | An invented difference. The null is a safety reference, not an automatic winner. |
| Naive Wilson | Uses separate Wilson intervals on the declared segment in a 140-day training window, then differences them. | How a simple interval-based approach behaves when its assumptions are exposed. |
| Simulation | Uses the existing synthetic simulation and its sensitivity envelope without allowing holdout truth into the prediction. | Whether the simulation stays disciplined when a plausible-looking range could tempt it to overstate certainty. |
The output is deliberately blunt. A result can say that the arms separated inside a fictional model. It cannot claim observed lift, revenue impact, a causal effect, ROI, customer preference, or a winner.
What the test measures
After truth is revealed, the rehearsal compares each arm's prediction with the planted holdout condition. It records the point prediction, interval, absolute error, and whether the interval covered the hidden truth.
Those are method checks. They are not marketing KPIs.
The integrity treatment is also explicit: chronological split, duplicate removal, missing-value exclusion, and unit normalization. Unknown stays unknown. The test does not turn an unavailable outcome into a zero so the math can continue.
What a pass means, and what it does not mean
A pass means the method handled the fictional condition it was designed to inspect, such as respecting the locked boundary, avoiding an invented difference under the null, or producing a range that covered planted truth.
That does not establish external validity. One fictional fixture cannot tell me how a real audience will react to a message, whether a media platform delivered balanced exposure, whether a CRM stage is reliable, or whether the finance system will recognize revenue the same way marketing counts a conversion.
A passed check means only that a method handled one known fictional condition before entering a commercial workflow.
The method sandbox stays separate from the result
The private Lab also keeps a separate method sandbox for CausalPy synthetic-control checks, Spotify's Confidence library as a reference implementation, and synthetic allocation work with Ax, Optuna, and Nevergrad. These are engineering tests run against synthetic inputs. No visitor traffic is allocated and no customer outcome is reported.
A method library, a simulation, and a live campaign answer different questions. Putting them in one screen may look sophisticated, but it makes it easier for a rehearsal to borrow the authority of a result.
How this becomes useful in MarTech and RevOps
The truth test is the safety check underneath the decision record.
For marketing, that record can make a team define the control, challenger, audience, primary outcome, denominator, measurement window, and stopping rule before media spend starts.
For RevOps, it can keep the source, period, freshness, and owner attached to each fact, rather than letting an attribution claim travel farther than the data behind it.
For sales, it can distinguish a better outreach hypothesis from an observed improvement in a real commercial motion.
Future read-only connectors can supply context from analytics, search, advertising, CRM, and finance systems. They cannot make the rehearsal more true by themselves. Each source still needs a job, a scope, a freshness rule, an owner, and a boundary between reading information and changing it. That is why connector governance belongs next to experiment design.
Frequently asked questions
What is a synthetic truth test?
A synthetic truth test is a controlled fictional dataset with a known, hidden condition. A method makes its prediction before the condition is revealed, then the prediction is compared with the planted truth. It tests the mechanics of a method, not customer behavior or campaign lift.
Why use preregistration in a synthetic experiment rehearsal?
Preregistration fixes the hypothesis, metric, denominator, threshold, and holdout boundary before hidden truth is revealed. The private lab records a SHA-256 hash of that record so a retained original can be compared with later definitions after the result is visible.
Does a synthetic truth test prove a marketing campaign will work?
No. It can show whether a method notices a planted condition in fictional data. It does not establish customer preference, observed lift, revenue impact, ROI, causal effect, or a winning campaign.
What is the null arm in the Lab's truth test?
The null arm is fixed at no change, with a true effect of zero. It is a safety reference that checks whether the system invents a difference where none was planted. It is not an automatic winner.
Current boundary
Private and synthetic: the truth test uses deterministic fictional fixtures and local rehearsal. It has zero provider calls, zero external writes, and zero allocated visitors. It is not a live campaign result or an authorization to make a real-world decision.
About the author
Jeff Brokaw is a CMO and Certified Chief AI Officer who builds commercial systems around AI, data, measurement, and the decisions that make those systems useful. He has been building AI systems commercially since 2016.