The summary sounded right. The evidence didn’t.
Leadership already had the dashboard data; preparing a useful weekly explanation was the expensive part. AI summaries seemed to solve that task until fluent commentary began turning plausible causes into confident explanations. A decline in conversion could acquire a story before anyone had inspected the affected campaigns.
Check the figures before explaining them
The analysis began with reconciled dashboards or datasets and a metric dictionary covering definitions, calculation, dates, attribution and exclusions. Material totals, ratios and comparisons were calculated outside the generative layer. AI received validated figures, reducing the opportunity for arithmetic errors to enter a persuasive narrative.
The prompt required references to the supporting metrics. A movement could be described precisely enough for a reviewer to inspect it, rather than relying on adjectives such as strong or weak without a numerical basis.
Separate what happened from why it might have happened
The draft showed the observation, possible explanations and the next check. Reviewers challenged claims of cause, checked metric definitions and recorded uncertainty.
A conversion could mean a form, a qualified lead or an opportunity. Making that definition explicit prevented the analysis from explaining a change in one measure as though it were a change in another.
Check the number. Then question the explanation.
The reporting workflow separated calculations, observations and possible causes. A reviewer could follow each claim back to a source and see which next check would confirm or challenge the interpretation.
| Statement | Supporting requirement | Allowed next step |
|---|---|---|
| A metric changed | Checked values, consistent periods and a defined denominator | Describe the movement precisely |
| A campaign may explain the change | Segmented evidence consistent with that explanation | Request the relevant diagnostic check |
| The change was caused by an intervention | A comparison capable of supporting that inference | Use causal language only to the extent justified |
| Budget should move | Commercial context, uncertainty and an accountable decision | Recommend a limited action and a follow-up check |
A fluent summary was not evidence. The useful output was a checked report with an actionable question.
What still needed human review
More model autonomy would have increased the amount of verification required, potentially erasing the preparation saving. I kept material calculations outside the generative layer and required the draft to distinguish an observed movement from a hypothesis about its cause.
The reviewer also needed to challenge a convenient explanation. A summary that reinforces the team’s existing belief can be numerically correct and analytically weak, so alternative explanations and missing evidence remained part of the review.
Eight reports, checked figures
Across eight weekly reports, first-draft analysis preparation fell from three hours to ninety minutes. Every material figure was independently checked, and each final report documented uncertainty and a named next action.
The eight-report result concerns first-draft analysis time, not a fully automated analytical function. Every material figure was checked, uncertainty was recorded, and an action had an owner. Those controls explain where the time saving stops and human responsibility begins.
Automation can draft the commentary; an analyst still has to defend the conclusion. Keeping calculations, evidence and uncertainty visible made the report easier to challenge before money moved.