Why an AI Analyst Returned the Wrong Number

When an AI analyst returns a number that looks wrong, rephrasing the question is rarely the best first move. It may produce a different answer, but it can also erase the evidence needed to find the defect.

Preserve the failed run first. Then trace the request through each analytical layer until you find the first incorrect assumption, definition, permission, query, source, calculation, or explanation. The first incorrect layer is the place to fix.

This approach turns an unsettling result into a reviewable analytical incident rather than a prompt-writing contest.

Preserve the failed run

Save enough context to reproduce and inspect the result:

• The exact question and answer
• The timestamp
• The user and access context
• The generated query, analytical plan, or tool calls
• The cited tables, records, or source extracts
• Relevant model, prompt, tool, and data versions

Anthropic distinguishes an agent’s trace from its outcome. A trace records the steps taken, while the outcome is the final state produced. The analyst’s statement that it found the answer is not proof that the result is correct.

Preservation matters because a later retry may use different wording, data, permissions, or execution choices. If the original evidence is lost, the team may be unable to determine whether the defect was semantic, technical, or merely explanatory.

Step 1: Confirm the intended question

Natural-language requests often compress several analytical decisions into one sentence. Write those decisions out explicitly:

• Metric
• Date range and effective date
• Entity or population
• Segment and filters
• Currency or unit
• Aggregation grain
• Comparison period

For example, a request for revenue might refer to booked, billed, recognized, collected, or attributed revenue. The system cannot resolve a disputed business definition through fluent language alone.

Do not change the wording yet. Record what the user intended and compare it with what the system interpreted.

Step 2: Check the approved metric definition

Find the governed definition that should have applied at the time of the question. Confirm its formula, inclusions, exclusions, dimensions, effective period, and owner.

dbt describes a semantic layer as a way to centralize metric definitions over governed models and manage joins so downstream tools can use consistent definitions. This reduces one class of inconsistency, but it does not guarantee that every source value or analytical result is correct.

Ask:

• Did the analyst use the approved metric?
• Was the definition valid for the requested period?
• Did it substitute a similarly named field or calculation?
• Was the definition itself disputed or outdated?

If the wrong definition entered the plan, fix that semantic layer before tuning the final response.

Step 3: Inspect access context

A result can be internally consistent and still be wrong for the user because permissions silently narrowed the available data.

Check the identity under which the analysis ran. Compare its access with the population implied by the question. Look for row, object, field, account, region, or business-unit restrictions.

dbt includes access permissions in its Semantic Layer, although that implementation should not be generalized to every analytics platform. The broader lesson is simpler: access context is part of the analytical question.

If the system could only see one region while the user requested a global total, prompt refinement will not solve the underlying authorization mismatch.

Step 4: Inspect the query or plan

Now examine how the request became executable logic. Review:

• Selected sources and fields
• Join keys and join types
• Filters and date predicates
• Grouping and aggregation
• Null handling
• Deduplication
• Currency or unit conversion
• Calculation order

Duplicate joins, incorrect grain, stale effective dates, and missing filters are categories to inspect, not assumed causes. Follow the actual query and compare each step with the intended metric contract.

Pay special attention to grain. A customer-level measure joined to multiple activity rows may behave differently from a transaction-level measure. The issue is not that one-to-many joins are always wrong. It is that aggregation must match the grain of the metric and the sources.

Step 5: Trace source lineage and freshness

If the query is logically correct, move down to the data.

Confirm which systems supplied the result, whether scheduled refreshes completed, and whether intermediate transformations ran successfully. Check the data’s effective timestamp, not only the dashboard or response timestamp.

Then reconcile the result to an appropriate trusted control total. The control might be a governed report, ledger extract, validated operational total, or another approved source for that metric. Document any known timing or scope differences rather than forcing an artificial match.

A stale load, source correction, changed mapping, or incomplete integration can make a correct query return an incorrect business answer.

Step 6: Separate the number from the explanation

Only after validating the metric, access, query, and source should you inspect the narrative layer.

The number may be correct while the explanation is wrong. The response might mislabel the period, imply a causal conclusion, omit a material filter, or express unjustified certainty.

For research-style agents, Anthropic recommends checking groundedness, coverage, and source quality. Where an objectively correct result exists, exact comparison can also be useful. Apply the appropriate check to each component:

• Is the value correct?
• Is the label correct?
• Is the explanation supported?
• Are limitations visible?
• Does the cited evidence cover the claim?

Do not allow a polished narrative to compensate for an unsupported calculation.

Turn the correction into a regression test

Once the first incorrect layer is fixed, preserve the incident as a test case. Include the original question, expected interpretation, access context, approved definition, expected result or validation method, and required explanation constraints.

NIST calls for documenting test sets, metrics, tools used in testing, validation methods, contextual interpretation, and ongoing monitoring. A regression case converts one failure into a durable control for future changes.

The goal is not to make every answer look certain. It is to ensure that ambiguity is clarified, unsupported requests are refused, correct calculations are traceable, and mistakes can be located without guesswork.

Brainiac’s Nexus work focuses on governed definitions, traceability, and reviewable analytical outputs. It does not eliminate source-data problems or promise perfect accuracy. It provides a stronger operating structure for finding the first wrong layer and fixing the right thing.