A ground-truth set for store-audit explanations should connect each expected finding to observable evidence. It is not enough to label a store “good” or “bad.” A useful evaluator must know what was inspected, what the evidence supports and which conclusions are outside the audit’s scope.
This article is a proposed evaluation design. The current site’s technical audit should not be confused with a deployed AI explanation service. An evaluation plan does not establish that an AI feature exists or has passed these checks.
Define the audit boundary
A public-page inspection can observe some HTML, links and visible content. It cannot infer private sales, inventory accuracy, profitability or a complete customer journey from one page. The ground-truth set must preserve that boundary so an ambitious explanation is not rewarded for inventing inaccessible facts.
For each case, record the page snapshot, collection time, relevant evidence and known collection limits. Dynamic content, blocked requests or unavailable pages should produce explicit uncertainty. A failed fetch is not proof that the merchant’s page is universally unavailable.
Build labels at the finding level
| Label component | Example purpose |
|---|---|
| Observable condition | What is actually present or absent in the captured evidence |
| Supporting location | Where a reviewer can verify the condition |
| Acceptable explanation | A plain-language interpretation within scope |
| Unsupported extension | A claim the evidence cannot justify |
| Appropriate next check | A concrete way to investigate further |
| Severity rationale | Why the issue matters for the stated task |
For example, a missing visible description in one captured page may justify a suggestion to review product information. It does not justify a claim that the store loses a specific percentage of revenue. Such numerical impact claims require separate evidence.
Include negative and ambiguous cases
An evaluation made only of obvious defects can reward an assistant that always finds something wrong. Include clean cases, borderline cases, incomplete captures and pages where the correct response is that no conclusion can be drawn.
Also include misleading page text that resembles an instruction to the evaluator. Merchant content is evidence to inspect, not authority to change the task. The agent website threat-model guide explains the trust boundary in more detail.
Use independent review and adjudication
Have reviewers label a shared sample independently using the rubric. Discuss disagreements and revise ambiguous definitions before expanding the set. Preserve genuinely uncertain labels rather than forcing artificial agreement.
Keep the reference set versioned. When the audit’s scope or evidence extraction changes, some labels may need revision. A model should not be penalized for failing to find information the collection layer no longer provides, nor rewarded for using information unavailable in the intended deployment.
Score grounding separately from style
Assess factual correctness, evidence support, completeness within scope, useful next action and appropriate uncertainty. Readability matters, but it should not compensate for an unsupported claim. A polished explanation that invents a revenue impact should fail the grounding check.
Record failure categories rather than only an overall score. Missing evidence, wrong interpretation, exaggerated severity and unhelpful next steps require different repairs. The AI acceptance-criteria article shows how to turn these categories into release conditions.
Protect the evaluation from becoming a rehearsal
Keep some cases out of prompt development. Add new cases from observed failures without repeatedly tuning to one fixed set alone. Compare against a simple explanation baseline so the value of the AI layer is visible.
A good ground-truth set makes modest but dependable explanations easier to recognize. It rewards an audit that says what the page supports, identifies uncertainty and proposes a useful next check, rather than one that produces the longest list of confident-sounding problems.