Designing an auditable multi-LLM evaluation pipeline

A benchmark needs an evidence model, not just a loop

ShelfSignal’s documented visibility pipeline asks 20 buyer questions of five assistants with five repetitions, yielding 500 answers per category run. Those numbers describe one sample design. They are not a reason to hide failed runs or assume every model is interchangeable.

The engineering problem is to preserve enough context to explain each figure later: the original question, model configuration, attempt, raw answer, extracted entities and the aggregation rule. This article describes design principles informed by that work. The record format below is illustrative, not a publication of ShelfSignal’s private production schema.

1. Keep the unit of observation explicit

A question, an answer, a citation and a brand mention are different objects. Store a run as its own observation, then derive the counts you need. If a question fails, preserve the failure separately: an error is not a valid answer with zero brand mentions.

{
  "question_id": "q-01",
  "run_index": 1,
  "assistant": "configured-assistant",
  "model_version": "recorded-at-run-time",
  "search_mode": "enabled-or-disabled",
  "market": "agreed-market",
  "status": "ok",
  "answer_ref": "stored-source-answer",
  "brands": ["Example Brand"],
  "citations": ["https://example.com/source"]
}

For reproducibility, also retain the exact prompt and timestamp. A friendly assistant label is not enough when model versions or search behaviour change. Do not infer missing configuration retrospectively.

2. Separate generation, extraction and aggregation

This separation makes a parser correction different from a new model run. It also lets a reviewer trace an aggregate back to the observation without rerunning a model and pretending the new text is the old evidence.

3. Repetition does not turn a model into a deterministic system

Repeated prompts expose variation. They do not guarantee that a later run has the same distribution: search results, model updates, retrieval and market availability can all change. Compare like with like and keep the per-assistant breakdown visible.

A zero count in a finite sample means the brand was not observed in that sample. It does not prove that the model has never encountered the brand. Similarly, several platforms agreeing on an answer is not independent proof that the recommendation is correct.

4. Keep the decision metric separate from the business outcome

A brand mention is not a page visit. A cited URL is not a click. A seller selected in a pre-payment simulation is not a completed transaction. Store these as separate measures rather than naming every stage a conversion.

For example, the visibility study counts answers naming a brand. Shopping simulations record a selected brand and seller under a brief. Website analytics records visits and accepted form requests. Actual enquiry receipt and revenue require their own confirmation. A useful report can connect the stages without pretending they are the same observation.

5. Make a rerun auditable

• Freeze questions, market, language, search mode, repetition count and identity mappings before a comparison.
• Record configuration and external changes instead of silently replacing them.
• Preserve unsuccessful runs and report coverage alongside results.
• Keep raw evidence private where it contains customer data; publish only authorised aggregates.
• Report small counts as counts and state what the sample does not establish.

The useful deliverable is a result another engineer can inspect, not a screenshot that happens to support the preferred story.

• The documented ShelfSignal project
• Public measurement definitions
• Discuss an AI product or evaluation pipeline