PC Arena Shopping Assistant

The product

A storefront for PC components with an assistant built into the page. I own the retrieval layer over the product catalogue and the tool-calling agent that checks specs, stock and pricing before it answers rather than describing hardware from memory.

Why retrieval is measured on its own

The assistant uses agentic retrieval: the model writes the arguments to a keyword search tool. That splits the system into two failure modes that look identical from the outside — the retriever failing to reach the right product, and the model failing to form a query the retriever can serve. Scoring the finished answer collapses both into one number and tells you nothing about which one broke.

So the retrieval layer gets its own evaluation. Ground truth is a 200-case golden set derived from the live catalogue, labelled per layer: retrieval, tool-args, ranking, precision, grounding and refusal.

Deterministic scoring, no judge model

Recall@5, MRR, nDCG@5 and Precision@5, computed by set intersection and log2. Nothing is model-graded, which is what makes the numbers stable enough to gate CI on — a run that moves means the retriever moved, not that a grader had a different day.

What it found

Across 62 known-item queries, Recall@5 ranges from 1.000 to 0.000 depending purely on how the query string is formed. Passing the user's raw question straight through as the search term returns zero results 62 times out of 62: the backend matches with a single ILIKE '%term%' and no token splitting, so a whole sentence matches nothing.

All of the retrieval risk in this system is query formulation. None of it is the retriever's reach. That is the measurement that justifies moving to vector search — and it is a different fix from the one a low end-to-end score would have suggested.

Recall alone would have hidden a second failure

Searching by brand scores Recall@5 1.000 and MRR 0.806. It finds the right product every time, then buries it among its siblings. A hit-rate metric calls that a pass; the ranking metrics caught it.

Two bugs it surfaced

Result ordering had no deterministic tiebreaker, so 16 of 116 cases ranked differently between the live database and a reference implementation — every disagreement an exact tie on the sort key.

Superlative queries (“cheapest GPU”) returned five results for a one-item answer, scoring 0.336 adjusted precision against 1.000 everywhere else.

What the numbers do not cover

The golden set's questions come from two templates, so it measures the retriever against two sentence shapes rather than real user language. Broadening that is the next piece of work.

The harness also has an oracle-args run that scores 1.000 on Recall@5, MRR and nDCG@5. I do not quote it: those labels were derived by replaying the same filter the run executes, so the score is circular. The 0.000 → 1.000 spread is the number that survives scrutiny.