AI systems · Essay
Challenges of retrieval in the assistant space
I've been sitting with Hamel Husain's write-up of Orion Weller's talk on retrieval and reasoning models, and the part that stuck was an argument about search interfaces. Weller puts Google's 1999 homepage next to a modern one and points out that the interaction hasn't moved: you type keywords, you get a ranked list. Then he shows a chat-style search product and makes the harder point. The interface changed. Retrieval didn't. The model sits on top, summarising results that a conventional engine picked, and the picking is where everything is decided.
Commerce has the same shape, with a worse version of the problem, because our conventional engine is a facet cabinet.
Four stages, and we're stuck on two
Weller's progression is keyword search, then semantic search, then instruction-based search, then something closer to prompting the retriever about how to behave.
Most e-commerce sits between the first two. Keyword matching with a semantic layer bolted on, business rules for merchandising, and a rerank pass at the end. Some teams have added an assistant on top of that stack. The assistant is polite about it, but it's asking the same 2015 retrieval system the same 2015 questions.
Instruction-based retrieval is different in kind. The instruction isn't a filter applied after results come back. It changes what counts as relevant during the first pass.
Reranking cannot save what retrieval never surfaced
This is the part that should worry anyone who has shipped "AI search" as a reranking layer.
Weller's example is a query about data privacy plus an instruction to find documents that use extended metaphors. A semantic retriever has no representation of "uses a metaphor," so the metaphorical document may sit thousands of ranks down. The reranker never sees it. Your reranker can only reorder the candidates you handed it, and you handed it the wrong ones.
Translate that: a customer asks for a birthday gift for someone who hates strong smells. The correct product is an unscented body oil from a brand with weak topical relevance to the word "gift." It's at rank 4,000. Your reranker is working on the top 200. No amount of reasoning at the rerank stage will find it, because that constraint never touched the retrieval step.
Now look at what the system still doesn't know even when it does surface a plausible shelf. This customer has never once spent above forty euros on a gift, which the price facet would catch only if she thought to set it herself. It's early December, and an unscented body oil sits very differently in a Christmas basket than it does in June. The recipient is the same sister who received a candle from this same site last year. None of that reaches retrieval. The typed query is the smallest part of the request, and we built the entire ranking system around it.
The fix isn't a better reranker. It's an instruction-aware retriever.
That's what Weller's Promptriever work goes after: bi-encoder embeddings trained so the instruction gets encoded into the query representation before the search happens.
Train the retriever on the expert, not the click
Once you accept that retrieval is where the decision happens, a second question opens up. Whose judgment is encoded in it?
Right now, the click. Every retailer employs people who know things the catalogue doesn't say. A beauty advisor knows two products with near-identical descriptions behave differently on reactive skin. A wine buyer knows which bottle survives a warm room. That knowledge enters the system, when it enters at all, as a curated edit or a merchandising boost applied after search returns. It sits on top, correcting a ranking it had no hand in producing.
Expert judgment can go into retrieval instead of on top of it. Weller's team built their training data by taking query-document pairs and generating sharper relevance criteria, along with negatives that were topically right and criteria-wrong. Put an expert where the generator was. A dermatologist says this cleanser suits reactive skin and that one doesn't, though both say "gentle" on the bottle. That's a labelled pair with a hard negative already attached. Enough of them and you have a fine-tuning set that teaches a retriever what suitability means in your category, rather than what similarity means in general English.
Rank1 points at a second route. It learned to reason by training on reasoning chains produced by a stronger model. Swap that for your own advisors. Ask twenty of them to write out why they'd put this product in front of this customer and not that one, and you have traces carrying knowledge that no general-purpose model was trained on and no competitor can scrape.
The objection is cost, and it dissolves faster than people expect. Nobody is labelling a million SKUs by hand. You need enough examples for the model to learn the principle, and then it generalises across the catalogue. Weller's models transfer well even with no in-domain training at all, so a few thousand expert-curated examples in one category is a modest bet.
One trade-off I've had to sit with. A deterministic rule engine built on an expert framework is provable. You can show precisely why a product was recommended, which stops being optional in anything touching health or skin. A learned retriever with expert judgment inside its weights covers far more of the catalogue but hands you no audit trail. That isn't a reason to avoid it. It's a reason to run both, with the deterministic layer owning the claims you make out loud, and the learned layer deciding what gets considered in the first place.
Three things commerce has that make this easier than it looks
Returns are labelled instruction negatives. Promptriever's training data trick is generating instruction negatives — documents relevant to the query but wrong for the instruction. That's an expensive synthetic step in general search. In commerce, we generate them for free every day. A customer searched for foundation, bought one, returned it because the shade was wrong. That's a product that matched the query and violated the constraint, labelled by someone who cared enough to ship it back. Complaint text in reviews is the same thing at scale. Most retailers are sitting on a training set for instruction-aware retrieval and treating it as a logistics cost line.
We already owe the customer a rationale. In general RAG, a reasoning trace is overhead you tolerate for better ranking. In commerce, especially in categories where suitability is the purchase decision, the explanation is a product feature we build anyway. Weller's Rank1 generates an auditable chain of reasoning to judge relevance. We need something to show the customer regardless. The compute serves two purposes at once, which changes the cost calculation considerably.
Products carry more structure than documents. A product has attributes, images, reviews, price history, stock position, and a relationship to what this specific person bought last spring. That's a richer object to reason over than a paragraph of text.
And two that make it harder
The rich-structure advantage evaporates if your catalogue data is thin, which for most retailers it is. An instruction like "fragrance-free" is unanswerable if the attribute isn't populated for two-thirds of the assortment. I've run the content programme that fixes this for a top slice of SKUs, and I can tell you the model was never the constraint. Instruction-based retrieval is bottlenecked on product data long before it's bottlenecked on embeddings.
The second problem is measurement, and it's nastier than it first appears. Weller's team hit low scores on an older benchmark and found the cause: their model was surfacing documents no human annotator had ever judged, because no previous system had ever retrieved them. They re-judged by hand and the model jumped to the top.
Now consider where e-commerce relevance labels come from. Clicks and purchases. Generated by the current ranker. Which means a genuinely better retriever will surface products with no engagement history, and your offline evaluation will score them as failures, because absence of evidence and evidence of absence look identical in a click log. Your metrics will actively punish the improvement. Anyone attempting this needs a human judgment set built before they start, or they'll kill the better system in an A/B test and never know why.
What I'd build
Split the stack by intent. A fast instruction-aware retriever handles the whole catalogue on every query, because that's where constraints have to enter. A slower reasoning reranker runs only on the top candidates, and only in high-intent moments where a customer is deciding rather than browsing. Weller is explicit that the reasoning models are powerful and slow; in commerce you cannot spend that budget on someone scrolling a category page, but you can absolutely spend it when they've asked a real question and are waiting for a real answer.
Then hand merchandisers the prompt. One finding from Promptriever that hasn't been metabolised by anyone I've spoken to in retail: you can change retrieval behaviour with natural language instead of tuning numeric thresholds. Not "set recall boost to 1.4" but "be generous with recall for gifting queries, be conservative for anything touching skin sensitivity." That's a different operating model for merchandising teams. It also needs versioning and audit before it goes anywhere near production, because a sentence someone edited on a Friday afternoon is now a ranking parameter.
The uncomfortable summary is that most of us have been improving the wrong layer. The assistant on top got better every quarter. The thing that decides which fifty products the assistant is allowed to talk about has barely changed since we launched it.
Based on P3: Optimizing Retrieval with Reasoning Models by Hamel Husain, featuring a talk by Orion Weller of Johns Hopkins University. Weller's Promptriever and Rank1 models and their training data are open source.