A nutrition estimation agent that answers inside a live chat, and an evaluation pipeline that had to be widened before it was allowed to get there.
What the agent does
A user describes a meal the way people actually describe meals - in passing, incompletely, and in reference to something they said earlier. The agent turns that into a nutrition estimate. Two things make this harder than a single prompt call:
- Short-term memory
- The estimate has to survive a conversation. When the next message says "same as yesterday, but bigger", the agent resolves it against what was already said instead of asking the user to repeat themselves.
- User context
- The same description means different quantities for different people. The agent reads the user's own history rather than treating every message as if it arrived from nobody.
Why the evaluation pipeline came first
Nutrition estimates are numbers, and a wrong number in a chat window looks exactly like a right one. There is no phrasing that signals to the user that the model guessed. That is the whole reason the order of work here ran the way it did: before the agent was connected to the production chat, the evaluation pipeline was expanded with new datasets and broader case coverage, so that the cases it would meet in production were represented in the set it was scored against first. Coverage was the thing being expanded, not the score.
What broke, and what that cost
The interesting failure was not in the model. The session queue - the infrastructure that keeps each conversation's turns in order while requests run asynchronously - had to be fixed before the production chat endpoint could be deployed. An agent that reasons correctly over a conversation it has half-received is still wrong, and it fails in a way no prompt change repairs. Work on this project split roughly between the agent's own behaviour and the plumbing that has to hold for that behaviour to mean anything.
Why not just look the food up
- Look the food up in a database
- A nutrition database is authoritative and answers a question users do not ask. They write half a bowl of yesterday's pasta, not a canonical portion of a catalogued dish. Lookup handles the named case and returns nothing for the ordinary one.
How it stays measured
The agent runs behind FastAPI as an async microservice, with WebSockets carrying the chat and traces going to LangSmith and Langfuse. Tracing here is not for debugging alone: it is what makes a regression visible as a change in behaviour on known cases rather than as a support ticket. The engagement is ongoing, which means the evaluation set keeps growing as production turns up cases the original set did not contain.
What it proves
The agent only reached production chat after its evaluation pipeline was expanded, with new datasets and broader case coverage added before release.
Category
AI Engineering
Built with
Python · FastAPI · LLMs · Prompt Engineering · LangSmith · Langfuse · Async Microservices · WebSockets · Docker
Need something similar?
The cheapest way in is two weeks. The first days work out which task would pay for itself in your processes; the rest builds that agent on your own data and measures it.