Expert Blockchain Chatbot

Where plain RAG runs out

Semantic search over a document collection answers questions shaped like what does this mean and how does this work. It does not answer how many, which is the largest, what is the current value, or has this address done anything unusual. Those need a query, not a similarity score, and a vector store has no way to say it cannot help. It returns the nearest passages either way, the model writes a fluent paragraph around them, and the number in that paragraph came from nowhere. In a domain where users ask for statistics constantly, that is not an edge case. It is most of the traffic.

What we added, and why each one

The retrieval stopped being one thing and became a set of them, each answering a class of question the others could not.

SQL for anything countable

Questions about statistics get turned into queries against the real tables instead of being answered from retrieved prose. A count that came out of a database is either right or wrong; a count that came out of a paragraph is neither.

Entity lookup

Named things - addresses, validators, contracts, proposals - resolve to the record rather than to whatever text mentioned them most similarly. Similarity is the wrong instrument for identity.

Real-time sources

A chain moves. Anything indexed is stale the moment it is indexed, so live data is fetched at answer time rather than re-embedded and hoped over.

How we knew any of it worked

Each addition was scored rather than assumed. An evaluation framework built on RAGAS graded answers on the things that go wrong in production RAG - whether the answer is grounded in what was retrieved, and whether what was retrieved was the right material - and the benchmarking ran across domains rather than on one favourable slice. That last part matters more than it sounds: a retriever tuned until it wins on the questions you thought of will lose on the ones you did not, and the only way to see that is to test it somewhere it was not tuned.

The part that was not technical

Twelve months of this involved several teams, and the architecture had to survive that. The routing decision - which retriever handles which question - is the piece everyone touches and nobody owns by default, so it was worth designing deliberately instead of letting it accumulate.

Why more embeddings would not have fixed it

Index more documents

When retrieval misses, the reflex is that coverage is thin, and here it was not: the corpus held the answer to every question people were asking. Adding to an index changes what can be found, never what can be computed, so no amount of it moves the needle on how many.

Put everything in the context window instead

Long contexts make it tempting to skip retrieval and paste the corpus in. It moves the count from the index into the model, where it becomes an arithmetic task performed on prose, unverifiably, and paid for again on every request. A query answers the same question exactly and the same way twice.

One build, two things settled

For a knowledge base: answers come from the source that can actually answer, and the system distinguishes between kinds of question rather than treating every one as a search. For evals: none of the additions were justified by a demo. Each one had a number attached before and after, which is the only reason we can say the retrieval got better rather than that it got more complicated.