AI Research Analyst · Sacha Tihanyi

I was interested in an AI research tool, effectively a research-focused agent, specifically constrained to a defined and strictly local information set. Privacy, transparency, and full design control over the hybrid search pipeline were all key motivations. I’ve had a lot of experience in global macro and financial markets research, and this tool was particularly interesting to test in that context, though it had to be general enough to be useful across any domain.

The frontend had to be non-distracting, basic, and “nice”, but as I have very little interest in the whole world of frontend development (crucial as it is), I leaned heavily on AI assistance to set that up. It’s a responsive React and TypeScript interface, featuring drag-and-drop uploads, document-status management, and a conversational chat experience with secure markdown rendering. The frontend is integrated with a FastAPI backend through REST APIs, and uses Vite and Tailwind CSS for development and styling.

The backend is where I focused my energy on developing a solid hybrid semantic and lexical search and retrieval algorithm, as well as the data transformation pipeline to feed it. For chunk, embedding, and metadata storage, I kept it lightweight with an SQLite database.

Interesting Lessons

Transformer architectures are not just for semantic search.

Leveraging the LLM in lexical search was a key idea that I wanted to test and implement. In thinking about how a user might query a document store for facts, ideas, or context important in reasoning tasks, it’s obvious that the particular way in which people naturally phrase queries, while appropriate for the semantic search leg, could be very brittle in terms of lexical search precision and recall. Simply doing a keyword search with only the entities pulled from the query would likely result in very weak recall in most cases.

Using an LLM as a ‘query planner’, with structured output, allowed me to convert user prompts into a structured, typed intermediate representation that identified word roles (semantically relevant vs. metadata), phrases or multi-word expressions, and useful lexical variants. This allowed me to design a lexical search that preserved the user’s intent while making keyword search less brittle through the inclusion of alternative phrasings and wordings in the target document set. The idea was to expand the range of expressions that could match the same underlying evidence, without expanding the set of ideas being searched for beyond the user’s original intent. The hope is higher recall without sacrificing precision.

This in turn facilitated the construction of multiple lexical search queries for a single user prompt, with the exact terms in each query depending on the output of the LLM ‘query planner’. These constructed queries were built to go from broad (recall) to more constrained (precision). The results of each lexical search were then merged via reciprocal rank fusion (RRF) and the combined results were subsequently merged, again using RRF, with those of a semantic vector search based on the user query.

Ultimately, this helped to flesh out questions like “What did Marcus Aurelius say was the greatest virtue in his Meditations?” by producing semantically informed lexical expansions that improved the robustness of keyword retrieval. In this illustrative instance, the LLM’s structured output ensured that the word “Meditations” and the name “Marcus Aurelius” were left out of the search because they were classified as contextual metadata that were unlikely to appear in the documents. In this case, Marcus Aurelius never gave his writings a name and didn’t reference himself in the third person in what was effectively his diary of thoughts.
Context engineering to improve precision and encourage useful reasoning across documents.

An obvious challenge in multi-document RAG is that providing more context can actually reduce precision. This was something I tested different context construction techniques to address while running searches across a dataset with documents covering many different subjects, some of which were completely unrelated to one another and to the search prompts.

Constructing context by keeping passages demarcated in semantically consistent XML tags, grouped by source and in order of retrieval, was very helpful. Clearly instructing the model to reason across documents while independently judging every passage’s relevance to the user query helped guide the model in combining corroborating evidence, retaining disagreements, and ignoring sections that were retrieved but irrelevant simply because of superficial keyword or minor semantic overlap.

This actually led to an interesting surprise in one run where I was testing searches over Tesla’s Q2 financial results using a document set with a lot of unrelated content (like the memoirs of Ulysses S. Grant, Meditations, the play Lysistrata by Aristophanes, an IMF Article IV report on China, etc.). The database of document chunks also included NVIDIA’s Q2 results, which were retrieved by the hybrid search targeting a question about Tesla’s results, due to some overlapping semantic and lexical content.

The model generated an interesting insight relevant to the user prompt regarding Tesla (asking it to highlight risks in the Q2 financials), raising an interesting point about potential competition risk, but also positive ecosystem tailwinds represented by what the model characterized as NVIDIA’s increasingly open, broadly adopted platforms.

This is representative of the potential for LLMs to make creative connections between seemingly disparate facts, so long as context is well-curated and delineated, and presented with a reasonable prompt to encourage such behaviour when useful.
Gating the RAG pipeline, saving tokens and sparing the context window.

After getting the retrieval and context engineering to a point where they were working well, it quickly became obvious that a gate was needed to intercept additional user questions for which there was already sufficient context present in the conversation to answer correctly. This would reduce latency from model calls during the search query planning step, avoid token usage from retrieving unnecessary context, and preserve context window space.

To bypass unnecessary RAG pipeline triggers, I added an LLM-based decision gate to decide whether the context already retrieved and present in the conversation history was sufficient to answer the new user question or discussion point. This provided an opportunity to use a new class of models, well represented by TypeSafe AI’s Jev System One model, which are purpose-built for very quick and relatively cheap binary decisions thanks to their constrained, non-recursive output.

I chose to set a fairly conservative trigger threshold for the RAG pipeline; if the model assigned a probability of less than 0.75 to the proposition that the already available context was sufficient to answer the new question, the RAG pipeline was triggered. This threshold incorporates the view that the cost of the research agent hallucinating an answer with unwarranted epistemic certainty was far higher than the cost of adding additional latency and increasing token usage, should no extra context actually be needed. This view was further reinforced by the fact that the LLM had already been instructed in the system prompt to ignore any context deemed irrelevant to the question at hand.

This tool has really generated quality-of-life improvements for me in terms of quickly conducting in-depth research over a set of local documents and leveraging the reasoning and synthesis capabilities provided by an LLM. The drag-and-drop pipeline of document conversion, chunking, embedding, and database storage is particularly convenient. I’d probably eventually look to extend this to more of a general agent architecture to facilitate supplementary research over the internet. There’s also the issue of slow text extraction from scanned, image-only PDFs, given that I’m keeping document ingestion, parsing, and embedding all local, and am not running any of it on hardware more powerful than my MacBook.