Making My Personal AI Assistant Safer and More Reliable

The first version could retrieve notes and answer questions. The next version needed to know when it should not answer, and when its context was not good enough.

Why the first version was not enough

The original bot had a simple flow: a research agent searched a vector store, then an assistant wrote the final response.

That worked for basic questions, but it left two problems. The bot could retrieve irrelevant notes, and it had no reliable way to handle a request for private information. The system also could not verify who was typing.

The new workflow

I split the process into specialized agents: a Safety Agent, a Query Agent, a Ranking Agent, and a PR Agent. Each agent has a smaller responsibility and a clearer reason for existing.

High-level structure of the assistant bot.

Here is a short demo of the workflow. The logs show how the agents pass the request through each stage.

Step 1: Safety Agent

The Safety Agent reviews each incoming question before retrieval begins. If a request asks for passwords, financial details, or other private information, the agent sends it to the PR Agent for a safe refusal.

This check is useful because the bot does not know whether the person using it is actually me. The workflow therefore treats sensitive requests as unsafe by default.

Step 2: Query and Ranking Agents

If the request passes the safety check, the Query Agent turns it into a search query and looks for relevant note chunks.

The Ranking Agent then checks whether the retrieved material is useful. If the results are weak, the workflow sends the request back for another search. The number of retries is bounded so a bad query cannot create an endless loop.

Step 3: PR Agent

The PR Agent turns the workflow’s result into a readable response. It handles both successful retrieval and the case where the system cannot find enough information.

Its job is not to add new facts. It makes the answer clearer, sets the right tone, and explains uncertainty when the retrieved notes are insufficient.

Why the agent boundaries help

The main benefit is not that several agents automatically make the model smarter. The benefit is that each stage has an explicit job and a testable decision.

The Safety Agent focuses on privacy, the Query Agent focuses on search, the Ranking Agent focuses on relevance, and the PR Agent focuses on communication. Together, they give the workflow more control than one large prompt.

What remains unfinished

There is still plenty to improve. Parent-document retrieval and a dedicated reranking model may improve search quality, but those additions make more sense after the basic workflow is stable.

I am also considering caching common questions and building a smoother interface. These changes should reduce repeated work and make the assistant easier to use, but they are separate from the safety and retrieval changes described here.

Try the project

The code and setup details are available in the GitHub repository.