Building a Personal AI Assistant from My Notes

I wanted a chatbot that knew my notes well enough to help me think, not just answer a one-off question.

The idea

My Obsidian vault is where I keep notes, plans, and half-formed ideas. Obsidian helps me organize that material, but I wanted another way to explore it: a conversation with an assistant that could find the right notes when I needed them.

There are already many AI tools that connect to Obsidian. Building my own was still worthwhile because the project gave me a way to understand the moving parts instead of treating the assistant as a black box.

A simple retrieval-based design

The assistant follows a familiar Retrieval-Augmented Generation (RAG) pattern. It searches my notes for relevant passages, then gives those passages to an agent that writes the answer.

High-level structure of the assistant bot.

When I ask a question, the system turns it into a HumanMessage and sends it to a research agent. The agent converts the question into an embedding and searches a vector store for similar note chunks.

The retrieved chunks are passed to the assistant agent, which uses them to draft a response. Inputs, retrieved context, and outputs are kept in the agent state, which acts like a shared whiteboard for the workflow.

Turning the vault into searchable data

The first step is loading the Markdown files with LangChain’s DirectoryLoader. Long notes are split into smaller chunks with RecursiveCharacterTextSplitter.

Each chunk is then converted into an embedding and stored in Chroma. This gives the assistant a way to find both a short idea from months ago and a longer plan I wrote last week.

The important design choice is to separate retrieval from response writing. If an answer looks wrong, I can inspect the retrieved notes instead of guessing which part of the system failed.

A glimpse of the result

The first version is far from perfect, but it is satisfying to see the system search the vault and produce an answer from my own material.

This is what the note structure looks like in my vault:

What I would improve next

The next improvements are mostly about reliability. I want to make retrieval faster and more precise, add a way to judge whether the retrieved context is useful, and give the response agent clearer rules for concise answers.

I am also considering a small web interface so I can use the assistant without opening a terminal. That became the next step in the series.

Try the project

The code and setup details are available in the GitHub repository. You can use it as a starting point for your own notes or simply inspect how the pieces fit together.