Building a Personal AI Assistant from My Notes

I wanted a chatbot that knew my notes well enough to help me think, not just answer a one-off question.

The idea

My Obsidian vault is where I keep notes, plans, and half-formed ideas. Obsidian helps me organize that material, but I wanted another way to explore it: a conversation with an assistant that could find the right notes when I needed them.

There are already many AI tools that connect to Obsidian. I still wanted to build my own so I could understand the moving parts instead of treating the assistant as a black box.

A simple retrieval-based design

The assistant follows a familiar Retrieval-Augmented Generation (RAG) pattern. It searches my notes for relevant passages, then gives those passages to an agent that writes the answer.

When I ask a question, the system turns it into a HumanMessage and sends it to a research agent. The agent converts the question into an embedding and searches a vector store for similar note chunks.

The retrieved chunks are passed to the assistant agent, which uses them to draft a response. Inputs, retrieved context, and outputs are kept in the agent state, which acts like a shared whiteboard for the workflow.

Turning the vault into searchable data

The first step is loading the Markdown files with LangChain’s DirectoryLoader. Long notes are split into smaller chunks with RecursiveCharacterTextSplitter.

Each chunk is then converted into an embedding and stored in Chroma. The assistant can then find both a short idea from months ago and a longer plan I wrote last week.

I kept retrieval separate from response writing. When an answer looks wrong, I can inspect the retrieved notes instead of guessing where the system failed.

A glimpse of the result

The first version is far from perfect, but it is satisfying to see the system search the vault and produce an answer from my own material.

This is what the note structure looks like in my vault:

What I would improve next

The next improvements are mostly about reliability. I want faster, more precise retrieval and a way to judge whether the retrieved context is useful. I also want clearer rules for concise answers.

I am also considering a small web interface so I can use the assistant without opening a terminal. That became the next step in the series.

Try the project

The code and setup details are in the GitHub repository. It is there if you want to adapt it to your own notes or inspect how the pieces fit together.